Target-based laser radar and camera calibration method and system, electronic device

Abstract

The application provides a target-based laser radar and camera calibration method and system, and an electronic device. The method comprises: acquiring laser radar point cloud data and camera image data of a target; screening the point cloud data to obtain target three-dimensional information of the target in a laser radar coordinate system; based on the image data, obtaining target three-dimensional information of the target in a camera coordinate system; based on a plurality of groups of preset external parameters of the target and the camera conversion, converting the two groups of target three-dimensional information in the laser radar coordinate system and the camera coordinate system to the same coordinate system; calculating the coincidence degree of the two groups of converted target three-dimensional information to obtain a first external parameter corresponding to the maximum coincidence degree. A high-automation laser radar and camera external parameter calibration method applicable to different scenes is realized.

Description

Technical Field

[0001] This invention relates to the field of camera calibration technology, and in particular to a target-based lidar and camera calibration method, system, and electronic device. Background Technology

[0002] In industries such as autonomous driving, robotics, and surveying, cameras and LiDAR are two widely used sensors. Cameras can provide high-resolution environmental information (such as color and texture information), but are easily affected by lighting conditions; LiDAR can provide accurate 3D distance and surface reflection intensity information, but its resolution is generally lower than that of cameras. Fusing the observations from these two complementary sensors can yield richer and more accurate results than single-source observations. However, the prerequisites for fusion are: 1. accurate camera intrinsic parameters; 2. known coordinate system transformation relationships (extrinsic parameters) between the camera and LiDAR.

[0003] Currently, most extrinsic parameter calibration methods between LiDAR and cameras lack automated processing and still require some degree of manual intervention, such as manually selecting the point cloud target position. Errors introduced by manual selection can reduce calibration accuracy and fail to meet the demands of mass production. Some calibration methods obtain the target's position range in the point cloud coordinate system by fixing the relative positions of the sensor and the target; however, this method is not directly applicable to different calibration sites and lacks flexibility. To address these issues, this invention proposes a highly automated, target-based extrinsic parameter calibration method for LiDAR and cameras, applicable to various sites.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a target-based lidar and camera calibration method, system, and electronic device.

[0006] This invention provides a target-based lidar and camera calibration method, the method comprising:

[0007] Acquire lidar point cloud data and camera image data of the target;

[0008] The point cloud data is filtered to obtain the target's three-dimensional information in the lidar coordinate system;

[0009] Based on the image data, obtain the target's three-dimensional information in the camera coordinate system;

[0010] Based on the preset multiple sets of external parameters for the conversion between the target and the camera, the two sets of three-dimensional target information in the lidar coordinate system and the camera coordinate system are converted to the same coordinate system;

[0011] Calculate the overlap of the two sets of target 3D information after conversion, and obtain the first extrinsic parameter corresponding to the maximum overlap.

[0012] According to the present invention, a target-based lidar and camera calibration method is provided, the method further comprising:

[0013] Based on the first extrinsic parameter and the target's three-dimensional information in the camera coordinate system, the first pose of the target in the lidar coordinate system is obtained.

[0014] Based on the first pose, determine the center point of the target;

[0015] Based on the target's external dimensions and the center point, the target's three-dimensional bounding box is obtained;

[0016] Obtain each laser point in the three-dimensional information of the target in the lidar coordinate system, and calculate the sum of the distances from all the laser points to the three-dimensional bounding box;

[0017] The first extrinsic parameter is optimized by minimizing the sum of the distances, thus forming the second extrinsic parameter.

[0018] According to the present invention, a target-based lidar and camera calibration method is provided, which obtains each laser point in the three-dimensional information of the target in the lidar coordinate system, including:

[0019] A screening sphere is formed with the center point of the target as the center and a preset first radius;

[0020] The laser points within the screening sphere are used as the laser points in the target's three-dimensional information in the lidar coordinate system.

[0021] According to the present invention, a target-based lidar and camera calibration method is provided, which obtains each laser point in the three-dimensional information of the target in the lidar coordinate system, including:

[0022] For the target, all laser points in the three-dimensional information of the target under the lidar coordinate system are fitted to form a point cloud surface;

[0023] Calculate the point-to-surface distance from all laser points in the target's three-dimensional information in the lidar coordinate system to the point cloud surface, and remove laser points whose point-to-surface distance is greater than a predetermined distance threshold;

[0024] The remaining laser points after removal are taken as the laser points in the target's three-dimensional information in the lidar coordinate system.

[0025] According to the present invention, a target-based lidar and camera calibration method is provided to acquire lidar point cloud data and camera image data of the target, including:

[0026] Multiple targets are set within the common field of view of the lidar and the camera;

[0027] The camera and the lidar are fixed as a whole, and the whole of the camera and lidar is allowed to move relative to the target to acquire multi-frame point cloud data and image data.

[0028] According to the present invention, a target-based lidar and camera calibration method is provided, which calculates the overlap degree of two sets of converted target 3D information and obtains the first extrinsic parameter corresponding to the maximum overlap degree, including:

[0029] Based on the timestamp information of the multi-frame point cloud data and image data, the multi-frame point cloud data and image data are paired according to time.

[0030] Based on the paired point cloud data and image data, the overlap of the two sets of target 3D information after conversion in each frame is calculated respectively.

[0031] Calculate the sum of overlap of all frames, and obtain the extrinsic parameter with the largest sum of overlap as the first extrinsic parameter.

[0032] According to the present invention, a target-based lidar and camera calibration method is provided, which filters the point cloud data to obtain the three-dimensional information of the target in the lidar coordinate system, including:

[0033] Based on the point cloud data, obtain the corresponding two-dimensional depth map;

[0034] Based on the depth map, obtain the angle difference information between each pixel and its neighboring pixels;

[0035] Based on the angle difference information, the point cloud data is clustered into several first point cloud clusters through connected component analysis;

[0036] The feature information of the first point cloud cluster is described by descriptors to obtain several feature description vectors for the several first point cloud clusters;

[0037] The aforementioned feature description vectors are matched with the predetermined target point cloud feature vectors to obtain the first successfully matched point cloud cluster, which serves as the target's three-dimensional information in the lidar coordinate system.

[0038] According to the target-based lidar and camera calibration method provided by the present invention, the overlap degree of two sets of converted target 3D information is calculated, including:

[0039] Based on the first successfully matched cloud cluster, obtain multiple corresponding points on the depth map;

[0040] Based on the target's three-dimensional information in the camera coordinate system, the center point of the target in the camera coordinate system is obtained.

[0041] Based on the multiple sets of external parameters, the center point of the target in the camera coordinate system is converted into multiple center points of the target in the lidar coordinate system;

[0042] Based on multiple center points of the target in the lidar coordinate system, the number of corresponding points within the target boundary range in the depth map is counted and used as the overlap degree of the two sets of target 3D information after conversion.

[0043] This invention also provides a target-based lidar and camera calibration system, the system comprising:

[0044] The acquisition module is used to acquire lidar point cloud data and camera image data of the target;

[0045] The point cloud filtering module is used to filter the point cloud data and obtain the target's three-dimensional information in the lidar coordinate system.

[0046] An image conversion module is used to obtain the target's three-dimensional information in the camera coordinate system based on the image data.

[0047] The conversion module is used to convert the two sets of three-dimensional target information in the lidar coordinate system and the camera coordinate system to the same coordinate system based on multiple preset external parameters of the target and the camera.

[0048] The calculation module is used to calculate the overlap of the two sets of target 3D information after conversion and obtain the first extrinsic parameter corresponding to the maximum overlap.

[0049] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the target-based lidar and camera calibration method as described in any of the preceding claims.

[0050] The present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the target-based lidar and camera calibration method as described in any of the preceding claims.

[0051] The target-based lidar and camera calibration method, system, and electronic equipment provided by this invention realize a highly automated lidar and camera extrinsic parameter calibration method applicable to different scenarios. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 A schematic flowchart of a target-based lidar and camera calibration method provided by the present invention;

[0054] Figure 2 A schematic diagram of a target-based lidar and camera calibration system provided for this invention;

[0055] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] The target-based lidar and camera calibration method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0058] Figure 1 A schematic flowchart of a target-based lidar and camera calibration method provided by the present invention is shown below. Figure 1 As shown, the present invention provides a target-based lidar and camera calibration method, which may include the following steps.

[0059] S100: Acquires lidar point cloud data and camera image data of the target.

[0060] Optionally, acquire lidar point cloud data and camera image data of the target, including:

[0061] Multiple targets are set up within the common field of view of the lidar and camera;

[0062] The camera and lidar are fixed as a whole, and the whole unit moves relative to the target to acquire multi-frame point cloud data and image data.

[0063] Preferably, three to five square targets with QR code (Apriltag) patterns are placed within the common field of view of the lidar and camera.

[0064] Preferably, the camera and lidar are moved back and forth along the direction facing the target pattern at a speed of about 5 km / h to collect lidar point cloud data and camera image data for about 30 seconds.

[0065] S200: Filter the point cloud data to obtain the target's three-dimensional information in the lidar coordinate system.

[0066] Optionally, the point cloud data is filtered to obtain the target's three-dimensional information in the lidar coordinate system, including:

[0067] Based on point cloud data, obtain the corresponding two-dimensional depth map (RangeImage);

[0068] Based on the depth map, obtain the angle difference information between each pixel and its neighboring pixels;

[0069] Based on the angle difference information, the point cloud data is clustered into several first point cloud clusters through connected component analysis;

[0070] The feature information of the first point cloud cluster is described by descriptors, and several feature description vectors about several first point cloud clusters are obtained.

[0071] Several feature description vectors are matched with predetermined target point cloud feature vectors to obtain the first successfully matched point cloud cluster, which serves as the target's three-dimensional information in the lidar coordinate system.

[0072] Preferably, a frame of point cloud data is converted into a two-dimensional depth map based on the vertical and horizontal resolution parameters of the lidar.

[0073] Preferably, the depth map is a sector coordinate system, and the above-mentioned depth map transformation is completed based on the vertical and horizontal angles in the point cloud parameters.

[0074] Preferably, the depth map is traversed in a four-neighbor manner. The angle difference information β between the neighboring pixels and the current pixel is calculated using the depth values ​​of the neighboring pixels and the current pixel. All pixels in the depth map are traversed, and then connected component analysis is used to cluster and segment the entire frame point cloud to obtain a series of point cloud clusters with classification information. For details, please refer to the paper with doi 10.1109 / IROS.2016.7759050.

[0075] Preferably, for the point cloud cluster data after the above clustering and segmentation, the point cloud cluster is rotated and transformed to its own principal axis direction (the coordinate system with the center of the point cloud cluster as the origin) by Principal Component Analysis (PCA). Based on the rotated point cloud, the feature information is described by the three-dimensional point cloud global descriptor (Multiview 2D Projection, M2DP), and a one-dimensional feature description vector can be obtained.

[0076] Preferably, the predetermined target point cloud feature vector is calculated in advance based on the target size information. The correlation between the feature description vector of each cluster and the predetermined target point cloud feature vector is calculated. The point cloud cluster is then classified in binary according to the vector correlation (i.e., it is determined whether the point cloud cluster corresponds to a target), so that a more accurate point cloud target detection result can be obtained.

[0077] S300: Based on image data, obtain the target's three-dimensional information in the camera coordinate system.

[0078] Preferably, the original image of the image data is converted into a grayscale image for QR code (Apriltag) detection to obtain the pixel coordinates of the target vertex; then, based on the camera intrinsic parameters, the Perspective-n-Point (PnP) method is used to calculate the target center position and orientation in the camera coordinate system, as well as the three-dimensional coordinates of the four vertices of the target.

[0079] S400, based on preset target and camera conversion multiple sets of extrinsic parameters, converts two sets of target 3D information in the lidar coordinate system and the camera coordinate system to the same coordinate system.

[0080] Preferably, based on the design values ​​of the lidar-camera extrinsic parameters, the angular design values ​​of the extrinsic parameters are multiplied by a series of perturbation rotation matrices to generate a candidate initial extrinsic parameter queue, i.e., multiple sets of extrinsic parameters. The perturbation rotation matrices are calculated by sampling the pitch angle, yaw angle, and roll angle with fixed small angle steps.

[0081] S500: Calculate the overlap of the two sets of target 3D information after conversion, and obtain the first external parameter corresponding to the maximum overlap.

[0082] Optionally, the overlap of the two sets of target 3D information after conversion is calculated, including:

[0083] Based on the first successfully matched cloud cluster, obtain multiple corresponding points on the depth map;

[0084] Based on the target's 3D information in the camera coordinate system, the center point of the target in the camera coordinate system is obtained.

[0085] Based on multiple sets of external parameters, the center point of the target in the camera coordinate system is transformed into multiple center points of the target in the lidar coordinate system;

[0086] Based on multiple center points of the target in the lidar coordinate system, the number of corresponding points within the target boundary range in the depth map is counted and used as the overlap degree of the two sets of target 3D information after conversion.

[0087] It should be noted that the multiple sets of extrinsic parameters obtained by the perturbation rotation matrix are the process of rasterizing the design values ​​of the lidar-camera extrinsic parameters. The first extrinsic parameter between the lidar and the camera is obtained by the raster search method. This first extrinsic parameter has a relatively lower accuracy than the second extrinsic parameter.

[0088] Optionally, the overlap degree of the two sets of target 3D information after conversion is calculated, and the first extrinsic parameter corresponding to the maximum overlap degree is obtained, including:

[0089] Based on the timestamp information of multi-frame point cloud data and image data, the multi-frame point cloud data and image data are paired according to time.

[0090] Based on paired point cloud data and image data, the overlap of the two sets of target 3D information after conversion in each frame is calculated.

[0091] Calculate the sum of overlap of all frames, and take the extrinsic parameter with the largest sum of overlap as the first extrinsic parameter.

[0092] Optionally, the method further includes:

[0093] Based on the first external parameter and the target's three-dimensional information in the camera coordinate system, the first pose of the target in the lidar coordinate system is obtained;

[0094] Based on the first pose, determine the center point of the target;

[0095] Based on the target's external dimensions and center point, obtain the target's three-dimensional bounding box;

[0096] Obtain each laser point in the three-dimensional information of the target in the lidar coordinate system, and calculate the sum of the distances from all laser points to the three-dimensional bounding box;

[0097] The optimization direction is to minimize the sum of distances, optimize the first extrinsic parameter, and form the second extrinsic parameter.

[0098] It should be noted that, based on the first pose, the center point of the target is determined, and the predicted coordinates of the target center point in the lidar coordinate system are obtained. Based on the first pose, the correspondence between the target point cloud cluster and the camera detection QR code (Apriltag) can be obtained.

[0099] Preferably, a three-dimensional bounding box is established based on the target width and thickness, with the center point of the target as the origin of the coordinate system.

[0100] Preferably, the optimization direction is to minimize the sum of distances, including optimizing the first extrinsic parameter using the Levenberg-Marquardt (LM) nonlinear optimization method to minimize the total distance. The second extrinsic parameter is a further fine-tuning based on the first extrinsic parameter. The second extrinsic parameter here has six degrees of freedom: x, y, z, pitch angle, yaw angle, and roll angle.

[0101] Furthermore, based on the optimized second external parameter, the point cloud in the lidar coordinate system is transformed to the target coordinate system, and the x, y, and z distances from the midpoint of the point cloud to the center of the target are counted. Points within the target boundary are recorded as valid in-points. The proportion of in-points is calculated based on the number of valid in-points and the number of initial point clouds, and the proportion of in-points is used as the standard for measuring the validity of the calibration results.

[0102] Optionally, each laser point in the three-dimensional information of the target in the lidar coordinate system is obtained, including:

[0103] A screening sphere is formed with the center point of the target as the center and a preset first radius.

[0104] The laser points within the selected sphere are used as the laser points in the target's three-dimensional information in the lidar coordinate system.

[0105] Preferably, before filtering, a KD-tree (k-dimensional tree) data structure is established for the entire frame point cloud.

[0106] Preferably, the first radius is slightly larger than the side length of the target. This prevents an excessively large first radius from encompassing point clouds belonging to other targets into the point cloud cluster of this target.

[0107] Optionally, each laser point in the three-dimensional information of the target in the lidar coordinate system is obtained, including:

[0108] For all laser points in the three-dimensional information of the target in the lidar coordinate system, a point cloud surface is formed by fitting the target.

[0109] Calculate the point-to-surface distances of all laser points in the target's 3D information in the lidar coordinate system to the point cloud surface, and remove laser points whose point-to-surface distances are greater than a predetermined distance threshold;

[0110] The remaining laser points after removal are taken as the laser points in the target's three-dimensional information in the lidar coordinate system.

[0111] Preferably, the point cloud surface is formed by fitting using the Random Sampling Consensus Algorithm (RANSAC).

[0112] This embodiment implements a highly automated method for extrinsic parameter calibration of LiDAR and cameras applicable to different scenarios. The implemented LiDAR point cloud target detection function can automatically complete the detection without relying on manual annotation. The grid search method is applicable to situations where the initial values ​​of extrinsic parameters have certain deviations. Compared with the solution method based on corner perspective n-point projection, this scheme obtains the second extrinsic parameter through direct optimization, which can eliminate the step of fitting corner points, and more observation points participate in the optimization, thus reducing the calculation time while ensuring optimization accuracy.

[0113] The target-based lidar and camera calibration system provided by the present invention will be described below. The target-based lidar and camera calibration system described below and the target-based lidar and camera calibration method described above can be referred to in correspondence.

[0114] Figure 2 A schematic diagram of a target-based lidar and camera calibration system is provided for this invention, as shown below. Figure 2 As shown, the present invention also provides a target-based lidar and camera calibration system, the system comprising:

[0115] The acquisition module is used to acquire lidar point cloud data and camera image data of the target;

[0116] The point cloud filtering module is used to filter point cloud data and obtain the three-dimensional information of the target in the lidar coordinate system.

[0117] The image conversion module is used to obtain the target's three-dimensional information in the camera coordinate system based on image data;

[0118] The conversion module is used to convert two sets of target 3D information in the lidar coordinate system and the camera coordinate system to the same coordinate system based on multiple preset external parameters for target and camera conversion.

[0119] The calculation module is used to calculate the overlap of the two sets of target 3D information after conversion and obtain the first extrinsic parameter corresponding to the maximum overlap.

[0120] This embodiment implements a highly automated method for calibrating the extrinsic parameters of LiDAR and cameras, applicable to different scenarios.

[0121] Figure 3 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a target-based lidar and camera calibration method, the method including:

[0122] Acquire lidar point cloud data and camera image data of the target;

[0123] The point cloud data is filtered to obtain the target's three-dimensional information in the lidar coordinate system;

[0124] Based on the image data, obtain the target's three-dimensional information in the camera coordinate system;

[0125] Based on the preset multiple sets of external parameters for the conversion between the target and the camera, the two sets of three-dimensional target information in the lidar coordinate system and the camera coordinate system are converted to the same coordinate system;

[0126] Calculate the overlap of the two sets of target 3D information after conversion, and obtain the first extrinsic parameter corresponding to the maximum overlap.

[0127] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the target-based lidar and camera calibration method provided by the above methods, the method comprising:

[0129] Acquire lidar point cloud data and camera image data of the target;

[0130] The point cloud data is filtered to obtain the target's three-dimensional information in the lidar coordinate system;

[0131] Based on the image data, obtain the target's three-dimensional information in the camera coordinate system;

[0132] Based on the preset multiple sets of external parameters for the conversion between the target and the camera, the two sets of three-dimensional target information in the lidar coordinate system and the camera coordinate system are converted to the same coordinate system;

[0133] Calculate the overlap of the two sets of target 3D information after conversion, and obtain the first extrinsic parameter corresponding to the maximum overlap.

[0134] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the target-based lidar and camera calibration methods provided above, the methods comprising:

[0135] Acquire lidar point cloud data and camera image data of the target;

[0136] The point cloud data is filtered to obtain the target's three-dimensional information in the lidar coordinate system;

[0137] Based on the image data, obtain the target's three-dimensional information in the camera coordinate system;

[0138] Based on the preset multiple sets of external parameters for the conversion between the target and the camera, the two sets of three-dimensional target information in the lidar coordinate system and the camera coordinate system are converted to the same coordinate system;

[0139] Calculate the overlap of the two sets of target 3D information after conversion, and obtain the first extrinsic parameter corresponding to the maximum overlap.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target-based lidar and camera calibration method, characterized in that, The method includes: Acquire lidar point cloud data and camera image data of the target; The point cloud data is filtered to obtain the target's three-dimensional information in the lidar coordinate system; Based on the image data, obtain the target's three-dimensional information in the camera coordinate system; Based on the preset multiple sets of external parameters for the conversion between the target and the camera, the two sets of three-dimensional target information in the lidar coordinate system and the camera coordinate system are converted to the same coordinate system; Calculate the overlap of the two sets of target 3D information after conversion, and obtain the first extrinsic parameter corresponding to the maximum overlap. Based on the first extrinsic parameter and the target's three-dimensional information in the camera coordinate system, the first pose of the target in the lidar coordinate system is obtained. Based on the first pose, determine the center point of the target; Based on the target's external dimensions and the center point, the target's three-dimensional bounding box is obtained; Obtain each laser point in the three-dimensional information of the target in the lidar coordinate system, and calculate the sum of the distances from all the laser points to the three-dimensional bounding box; The first extrinsic parameter is optimized by minimizing the sum of the distances, thus forming the second extrinsic parameter.

2. The target-based lidar and camera calibration method according to claim 1, characterized in that, Obtaining each laser point in the three-dimensional information of the target in the lidar coordinate system includes: A screening sphere is formed with the center point of the target as the center and a preset first radius; The laser points within the screening sphere are used as the laser points in the target's three-dimensional information in the lidar coordinate system.

3. The target-based lidar and camera calibration method according to claim 1 or 2, characterized in that, Obtaining each laser point in the three-dimensional information of the target in the lidar coordinate system includes: For the target, all laser points in the three-dimensional information of the target under the lidar coordinate system are fitted to form a point cloud surface; Calculate the point-to-surface distance from all laser points in the target's three-dimensional information in the lidar coordinate system to the point cloud surface, and remove laser points whose point-to-surface distance is greater than a predetermined distance threshold; The remaining laser points after removal are taken as the laser points in the target's three-dimensional information in the lidar coordinate system.

4. The target-based lidar and camera calibration method according to any one of claims 1-3, characterized in that, Acquire lidar point cloud data and camera image data of the target, including: Multiple targets are set within the common field of view of the lidar and the camera; The camera and the lidar are fixed as a whole, and the whole of the camera and lidar is allowed to move relative to the target to acquire multi-frame point cloud data and image data.

5. The target-based lidar and camera calibration method according to claim 4, characterized in that, Calculate the overlap between the two sets of target 3D information after conversion, and obtain the first extrinsic parameter corresponding to the maximum overlap, including: Based on the timestamp information of the multi-frame point cloud data and image data, the multi-frame point cloud data and image data are paired according to time. Based on the paired point cloud data and image data, the overlap of the two sets of target 3D information after conversion in each frame is calculated respectively. Calculate the sum of overlap of all frames, and obtain the extrinsic parameter with the largest sum of overlap as the first extrinsic parameter.

6. The target-based lidar and camera calibration method according to claim 1, characterized in that, The point cloud data is filtered to obtain the target's three-dimensional information in the lidar coordinate system, including: Based on the point cloud data, obtain the corresponding two-dimensional depth map; Based on the depth map, obtain the angle difference information between each pixel and its neighboring pixels; Based on the angle difference information, the point cloud data is clustered into several first point cloud clusters through connected component analysis; The feature information of the first point cloud cluster is described by descriptors to obtain several feature description vectors for the several first point cloud clusters; The aforementioned feature description vectors are matched with the predetermined target point cloud feature vectors to obtain the first successfully matched point cloud cluster, which serves as the target's three-dimensional information in the lidar coordinate system.

7. The target-based lidar and camera calibration method according to claim 6, characterized in that, Calculate the overlap between the two sets of target 3D information after conversion, including: Based on the first successfully matched cloud cluster, obtain multiple corresponding points on the depth map; Based on the target's three-dimensional information in the camera coordinate system, the center point of the target in the camera coordinate system is obtained. Based on the multiple sets of external parameters, the center point of the target in the camera coordinate system is converted into multiple center points of the target in the lidar coordinate system; Based on multiple center points of the target in the lidar coordinate system, the number of corresponding points within the target boundary range in the depth map is counted and used as the overlap degree of the two sets of target 3D information after conversion.

8. A target-based lidar and camera calibration system, characterized in that, The system includes: The acquisition module is used to acquire lidar point cloud data and camera image data of the target; The point cloud filtering module is used to filter the point cloud data and obtain the target's three-dimensional information in the lidar coordinate system. An image conversion module is used to obtain the target's three-dimensional information in the camera coordinate system based on the image data. The conversion module is used to convert the two sets of three-dimensional target information in the lidar coordinate system and the camera coordinate system to the same coordinate system based on multiple preset external parameters of the target and the camera. The calculation module is used to calculate the overlap degree of the two sets of target 3D information after conversion, obtain the first extrinsic parameter corresponding to the maximum overlap degree, obtain the first pose of the target in the lidar coordinate system based on the first extrinsic parameter and the target 3D information of the target in the camera coordinate system, determine the center point of the target based on the first pose, obtain the 3D bounding box of the target based on the target's external dimensions and the center point, obtain each laser point in the target 3D information of the target in the lidar coordinate system, calculate the sum of the distances from all the laser points to the 3D bounding box, and optimize the first extrinsic parameter with the minimization of the sum of the distances as the optimization direction to form the second extrinsic parameter.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the target-based lidar and camera calibration method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the target-based lidar and camera calibration method as described in any one of claims 1-7.

Images