A laser radar and camera joint calibration method and system

Abstract

The present invention belongs to the technical field of sensor joint calibration, and more specifically, relates to a method for extracting corner points of a calibration plate in laser radar point cloud data, including: extracting edge points of a calibration plate in point cloud data; clustering edge points, and screening the obtained clusters to obtain initial clusters, recorded as potential calibration plate objects; denoising the potential calibration plate objects to obtain denoised potential calibration plate objects; projecting the denoised potential calibration plate objects to a plane, and further screening the initial clusters in combination with the actual area of ​​the calibration plate to obtain target clusters; performing linear fitting on the edges of the target clusters through a linear fitting algorithm to obtain multiple edge lines, solving the intersection of the multiple edge lines, and obtaining the corner points of the calibration plate. The corner point extraction of the present invention is more accurate, making the joint calibration of the laser radar and the camera more reliable.

Description

Technical Field

[0001] The present invention belongs to the technical field of sensor joint calibration, and more specifically, relates to a joint calibration method and system for a laser radar and a camera. Background Art

[0002] An important constraint on the development of robotics and autonomous driving technology is the ability to perceive the environment, usually through sensors. At present, the sensors that perceive environmental information mainly include cameras and lidar.

[0003] The camera uses pictures or video streams to perform two-dimensional imaging of the environment to obtain continuous color, lighting, and semantic information, and then obtains information that represents the environment through corresponding algorithm processing. Its technology is relatively mature and low in cost. However, camera imaging has high requirements for lighting conditions. An environment that is too bright or too dark will seriously affect the results of the algorithm. In addition, it is difficult to accurately measure the environment due to the fact that the imaging principle is insensitive to scale information. LiDAR perceives the environment by emitting a laser beam, then capturing the signal (echo) reflected by the object, processing the reflected signal and solving the ranging result. It has high measurement accuracy. At the same time, since LiDAR is an active sensing sensor, the emitted laser beam can be in a carefully selected frequency band and has good anti-interference ability. Therefore, it can be used normally in environments with drastic changes in light intensity, excessive brightness or no light. However, LiDAR is not sensitive to environments with similar structures, and point cloud data is generally sparse and discontinuous (general mechanical LiDAR). At present, the cost of both mechanical LiDAR and solid-state LiDAR is high. For the above reasons, the solution currently adopted by autonomous driving or robots is to use a combination of cameras and lidars to fuse and complement their data to obtain more accurate environmental perception results. The fusion of the two data requires knowing the exact positional relationship between the camera and the lidar, that is, calibration, which is crucial to data fusion and subsequent perception results.

[0004] The calibration of cameras and lidars can usually be divided into two categories: based on calibration objects and object-based calibration objects. For application scenarios that pursue accuracy, the method based on calibration objects is still the mainstream at this stage. However, most of the current methods based on calibration objects cannot complete the calibration process adaptively and fully automatically, and still require manual intervention. A joint calibration method for a laser radar and a camera provided in the prior art, when a target plate is located in the common field of view of the laser radar and the camera, the laser radar and the camera respectively collect calibration scene data to obtain point cloud data and image data, wherein each target plate has at least three straight edges and at least three target plate corner points, and any two adjacent straight edges intersect at the corresponding target plate corner point; extract the point cloud corner points corresponding to each target plate corner point on the target plate in the point cloud data to obtain the three-dimensional coordinates of each target plate corner point; extract the image corner points corresponding to each target plate corner point on the target plate in the image data to obtain the two-dimensional coordinates of each target plate corner point; and solve the point pair data composed of the three-dimensional coordinates and the two-dimensional coordinates of each target plate corner point through the PnP (multi-point perspective) method to obtain the external parameters between the laser radar and the camera. However, such a scheme has low extraction accuracy for corner points in complex environments, especially for mechanical scanning laser radars, which leads to insufficient final calibration accuracy, and ultimately affects the environmental perception capability of the sensor system. Summary of the invention

[0005] In order to overcome at least one defect in the above-mentioned prior art, the present invention provides a method for extracting corner points of a calibration plate in lidar point cloud data and a joint calibration method of a lidar and a camera, which establishes a noise model to remove the ranging error of the point cloud data, then fits the plane after backfilling the points included in the cluster, and further screens the points through two indicators: plane fitting degree and area occupied by the points within the area; it also optimizes edge extraction based on reflection intensity, thereby realizing automatic screening of calibration objects in complex environments, with good adaptability and high precision.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for extracting corner points of a calibration plate in laser radar point cloud data is provided, comprising the following steps:

[0008] Extract edge points of the calibration plate from the point cloud data;

[0009] Cluster the edge points and screen the obtained clusters to obtain initial clusters, which are recorded as potential calibration plate objects;

[0010] Denoising the potential calibration plate object to obtain a denoised potential calibration plate object;

[0011] The potential calibration plate objects after denoising are projected onto a plane, and the initial clusters are further screened based on the actual area of ​​the calibration plate to obtain the target clusters.

[0012] The edges of the target cluster are fitted with a straight line through a straight line fitting algorithm to obtain multiple edge lines, and the intersection points of the multiple edge lines are solved to obtain the corner points of the calibration plate.

[0013] In this scheme, when extracting the corner points of the calibration plate in the point cloud data, the potential calibration plate objects are denoised and the denoised potential calibration plate objects are further screened according to the actual area of ​​the calibration plate, so as to obtain more accurate calibration plate corner points later; this can further improve the calibration accuracy of the lidar and camera.

[0014] Preferably, the above-mentioned step of extracting edge points of the calibration plate in the point cloud data specifically includes the following steps:

[0015] Adjust the point cloud order to row-first;

[0016] Record the number of points in each row of the current frame;

[0017] All points are classified based on the single-point smoothness evaluation amount, all left edge points and right edge points are obtained, and an edge point set is formed.

[0018] Preferably, the above-mentioned clustering of edge points is specifically performed by clustering in the edge point set through a single connection algorithm, and each cluster stores at least the following information: the axis in the Cartesian coordinate system of the lidar, the maximum and minimum values ​​between the axes, the index interval of the points on each row, and the coordinates of all points in the cluster.

[0019] Preferably, the above-mentioned laser radar is a mechanical rotating laser radar with repeated scanning, and the mutation is updated when clustering is performed through a single connection algorithm to obtain the index interval of the points on each row.

[0020] Preferably, the above-mentioned screening of the obtained clusters to obtain the initial clusters specifically comprises the following steps: obtaining each cluster bounding box by the maximum and minimum values ​​on the x-axis, y-axis and z-axis in the Cartesian coordinate system of the laser radar, and performing preliminary screening according to the cluster bounding box;

[0021] For the clusters that have been initially screened, fill in all the original points according to the index interval of the points on each row;

[0022] Perform plane fitting for each cluster, and obtain the three-dimensional plane normal vector of the calibration plate in the LiDAR Cartesian coordinate system and all the local points contained in the plane.

[0023] The clusters whose in-office point ratio is greater than the first set threshold are screened to obtain the initial clusters.

[0024] Preferably, the above-mentioned denoising of the potential calibration plate object is specifically to denoise the distance measurement result of the midpoint of the potential calibration plate object by using the three-dimensional plane normal vector, and then calculate the coordinates of the denoised point.

[0025] Preferably, after the noise reduction is performed on the potential calibration plate object, the following steps are further performed: for each potential calibration plate object, all points are binarized based on the reflection intensity;

[0026] The edge points of potential calibration plate objects are extracted based on the binarization results.

[0027] Preferably, the above-mentioned projecting the potential calibration plate object after noise reduction onto a plane, and further screening the initial clusters in combination with the actual area of ​​the calibration plate to obtain the target clusters specifically comprises the following steps:

[0028] Find the normal vector of a 3D plane Axis-angle transformation vector to the z-axis Transform vectors according to axis angles Find the corresponding three-dimensional rotation matrix R;

[0029] The translation vector is obtained by taking the mean of all the internal points in the potential calibration plate object. Using the three-dimensional rotation matrix R and translation vector Composing the projection transformation matrix Where R T Represented as the transposed matrix of R, all the internal points in the potential calibration plate object are projected onto the two-dimensional plane; the convex hull of the internal points of the internal points is calculated by the convex hull algorithm And find the convex hull Area Compare Area The initial clusters are further screened based on the actual area of ​​the calibration plate to obtain the target clusters.

[0030] Preferably, the method further includes optimizing the obtained corner points of the calibration plate, which specifically includes the following steps:

[0031] Use the actual size of the calibration plate to process the corner points of the obtained calibration plate:

[0032]

[0033]

[0034] Among them, the matrix is a two-dimensional transformation matrix that satisfies the Lie group SE(2), a 00 、a 01 、a 10 、a 11 , b0, b1 are elements of matrix T, matrix T *is the optimization result of matrix T, c * is the optimized corner point, c i is the corner point of the calibration plate obtained, is the real corner point matrix template constructed according to the size information of the calibration plate, and i is the serial number of the corner point currently being processed;

[0035] Using the projection transformation matrix Convert the optimized corner points into three-dimensional corner point coordinates.

[0036] A joint calibration method of a laser radar and a camera is also provided, comprising the following steps:

[0037] Extract the calibration plates from the lidar point cloud data and camera image data respectively;

[0038] The calibration plate corner points in the laser radar point cloud data are extracted by using the calibration plate corner point extraction method in the laser radar point cloud data;

[0039] Extract the corner points of the calibration plate from the camera image data;

[0040] The joint calibration of the lidar and the camera is obtained by using the corner points of the calibration plate in the point cloud data and the corner points of the calibration plate in the image data through the PnP algorithm.

[0041] Compared with the prior art, the beneficial effects are:

[0042] 1) After the initial screening, all the original points of the cluster are backfilled and plane fitting is performed, and screening is performed according to the results of plane fitting to achieve automatic extraction of calibration plates in complex environments;

[0043] 2) De-noise the potential calibration plate objects to adapt to the sparse point cloud data and improve the accuracy of corner point extraction;

[0044] 3) The edge points of potential calibration plate objects are extracted using a method based on reflection intensity binarization to further improve the reliability of corner point extraction;

[0045] 4) Optimize the obtained corner points of the calibration plate based on the actual size of the calibration plate to improve the overall calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of extracting a calibration plate from point cloud data in the method for extracting corner points of a calibration plate from laser radar point cloud data in Example 1 of the present invention;

[0047] Figure 2 is a schematic diagram of multiple edge lines and corresponding corner points obtained by fitting the calibration plate corner point extraction method in the laser radar point cloud data in Example 1 of the present invention;

[0048] Figure 3 It is a schematic diagram of the calibration plate edge and corresponding corner points obtained by optimizing the obtained calibration plate corner points in the calibration plate corner point extraction method in the laser radar point cloud data in Example 1 of the present invention;

[0049] Figure 4 It is a schematic diagram of extracting edge points of potential calibration plate objects based on binarization results in the calibration plate corner point extraction method in laser radar point cloud data in embodiment 2 of the present invention;

[0050] Figure 5 is a schematic flow chart of a joint calibration method of a laser radar and a camera in Embodiment 3 of the present invention;

[0051] Figure 6 is a schematic block diagram of the overall connection of the joint calibration system of the laser radar and the camera in Example 3 of the present invention;

[0052] Figure 7 is a schematic block diagram of the connection of an image data system of a joint calibration system of a laser radar and a camera in Embodiment 3 of the present invention;

[0053] Figure 8 It is a schematic block diagram of the connection of the point cloud data system of the joint calibration system of the laser radar and the camera in Example 3 of the present invention. DETAILED DESCRIPTION

[0054] The drawings are only for illustrative purposes and should not be construed as limiting the present invention. To better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the drawings. The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present invention.

[0055] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "long", "short" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limitations on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0056] The technical solution of the present invention is further described in detail below through specific embodiments and in conjunction with the accompanying drawings:

[0057] Embodiment 1:

[0058] like Figures 1 to 3 The first embodiment of the method for extracting corner points of a calibration plate in laser radar point cloud data is shown, and includes the following steps:

[0059] S1: Extract edge points of the calibration plate in the point cloud data;

[0060] S2: Cluster the edge points and screen the obtained clusters to obtain initial clusters, which are recorded as potential calibration plate objects;

[0061] S3: Denoising the potential calibration plate object to obtain the denoised potential calibration plate object;

[0062] S4: Project the denoised potential calibration plate object onto a plane, and further screen the initial clusters based on the actual area of ​​the calibration plate to obtain the target clusters;

[0063] S5: A straight line fitting algorithm is used to perform straight line fitting on the edge of the target cluster to obtain multiple edge lines, and the intersection points of the multiple edge lines are solved to obtain the corner points of the calibration plate.

[0064] It should be noted that the size of the calibration plate is a known quantity, which can be measured before calibration, which is well known to those skilled in the art; the calibration plate is preferably supported by a transparent structure, which can avoid unnecessary reflection of the lidar signal.

[0065] In addition, the order of each step in this embodiment is for reference only and cannot be understood as a limitation of this solution. Those skilled in the art can make reasonable changes to the order to achieve the same function as this embodiment.

[0066] In addition, before extracting the edge of the calibration plate in the point cloud data, the calibration plate in the point cloud data should also be identified, which is something that is easy to think of.

[0067] In step S1 of this embodiment, extracting edge points of the calibration plate in the point cloud data specifically includes the following steps:

[0068] S11: Adjust the point cloud order to be row-first;

[0069] S12: Record the number of points in each row of the current frame Where row is the row identifier;

[0070] S13: Classify all points based on the single-point smoothness evaluation amount, obtain all left edge points and right edge points, and form an edge point set; specifically, the smoothness evaluation amount is and in Characterizes point p iThe smoothing of the neighborhood K on the left and the neighborhood K after n points on the right, K and n are the number of points. In a specific embodiment, select and based on and Establish the following judgment relationship:

[0071] when and When both are greater than the thickness of the calibration plate, it is considered that point p i To highlight the point;

[0072] when Greater than the thickness of the calibration plate but When the thickness of the point p is less than the calibration plate thickness, it is considered that point p i is the left edge point;

[0073] when Greater than the thickness of the calibration plate but When the thickness of the point p is less than the calibration plate thickness, it is considered that point p i+n is the right edge point;

[0074] when and When the thickness of the calibration plate is less than that of the point p i is a non-edge point;

[0075] Perform the above judgment on all points, find all left edge points and right edge points, and put the left edge points and right edge points into the edge point set

[0076] The thickness of the calibration plate is a known parameter, which can be known when it is used, and this is well known to those skilled in the art.

[0077] In step S2 of this embodiment, the edge points are clustered by using a single connection algorithm to cluster the edge points. Clustering is performed in , and each cluster stores at least the following information: the axis in the LiDAR Cartesian coordinate system, the maximum and minimum values ​​between the axes, the index interval of the points on each row, and the coordinates of all points in the cluster; among the parameters of the single connection algorithm, the neighboring threshold τ can be set to one-quarter of the diagonal of the calibration plate. Of course, clustering can also be performed using a multi-connection algorithm or DBSCAN (density-based clustering algorithm) or other clustering algorithms, which are not limited here.

[0078] The point cloud data in this embodiment is obtained by a mechanical rotating laser radar that repeatedly scans. The index of each point on each row always starts from 0 in the front and increases clockwise to the maximum value near 0. At this time, the mutation is updated when the index interval of each point on each row is obtained by clustering through the single connection algorithm. In a specific implementation, an additional variable crossed is saved in each cluster to record the current index interval [v min ,v max ] Whether it crosses the mutation area in front, the default is False, that is, it does not cross the mutation area. At this time, the index interval is still linear. When the new index value v is taken, if the variable crossed = True, it means that the index interval has crossed the mutation area in front, and the current index value v has exceeded the number of current row points. half of Then through Get the adjusted index value v * ,like, Then keep v * =v; the adjusted index value is v * The following situations may occur:

[0079] 1) Adjusted index value v * Subtract the minimum value v of the current index interval min After that, it is still greater than the number of points in the current row S [row] half of This means that the original v has passed halfway and the original index interval [v min ,v max The maximum value v of min , minimum value v min , which means that the original v and the original index interval both contain index 0, so it is necessary to mark crossed = True and update the minimum value v of the index interval min =min(v * ,v min );

[0080] 2) Adjusted index value v * Subtract the minimum value v of the current index interval min After that, it is less than the number of points in the current row S [row] half of This means that the original v is less than halfway, but the original index interval [v min ,v max ] has exceeded half of its maximum and minimum values, which means that the original v and the original index interval both include index 0, so we need to mark crossed = True and update the minimum value v of the index interval. min =v min -S[row] 、The maximum value of the index interval v max =max(v * ,v max -S [row] );

[0081] If the adjusted index value is v * If it is not the case, then the adjusted index value is v * and the original index interval [v min ,v max ] does not cross the mutation zone, so it only needs to be compared linearly. Of course, it is worth noting that if the point cloud data is not obtained by mechanical laser radar, but by solid-state laser radar, this step is not necessary.

[0082] In step S2 of this embodiment, screening the obtained clusters to obtain initial clusters specifically includes the following steps:

[0083] S21: Obtain each cluster bounding box through the maximum and minimum values ​​on the x-axis, y-axis and z-axis in the LiDAR Cartesian coordinate system, and perform preliminary screening based on the cluster bounding boxes; specifically, exclude clusters that do not meet the volume range. In this embodiment, clusters with a volume smaller than the actual volume of the calibration plate or larger than 4 times the actual volume of the calibration plate are excluded. Of course, the exclusion conditions can be set according to actual conditions and are not limited here; S22: Fill in all the original points in the clusters after preliminary screening according to the index interval of the points on each row; S23: Perform plane fitting on each cluster, and obtain the three-dimensional plane normal vector of the fitted calibration plate in the LiDAR Cartesian coordinate system. and all the internal points contained in the plane;

[0084] S24: Filter clusters with a ratio of in-office points greater than 80% to obtain initial clusters. The ratio of in-office points greater than 80% is only a reference threshold, and the first set threshold can be changed as needed during the specific implementation process.

[0085] Since the main noise source of laser radar is ranging error, in this embodiment, the noise reduction of the potential calibration plate object is specifically to reduce the noise of the ranging result of the midpoint of the potential calibration plate object, and then calculate the coordinates of the noise-reduced point. Specifically, suppose any point in the Cartesian coordinate system of the point cloud data C The original measurement p = (x, y, z) is usually described in spherical coordinates S p=(α,ω,R), where is the azimuth, is the lift angle, R is the distance measurement result, x, y, z are the values ​​of the point in the Cartesian coordinate system of the point cloud data; it is considered that the distance measurement result in is the true value, δR is the error, then according to the three-dimensional plane normal vector Find the true value of the distance measurement result Therefore, the coordinates of the denoised points can be obtained

[0086] C p * =x * ,y * ,z * The coordinates of are:

[0087]

[0088] In step S4 of this embodiment, the potential calibration plate object after noise reduction is projected onto a plane, and the initial clusters are further screened in combination with the actual area of ​​the calibration plate to obtain the target clusters, which specifically includes the following steps:

[0089] S41: Find the normal vector of a three-dimensional plane Axis-angle transformation vector to the z-axis

[0090] S42: Transform vectors according to axis angles Find the corresponding three-dimensional rotation matrix R;

[0091] S43: Calculate the translation vector by averaging the coordinates of all internal points in the potential calibration plate object on the x-axis, y-axis and z-axis respectively.

[0092] S44: Using the three-dimensional rotation matrix R and translation vector Composing the projection transformation matrix Among them, R T Represented as the transposed matrix of R, all local points in the potential calibration plate object are projected onto the two-dimensional plane;

[0093] S45: Find the convex hull of the inner points of the inner points by using the convex hull algorithm And find the convex hull Area

[0094]

[0095] Among them, x, y, and z are the values ​​of the point in the Cartesian coordinate system of the point cloud data. Convex hull contains the number of points, i is the sequence number of the point currently being processed;

[0096] S46: Comparison of area The initial clusters are further screened according to the actual area of ​​the calibration plate to obtain the target clusters; specifically, the initial clusters whose area A is smaller than three quarters of the actual area of ​​the calibration plate or larger than four fifths of the actual area of ​​the calibration plate are removed.

[0097] Since there will be certain errors in straight line fitting and certain errors in corner points, this embodiment also includes step S6:

[0098] S6: Optimize the obtained corner points of the calibration plate;

[0099] Step S6 specifically includes the following steps:

[0100] S61: Process the corner points of the obtained calibration plate using the real size of the calibration plate:

[0101]

[0102]

[0103] Among them, the matrix is a two-dimensional transformation matrix that satisfies the Lie group SE(2), a 00 、a 01 、a 10 、a 11 , b0, b1 are elements of matrix T, matrix T * is the optimization result of matrix T, c * is the optimized corner point, c i is the corner point of the calibration plate obtained, is the real corner point matrix template constructed according to the size information of the calibration plate, and i is the serial number of the corner point currently being processed;

[0104] S62: Using the Projection Transformation Matrix Inverse transformation Convert the optimized corner points into three-dimensional corner point coordinates.

[0105] Among them, this embodiment uses a calibration plate with 7 rows and 6 columns, a total of 42 corner points. Each small grid is a square with a side length of 5 cm. The outer edges of the chessboard are 5 cm wide white edges and 5 cm wide black edges respectively. The lower left corner is taken as the first corner point, and each corner point is arranged in columns first and then rows to obtain a matrix containing the coordinates of all corner points. The final three-dimensional corner point coordinates are

[0106] After obtaining the corner points of the point cloud data calibration plate and the corner points of the image data calibration plate through the above steps and the one-to-one correspondence between the two, the PnP (multi-point perspective imaging) algorithm can be used to solve the joint calibration of the lidar and the camera.

[0107] In this scheme, when extracting the corner points of the calibration plate in the point cloud data, the potential calibration plate objects are denoised and the denoised potential calibration plate objects are further screened according to the actual area of ​​the calibration plate, so as to obtain more accurate calibration plate corner points later; this can further improve the calibration accuracy of the lidar and camera, especially for mechanical scanning lidar with sparse point cloud data.

[0108] Embodiment 2:

[0109] like Figure 4 The second embodiment of the joint calibration method of the laser radar and the camera is shown. The difference between this embodiment and the first embodiment is that after the noise reduction of the potential calibration plate object, the edge extraction based on the laser reflection intensity is also performed in this embodiment, which specifically includes the following steps:

[0110] For each potential calibration plate object, all points are binarized based on the reflection intensity;

[0111] The edge points of potential calibration plate objects are extracted based on the binarization results.

[0112] Specifically, for each potential calibration plate object, the reflection intensity of all points in it is counted and arranged in order from low to high, and the first 20% is taken as the segmentation threshold to binarize all points in the potential calibration plate object, that is, the reflection intensity higher than the segmentation threshold is set to the highest intensity, and the reflection intensity lower than the segmentation threshold is set to the lowest intensity; then traverse each row of points from left to right, for point p i , if point p i-1 Compare points i and point p i+1 The reflection intensity is high, or point p i+1 Compare points i and point p i-1 The reflection intensity of all points is high, then point p is judged as an edge point of the reflection intensity.

[0113] Embodiment 3:

[0114] like Figure 5 An embodiment of a joint calibration method of a laser radar and a camera is shown, comprising the following steps:

[0115] Extract the calibration plates from the lidar point cloud data and camera image data respectively;

[0116] The calibration plate corner points in the laser radar point cloud data are extracted by using the calibration plate corner point extraction method in the laser radar point cloud data in Example 1 or Example 2;

[0117] Extract the corner points of the calibration plate from the camera image data;

[0118] The joint calibration of the lidar and the camera is obtained by using the corner points of the calibration plate in the point cloud data and the corner points of the calibration plate in the image data through the PnP algorithm.

[0119] In a specific implementation, the joint calibration of the laser radar and the camera is obtained by using the PnP algorithm using the calibration plate corner points in the point cloud data and the calibration plate corner points in the image data. Specifically, an iterative algorithm based on RANSAC is used, including the following steps:

[0120] The calibration plate corner points (which are 3D corner points) in the series of point cloud data obtained in the above steps and the calibration plate corner points (which are 2D corner points) in the camera image data form 3D-2D corner point pairs;

[0121] In each iteration, four pairs of 3D-2D corner points are selected and the rough coordinate transformation of the 3D-2D corner point pairs is obtained by solving the EPnP algorithm.

[0122] Project all 3D corner points onto the camera image coordinate system based on the rough coordinate transformation of the 3D-2D corner point pairs, and calculate the reprojection error;

[0123] Separate the inliers and outliers based on the reprojection error, that is, the 3D-2D corner point pairs with errors less than a certain threshold are inliers, and the 3D-2D corner point pairs with errors greater than a certain threshold are outliers;

[0124] For all local points, the EPnP algorithm is used to solve the better coordinate transformation and update the reprojection error. If the solution to the better coordinate transformation fails, the initial result is retained and multiple iterations are performed until the number of times reaches the requirement or the reprojection error is small enough.

[0125] At this point, the optimal coordinate transformation obtained is the optimal transformation relationship from the lidar coordinate system to the camera coordinate system in the current frame data, that is, the calibration of the lidar and the camera is completed.

[0126] Of course, the use of the RANSAC-based iterative algorithm for solving in this embodiment is only a reference implementation method and cannot be understood as a limitation of this solution. Those skilled in the art can certainly use other corresponding algorithms to complete this operation; in addition, the number of three-dimensional-two-dimensional corner point pairs cannot be understood as a limitation.

[0127] In this embodiment, before extracting the calibration plate corner points in the camera image data, the camera image data is distorted and corrected so that the extracted calibration plate corner points can correspond to the calibration plate corner points of the point cloud data, thereby avoiding affecting the calibration.

[0128] In this embodiment, the corner points of the calibration plate in the camera image data can be extracted by jointly obtaining the corner points through the two tools findChessboardCorners() and cornerSubPix() provided in the OpenCV tool library; of course, those skilled in the art can also extract the camera image data in other ways, which are not limited here.

[0129] like Figures 6 to 8 The embodiment of a joint calibration system of a laser radar and a camera is shown, which is used to implement the joint calibration method of the laser radar and the camera, and includes a calibration module and a point cloud data system and an image data system which are both communicatively connected thereto; the point cloud data system includes a laser radar and an edge point extraction module, a clustering screening module, a noise reduction module, a screening module, and a fitting module which are communicatively connected thereto in sequence, the edge point extraction module is communicatively connected to the laser radar, and the fitting module is communicatively connected to the calibration module; the image data system includes a camera and an image corner point extraction module which is communicatively connected to the camera, and the image corner point extraction module is also communicatively connected to the calibration module;

[0130] The laser radar and camera are used to scan the environment of the same calibration plate and obtain point cloud data and image data respectively;

[0131] The edge point extraction module is used to extract the edge points of the calibration plate in the point cloud data;

[0132] The clustering and screening module is used to cluster edge points and screen the obtained clusters to obtain potential calibration plate objects;

[0133] The denoising module is used to denoise the potential calibration plate object;

[0134] The screening module is used to project the potential calibration plate objects after denoising onto a plane, and further screen the initial clusters based on the actual area of ​​the calibration plate to obtain the target clusters;

[0135] The fitting module is used to obtain multiple edge lines by performing straight line fitting on the edge of the target cluster through a straight line fitting algorithm, and to solve the intersection points of the multiple edge lines to obtain the corner points of the calibration plate;

[0136] The image corner point module is used to extract the corner points of the calibration plate in the camera image data;

[0137] The calibration module is used to obtain the corner points of the calibration plate obtained from the point cloud data and the image data, and to obtain the joint calibration of the lidar and the camera using the PnP algorithm.

[0138] The laser radar in this embodiment can be a solid-state laser radar or a mechanical rotating laser radar, which is not limited here.

[0139] The camera in this embodiment may be a visible light grayscale camera, a color camera, or an infrared camera, etc., which is not limited here.

[0140] The number of laser radars and cameras in this embodiment is not limited to one, and can be any number, as long as the laser radars and cameras have a corresponding calibration plate field of view. This is well known to those skilled in the art and is not limited here.

[0141] The present invention is described with reference to the flowcharts or block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each process or block in the flowchart or block diagram, as well as the combination of processes or blocks in the flowchart or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0143] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for extracting corner points of a calibration plate in laser radar point cloud data, characterized in that: The following steps are involved: Extract edge points of the calibration plate from the point cloud data; Cluster the edge points and screen the obtained clusters to obtain initial clusters, which are recorded as potential calibration plate objects; Denoising the potential calibration plate object to obtain a denoised potential calibration plate object; The potential calibration plate objects after denoising are projected onto a plane, and the initial clusters are further screened based on the actual area of ​​the calibration plate to obtain the target clusters. The edges of the target cluster are fitted with a straight line by a straight line fitting algorithm to obtain multiple edge lines, and the intersection points of the multiple edge lines are solved to obtain the corner points of the calibration plate; The step of screening the obtained clusters to obtain the initial clusters specifically includes the following steps: The maximum and minimum values ​​on the x-axis, y-axis and z-axis in the LiDAR Cartesian coordinate system are used to obtain each cluster bounding box, and preliminary screening is performed based on the cluster bounding boxes; For the clusters that have been initially screened, fill in all the original points according to the index interval of the points on each row; Perform plane fitting for each cluster, and obtain the three-dimensional plane normal vector of the calibration plate in the LiDAR Cartesian coordinate system and all the local points contained in the plane. The clusters whose in-office point ratio is greater than the first set threshold are screened to obtain the initial clusters; The process of projecting the denoised potential calibration plate object onto a plane and further screening the initial clusters in combination with the actual area of ​​the calibration plate to obtain the target clusters specifically includes the following steps: Find the axis-angle transformation vector from the normal vector of a three-dimensional plane to the z-axis Transform vectors according to axis angles Find the corresponding three-dimensional rotation matrix R; The translation vector is obtained by taking the mean of all the internal points in the potential calibration plate object. Using the three-dimensional rotation matrix R and translation vector Composing the projection transformation matrix Where R T Represented as the transposed matrix of R, all local points in the potential calibration plate object are projected onto the two-dimensional plane; Find the convex hull of the inner points of the inner points by using the convex hull algorithm And find the convex hull Area Compare Area The initial clusters are further screened based on the actual area of ​​the calibration plate to obtain the target clusters.

2. The method for extracting corner points of a calibration plate in laser radar point cloud data according to claim 1, characterized in that: The step of extracting edge points of the calibration plate from the point cloud data specifically comprises the following steps: Adjust the point cloud order to row-first; Record the number of points in each row of the current frame; All points are classified based on the single-point smoothness evaluation amount, all left edge points and right edge points are obtained, and an edge point set is formed.

3. The method for extracting corner points of a calibration plate in laser radar point cloud data according to claim 2, characterized in that: The clustering of edge points is specifically to cluster the edge point set through a single-link algorithm or a multiple-link algorithm or DBSCAN, and each cluster stores at least the following information: the maximum and minimum values ​​on the x-axis, y-axis and z-axis in the Cartesian coordinate system of the lidar, the index interval of the points on each row, and the coordinates of all points in the cluster.

4. The method for extracting corner points of a calibration plate in laser radar point cloud data according to claim 3, characterized in that: The denoising of the potential calibration plate object specifically comprises using the three-dimensional plane normal vector to denoise the distance measurement result of the midpoint of the potential calibration plate object, and then calculating the coordinates of the denoised point.

5. The method for extracting corner points of a calibration plate in laser radar point cloud data according to claim 4, characterized in that: After denoising the potential calibration plate objects, the following steps are performed: For each potential calibration plate object, all points are binarized based on the reflection intensity; The edge points of potential calibration plate objects are extracted based on the binarization results.

6. The method for extracting corner points of a calibration plate in laser radar point cloud data according to claim 5, characterized in that: The method further includes optimizing the obtained corner points of the calibration plate, which specifically includes the following steps: Use the actual size of the calibration plate to process the corner points of the obtained calibration plate: Among them, the matrix is a two-dimensional transformation matrix that satisfies the Lie group SE(2), a 00 、a 01 、a 10 、a 11 , b0, b1 are elements of matrix T, matrix T * is the optimization result of matrix T, c * is the optimized corner point, c i is the corner point of the calibration plate obtained, is the real corner point matrix template constructed according to the size information of the calibration plate, and i is the serial number of the corner point currently being processed; Using the projection transformation matrix Convert the optimized corner points into three-dimensional corner point coordinates.

7. A joint calibration method for laser radar and camera, characterized in that: The following steps are involved: Extract the calibration plates from the lidar point cloud data and camera image data respectively; Extracting the calibration plate corner points in the laser radar point cloud data by using the calibration plate corner point extraction method in the laser radar point cloud data according to any one of claims 1 to 6; Extract the corner points of the calibration plate from the camera image data; The joint calibration of the lidar and the camera is obtained by using the corner points of the calibration plate in the point cloud data and the corner points of the calibration plate in the image data through the PnP algorithm.

8. The joint calibration method of a laser radar and a camera according to claim 7, characterized in that: The joint calibration of the laser radar and the camera is obtained by using the PnP algorithm using the calibration plate corner points in the point cloud data and the calibration plate corner points in the image data. The RANSAC-based iterative algorithm is specifically used, and includes the following steps: The calibration plate corner points in the point cloud data and the calibration plate corner points in the image data are combined into 3D-2D corner point pairs; several pairs of 3D-2D corner point pairs are selected in each iteration, and the rough coordinate transformation of the 3D-2D corner point pairs is obtained by solving the EPnP algorithm; Projecting all three-dimensional corner points onto the camera image coordinate system according to the rough coordinate transformation of the three-dimensional-two-dimensional corner point pairs, and calculating the reprojection error; Separate inliers and outliers based on reprojection errors; The EPnP algorithm is used to solve the better coordinate transformation for all local points and update the reprojection error. The optimal coordinate transformation is the calibration of the lidar and the camera.

Images