A Point Cloud and Image Registration Method and Device Based on the Alignment of Calibration Board Corner Points

Abstract

The present invention relates to a point cloud and image registration method for calibrating board corner alignment, which includes placing a visible light camera and a lidar in the same position, collecting scene data of a calibration board placed at different preset positions, and obtaining multiple groups of image data and point cloud data of the calibration board; detecting the corner points of the calibration board in the image data, and obtaining the 2D pixel coordinates of the four corner points of the calibration board according to the relative distances between the calibration board corner points and the preset checkerboard endpoints; after segmenting the calibration board point cloud from the point cloud data, projecting it onto a unified plane and then projecting it onto the yoz plane; according to the radar line scanning principle, setting classification conditions to segment the calibration board point cloud coordinates into multiple clustering regions; obtaining four edge points according to the point farthest from the center of the abscissa in each clustering region and another point farthest from this point, and fitting them into four edge lines; calculating the intersection points of the edge lines to obtain the 3D point cloud coordinates, so as to use EPnP and RANSAC with the 2D pixel coordinates to obtain the optimal transformation matrix and complete the point cloud and image registration.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud and image registration, and in particular to a point cloud and image registration method and device based on calibration plate corner point alignment. Background Art

[0002] The development of environmental perception and understanding technologies represented by the field of autonomous driving applications is rapid. In the face of complex and changing scenarios, the use of multi-source optical data fusion can make up for the deficiencies in information expression of single-source optical data and improve the reliability and safety during vehicle operation. Among them, sensors generally mainly include video cameras and lidar. The key task is to design a registration method for multi-sensors to keep the alignment of the same target in different data forms as much as possible. This not only helps to establish a unified multi-source data representation space, but also simplifies the data fusion process.

[0003] The information matching between video sensors and radar sensors is a common problem that needs to be solved in multi-modal data fusion. Only by unifying the multi-source data of each sensor to the same spatial reference through certain technical means can the segmentation and positioning of key elements be achieved more accurately and effectively. The unification of the spatial reference can determine the relative position relationship of each sensor through calibration technology, so as to calculate the exact position of each sensor in a specific coordinate system according to the initial coordinates in the high-precision map. In autonomous driving tasks, the calibration of data is often combined with the positioning system to form a "hand-eye calibration" task.

[0004] Bender et al. used the sensor installation position information and detection sensitivity information, taking the visible light image and the point cloud depth image as the "eye" and the inertial navigation system based on the earth coordinate system as the "hand" for data calibration. Ishikawa et al. extended the data calibration to form a "hand-eye calibration" task with any positioning system, performed feature matching on multi-view visible light images, and used multi-frame matching for lidar to achieve data calibration. After data calibration, due to various reasons such as measurement error, installation error, and perturbation error, there are still offsets between multi-source optical data, which need to be further adjusted through registration. However, the heterogeneity of multi-source optical data makes it difficult to find homogeneous feature points, and the rise of neural networks has well solved this problem, but such models require strong supervision to train a large number of parameters, which limits their application development.

[0005] Currently, most of the commonly used video and radar data fusion methods in the actual application field are based on decision-level fusion, that is, they have low requirements for registration accuracy. Decision-level fusion often requires a good decision-making mechanism to select different detection results. With the rise of feature-level and pixel-level fusion, data registration has become the primary prerequisite for fusion. Existing methods mainly include co-location calibration and feature matching: the former calibrates according to the positional relationship of fixed sensors and requires high calibration accuracy; the latter realizes registration by finding corresponding homologous feature points in two types of data and constructing a transformation matrix, and requires the design of a relatively accurate feature matching algorithm.

[0006] In summary, the existing video and radar data fusion methods have low calibration efficiency, complex feature matching, and poor registration accuracy. Summary of the Invention

[0007] Therefore, the technical problem to be solved by the present invention is to overcome the problems of low registration and calibration efficiency, complex feature matching, and poor registration accuracy of point cloud and image in the prior art.

[0008] To solve the above technical problem, the present invention provides a point cloud and image registration method based on calibration board corner points, including:

[0009] Place the visible light camera and the lidar in the same position, collect the scene data of the calibration board placed at different preset positions, and obtain multiple groups of image data and point cloud data of the calibration board;

[0010] Use the Harris algorithm to detect the corner points of the calibration board in the image data, and obtain the 2D pixel coordinates of the 4 corner points of the calibration board according to the relative distances between the calibration board corner points and the endpoints of the checkerboard;

[0011] Use the pass-through filter to segment the calibration board point cloud from the point cloud data;

[0012] After projecting all the calibration board point cloud coordinates into a preset three-dimensional plane, set the x coordinate to 0, project it onto the yOz plane, and according to the lidar line scanning principle, use the Euclidean distance in the z direction as the classification condition to segment the calibration board point cloud coordinates into multiple clustering regions;

[0013] According to the points farthest from the abscissa center in all clustering regions and another point farthest from this point, obtain 4 edge points corresponding to the lower right edge, upper right edge, lower left edge, and upper left edge, and fit them into 4 edge lines;

[0014] By calculating the intersection points of the edge lines, obtain the 3D point cloud coordinates of the 4 corner points of the calibration board;

[0015] Based on the 2D pixel coordinates and the 3D point cloud coordinates of the four corner points of the calibration board, using EPnP and RANSAC, an optimal transformation matrix from the 3D point cloud coordinates to the 2D pixel coordinates is obtained to realize the reprojection of the 3D point cloud coordinates and complete the registration of the point cloud and the image.

[0016] In an embodiment of the present invention, before detecting the corner points of the calibration board in the image data by using the Harris algorithm, it further includes: calibrating the visible light camera according to multiple groups of image data of the calibration board, and performing distortion correction on the image data to obtain corrected image data.

[0017] In an embodiment of the present invention, the obtaining of the 2D pixel coordinates of the four corner points of the calibration board according to the relative distances between the corner points of the calibration board and the end points of the checkerboard includes:

[0018] Using the Harris algorithm to detect the image data to obtain the corner points of the checkerboard;

[0019] According to the four corner points at the end points of the checkerboard and the two adjacent corner points of each corner point, calculate the distance scales from the corner points of the calibration board and the two adjacent corner points to the end points of the checkerboard in the horizontal and vertical directions respectively;

[0020] Taking the end point of the checkerboard as the origin, and taking the pixels of the end point of the checkerboard and the two adjacent corner points as the unit distances of the x-axis and y-axis respectively, construct a vector coordinate system; according to the vector relationship, obtain the 2D pixel coordinates of the four corner points.

[0021] In an embodiment of the present invention, the distance scales from the corner points of the calibration board and the two adjacent end points to the end points of the checkerboard in the horizontal and vertical directions are expressed as:

[0022] Horizontal x-axis distance scale:

[0023] Vertical y-axis distance scale:

[0024] Wherein, C represents a corner point of the calibration board, the end point of the checkerboard is O, and the triangle CDO forms a right triangle. The two adjacent end points with the end point O are A and B respectively, and the directions of OA and OB are the horizontal and vertical directions respectively. Let x1 and y1 represent the pixel spacings of OA and OB, and record their corresponding actual spacings as X1 and Y1 respectively. Let x and y represent the pixel spacings between O and C in the horizontal and vertical directions, and record their corresponding actual spacings as X and Y respectively.

[0025] In an embodiment of the present invention, the taking the end point of the checkerboard as the origin, and taking the pixels of the end point of the checkerboard and the two adjacent corner points as the unit distances of the x-axis and y-axis respectively, constructing a vector coordinate system; according to the vector relationship, obtaining the 2D pixel coordinates of the four corner points includes:

[0026] Construct a vector coordinate system with the endpoint O as the origin and the pixel pitch as the unit. Then, the vector and the vector are in opposite directions, and the vector and the vector are in opposite directions;

[0027] Let the coordinates of the detected endpoints A, B, and O be (m1, n1), (m2, n2), and (m, n) respectively. Then, there is According to the principle of similar triangles, it can be obtained that:

[0028]

[0029]

[0030] According to the vector relationship the coordinates (c x , c y ) of the calibration board corner point C can be obtained and expressed as:

[0031] c x = m - D x (m1 - m) - D y (m2 - m);

[0032] c y = n - D x (n1 - n) - D y (n2 - n).

[0033] In an embodiment of the present invention, after using straight-through filtering to segment the calibration board point cloud from the point cloud data, it further includes removing noise points through radius filtering.

[0034] In an embodiment of the present invention, after projecting all the calibration board point cloud coordinates into a preset three-dimensional plane, the x coordinates are all set to 0, projected onto the yOz plane, and according to the radar line scanning principle, with the Euclidean distance in the z direction as the classification condition, the calibration board point cloud coordinates are segmented into multiple clustering regions, including:

[0035] Using the RANSAC method and the spatial plane equation Ax + By + Cz + D = 0, estimate the optimal plane of the calibration board point cloud, and project all the calibration board point cloud coordinates into the same plane, expressed as:

[0036]

[0037] where x, y, and z represent the calibration board point cloud coordinates, and x', y', and z' represent the coordinates projected onto the same plane;

[0038] Set the x - coordinates of the projected calibration board point cloud to 0, project it onto the yOz plane, and according to the principle of radar line scanning, use the DBSACN algorithm to segment the calibration board point cloud coordinates into multiple clustering regions based on the Euclidean distance in the z - direction as the classification condition.

[0039] In an embodiment of the present invention, calculate the point farthest from the abscissa center in each region, and another point farthest from this point, obtain 4 edge points corresponding to the lower - right edge, upper - right edge, lower - left edge, and upper - left edge, and fit them into 4 edge lines; by calculating the intersection points of the edge lines, obtain the 3D point cloud coordinates of the 4 corner points of the calibration board, including:

[0040] Calculate the point farthest from the abscissa center and another point farthest from this point for each clustering region respectively;

[0041] For the two points obtained in each clustering region, after arranging them in descending order of the y - coordinate value and then in ascending order of the z - coordinate value;

[0042] Calculate the maximum index and minimum index of the y - coordinate respectively. The points with y - coordinate less than the maximum index belong to the lower - right edge, the points with y - coordinate not less than the maximum index belong to the upper - right edge, the points with y - coordinate less than the minimum index belong to the lower - left edge, and the points with y - coordinate not less than the minimum index belong to the upper - left edge;

[0043] Respectively take the edge points corresponding to the lower - right edge, the upper - right edge, the lower - left edge, and the upper - left edge, and use the RANSAC method to fit to obtain 4 edge lines;

[0044] Calculate the intersection points of the edge lines to obtain the projected coordinates of the calibration board corner points on the yOz plane; substitute them into the spatial plane equation to obtain the 3D point cloud coordinates of the calibration board corner points.

[0045] In an embodiment of the present invention, based on the 2D pixel coordinates and the 3D point cloud coordinates of the 4 corner points of the calibration board, use EPnP and RANSAC to obtain the optimal transformation matrix from the 3D point cloud coordinates to the 2D pixel coordinates, including:

[0046] Use multiple groups of 2D pixel coordinates to form a 2D matrix, and the 3D point cloud coordinates of the same number of groups to form a 3D matrix, and use the EPnP algorithm to obtain the initial transformation matrix from the 2D matrix to the 3D matrix;

[0047] Use the RANSAC method to iterate the initial transformation matrix to obtain the optimal transformation matrix.

[0048] The embodiment of the present invention also provides a point cloud and image registration device based on the alignment of calibration board corner points, including:

[0049] A visible - light camera, placed at a preset position, for collecting image data of the calibration board within the field of view;

[0050] A lidar, placed at the same position as the visible light camera, is used to collect the point cloud data of the calibration board within the field of view;

[0051] A host computer, communicatively connected to the visible light camera and the lidar, is used to obtain the image data and the point cloud data, obtain the 2D pixel coordinates and 3D point cloud coordinates of the four corner points of the calibration board, and obtain the optimal transformation matrix from the 3D point cloud coordinates to the 2D pixel coordinates, so as to realize the conversion from the 3D point cloud coordinates to the 2D pixel coordinates and complete the registration.

[0052] The above technical solution of the present invention has the following advantages compared with the prior art:

[0053] The point cloud and image registration method for aligning the corner points of the calibration board according to the present invention extracts reliable calibration board corner points in the visible light image and the radar point cloud to realize the registration of the video and radar spaces; utilizes the line scanning characteristic of the lidar, divides the calibration board point cloud by clustering according to the height threshold, calculates the 3D point cloud coordinates of the calibration board through the edge points fitted by the clustering region, which is simple, convenient and accurate in coordinates; obtains the 2D pixel coordinates of the calibration board according to the known sizes and relative relationships of the calibration board and the checkerboard, which better corresponds one by one with the point cloud coordinates; after obtaining the 3D point cloud coordinates and 2D pixel coordinates, combines EPnP and RANSAC to obtain the transformation matrix for converting the point cloud coordinates to the pixel coordinates, thereby realizing the registration of the point cloud and the image. The implementation is simple, does not require an accurate registration algorithm, has a high registration accuracy, and has high robustness and applicability. Description of the Drawings

[0054] In order to make the content of the present invention easier to be clearly understood, the following further details the present invention according to the specific embodiments of the present invention in conjunction with the drawings, where

[0055] Figure 1 is the flowchart of the steps of the point cloud and image registration method based on aligning the corner points of the calibration board provided by the present invention;

[0056] Figure 2 is the flowchart of the steps of the method for calculating the 2D pixel coordinates of the calibration board corner points provided by the present invention;

[0057] Figure 3 is the schematic diagram of the relationship of the calibration board corner points provided by the present invention;

[0058] Figure 4 is the schematic diagram of the segmentation result of the calibration board point cloud clustering region provided by the present invention;

[0059] Figure 5 is the schematic diagram of the contour points extracted by the present invention;

[0060] Figure 6 It is a schematic diagram of the fitted edge line provided by the present invention;

[0061] Figure 7 It is a schematic diagram of the fitting result of the calibration plate corner points provided by the present invention. Specific embodiments

[0062] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the illustrated embodiments are not intended to limit the present invention.

[0063] Referring to Figure 1 As shown, the flowchart of the steps of the point cloud and image registration method based on calibration plate corner points of the present invention specifically includes:

[0064] S1: Place the visible light camera and the lidar in the same position, collect the scene data of the calibration plate placed at different preset positions, and obtain multiple groups of image data and point cloud data of the calibration plate;

[0065] S2: Use the Harris algorithm to detect the calibration plate corner points of the image data, and obtain the 2D pixel coordinates of the 4 corner points of the calibration plate according to the relative distances between the calibration plate corner points and the preset checkerboard endpoints;

[0066] S3: Use the pass-through filter to segment the calibration plate point cloud from the point cloud data; and remove the noise points through radius filtering;

[0067] S4: After projecting all the calibration plate point cloud coordinates onto a unified plane, set the x coordinates all to 0, project them onto the yOz plane, and according to the lidar line scanning principle, use the Euclidean distance in the z direction as the classification condition to segment the calibration plate point cloud coordinates into multiple clustering regions;

[0068] Use the RANSAC method and the spatial plane equation Ax + By + Cz + D = 0 to estimate the optimal plane of the calibration plate point cloud, and project all the calibration plate point cloud coordinates onto a unified plane, expressed as:

[0069]

[0070] Among them, x, y, z represent the calibration plate point cloud coordinates, and x', y', z' represent the coordinates projected onto the same plane;

[0071] Set the x coordinates of the projected calibration plate point cloud all to 0, project them onto the yOz plane, and according to the lidar line scanning principle, use the Euclidean distance in the z direction as the classification condition, and use the DBSACN algorithm to segment the calibration plate point cloud coordinates into multiple clustering regions; the segmentation result is shown in Figure 4 As shown;

[0072] S5: Calculate the point farthest from the abscissa center in each region, and another point farthest from this point, obtain 4 edge points corresponding to the lower-right, upper-right, lower-left, and upper-left edges, and fit them into 4 edge lines;

[0073] S6: By calculating the intersection points of the edge lines, obtain the 3D point cloud coordinates of the 4 corner points of the calibration board;

[0074] Calculate the intersection points of the edge lines to obtain the projection coordinates of the calibration board corner points on the yOz plane; Substitute them into the spatial plane equation to obtain the 3D point cloud coordinates of the calibration board corner points;

[0075] S7: Based on the 2D pixel coordinates and the 3D point cloud coordinates of the 4 corner points of the calibration board, use EPnP and RANSAC to obtain the optimal transformation matrix from the 3D point cloud coordinates to the 2D pixel coordinates, realize the reprojection of the 3D point cloud coordinates, and complete the registration of the point cloud and the image.

[0076] Use multiple groups of 2D pixel coordinates to form a 2D matrix, and the 3D point cloud coordinates of the same number of groups to form a 3D matrix. Use the EPnP algorithm to obtain the initial transformation matrix from the 2D matrix to the 3D matrix; Use the RANSAC method to iterate the initial transformation matrix to obtain the optimal transformation matrix.

[0077] Specifically, referring to Figure 2 As shown, in step S2, it includes:

[0078] S21: According to multiple groups of image data of the calibration board, complete the calibration of the visible light camera, and perform distortion removal processing on the image data to obtain corrected image data;

[0079] S22: Use the Harris algorithm to detect the checkerboard corner points of the corrected image data;

[0080] S23: Take the 4 corner points detected at the endpoints of the checkerboard, and two adjacent corner points of each corner point, and calculate the distance scales from the calibration board corner points and the two adjacent corner points to the checkerboard endpoints in the horizontal and vertical directions respectively;

[0081] Specifically, referring to Figure 3 As shown, define a checkerboard endpoint as O, the calibration board corner point as C, and the triangle CDO forms a right triangle. The two adjacent corner points of the endpoint O are A and B respectively, and the directions of OA and OB are the horizontal and vertical directions respectively. Let x1 and y1 be the pixel spacings of OA and OB, and record their corresponding real spacings as X1 and Y1 respectively. Let x and y be the horizontal and vertical pixel spacings between O and C, and record their corresponding real spacings as X and Y respectively. The distance scale relationship can be obtained:

[0082] Horizontal x-axis distance scale:

[0083] Longitudinal y-axis distance scale:

[0084] S24: Taking the end point O of the checkerboard grid as the origin, and taking the pixels of the end point of the checkerboard grid and two adjacent corner points as the unit distances of the x-axis and y-axis respectively, construct a vector coordinate system; according to the vector relationship, obtain the 2D pixel coordinates of the four corner points.

[0085] It can be known that the vector and the vector are in opposite directions, and the vector and the vector are in opposite directions. Assuming that the coordinates of the detected points A, B, and O are (m1, n1), (m2, n2), and (m, n) respectively, then there are According to the principle of similar triangles, it can be obtained that:

[0086]

[0087]

[0088] Define the coordinates of point C as (c x , c y ). According to the vector relationship it can be obtained that:

[0089] c x = m - D x (m1 - m) - D y (m2 - m),

[0090] c y = n - D x (n1 - n) - D y (n2 - n),

[0091] Similarly, the 2D pixel coordinates of the other 3 calibration plate corner points can be obtained.

[0092] Refer to Figure 5 as shown, it is a schematic diagram of the extracted contour points; refer to Figure 6 as shown, it is a schematic diagram of the fitted edge line; specifically, in step S5, it includes:

[0093] S51: Calculate the point farthest from the center of the abscissa and another point farthest from this point for each clustering region respectively;

[0094] S52: For the two points obtained in each clustering region, after arranging them in descending order of the y coordinate value, then arrange them in ascending order of the z coordinate value;

[0095] S53: Calculate the maximum index and minimum index of the y coordinate respectively. The points with y coordinates less than the maximum index belong to the lower right edge, the points with y coordinates not less than the maximum index belong to the upper right edge, the points with y coordinates less than the minimum index belong to the lower left edge, and the points with y coordinates not less than the minimum index belong to the upper left edge;

[0096] S54: Take the edge points corresponding to the lower right edge, the upper right edge, the lower left edge, and the upper left edge respectively, and use the RANSAC method for fitting to obtain four edge lines.

[0097] Refer to Figure 7 As shown, it is a schematic diagram of the calibration board corner point fitting result obtained by using the point cloud and image registration method based on calibration board corner point alignment provided by the present invention; Therefore, the point cloud and image registration method based on calibration board corner point alignment described in the present invention extracts reliable calibration board corner points in visible light images and radar point clouds to achieve the registration of video and radar spaces; Utilize the line scanning characteristic of the lidar, divide the calibration board point cloud by clustering according to the height threshold, calculate the 3D point cloud coordinates of the calibration board through the edge points fitted by the clustering region, which is simple, convenient and accurate in coordinates; According to the known calibration board, checkerboard size and relative relationship, obtain the 2D pixel coordinates of the calibration board, which better corresponds one-to-one with the point cloud coordinates; After obtaining the 3D point cloud coordinates and 2D pixel coordinates, use the combination of EPnP and RANSAC to obtain the transformation matrix for converting the point cloud coordinates to pixel coordinates, thereby realizing the registration of the point cloud and the image. The implementation is simple, does not require an accurate registration algorithm, has a high registration accuracy, and has high robustness and applicability.

[0098] Based on the above embodiments, the embodiments of the present invention further provide a point cloud and image registration device based on calibration board corner point alignment, including:

[0099] A visible light camera, placed at a preset position, for collecting image data of the calibration board within the field of view;

[0100] A lidar, placed at the same position as the visible light camera, for collecting point cloud data of the calibration board within the field of view;

[0101] A host computer, communicatively connected to the visible light camera and the lidar, for obtaining the image data and the point cloud data, obtaining the 2D pixel coordinates and 3D point cloud coordinates of the four corners of the calibration board, and obtaining the optimal transformation matrix from the 3D point cloud coordinates to the 2D pixel coordinates to realize the conversion from the 3D point cloud coordinates to the 2D pixel coordinates and complete the registration.

[0102] The point cloud and image registration device based on calibration board corner alignment provided by the present invention utilizes the point cloud and image registration method based on calibration board corner alignment and the calibration board point cloud corner extraction method based on height aggregation and edge fitting to extract the 3D point cloud coordinates of the calibration board corners, which is simple, convenient and accurate in coordinates; based on the 2D pixel coordinates and 3D point cloud coordinates of the calibration board corners, a method combining EPnP and RANSAC is adopted to realize the registration of the point cloud and the image. The implementation is simple, does not require an accurate registration algorithm, and has high registration accuracy; moreover, the present invention is applicable to scenarios where the relative positions of sensors are fixed, realizing high-robustness and applicable video and radar space registration.

[0103] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0105] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for realizing the process specified in Figure 1One process or multiple processes and / or boxes Figure 1 Steps of the functions specified in one box or multiple boxes.

[0107] Obviously, the above embodiments are only examples for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or variations can be made based on the above description. It is not necessary and impossible to enumerate all implementation manners here. And the obvious changes or variations derived therefrom are still within the protection scope of the present invention.

Claims

1. A point cloud and image registration method based on the alignment of calibration board corner points, characterized in that, include: Place the visible light camera and the laser radar at the same location, collect scene data of calibration plates placed at different preset positions, and obtain multiple sets of image data and point cloud data of the calibration plates; The Harris algorithm is used to detect the corner points of the calibration plate of the image data, and the 2D pixel coordinates of the four corner points of the calibration plate are obtained according to the relative distances between the corner points of the calibration plate and the end points of the chessboard; Segmenting the calibration plate point cloud from the point cloud data using straight-through filtering; After projecting all the calibration plate point cloud coordinates into the preset three-dimensional plane, the x coordinates are all set to 0, projected to the yOz plane, and based on the radar line scan principle, the calibration plate point cloud coordinates are divided into multiple clustering areas using the Euclidean distance in the z direction as the classification condition; According to the point farthest from the center of the horizontal axis in all clustering areas and another point farthest from the point, four edge points corresponding to the lower right edge, upper right edge, lower left edge and upper left edge are obtained and fitted into four edge lines; By calculating the intersection of the edge lines, the 3D point cloud coordinates of the four corner points of the calibration plate are obtained; Based on the 2D pixel coordinates of the four corner points of the calibration plate and the 3D point cloud coordinates, EPnP and RANSAC are used to obtain the optimal transformation matrix from 3D point cloud coordinates to 2D pixel coordinates, realize the reprojection of 3D point cloud coordinates, and complete the point cloud and image registration.

2. The point cloud and image registration method based on calibration board corner point alignment according to claim 1, wherein Before using the Harris algorithm to detect the corner points of the calibration plate of the image data, the method also includes: calibrating the visible light camera according to multiple groups of image data of the calibration plate, and performing dedistortion processing on the image data to obtain corrected image data.

3. The method for registering point cloud and image based on calibration board corner point alignment according to claim 1, wherein The method of obtaining the 2D pixel coordinates of the four corner points of the calibration plate according to the relative distances between the corner points of the calibration plate and the endpoints of the chessboard includes: Use Harris algorithm to detect image data and obtain checkerboard corner points; According to the four corner points at the end points of the chessboard and the two adjacent corner points of each corner point, the distance scales from the corner point of the calibration plate and the two adjacent corner points to the end points of the chessboard in the horizontal and vertical directions are calculated respectively; A vector coordinate system is constructed with the endpoint of the chessboard as the origin and the pixels of the endpoint of the chessboard and the two adjacent corner points as the unit distances of the x-axis and y-axis respectively; based on the vector relationship, the 2D pixel coordinates of the four corner points are obtained.

4. The method for registering point cloud and image based on the alignment of calibration board corner points according to claim 3, wherein The distance scale of the corner point and two adjacent endpoints of the calibration plate to the endpoints of the chessboard in the horizontal and vertical directions is expressed as: Horizontal x-axis distance scale: Vertical y-axis distance scale: Among them, C represents a corner point of the calibration plate, the endpoint of the chessboard is O, the triangle CDO forms a right triangle, the two adjacent endpoints of endpoint O are A and B, and the directions of OA and OB are horizontal and vertical respectively; x1 and y1 represent the pixel spacing between OA and OB, and their corresponding real spacings are X1 and Y1 respectively, and x and y represent the horizontal and vertical pixel spacing between O and C, and their corresponding real spacings are X and Y respectively.

5. The method for registering point cloud and image based on the alignment of calibration board corner points according to claim 4, wherein The vector coordinate system is constructed by taking the chessboard endpoint as the origin and the pixels between the chessboard endpoint and two adjacent corner points as the unit distances of the x-axis and y-axis respectively; According to the vector relationship, the 2D pixel coordinates of the four corner points are obtained, including: Construct a vector coordinate system with the endpoint O as the origin and the pixel pitch as the unit. Then the vector and the vector are in opposite directions, and the vector and the vector are in opposite directions; Let the coordinates of the detected endpoints A, B, and O be (m1, n1), (m2, n2), and (m, n) respectively. Then we have According to the principle of similar triangles, we can obtain: According to the vector relationship the coordinates (c x , c y ) of the calibration board corner point C can be obtained and expressed as: c x = m - D x (m1 - m) - D y (m2 - m); c y = n - D x (n1 - n) - D y (n2 - n).

6. The method for registering point cloud and image based on the alignment of calibration board corner points according to claim 1, characterized in that After segmenting the calibration plate point cloud from the point cloud data by using straight-through filtering, the method further includes removing noise points by using radius filtering.

7. The method for point cloud and image registration based on calibration board corner point alignment according to claim 1, wherein After projecting all the calibration board point cloud coordinates into a preset three-dimensional plane, set the x coordinates to 0, project them onto the yOz plane, and according to the principle of radar line scanning, use the Euclidean distance in the z direction as the classification condition to divide the calibration board point cloud coordinates into multiple clustering regions, including: Using the RANSAC method and the spatial plane equation Ax + By + Cz + D = 0, estimate the optimal plane of the calibration board point cloud, and project all the calibration board point cloud coordinates into a unified plane, expressed as: where x, y, and z represent the calibration board point cloud coordinates, and x', y', and z' represent the coordinates projected onto the same plane; Set the x coordinates of the projected calibration board point cloud to 0, project them onto the yOz plane, and according to the principle of radar line scanning, use the Euclidean distance in the z direction as the classification condition, and use the DBSACN algorithm to divide the calibration board point cloud coordinates into multiple clustering regions.

8. The method for registering point cloud and image based on calibration board corner point alignment according to claim 7, characterized in that Calculate the point farthest from the abscissa center in each region, and another point farthest from this point, obtain 4 edge points corresponding to the lower right edge, upper right edge, lower left edge, and upper left edge, and fit them into 4 edge lines; By calculating the intersection points of the edge lines, obtain the 3D point cloud coordinates of the 4 corner points of the calibration board, including: For each clustering region, calculate the point farthest from the abscissa center and another point farthest from this point; Arrange the two points obtained in each clustering region in descending order of the y coordinate value, and then in ascending order of the z coordinate value; Calculate the maximum index and minimum index of the y coordinate respectively. The points less than the maximum index belong to the lower right edge, the points not less than the maximum index belong to the upper right edge, the points less than the minimum index belong to the lower left edge, and the points not less than the minimum index belong to the upper left edge; Respectively take the edge points corresponding to the lower right edge, the upper right edge, the lower left edge, and the upper left edge, and use the RANSAC method to fit to obtain 4 edge lines; Calculate the intersection points of the edge lines to obtain the projected coordinates of the calibration board corner points in the yOz plane; substitute them into the spatial plane equation to obtain the 3D point cloud coordinates of the calibration board corner points.

9. The point cloud and image registration method based on calibration board corner point alignment according to claim 1, characterized in that Based on the 2D pixel coordinates and the 3D point cloud coordinates of the 4 corner points of the calibration board, use EPnP and RANSAC to obtain the optimal transformation matrix from the 3D point cloud coordinates to the 2D pixel coordinates, including: Use multiple groups of 2D pixel coordinates to form a 2D matrix, and the 3D point cloud coordinates of the same number of groups to form a 3D matrix, and use the EPnP algorithm to obtain the initial transformation matrix from the 2D matrix to the 3D matrix; Use the RANSAC method to iterate the initial transformation matrix to obtain the optimal transformation matrix.

10. A point cloud and image registration device based on calibration board corner point alignment that references the method according to any one of claims 1-9, characterized in that Including: A visible light camera placed at a preset position for collecting image data of the calibration board within the field of view; A lidar placed at the same position as the visible light camera for collecting point cloud data of the calibration board within the field of view; A host computer communicatively connected to the visible light camera and the lidar for obtaining the image data and the point cloud data, obtaining the 2D pixel coordinates and the 3D point cloud coordinates of the 4 corner points of the calibration board, and obtaining the optimal transformation matrix from the 3D point cloud coordinates to the 2D pixel coordinates to achieve the conversion from the 3D point cloud coordinates to the 2D pixel coordinates and complete the registration.

Images