A stereo calibration plate and calibration method for joint calibration of laser radar and camera

Abstract

A three-dimensional calibration plate for joint calibration of a laser radar and a camera, comprising a base plate and a regular tetrahedron located at the center of the base plate, the four corners of the base plate and the upper bottom surface of the tetrahedron having image ArUco marks, which are used for automatic recognition of laser radar feature points and automatic matching with image ArUco marked corner points, wherein the image ArUco marks are used for automatic recognition of image marks. Also included is a calibration method for joint calibration of a laser radar and a camera. The present invention obtains the points falling on the side edges and the bottom edge lines of the prism through the intersection of two straight lines of the same laser beam on adjacent planes, and then obtains the straight line equations of the side edges and the bottom edge lines, and then obtains the coordinates of the vertices of the tetrahedron in the laser radar coordinate system by the intersection of the side edges and the bottom edge lines, which can effectively avoid the error caused by the method of determining the edge points by the distance jump of the laser radar and then fitting the vertices, thereby improving the accuracy of laser radar feature point detection.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-sensor data fusion, and in particular to a stereo calibration plate and a calibration method for joint calibration of a laser radar and a camera. Background Art

[0002] Calibration is the basic condition for multi-sensor information fusion, which mainly solves the problems of multi-sensor time synchronization and spatial synchronization. Time synchronization can be solved by sensor hardware triggering, while spatial synchronization mainly solves their external parameters through the common view information or motion constraints of multiple sensors.

[0003] In the field of mobile robots and autonomous driving, cameras and LiDAR are widely used sensors because they have good complementary characteristics. Using images taken by cameras for target recognition is more advantageous, but images do not have depth information and have certain lighting conditions, while LiDAR has the performance of depth detection and can also be used in dark environments. Therefore, how to quickly and accurately obtain the position conversion relationship between the camera and LiDAR coordinate systems has become an important issue in the field of multi-sensor data fusion.

[0004] Some existing offline calibration methods for lidar and cameras still use a checkerboard as a calibration plate, and extract point features or surface features of the calibration plate on this basis for matching to calculate the external parameters of the lidar and camera. There are two problems with point feature matching to calculate external parameters. One is that the laser point of the lidar is very likely not to hit the corner point of the calibration plate. If the corner point is determined by manual click, the subjectivity is very large and will inevitably lead to uncontrollable errors. If the edge point found by the lidar distance jump is used to fit the vertex of the calibration plate, it will also introduce a large error, because the laser point is very likely not to hit the edge of the calibration plate. The second problem is that there is a singularity between 0° and 180° of the checkerboard, which may lead to incorrect matching. The problem with surface feature matching is that when extracting the normal vector of the calibration plate plane in the point cloud data, it will be affected by the similar point cloud around the calibration plate. For example, if the laser point hitting the hand holding the calibration plate participates in the calculation of the normal vector of the calibration plate plane, it will cause the normal vector of the calibration plate to deviate from the true value. Summary of the invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a stereo calibration plate and a calibration method for joint calibration of a laser radar and a camera to solve the problems raised in the above-mentioned background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a stereo calibration plate for joint calibration of laser radar and camera, the stereo calibration plate comprising a base plate and a regular tetrahedron located at the center of the base plate, the four corners of the base plate and the upper bottom surface of the tetrahedron have image ArUco marks, the stereo calibration plate is used for automatic recognition of laser radar feature points and automatic matching with image ArUco marked corner points, and the image ArUco marks are used for automatic recognition of image marks.

[0007] The black border of the image ArUco mark helps to quickly detect the image. The eight vertices of the pyramid on the stereo calibration board help to accurately extract the feature points of the lidar. At the same time, the eight vertices of the pyramid on the stereo calibration board are also designed as the corner points of the image ArUco mark, which helps to accurately match the laser point with the corner points of the image ArUco mark. At the same time, the 5 image ArUco marks of the stereo calibration board can also be used to calibrate the camera intrinsic parameters.

[0008] The base plate of the stereo calibration plate is a square plate with ArUco marks on the four corners. There is a regular tetrahedron in the center of the stereo calibration plate. The angle between the side of the tetrahedron and the base plate is 45°. The side of the tetrahedron is 4 white congruent isosceles trapezoids. The bottom surface of the tetrahedron in the center of the stereo calibration plate is composed of an ArUco mark.

[0009] The image ArUco marker includes a first ArUco marker, a second ArUco marker, a third ArUco marker, a fourth ArUco marker and a fifth ArUco marker, wherein the five ArUco markers have equal areas and different IDs, and the five ArUco markers include a black border and a binary matrix, wherein the binary matrix expresses the ArUco marker ID.

[0010] The first to fourth ArUco marks are located at the four corners of the bottom plate and the corner points facing the inside of the bottom plate just intersect with the vertices of the bottom surface of the quadrangular pyramid. The fifth ArUco mark is located on the bottom surface of the quadrangular pyramid and fills the entire area.

[0011] The diagonal line of the ArUco mark located on the diagonal line of the bottom plate, the projection line of the edge line of the regular tetrahedron, and the diagonal line of the fifth ArUco mark are in a straight line.

[0012] The bottom plate of the stereo calibration plate is located in the plane of z=0, the upper left corner of the bottom plate is the origin of the world coordinate system, the x-axis is downward, the y-axis is rightward, and the z-axis is upward. The upper bottom surface of the central tetrahedron of the stereo calibration plate is parallel to the bottom plate and is located in the plane of z=h, where h is the height of the tetrahedron and the coordinate unit is meter.

[0013] The world coordinates of the twenty corner points corresponding to the ArUco marks of the five images of the stereo calibration plate are determined and known.

[0014] A calibration method for joint calibration of a laser radar and a camera, characterized in that: the joint calibration method specifically includes:

[0015] Step 1: Enter the side length of the bottom plate of the stereo calibration plate, the side length of the image ArUco mark, and the height of the four-sided pyramid on the calibration plate into the calibration system. The calibration system includes ros startup module, data visualization module, data acquisition module, image marker corner point detection module, camera intrinsic parameter calibration module, laser radar pyramid edge detection module, laser radar pyramid vertex detection module, data screening module, feature matching module and optimization solution module;

[0016] Step 2: The ros startup module starts the camera and lidar, and enters the corresponding topic into the data acquisition module;

[0017] Step 3: Hold the calibration plate facing the camera and move it to collect data. The data visualization module will display the lidar point cloud data and camera image data simultaneously to determine whether the position of the calibration plate is reasonable.

[0018] Step 4: The image marker corner point detection module identifies the image ArUco markers and outputs the pixel coordinates of the four corner points of each ArUco marker in ascending order of the ArUco marker ID. When outputting the corner point coordinates of each ArUco marker, the order of output is clockwise starting from the pixel coordinates of the upper left corner point of the marker.

[0019] Step 5: After the image marker corner detection module detects the corner points of the image ArUco marker, the camera intrinsic calibration module establishes a correspondence between its pixel coordinates and the world coordinates on the calibration board where it is located, and calculates and solves the camera's intrinsic parameter matrix;

[0020] Step 6: The result of internal parameter calibration is transferred to the optimization solution module;

[0021] Step 7: The laser radar pyramid edge detection module detects the straight line equations of the side edges and upper and lower bottom edges of the pyramid on the calibration plate;

[0022] Step 8: The laser radar pyramid vertex detection module detects the coordinates of the eight vertices of the pyramid on the calibration plate in the laser radar coordinate system;

[0023] Step 9: The data screening module removes unusable data frames and retains usable data frames for joint calibration;

[0024] Step 10: The feature matching module establishes a corresponding relationship between the coordinates of the corner points marked by the image ArUco in the camera coordinate system and the coordinates of the vertices of the pyramid in the lidar coordinate system;

[0025] Step 11: Start the optimization solution module to calculate external parameters and output the calibration results;

[0026] The step 3 of judging whether the position of the calibration plate is reasonable is specifically as follows: if the calibration plate is in the camera field of view, it is considered reasonable; if the calibration plate is beyond the camera field of view, it is considered unreasonable; if at least two laser beams fall on each side of the quadrangular pyramid at the center of the calibration plate, it is considered reasonable; otherwise, it is considered unreasonable;

[0027] The specific steps of step 4 are as follows: the type of ArUco mark is DICT_4 × 4_50, starting from the upper left corner of the calibration plate bottom plate and in clockwise order, the IDs of ArUco marks are 1, 2, 3, 4, and the ID of the ArUco mark on the bottom surface of the quadrangular platform is 5;

[0028] The specific steps of the laser radar prism edge detection in step seven include:

[0029] Step 1.1: According to the laser points of the same laser beam on two adjacent planes, the linear equations of the laser beam on two adjacent planes are fitted as follows: y = m1x + n1, y = m2x + n2;

[0030] Step 1.2: Establish the system of equations The solution of this system of equations [x1, y1] is a point that falls on the side edge of the pyramid or on the edge line of the upper and lower bases of the pyramid;

[0031] Step 1.3: According to two or more laser points falling on the same side edge or upper and lower bottom edge of the tetrahedron, fit the straight line equation of the side edge and upper and lower bottom edge of the tetrahedron: y = a i x+b i , 0≤i≤12, i corresponds to the 4 side edges and 8 edge lines of the upper and lower bottom surfaces of the four-sided pyramid on the calibration plate;

[0032] The laser radar prism vertex detection in step eight is specifically as follows: the straight line equation y=a between the prism side edge and the upper and lower bottom surface edges of the prism calculated in step seven i x+b i The coordinates of the eight vertices of the prism in the laser radar coordinate system are obtained by intersecting the equation of the straight line where the side edge is located with the equation of the straight line where the bottom edge is located.

[0033] The unavailable data frames and available data frames of step nine are specifically: the unavailable data frames include image frames with incomplete detection of image ArUco marked corner points and laser radar point cloud frames with incomplete detection of vertices of the central quadrangular pyramid of the calibration plate;

[0034] The feature matching of step 10 is specifically as follows: the third corner point of the ArUco mark with ID=1 is matched with the upper left vertex of the bottom surface of the quadrangular pyramid, the fourth corner point of the ArUco mark with ID=2 is matched with the upper right vertex of the bottom surface of the quadrangular pyramid, the first corner point of the ArUco mark with ID=3 is matched with the lower right vertex of the bottom surface of the quadrangular pyramid, the second corner point of the ArUco mark with ID=4 is matched with the lower left vertex of the bottom surface of the quadrangular pyramid, the first corner point of the ArUco mark with ID=5 is matched with the upper left vertex of the top surface of the quadrangular pyramid, the second corner point of the ArUco mark with ID=5 is matched with the upper right vertex of the top surface of the quadrangular pyramid, the third corner point of the ArUco mark with ID=5 is matched with the lower right vertex of the top surface of the quadrangular pyramid, and the fourth corner point of the ArUco mark with ID=5 is matched with the lower left vertex of the top surface of the quadrangular pyramid. That is, the image feature [u j ,v j ] T With the laser radar feature [x lj ,y lj ,z lj ] T The corresponding matching in sequence, 1≤j≤8, represents 8 radar feature points and their corresponding ArUco marked corner points. The “corner point” mentioned above refers to the order of sorting in clockwise order starting from the upper left corner point of the ArUco mark of the image;

[0035] The specific steps of calculating the external parameters by the optimization solution module in step 11 include:

[0036] Step 2.1: Based on the camera intrinsic calibration results obtained by the camera intrinsic calibration module, separate the external parameters from the homography matrix of the image ArUco corner point world coordinates to pixel coordinates, so as to obtain the coordinates of the image ArUco corner point in the camera coordinate system as [x cj ,y cj ,z cj ] T , the homography matrix refers to the transformation matrix from the world coordinates of the image corner points to the pixel coordinates in the Zhang Zhengyou method;

[0037] Step 2.2: According to the correspondence between the image feature points obtained by the feature matching module and the laser radar feature points, constraints are established. The constraint equation is as follows:

[0038]

[0039] where [x cj ,y cj ,z cj ,1] TIndicates the coordinates of the corner points of the image ArUco markers with feature matching in the camera coordinate system, [x lj ,y lj ,z lj ,1] T Indicates the coordinates of the LiDAR feature points corresponding to the corner points of the image ArUco markers in the LiDAR coordinate system. [R|t] l2c is an external parameter, which represents the transformation from the laser radar coordinate system to the camera coordinate system;

[0040] Step 2.3: Minimize the reprojection error to establish the least squares problem. The specific formula of the objective function of the optimization model is:

[0041]

[0042] where p k represents the image feature points established in the feature matching stage, q k represents the lidar feature points corresponding to the image features in the feature matching stage, and N represents the number of feature point pairs established in the feature matching stage.

[0043] Compared with the prior art, the beneficial effect achieved by the present invention is as follows: the present invention creates a three-dimensional calibration plate for joint calibration of a laser radar and a camera, on which the points falling on the side edges and the bottom surface edge lines of the prism can be obtained by the intersection of two straight lines of the same laser beam on adjacent planes, and then the straight line equations of the side edges and the bottom surface edge lines can be obtained, and then the coordinates of the vertices of the four-sided pyramid corresponding to the laser radar coordinate system can be obtained by the intersection of the side edges and the bottom surface edge lines, which can effectively avoid the errors caused by the method of determining the edge points by the distance jump of the laser radar and then fitting the vertices, thereby improving the accuracy of the laser radar feature point detection.

[0044] The black border in the image Aruco marker helps to quickly detect the image and enhance the accuracy of image ArUco marker acquisition. Different binary information can be set to represent the image Aruco marker ID to determine the direction of the calibration plate to avoid the singularity caused by the 0° and 180° placement of the calibration plate, and it is more conducive to the automatic matching of image features and lidar features. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0046] Figure 1 It is a structural schematic diagram of a three-dimensional calibration plate for joint calibration of a laser radar and a camera according to the present invention;

[0047] Figure 2It is a calibration process diagram for joint calibration of LiDAR and camera;

[0048] Among them, 1. the first ArUco marker; 2. the second ArUco marker; 3. the third ArUco marker; 4. the fourth ArUco marker; 5. the fifth ArUco marker; 6. the bottom plate; 7. the quadrangular pyramid. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] See also Figure 1-Figure 2 , the present invention provides a technical solution:

[0051] Embodiment 1:

[0052] A stereo calibration plate for joint calibration of a laser radar and a camera, comprising a base plate and a regular tetrahedron located at the center of the base plate, wherein the four corners of the base plate and the upper bottom surface of the tetrahedron are provided with image ArUco marks, wherein the stereo calibration plate is used for automatic recognition of laser radar feature points and automatic matching with image ArUco marked corner points, and the image ArUco marks are used for automatic recognition of image marks.

[0053] The black border of the image ArUco mark helps to quickly detect the image. The eight vertices of the pyramid on the stereo calibration board help to accurately extract the feature points of the lidar. At the same time, the eight vertices of the pyramid on the stereo calibration board are also designed as the corner points of the image ArUco mark, which helps to accurately match the laser point with the corner points of the image ArUco mark. At the same time, the 5 image ArUco marks of the stereo calibration board can also be used to calibrate the camera intrinsic parameters.

[0054] The base plate of the stereo calibration plate is a square plate with ArUco marks on the four corners. There is a regular tetrahedron in the center of the stereo calibration plate. The angle between the side of the tetrahedron and the base plate is 45°. The side of the tetrahedron is 4 white congruent isosceles trapezoids. The bottom surface of the tetrahedron in the center of the stereo calibration plate is composed of an ArUco mark.

[0055] The image ArUco marker includes a first ArUco marker, a second ArUco marker, a third ArUco marker, a fourth ArUco marker and a fifth ArUco marker, wherein the five ArUco markers have equal areas and different IDs, and the five ArUco markers include a black border and a binary matrix, wherein the binary matrix expresses the ArUco marker ID.

[0056] The first to fourth ArUco marks are located at the four corners of the bottom plate and the corner points facing the inside of the bottom plate just intersect with the vertices of the bottom surface of the quadrangular pyramid. The fifth ArUco mark is located on the bottom surface of the quadrangular pyramid and fills the entire area.

[0057] The diagonal of the ArUco mark on the diagonal of the bottom plate is on the same straight line as the projection line of the ridgeline of the regular tetrahedron and the diagonal of the fifth ArUco mark. A diagonal of the first and third ArUco marks, a diagonal of the bottom plate, and a diagonal of the fifth ArUco mark are on the same straight line. A diagonal of the second and fourth ArUco marks, another diagonal of the bottom plate, and another diagonal of the fifth ArUco mark are on the same straight line.

[0058] The bottom plate of the stereo calibration plate is located in the plane of z=0, the upper left corner of the bottom plate is the origin of the world coordinate system, the x-axis is downward, the y-axis is rightward, and the z-axis is upward. The upper bottom surface of the central tetrahedron of the stereo calibration plate is parallel to the bottom plate and is located in the plane of z=h, where h is the height of the tetrahedron and the coordinate unit is meter.

[0059] The world coordinates of the twenty corner points corresponding to the ArUco marks of the five images of the stereo calibration plate are determined and known.

[0060] A calibration method for joint calibration of a laser radar and a camera, characterized in that: the joint calibration method specifically includes:

[0061] Step 1: Enter the side length of the bottom plate of the stereo calibration plate, the side length of the image ArUco mark, and the height of the four-sided pyramid on the calibration plate into the calibration system. The calibration system includes ros startup module, data visualization module, data acquisition module, image marker corner point detection module, camera intrinsic parameter calibration module, laser radar pyramid edge detection module, laser radar pyramid vertex detection module, data screening module, feature matching module and optimization solution module;

[0062] Step 2: The ros startup module starts the camera and lidar, and enters the corresponding topic into the data acquisition module;

[0063] Step 3: Hold the calibration plate facing the camera and move it to collect data. The data visualization module will display the lidar point cloud data and camera image data simultaneously to determine whether the position of the calibration plate is reasonable.

[0064] Step 4: The image marker corner point detection module identifies the image ArUco markers and outputs the pixel coordinates of the four corner points of each ArUco in ascending order according to the ID of the ArUco marker. When outputting the corner point coordinates of each ArUco marker, the order of output is clockwise starting from the pixel coordinates of the upper left corner point of the marker.

[0065] Step 5: After the image marker corner detection module detects the corner points of the image ArUco marker, the camera intrinsic calibration module establishes a correspondence between its pixel coordinates and the world coordinates on the calibration board where it is located, and calculates and solves the camera's intrinsic parameter matrix;

[0066] Step 6: The result of internal parameter calibration is transferred to the optimization solution module;

[0067] Step 7: The laser radar pyramid edge detection module detects the straight line equations of the side edges and upper and lower bottom edges of the pyramid on the calibration plate;

[0068] Step 8: The laser radar pyramid vertex detection module detects the coordinates of the eight vertices of the pyramid on the calibration plate in the laser radar coordinate system;

[0069] Step 9: The data screening module removes unusable data frames and retains usable data frames for joint calibration;

[0070] Step 10: The feature matching module establishes a corresponding relationship between the coordinates of the corner points marked by the image ArUco in the camera coordinate system and the coordinates of the vertices of the pyramid in the lidar coordinate system;

[0071] Step 11: Start the optimization solution module to calculate external parameters and output the calibration results;

[0072] The step 3 of judging whether the position of the calibration plate is reasonable is specifically as follows: if the calibration plate is in the camera field of view, it is considered reasonable; if the calibration plate is beyond the camera field of view, it is considered unreasonable; if at least two laser beams fall on each side of the quadrangular pyramid at the center of the calibration plate, it is considered reasonable; otherwise, it is considered unreasonable;

[0073] The specific steps of step 4 are as follows: the type of ArUco mark is DICT_4 × 4_50, starting from the upper left corner of the calibration plate bottom plate and in clockwise order, the IDs of ArUco marks are 1, 2, 3, 4, and the ID of the ArUco mark on the bottom surface of the quadrangular platform is 5;

[0074] The specific steps of the laser radar prism edge detection in step seven include:

[0075] Step 1.1: According to the laser points of the same laser beam on two adjacent planes, the linear equations of the laser beam on two adjacent planes are fitted as follows: y = m1x + n1, y = m2x + n2;

[0076] Step 1.2: Establish the system of equations The solution of this system of equations [x1, y1] is a point that falls on the side edge of the pyramid or on the edge line of the upper and lower bases of the pyramid;

[0077] Step 1.3: According to two or more laser points falling on the same side edge or upper and lower bottom edge of the tetrahedron, fit the straight line equation of the side edge and upper and lower bottom edge of the tetrahedron: y = a i x+b i , 0≤i≤12, i corresponds to the 4 side edges and 8 edge lines of the upper and lower bottom surfaces of the four-sided pyramid on the calibration plate;

[0078] The laser radar prism vertex detection in step eight is specifically as follows: the straight line equation y=a between the prism side edge and the upper and lower bottom surface edges of the prism calculated in step seven i x+b i The coordinates of the eight vertices of the prism in the laser radar coordinate system are obtained by intersecting the equation of the straight line where the side edge is located with the equation of the straight line where the bottom edge is located.

[0079] The unavailable data frames and available data frames of step nine are specifically: the unavailable data frames include image frames with incomplete detection of image ArUco marked corner points and laser radar point cloud frames with incomplete detection of vertices of the central quadrangular pyramid of the calibration plate;

[0080] The feature matching of step 10 is specifically as follows: the third corner point of the ArUco mark with ID=1 is matched with the upper left vertex of the bottom surface of the quadrangular pyramid, the fourth corner point of the ArUco mark with ID=2 is matched with the upper right vertex of the bottom surface of the quadrangular pyramid, the first corner point of the ArUco mark with ID=3 is matched with the lower right vertex of the bottom surface of the quadrangular pyramid, the second corner point of the ArUco mark with ID=4 is matched with the lower left vertex of the bottom surface of the quadrangular pyramid, the first corner point of the ArUco mark with ID=5 is matched with the upper left vertex of the top surface of the quadrangular pyramid, the second corner point of the ArUco mark with ID=5 is matched with the upper right vertex of the top surface of the quadrangular pyramid, the third corner point of the ArUco mark with ID=5 is matched with the lower right vertex of the top surface of the quadrangular pyramid, and the fourth corner point of the ArUco mark with ID=5 is matched with the lower left vertex of the top surface of the quadrangular pyramid. That is, the image feature [u j ,v j ] T With the laser radar feature [x lj ,y lj ,zlj ] T The corresponding matching in sequence, 1≤j≤8, represents 8 radar feature points and their corresponding ArUco marked corner points. The “corner point” mentioned above refers to the order of sorting in clockwise order starting from the upper left corner point of the ArUco mark of the image;

[0081] The specific steps of calculating the external parameters by the optimization solution module in step 11 include:

[0082] Step 2.1: Based on the camera intrinsic calibration results obtained by the camera intrinsic calibration module, separate the external parameters from the homography matrix of the image ArUco corner point world coordinates to pixel coordinates, so as to obtain the coordinates of the image ArUco corner point in the camera coordinate system as [x cj ,y cj ,z cj ] T , the homography matrix refers to the transformation matrix from the world coordinates of the image corner points to the pixel coordinates in the Zhang Zhengyou method;

[0083] Step 2.2: According to the correspondence between the image feature points obtained by the feature matching module and the laser radar feature points, constraints are established. The constraint equation is as follows:

[0084]

[0085] where [x cj ,y cj ,z cj ,1] T Indicates the coordinates of the corner points of the image ArUco markers with feature matching in the camera coordinate system, [x lj ,y lj ,z lj ,1] T Indicates the coordinates of the LiDAR feature points corresponding to the corner points of the image ArUco markers in the LiDAR coordinate system. [R|t] l2c is an external parameter, which represents the transformation from the laser radar coordinate system to the camera coordinate system;

[0086] Step 2.3: Minimize the reprojection error to establish the least squares problem. The specific formula of the objective function of the optimization model is:

[0087]

[0088] where p k represents the image feature points established in the feature matching stage, q k represents the lidar feature points corresponding to the image features in the feature matching stage, and N represents the number of feature point pairs established in the feature matching stage.

[0089] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0090] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional calibration plate for joint calibration of laser radar and camera, characterized in that: It includes a base plate and a regular tetrahedron located at the center of the base plate, and the four corners of the base plate and the upper bottom surface of the tetrahedron have image ArUco marks for automatic recognition of laser radar feature points and automatic matching with image ArUco marked corner points, and the image ArUco marks are used for automatic recognition of image marks; The black border of the image ArUco mark helps to quickly detect the image. The eight vertices of the pyramid help to accurately extract the feature points of the laser radar. At the same time, the eight vertices of the pyramid are also designed as the corner points of the image ArUco mark, so that the laser point and the corner points of the image ArUco mark can be accurately matched. At the same time, the five image ArUco marks of the stereo calibration plate can also be used to calibrate the camera's intrinsic parameters. The base plate is a square plate with ArUco marks on the four corners, the angle between the side of the quadrangular pyramid and the base plate is 45°, the side of the quadrangular pyramid is 4 white congruent isosceles trapezoids, and the bottom surface of the quadrangular pyramid is composed of an ArUco mark; The image ArUco marker includes a first ArUco marker, a second ArUco marker, a third ArUco marker, a fourth ArUco marker and a fifth ArUco marker, the five ArUco markers have equal areas but different IDs, the five ArUco markers include a black border and a binary matrix, and the binary matrix expresses the ArUco marker ID; The first to fourth ArUco marks are located at the four corners of the bottom plate and the corner points facing the inside of the bottom plate just intersect with the vertices of the bottom surface of the quadrangular pyramid; the fifth ArUco mark is located on the bottom surface of the quadrangular pyramid and fills the entire area; The diagonal line of the ArUco mark located on the diagonal line of the bottom plate, the projection line of the edge line of the regular tetrahedron, and the diagonal line of the fifth ArUco mark are in a straight line; The bottom plate is located in the plane of z=0, the upper left corner of the bottom plate is the origin of the world coordinate system, the x-axis is downward, the y-axis is rightward, and the z-axis is upward. The upper bottom surface of the tetrahedron is parallel to the bottom plate and is located in the plane of z=h, where h is the height of the tetrahedron, and the coordinate unit is meter; The world coordinates of the twenty corner points corresponding to the ArUco marks of the five images of the stereo calibration plate are determined and known.

2. A calibration method for joint calibration of a laser radar and a camera using the stereo calibration plate for joint calibration of a laser radar and a camera according to claim 1, characterized in that: The steps include: Step 1: Enter the side length of the bottom plate of the stereo calibration plate, the side length of the image ArUco mark, and the height of the four-sided pyramid on the calibration plate into the calibration system. The calibration system includes ros startup module, data visualization module, data acquisition module, image marker corner point detection module, camera intrinsic parameter calibration module, laser radar pyramid edge detection module, laser radar pyramid vertex detection module, data screening module, feature matching module and optimization solution module; Step 2: The ros startup module starts the camera and lidar, and enters the corresponding topic into the data acquisition module; Step 3: Hold the calibration plate facing the camera and move it to collect data. The data visualization module will display the lidar point cloud data and camera image data simultaneously to determine whether the position of the calibration plate is reasonable. Step 4: The image marker corner point detection module identifies the image ArUco markers and outputs the pixel coordinates of the four corner points of each ArUco marker in ascending order of the ArUco marker ID. When outputting the corner point coordinates of each ArUco marker, the order of output is clockwise starting from the pixel coordinates of the upper left corner point of the marker. Step 5: After the image marker corner detection module detects the corner points of the image ArUco marker, the camera intrinsic calibration module establishes a correspondence between its pixel coordinates and the world coordinates on the calibration board where it is located, and calculates and solves the camera's intrinsic parameter matrix; Step 6: The result of internal parameter calibration is transferred to the optimization solution module; Step 7: The laser radar pyramid edge detection module detects the straight line equations of the side edges and upper and lower bottom edges of the pyramid on the calibration plate; Step 8: The laser radar pyramid vertex detection module detects the coordinates of the eight vertices of the pyramid on the calibration plate in the laser radar coordinate system; Step 9: The data screening module removes unusable data frames and retains usable data frames for joint calibration; Step 10 : The feature matching module establishes a corresponding relationship between the coordinates of the corner points marked by the image ArUco in the camera coordinate system and the coordinates of the vertices of the pyramid in the lidar coordinate system; Step 11: Start the optimization solution module to calculate external parameters and output the calibration results.

3. The method according to claim 2, characterized in that: The determination of whether the position of the calibration plate is reasonable in step three is specifically as follows: the calibration plate is considered reasonable if it is within the camera's field of view, and is considered unreasonable if it is beyond the camera's field of view. The calibration plate is considered reasonable if at least two laser beams fall on each side of the quadrangle at the center of the calibration plate, and is considered unreasonable otherwise.

4. The method according to claim 2, characterized in that: The specific steps of step 4 are as follows: the type of ArUco marker is DICT_4×4_50, and starting from the upper left corner of the bottom plate of the calibration plate, the IDs of the ArUco markers are 1, 2, 3, and 4 in a clockwise order. The ID of the ArUco marker on the bottom surface of the quadrangular platform is 5.

5. The method according to claim 2, characterized in that: The specific steps of the laser radar pyramid edge detection described in step 7 include: Step 1.1: According to the laser points of the same laser beam on two adjacent planes, the linear equations of the laser beam on two adjacent planes are fitted as follows: y=m1x+n1, y=m2x+n2; Step 1.2: Establish a system of equations , the solution of this system of equations [x1,y1] is a point that falls on the side edge of the pyramid or on the edge line of the upper and lower bases of the pyramid; Step 1.3: According to two or more laser points falling on the same side edge or upper and lower bottom edge of the tetrahedron, fit the straight line equation of the side edge and upper and lower bottom edge of the tetrahedron: y=a i x+b i , 0≤i≤12, i corresponds to the 4 side edges and 8 edge lines of the upper and lower bottom surfaces of the tetrahedron on the calibration plate.

6. The method according to claim 2, characterized in that: The laser radar prism vertex detection in step eight is specifically as follows: the straight line equation y=a between the prism side edge and the upper and lower bottom edges of the prism calculated in step seven i x+b i The coordinates of the eight vertices of the prism in the laser radar coordinate system are obtained by intersecting the equation of the straight line where the side edge is located with the equation of the straight line where the bottom edge is located.

7. The method according to claim 2, characterized in that: The unavailable data frames and available data frames described in step nine are specifically: the unavailable data frames include image frames with incomplete detection of image ArUco marked corner points and laser radar point cloud frames with incomplete detection of vertices of the central pyramid of the calibration plate.

8. The method according to claim 2, characterized in that: The feature matching described in step ten is specifically as follows: the third corner point of the ArUco mark with ID=1 is matched with the upper left vertex of the bottom surface of the tetrahedron, the fourth corner point of the ArUco mark with ID=2 is matched with the upper right vertex of the bottom surface of the tetrahedron, the first corner point of the ArUco mark with ID=3 is matched with the lower right vertex of the bottom surface of the tetrahedron, the second corner point of the ArUco mark with ID=4 is matched with the lower left vertex of the bottom surface of the tetrahedron, the first corner point of the ArUco mark with ID=5 is matched with the upper left vertex of the top surface of the tetrahedron, the second corner point of the ArUco mark with ID=5 is matched with the upper right vertex of the top surface of the tetrahedron, the third corner point of the ArUco mark with ID=5 is matched with the lower right vertex of the top surface of the tetrahedron, and the fourth corner point of the ArUco mark with ID=5 is matched with the lower left vertex of the top surface of the tetrahedron; that is, the image feature [u j ,v j ] T With the laser radar feature [x lj ,y lj ,z lj ] T The corresponding matching in sequence, 1≤j≤8, represents the 8 radar feature points and their corresponding ArUco marked corner points. The first to fourth corner points refer to the upper left corner point of the image ArUco mark as the starting point and are sorted in clockwise order.

9. The method according to claim 2, characterized in that: The specific steps of calculating the external parameters by the optimization solution module described in step 11 include: Step 2.1: Based on the camera intrinsic calibration results obtained by the camera intrinsic calibration module, separate the external parameters from the homography matrix of the image ArUco corner point world coordinates to pixel coordinates, so as to obtain the coordinates of the image ArUco corner point in the camera coordinate system as [x cj ,y cj ,z cj ] T , the homography matrix refers to the transformation matrix from the world coordinates of the image corner points to the pixel coordinates in the Zhang Zhengyou method; Step 2.2: According to the correspondence between the image feature points obtained by the feature matching module and the laser radar feature points, constraints are established. The constraint equation is as follows: ， where [x cj ,y cj ,z cj ,1] T Indicates the coordinates of the corner points of the image ArUco markers with feature matching in the camera coordinate system, [x lj ,y lj ,z lj ,1] T Indicates the coordinates of the LiDAR feature points corresponding to the ArUco marked corner points of the image with good feature matching in the LiDAR coordinate system; Step 2.3: Minimize the reprojection error to establish the least squares problem. The specific formula of the objective function of the optimization model is: （2） where p k represents the image feature points established in the feature matching stage, q k represents the lidar feature points corresponding to the image features in the feature matching stage, and N represents the number of feature point pairs established in the feature matching stage.

Images