Calibration Method, System, Device and Storage Medium for LiDAR and Camera

Abstract

The present invention provides a laser radar and camera calibration method, system, device and storage medium. The method includes the following steps: providing a calibration carrier, the surface of the calibration carrier including black squares, white squares and through holes arranged in a grid, aligning the laser radar and the camera with the calibration carrier, collecting laser point cloud data at different postures i, and obtaining the intersection corner points #imgabs0# of the black and white grids on the calibration plate in the image; obtaining the camera intrinsic parameter matrix K and distortion parameter s based on the corner point coordinates #imgabs1# of the calibration plate at different postures in the obtained image; and calculating the extrinsic parameter matrix E from the laser radar coordinate system to the camera coordinate system based on the correspondence between the image corner point coordinates #imgabs2# and the vertex coordinates #imgabs3# of the through holes in the laser coordinate system at each posture i. i , from E i An external parameter matrix E with the smallest reprojection error is selected. The present invention can automatically perform calibration to ensure that the laser sensor and the binocular camera are in the same coordinate system, greatly simplify the calibration process, and improve the accuracy and efficiency of calibration.

Description

Technical Field

[0001] The present invention relates to the field of AI optical recognition, and more specifically, to a calibration method, system, device, and storage medium for laser radar and camera. Background Art

[0002] With the continuous development of global trade, the workload density at port container yards is increasing. This increases costs and creates a more demanding workload for operators. However, with the continuous advancement of autonomous driving technology, the emergence of autonomous container trucks has emerged as a solution to this problem. In the past, autonomous container trucks were typically equipped with GPS and monocular cameras. Because they lack three-dimensional information, these sensors struggle to perceive distance and position. Today, many autonomous vehicles have addressed this limitation by incorporating cameras and lidar sensors. However, multiple sensors can exhibit azimuthal deviations, which can add excessive noise to high-precision decision-making. Manual calibration not only fails to meet the high-precision positioning requirements for autonomous container trucks in path planning, but also requires a significant amount of high-value labor.

[0003] Therefore, the present invention provides a laser radar and camera calibration method, system, device and storage medium. Summary of the Invention

[0004] In response to the problems in the prior art, the purpose of the present invention is to provide a calibration method, system, equipment and storage medium for lidar and camera, which overcomes the difficulties of the prior art, can automatically perform calibration, ensure that the laser sensor and binocular camera are in the same coordinate system, and greatly simplify the calibration process, thereby improving the accuracy and efficiency of calibration.

[0005] An embodiment of the present invention provides a method for calibrating a laser radar and a camera, comprising the following steps:

[0006] S110, providing a calibration carrier, wherein the surface of the calibration carrier includes black squares and white squares arranged in a grid and through holes for laser to pass through;

[0007] S120, fix the laser radar and camera to align with the calibration carrier, change the calibration plate posture, collect laser point cloud data at different postures i, and obtain the intersection corner points of the black and white grids on the calibration plate in the image Subscript j indicates the jth corner point;

[0008] S130, coordinates of corner points of the calibration plate at different positions in the image obtained in step S120 Obtain the camera intrinsic parameter matrix K and distortion parameter s;

[0009] S140, according to the image corner coordinates under each posture i And the vertex coordinates of the through hole in the laser coordinate system Correspondence, calculate the external parameter matrix E from the lidar coordinate system to the camera coordinate system i ;

[0010] S150, from E i Select an extrinsic parameter matrix E with the smallest reprojection error.

[0011] Preferably, the calibration carrier is a checkerboard sticker with a total of a×b squares, and n rectangular through holes with known side lengths are hollowed out at the positions of some white squares, and adjacent rectangular through holes are arranged at least in alternate rows and columns, and n is a natural number less than the total number of white squares in the checkerboard sticker.

[0012] Preferably, in step S140, under different calibration carrier positions i, the coordinates of the vertices of the through-holes of the calibration plate in the laser radar coordinate system are The calculation steps are as follows:

[0013] S141, each laser point All three-dimensional coordinate information is collected and a reflection intensity information The initial calibration carrier position is recorded as Q0. By estimating the position of the calibration plate in the laser coordinate system, the laser points other than those on the calibration plate are filtered for the first time using the preset coordinate threshold to obtain the first residual lidar point cloud set U. i0 ;

[0014] S142, the first remaining laser radar point cloud set U i0 Calculate the average value of the three-dimensional coordinates of the laser points on the calibration plate, record it as P0 (x0, y0, z0), and set up a sphere with this as the center and W times the length of the short side of the calibration plate as the diameter. Perform a second filtering to remove the laser points outside the sphere and obtain the second residual lidar point cloud set U i1 , with U i1 The coordinates of all remaining laser points in the plane are fitted to the plane where the calibration carrier is located, and the plane parameters are recorded as l(a, b, c, d). The value range of W is 0.6 to 0.9;

[0015] S143, the first remaining laser radar point cloud set U i0 The distance from the laser point to the fitting plane l is less than or equal to the preset threshold Ds as the filtering condition, and the third filtering is performed to obtain the lidar point cloud cluster U i ;

[0016] S144, according to the fitting plane l, obtain a transformation matrix M i1 , for the lidar point cloud cluster U projected on the calibration plate iAfter this transformation, it is moved to the YOZ plane in the laser coordinate system, and the position of the calibration plate after the movement is recorded as Q i , record the transformation to the point cloud cluster U at this position i The vertex coordinates of the through hole in the internal calibration plate are but

[0017] S145. Construct a virtual calibration plate identical to the calibration carrier on the YOZ plane in the laser coordinate system. In this position, the lower left corner of the virtual calibration plate coincides with point O, and the two sides are located on the Y axis and the Z axis respectively. The corner point of the through hole on the plate is According to the preset matching error function Cost calculation, a transformation matrix M is obtained i2 , the transformation matrix M i2 Contains rotation around the x-axis and translation along the y-axis and z-axis, so that the point cloud cluster U i After the matrix M i2 After the transformation, it coincides with the virtual calibration plate, and then satisfies

[0018] S146, through the obtained transformation matrix M i1 、M i2 Obtain the original coordinates of the corner points of the through hole in the laser coordinate system

[0019] Preferably, the step S142 includes: setting the laser coordinate system with the vertical upward direction as the positive direction of the z axis, the front of the laser radar as the positive direction of the x axis, and the left direction of the laser radar as the positive direction of the y axis, wherein the x and y axes form a horizontal plane, and using the Ransac algorithm to solve the laser radar point cloud set U i1 The plane equation of the plane where it is located, let the plane equation be aX+bY+cZ+d=0, and (a, b, c) are unit vectors, from U i1 Randomly select the coordinates of 4 laser points, solve the values of a, b, c, d, and use U i1 Substitute all laser points in the current aX+bY+cZ+d=0, and the statistical distance from this plane is less than the threshold D f The number of laser points n p ; Repeat m times, select n p The maximum values of a, b, c, and d are the best fitting plane parameters l(a, b, c, d).

[0020] Preferably, the laser radar point cloud cluster U i The plane parameters of the plane are l(a, b, c, d), then the normal vector corresponding to the plane is (a, b, c), then the rotation matrix corresponding to the plane is

[0021] After transformation, the point cloud cluster U i Transform to be parallel to YOZ; record the laser point cloud cluster U i The mean coordinates of all lidar points in Then let M i1 =(R i1 , t i1 ), so that the laser point cloud cluster U i M i1 After the transformation, it coincides with the YOZ plane, and the laser point cloud cluster after the transformation is recorded as U′ i .

[0022] Preferably, after the transformation matrix M i1 The transformed laser point cloud cluster U′ i , apply translation and rotation, and record the parameters as (dy, dz, dθ), where dy represents the translation along the y-axis, dz represents the translation along the z-axis, and dθ represents the rotation around the x-axis. The rotation matrix corresponding to dθ is The displacement matrix is make through Transformed laser point cloud cluster U′ i After changing position, it becomes U′ i ', will U' i The laser points in the 'are projected onto the YOZ plane, and then the number of laser points in the square through-hole area and the non-through-hole area on the virtual calibration plate are counted as num A 、num B , let the weights be weight A 、weight B , weight A 、weight B The ratio is set to Then we get the following objective function:

[0023] Cost=num A *weight A +num B *weight B ;

[0024] Used to determine the laser point cloud U′ after the transformed position i ', whether it can accurately coincide with the virtual calibration plate to obtain the optimal translation and rotation parameters corresponding to the minimum objective function Cost value; the value range of the translation and rotation parameters is determined by the relative position of the calibration carrier and the laser radar. The translation distance generally does not exceed half of the length of the short side of the calibration carrier. The rotation angle is not more than 30° by placing the calibration carrier in advance. The grid search method is used to find the optimal translation and rotation parameters that minimize the Cost value within the value range, which is recorded as the transformation matrix M i2 .

[0025] Preferably, the coordinates of the corner points of the calibration plate at different positions in the image obtained in step S120 in step S130 are Obtaining the camera intrinsic parameter matrix K and distortion parameter s includes the following steps:

[0026] S131, collecting the coordinates of the black and white grid corner points of the calibration carrier in the image at different postures

[0027] S132. Calculate the camera's intrinsic parameters according to Zhang's calibration method to obtain the camera's intrinsic parameter matrix K and distortion parameters s(k1, k2, p1, p2, k3);

[0028] S133, record the three-dimensional coordinates of each corner point in the camera coordinate system as According to the pinhole model, the transformation from the camera three-dimensional space to the image plane is given by the formula Indicates that f is the focal length of the camera, u0, v0 is the difference between the projection of the camera optical center on the image plane and the image center, Represents the coordinates of the image point before dedistortion processing;

[0029] S134, obtain

[0030] S135. Considering the camera lens distortion, according to the camera distortion model, the camera dedistortion formula Recorded as

[0031] S136, combined with formula (1)(2)(3), the black and white grid corner points of the calibration plate in the camera's three-dimensional coordinate system The coordinates projected onto the image are Right now

[0032] Preferably, in step S150, i Selecting an external parameter matrix E with the minimum reprojection error includes the following steps:

[0033] S151, let the external parameter matrix from the lidar coordinate system to the image coordinate system at each posture i be E i ,but

[0034] S152, combining formulas (4) and (5), project the three-dimensional point in the laser radar coordinate system onto the image plane, which is recorded as The projection error is recorded as Then the external parameter matrix E is solved by iterative optimization based on the projection error i ;

[0035] S153, the external parameter matrix E obtained under different postures i The corresponding reprojection error Sort by the external parameter matrix E with the smallest reprojection error. i As the optimal external parameter matrix E.

[0036] Preferably, the method further comprises the following steps:

[0037] S160. Obtain a first transformation matrix from the camera coordinate system to the image coordinate system based on the intrinsic parameter matrix K of the camera, obtain a second transformation matrix from the laser coordinate system to the camera coordinate system based on the extrinsic parameter matrix E of the camera, and obtain a third transformation matrix from the laser coordinate system to the image coordinate system through the first transformation matrix and the second transformation matrix.

[0038] An embodiment of the present invention further provides a laser radar and camera calibration system for implementing the above-mentioned laser radar and camera calibration method. The laser radar and camera calibration system includes:

[0039] The calibration carrier module provides a calibration carrier, wherein the surface of the calibration carrier includes black squares and white squares arranged in a grid and through holes for laser to pass through;

[0040] The laser point cloud module fixes the laser radar and the camera to align with the calibration carrier, changes the calibration plate posture, collects laser point cloud data at different postures i, and obtains the intersection corners of the black and white grids on the calibration plate in the image Subscript j indicates the jth corner point;

[0041] The internal parameter matrix module is used to calculate the corner coordinates of the calibration plate at different positions in the image obtained by the laser point cloud module. Obtain the camera intrinsic parameter matrix K and distortion parameter s;

[0042] The external parameter matrix module is based on the image corner coordinates at each pose i And the vertex coordinates of the through hole in the laser coordinate system Correspondence, calculate the external parameter matrix E from the lidar coordinate system to the camera coordinate system i ;as well as

[0043] External parameter sorting module, from E i Select an extrinsic parameter matrix E with the smallest reprojection error.

[0044] An embodiment of the present invention further provides a laser radar and camera calibration device, comprising:

[0045] processor;

[0046] a memory storing executable instructions for the processor;

[0047] The processor is configured to execute the steps of the above-mentioned laser radar and camera calibration method by executing executable instructions.

[0048] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the above-mentioned laser radar and camera calibration method.

[0049] The laser radar and camera calibration method, system, device and storage medium of the present invention can automatically perform calibration, ensure that the laser sensor and binocular camera are in the same coordinate system, greatly simplify the calibration process, and improve the accuracy and efficiency of calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Other features, objects and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0051] Figure 1 It is a flow chart of the calibration method of the laser radar and camera of the present invention.

[0052] Figures 2 to 6 It is a schematic diagram of the implementation process of the laser radar and camera calibration method of the present invention.

[0053] Figure 7 It is a structural diagram of the laser radar and camera calibration system of the present invention.

[0054] Figure 8 It is a schematic diagram of the structure of the laser radar and camera calibration equipment of the present invention. And

[0055] Figure 9 It is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.

[0056] Reference numerals

[0057] 1 Calibration carrier

[0058] 11 black squares

[0059] 12 white squares

[0060] 13 rectangular through hole

[0061] 14 corner points

[0062] 15 vertices

[0063] 2 LiDAR

[0064] 3 Binocular Camera DETAILED DESCRIPTION

[0065] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in the present application. The present application can also be implemented or applied through different specific embodiments. The details in the present application can also be modified or changed according to different viewpoints and application systems without departing from the spirit of the present application. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other unless they conflict.

[0066] The following is a detailed description of the embodiments of the present application with reference to the accompanying drawings so that those skilled in the art can easily implement the present application. The present application can be embodied in many different forms and is not limited to the embodiments described herein.

[0067] In the description of this application, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this application, as well as features of different embodiments or examples, unless otherwise contradictory.

[0068] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include at least one such feature. In the context of this application, "plurality" means two or more, unless otherwise specifically defined.

[0069] In order to clearly describe the present application, components not related to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0070] Throughout this specification, when a device is said to be "connected" to another device, this includes not only "direct connection" but also "indirect connection" with other elements interposed therebetween. Furthermore, when a device is said to "include" a certain component, unless otherwise stated, this does not exclude the inclusion of other components but rather implies that the device may include other components.

[0071] When a device is said to be "on" another device, it may be directly on the other device, but there may also be other devices between it. In contrast, when a device is said to be "directly on" another device, there are no other devices between it.

[0072] Although the terms first, second, etc. are used in some instances herein to represent various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, the first interface and the second interface, etc. are represented. Furthermore, as used in this article, the singular forms "one," "an," and "the" are intended to also include the plural forms, unless there is a contrary indication in the context. It should be further understood that the terms "comprise," "include," and "include" indicate the presence of features, steps, operations, elements, components, items, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, types, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B, or C" or "A, B, and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B, and C." Exceptions to this definition only occur when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0073] The technical terms used herein are intended only to refer to specific embodiments and are not intended to limit this application. The singular form used herein also includes the plural form unless the statement explicitly indicates otherwise. The term "comprising" as used in this specification is intended to specify specific features, regions, integers, steps, operations, elements, and / or components and does not exclude the presence or addition of other features, regions, integers, steps, operations, elements, and / or components.

[0074] Although not defined differently, all terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art to which this application belongs. Terms defined in commonly used dictionaries are to be interpreted as having meanings consistent with the relevant technical literature and current teachings, and unless otherwise defined, they should not be overly interpreted as ideal or highly formalized meanings.

[0075] Figure 1 This is a flow chart of the laser radar and camera calibration method of the present invention. Figure 1 As shown, the calibration method of the laser radar and camera of the present invention includes the following steps:

[0076] S110 , providing a calibration carrier, wherein the surface of the calibration carrier includes black squares and white squares arranged in a grid and through holes for laser to pass through.

[0077] S120, fix the laser radar and camera to align with the calibration carrier, change the calibration plate posture, collect laser point cloud data at different postures i, and obtain the intersection corner points of the black and white grids on the calibration plate in the image The subscript j represents the jth corner point.

[0078] S130: coordinates of corner points of the calibration plate at different positions in the image obtained in step S120 Obtain the camera intrinsic parameter matrix K and distortion parameter s.

[0079] S140, according to the image corner coordinates under each posture i And the vertex coordinates of the through hole in the laser coordinate system Correspondence, calculate the external parameter matrix E from the lidar coordinate system to the camera coordinate system i .

[0080] S150, from E i Select an extrinsic parameter matrix E with the smallest reprojection error.

[0081] In a preferred embodiment, the calibration carrier is a checkerboard tile with a total of a×b squares, and n rectangular through holes with known side lengths are hollowed out at the locations of some white squares. Adjacent rectangular through holes are arranged in at least alternate rows and columns, and n is a natural number less than the total number of white squares in the checkerboard tile.

[0082] In a preferred embodiment, in step S140, the coordinates of the vertices of the through-hole of the calibration plate in the laser radar coordinate system under different calibration carrier positions i are The calculation steps are as follows:

[0083] S141, each laser point All three-dimensional coordinate information is collected and a reflection intensity information The initial calibration carrier position is recorded as Q0. By estimating the position of the calibration plate in the laser coordinate system, the laser points other than those on the calibration plate are filtered for the first time using the preset coordinate threshold to obtain the first residual lidar point cloud set U. i0 .

[0084] S142, the first remaining laser radar point cloud set U i0 Calculate the average value of the three-dimensional coordinates of the laser points on the calibration plate, record it as P0 (x0, y0, z0), use it as the center of the sphere and set up a sphere with W times the length of the short side of the calibration plate as the diameter. Perform a second filtering to remove the laser points outside the sphere and obtain the second residual lidar point cloud set U i1 , with U i1 The coordinates of all the remaining laser points in the plane are fitted to the plane where the calibration carrier is located, and the plane parameters are recorded as l (a, b, c, d). The value range of W is 0.6 to 0.9.

[0085] S143, the first remaining laser radar point cloud set U i0The distance from the laser point to the fitting plane l is less than or equal to the preset threshold Ds as the filtering condition, and the third filtering is performed to obtain the lidar point cloud cluster U i .

[0086] S144. Obtain a transformation matrix M based on the fitting plane l. i1 , for the lidar point cloud cluster U projected on the calibration plate i After this transformation, it is moved to the YOZ plane in the laser coordinate system, and the position of the calibration plate after the movement is recorded as Q i , record the transformation to the point cloud cluster U at this position i The vertex coordinates of the through hole in the internal calibration plate are but

[0087] S145. Construct a virtual calibration plate identical to the calibration carrier on the YOZ plane in the laser coordinate system. When the virtual calibration plate is in this position, the lower left corner coincides with point O, and the two sides are located on the Y axis and the Z axis respectively. The corner point of the through hole on it is According to the preset matching error function Cost calculation, a transformation matrix M is obtained i2 , transformation matrix M i2 Contains rotation around the x-axis and translation along the y-axis and z-axis, so that the point cloud cluster U i After the matrix M i2 After transformation, it coincides with the virtual calibration plate, then it satisfies

[0088] S146, through the obtained transformation matrix M i1 、M i2 Get the original coordinates of the corner points of the through hole in the laser coordinate system

[0089] In a preferred embodiment, step S142 includes: setting the laser coordinate system vertically upward as the positive direction of the z axis, the front of the laser radar as the positive direction of the x axis, and the left direction of the laser radar as the positive direction of the y axis, where the x and y axes form a horizontal plane, and using the Ransac algorithm to solve the laser radar point cloud set U i1The plane equation of the plane in which the problem lies is calculated. The Random Sampling Consensus (RANSAC) algorithm uses an iterative approach to estimate the parameters of a mathematical model from a set of observed data that contains outliers. The RANSAC algorithm assumes that the data contains both correct data and outliers (or noise). Correct data are referred to as inliers, and outliers as outliers. RANSAC also assumes that, given a set of correct data, there exists a method to calculate the model parameters that fit this data. The core concepts of this algorithm are randomness and assumptions. Randomness involves randomly selecting sampled data based on the probability of correct data occurring. According to the law of large numbers, random simulation can approximate the correct result. Assumptions assume that the selected sampled data are correct. These correct data are then used to calculate other points using the model that satisfies the problem, and the results are then scored.

[0090] The RANSAC algorithm is widely used in computer vision and mathematics, for applications such as line fitting, plane fitting, calculating transformation matrices between images or point clouds, and computing basic matrices. This article introduces RANSAC and includes two sections of MATLAB code for line fitting and plane fitting. OpenCV provides corresponding function interfaces for solving matrix problems based on the RANSAC framework in computer vision.

[0091] Basic idea of RANSAC algorithm

[0092] (1) To obtain a straight line model, two points are required to uniquely determine the equation of the line. Therefore, the first step is to randomly select two points.

[0093] (2) Through these two points, the model equation y=ax+b represented by these two points can be calculated.

[0094] (3) Fit all data points into this model and calculate the error.

[0095] (4) Find all points that meet the error threshold.

[0096] (5) Then we repeat the process (1) to (4) until a certain number of iterations are reached, and then we select the model with the most support as the solution to the problem. As shown in the figure below:

[0097] As you can see, although the ratio of outliers to inliers in this dataset is almost equal, the RANSAC algorithm still finds the most appropriate solution. If this problem were optimized using least squares, the result would undoubtedly be incorrect due to the interference of the noisy data, as shown in the figure below. This is because least squares is a cost optimization problem that takes outliers into account, while RANSAC optimizes using inliers. Experimental verification has shown that for datasets containing 80% error, RANSAC performs far better than direct least squares.

[0098] In this embodiment, the plane equation is aX+bY+cZ+d=0, and (a, b, c) is a unit vector. i1 Randomly select the coordinates of 4 laser points, solve the values of a, b, c, d, and use U i1 Substitute all laser points in the current aX+bY+cZ+d=0, and the statistical distance from this plane is less than the threshold D f The number of laser points n p Repeat m times and select n p The maximum values of a, b, c, and d are the best fitting plane parameters l(a, b, c, d).

[0099] In a preferred embodiment, the lidar point cloud cluster U i The plane parameters of the plane are l(a, b, c, d), then the normal vector corresponding to the plane is (a, b, c), then the rotation matrix corresponding to the plane is

[0100] After transformation, the point cloud cluster U i Transform to be parallel to YOZ. Note the laser point cloud cluster U i The mean coordinates of all lidar points in Then let M i1 =(R i1 , t i1 ), which can make the laser point cloud cluster U i M i1 After the transformation, it can roughly coincide with the YOZ plane, and the laser point cloud cluster after the transformation position is recorded as U′ i .

[0101] In a preferred embodiment, the transformation matrix M i1 Transformed laser point cloud cluster U i , apply translation and rotation, and record the parameters as (dy, dz, dθ), where dy represents the translation along the y-axis, dz represents the translation along the z-axis, and dθ represents the rotation around the x-axis. The rotation matrix corresponding to dθ is The displacement matrix is make through Transformed laser point cloud cluster U′ i After changing position, it becomes U′ i ', will U' i 'The laser points in the area are projected onto the YOZ surface, and then the number of laser points in the square through-hole area and the non-through-hole area on the virtual calibration board are counted as num A 、num B , let the weights be weight A 、weight B , weight A 、weight B The ratio is set to Then we get the following objective function:

[0102] Cost=num A *weight A +num B *weight B .

[0103] Used to determine the laser point cloud U′ after the transformed position i ', whether it can accurately coincide with the virtual calibration plate, we know that the minimum objective function Cost value corresponds to the optimal translation and rotation parameters. The value range of the translation and rotation parameters can be determined by the relative position of the calibration carrier and the lidar. The translation distance is generally not more than half of the short side length of the calibration carrier. The rotation angle can be adjusted by placing the calibration carrier in advance so that it does not exceed 30°. The grid search method is used to find the optimal translation and rotation parameters that minimize the Cost value within the value range, which is recorded as the transformation matrix M i2 .

[0104] In a preferred embodiment, in step S130, the coordinates of the corner points of the calibration plate at different positions in the image obtained in step S120 are Obtaining the camera intrinsic parameter matrix K and distortion parameter s includes the following steps:

[0105] S131, collect the coordinates of the black and white grid corner points of the calibration carrier in the image under different postures

[0106] S132. Calculate the intrinsic parameters of the camera according to Zhang's calibration method to obtain the camera's intrinsic parameter matrix K and distortion parameters s(k1, k2, p1, p2, k3).

[0107] S133, record the three-dimensional coordinates of each corner point in the camera coordinate system as According to the pinhole model, the transformation from the camera three-dimensional space to the image plane can be expressed by the formula Indicates that f is the focal length of the camera, u0, v0 is the difference between the projection of the camera optical center on the image plane and the image center, Represents the coordinates of the image point before dedistortion.

[0108] S134, obtain

[0109] S135. Considering the camera lens distortion, according to the camera distortion model, the camera dedistortion formula Recorded as

[0110] S136, combined with formula (1)(2)(3), the black and white grid corner points of the calibration plate in the camera's three-dimensional coordinate system can be The coordinates projected onto the image are Right now

[0111] In a preferred embodiment, in step S150, i Selecting an external parameter matrix E with the minimum reprojection error includes the following steps:

[0112] S151, let the external parameter matrix from the lidar coordinate system to the image coordinate system at each posture i be E i ,but

[0113] S152, combined with formula (4) and (5), the three-dimensional point in the laser radar coordinate system can be projected onto the image plane, which is recorded as The projection error can be expressed as According to the projection error, the external parameter matrix E can be solved by iterative optimization i .

[0114] S153, the external parameter matrix E obtained under different postures i The corresponding reprojection error Sort by the external parameter matrix E with the smallest reprojection error. i As the optimal external parameter matrix E.

[0115] In a preferred embodiment, the following steps are also included:

[0116] S160. Obtain a first transformation matrix from the camera coordinate system to the image coordinate system based on the intrinsic parameter matrix K of the camera, obtain a second transformation matrix from the laser coordinate system to the camera coordinate system based on the extrinsic parameter matrix E of the camera, and obtain a third transformation matrix from the laser coordinate system to the image coordinate system through the first transformation matrix and the second transformation matrix.

[0117] Figures 2 to 6 FIG. 1 is a schematic diagram of the implementation process of the laser radar and camera calibration method of the present invention. Figures 2 to 6 As shown, the implementation process of the present invention is as follows:

[0118] like Figure 2 As shown, prepare a calibration carrier 1. It features an 8×16 checkerboard grid. All squares are standard squares with a side length of 0.1m. There are 64 black squares 11 and 64 white squares. 32 of the white squares are hollowed out to form 32 rectangular holes 13 for laser light to pass through, along with the remaining 32 white squares 12. Cut out the 32 rectangular holes 13, starting from the lower left corner, with adjacent rectangular holes 13 arranged in alternate rows and columns. Place the calibration carrier in front of the laser sensor and camera. Ideally, the background of the calibration carrier should be white to minimize the impact of the square holes on camera calibration. In the present invention, the points where the four corners of the white square 12 and the four corners of the black square 11 in the image obtained by the binocular camera intersect are used as corner points 14, and the four corners of the hollow pattern of the rectangular through hole 13 in the point cloud obtained by the laser radar are used as vertices 15 (in the world coordinate system, the corner points 14 and the vertices 15 completely overlap in three-dimensional space).

[0119] like Figure 3 As shown, the laser radar 2 and the binocular camera 3 are fixed to the calibration carrier 1, the calibration plate posture is changed, the laser point cloud data at different postures i are collected, and the intersection corners of the black and white grids on the calibration plate in the image are obtained. The subscript j represents the jth corner point.

[0120] like Figure 4 As shown, the laser radar coordinate system and camera coordinate system (X cam , Y cam , Z cam ) and the image coordinate system (Y img , Z img ). The coordinates of the corner points of the calibration plate at different positions in the image obtained by the prior art according to step S120 are The camera internal parameter matrix K and distortion parameter s are obtained, which will not be described here. For example: First, the coordinates of the black and white grid corners of the calibration carrier in the image at different postures are collected. According to Zhang’s calibration method, the camera’s intrinsic parameters are calculated to obtain the camera’s intrinsic parameter matrix K and distortion parameters s(k1, k2, p1, p2, k3). The three-dimensional coordinates of each corner point in the camera coordinate system are recorded as According to the pinhole model, the transformation from the camera three-dimensional space to the image plane can be expressed by the formula Indicates that f is the focal length of the camera, u0, v0 is the difference between the projection of the camera optical center on the image plane and the image center, Represents the coordinates of the image point before dedistortion. Considering the camera lens distortion, according to the camera distortion model, the camera distortion formula Recorded as Combined with formula (1)(2)(3), the black and white grid corner points of the calibration plate in the camera's three-dimensional coordinate system can be The coordinates projected onto the image are Right now

[0121] like Figure 5 、 6 As shown, according to the image corner coordinates under each pose i And the vertex coordinates of the through hole in the laser coordinate system Correspondence, calculate the external parameter matrix E from the lidar coordinate system to the camera coordinate system i For example: Each laser point All three-dimensional coordinate information is collected and a reflection intensity information The initial calibration carrier position is recorded as Q0. By estimating the position of the calibration plate in the laser coordinate system, the laser points other than those on the calibration plate are filtered for the first time using the preset coordinate threshold to obtain the first residual lidar point cloud set U. i0 The first remaining lidar point cloud set U i0 Calculate the average value of the three-dimensional coordinates of the laser points on the calibration plate, record it as P0 (x0, y0, z0), and set up a sphere with this as the center and 0.8 times the length of the short side of the calibration plate as the diameter. Perform a second filtering to remove the laser points outside the sphere and obtain the second residual lidar point cloud set U i1 , with U i1 The remaining laser point coordinates in the plane are fitted to the plane where the calibration carrier is located, and the plane parameters are recorded as l(a, b, c, d). The laser coordinate system is vertically upward as the positive direction of the z axis (Z laser ), the front of the laser radar is the positive direction of the x-axis (X laser ), the left direction of the laser radar is the positive direction of the y-axis (Y laser ), where the x and y axes form a horizontal plane, and the Ransac algorithm is used to solve the lidar point cloud set U i1 The plane equation of the plane where it is located, let the plane equation be aX+bY+cZ+d=0, and (a, b, c) are unit vectors, from U i1 Randomly select the coordinates of 4 laser points, solve the values of a, b, c, d, and use U i1 Substitute all laser points in the current aX+bY+cZ+d=0, and the statistical distance from this plane is less than the threshold D f The number of laser points n p Repeat m times and select n p The maximum a, b, c, d values are the best fitting plane parameters l(a, b, c, d). i0The distance from the laser point to the fitting plane l is less than or equal to the preset threshold Ds as the filtering condition, and the third filtering is performed to obtain the lidar point cloud cluster U i According to the fitting plane l, a transformation matrix M is obtained i1 , for the lidar point cloud cluster U projected on the calibration plate i After this transformation, it is moved to the YOZ plane in the laser coordinate system, and the position of the calibration plate after the movement is recorded as Q i , record the transformation to the point cloud cluster U at this position i The vertex coordinates of the through hole in the internal calibration plate are but LiDAR point cloud cluster U i The plane parameters of the plane are l(a, b, c, d), then the normal vector corresponding to the plane is (a, b, c), then the rotation matrix corresponding to the plane is After transformation, the point cloud cluster U i Transform to be parallel to YOZ. Note the laser point cloud cluster U i The mean coordinates of all lidar points in Then let M i1 =(R i1 , t i1 ), which can make the laser point cloud cluster U i M i1 After the transformation, it can roughly coincide with the YOZ plane, and the laser point cloud cluster after the transformation position is recorded as U′ i A virtual calibration plate identical to the calibration carrier is constructed on the YOZ plane in the laser coordinate system. In this position, the lower left corner of the virtual calibration plate coincides with point O, and the two sides are located on the Y axis and the Z axis respectively. The corner point of the through hole is According to the preset matching error function Cost calculation, a transformation matrix M is obtained i2 , transformation matrix M i2 Contains rotation around the x-axis and translation along the y-axis and z-axis, so that the point cloud cluster U i After the matrix M i2 After transformation, it coincides with the virtual calibration plate, then it satisfies By transforming the matrix M i1 Transformed laser point cloud cluster U′ i , apply translation and rotation, and record the parameters as (dy, dz, dθ), where dy represents the translation along the y-axis, dz represents the translation along the z-axis, and dθ represents the rotation around the x-axis. The rotation matrix corresponding to dθ is The displacement matrix is make Sutra Transformed laser point cloud cluster U′ i After changing position, it becomes U′ i ', will U' i'The laser points in the area are projected onto the YOZ surface, and then the number of laser points in the square through-hole area and the non-through-hole area on the virtual calibration board are counted as num A 、num B , let the weights be weight A 、weight B , weight A 、weight B The ratio is set to Then we get the following objective function:

[0122] Cost=num A *weight A +num B *weight B .

[0123] Used to determine the laser point cloud U after the transformation i ', whether it can accurately coincide with the virtual calibration plate, we know that the minimum objective function Cost value corresponds to the optimal translation and rotation parameters. The value range of the translation and rotation parameters can be determined by the relative position of the calibration carrier and the lidar. The translation distance is generally not more than half of the short side length of the calibration carrier. The rotation angle can be adjusted by placing the calibration carrier in advance so that it does not exceed 30°. The grid search method is used to find the optimal translation and rotation parameters that minimize the Cost value within the value range, which is recorded as the transformation matrix M i2 . Through the obtained transformation matrix M i1 、M i2 Get the original coordinates of the corner points of the through hole in the laser coordinate system

[0124] The external parameter matrix from the lidar coordinate system to the image coordinate system at each posture i is recorded as E i ,but Combining formulas (4) and (5), the three-dimensional point in the laser radar coordinate system can be projected onto the image plane, which can be expressed as The projection error can be expressed as According to the projection error, the external parameter matrix E can be solved by iterative optimization i The external parameter matrix E obtained under different postures i The corresponding reprojection error Sort by the external parameter matrix E with the smallest reprojection error. i As the optimal external parameter matrix E.

[0125] Finally, the existing technology is used to obtain the first transformation matrix from the camera coordinate system to the image coordinate system according to the camera's intrinsic parameter matrix K, and the second transformation matrix from the laser coordinate system to the camera coordinate system is obtained according to the camera's extrinsic parameter matrix E. Through the first transformation matrix and the second transformation matrix, the third transformation matrix from the laser coordinate system to the image coordinate system is obtained. The relevant conversion process uses the existing conversion formula and will not be repeated here.

[0126] The present invention estimates the position of a calibration carrier in the laser coordinate system by its placement. A laser sensor then emits laser light toward the calibration carrier, filtering out laser points other than those on the calibration plate. After filtering, the majority of the laser points remain on the calibration plate, while the remaining laser light passes through the square holes. The laser sensor then acquires the coordinates of the laser points on the calibration carrier.

[0127] Calculate the average value of the three-dimensional coordinates of the laser points on the calibration plate, record it as P0 (x0, y0, z0), set up a sphere with this as the center and 0.8 times the length of the short side of the calibration plate as the diameter, further filter the laser points outside the sphere, and use the coordinates of the laser points inside the sphere to perform plane fitting on the plane where the calibration carrier is located, as shown in the figure. Figure 5 As shown in the figure, the purpose of doing this is to avoid the laser point at the edge of the calibration plate from affecting the fitting process.

[0128] The calibration method proposed in this paper can effectively calibrate the laser sensor and camera. The characteristics of the calibration process are summarized as follows:

[0129] During the calibration process, the present invention filters the laser points and finds the center of gravity of the laser point distribution. Only the coordinates of the laser points within a specific range with the center of gravity as the center of the circle are used for plane fitting, thereby avoiding the adverse effects of laser points located at the edge of the calibration plate on the plane fitting.

[0130] During the calibration process, the present invention only needs to use a special calibration plate, which reduces the calibration cost and workload;

[0131] The present invention converts the solution of the coordinates of the through-hole corner points in the laser radar into a voting calculation with several iterations and a 3-degree-of-freedom grid search, which greatly reduces the amount of calculation.

[0132] The calibration process proposed by the present invention does not require human-computer interaction, and the transformation matrix from the lidar to the camera can be solved automatically, which simplifies the calibration process and improves the calibration efficiency.

[0133] Figure 7 Schematic diagram of the structure of the laser radar and camera calibration system of the present invention. Figure 7 As shown, the laser radar and camera calibration system of the present invention includes:

[0134] The calibration carrier module 51 provides a calibration carrier, the surface of which includes black squares and white squares arranged in a grid and through holes for laser to pass through;

[0135] Laser point cloud module 52, fixes the laser radar and camera to align with the calibration carrier, changes the calibration plate posture, collects laser point cloud data at different postures i, and obtains the intersection corner points of the black and white grids on the calibration plate in the image Subscript j indicates the jth corner point;

[0136] The internal parameter matrix module 53 is used to calculate the corner coordinates of the calibration plate at different positions in the image obtained by the laser point cloud module 52. Obtain the camera intrinsic parameter matrix K and distortion parameter s;

[0137] The external parameter matrix module 54 is based on the image corner coordinates at each posture i. And the vertex coordinates of the through hole in the laser coordinate system Correspondence, calculate the external parameter matrix E from the lidar coordinate system to the camera coordinate system i ;as well as

[0138] External parameter sorting module 55, from E i Select an extrinsic parameter matrix E with the smallest reprojection error.

[0139] The laser radar and camera calibration system of the present invention can automatically perform calibration, ensuring that the laser sensor and the binocular camera are in the same coordinate system, greatly simplifying the calibration process, and improving the accuracy and efficiency of calibration.

[0140] An embodiment of the present invention further provides a laser radar and camera calibration device, comprising a processor and a memory storing executable instructions for the processor. The processor is configured to execute the executable instructions to perform the steps of a laser radar and camera calibration method.

[0141] As described above, the laser radar and camera calibration equipment of the present invention can automatically perform calibration, ensuring that the laser sensor and the binocular camera are in the same coordinate system, and greatly simplifying the calibration process, thereby improving the accuracy and efficiency of calibration.

[0142] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Accordingly, various aspects of the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as "circuits," "modules," or "platforms."

[0143] Figure 8 This is a schematic diagram of the structure of the laser radar and camera calibration equipment of the present invention. Figure 8 An electronic device 600 according to this embodiment of the present invention will be described. Figure 8 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0144] like Figure 8 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), and a display unit 640.

[0145] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present invention described in the electronic prescription circulation processing method section of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0146] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0147] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0148] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0149] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0150] An embodiment of the present invention further provides a computer-readable storage medium for storing a program that, when executed, implements the steps of the laser radar and camera calibration method. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the electronic prescription circulation processing method section above.

[0151] As shown above, when the program of the computer-readable storage medium of this embodiment is executed, it can automatically perform calibration to ensure that the laser sensor and the binocular camera are in the same coordinate system, greatly simplify the calibration process, and improve the accuracy and efficiency of the calibration.

[0152] Figure 9 Schematic diagram of the structure of the computer readable storage medium of the present invention. Figure 9 , a program product 800 for implementing the above method according to an embodiment of the present invention is described. The program product 800 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0153] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0154] Computer-readable storage media may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0155] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0156] In summary, the laser radar and camera calibration method, system, device and storage medium of the present invention can automatically perform calibration, ensure that the laser sensor and binocular camera are in the same coordinate system, greatly simplify the calibration process, and improve the accuracy and efficiency of calibration.

[0157] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A laser radar and camera calibration method, characterized in that: The following steps are involved: S110, providing a calibration carrier, wherein the surface of the calibration carrier includes black squares and white squares arranged in a grid and through holes for laser to pass through; S120, fix the laser radar and camera to align with the calibration carrier, change the calibration plate posture, collect laser point cloud data at different postures i, and obtain the intersection corner points of the black and white grids on the calibration plate in the image Subscript j indicates the jth corner point; S130: coordinates of corner points of the calibration plate at different positions in the image obtained in step S120 Obtain the camera intrinsic parameter matrix K and distortion parameter s; S140, according to the image corner coordinates under each posture i And the vertex coordinates of the through hole in the laser coordinate system Correspondence, calculate the external parameter matrix E from the lidar coordinate system to the camera coordinate system i ; Under different calibration carrier poses i, the coordinates of the vertex of the calibration plate through hole in the lidar coordinate system The calculation steps are as follows: Each laser point All three-dimensional coordinate information is collected and a reflection intensity information The initial calibration carrier position is recorded as Q0. By estimating the position of the calibration plate in the laser coordinate system, the laser points other than those on the calibration plate are filtered for the first time using the preset coordinate threshold to obtain the first residual lidar point cloud set U. i0 ; The first remaining lidar point cloud set U i0 Calculate the average value of the three-dimensional coordinates of the laser points on the calibration plate, record it as P0 (x0, y0, z0), and set up a sphere with this as the center and W times the length of the short side of the calibration plate as the diameter. Perform a second filtering to remove the laser points outside the sphere and obtain the second residual lidar point cloud set U i1 , with U i1 The remaining laser point coordinates in the plane where the calibration carrier is located are fitted to the plane, and the plane parameters are recorded as l (a, b, c, d). The value range of W is 0.6 to 0.9; the laser coordinate system is vertically upward as the positive direction of the z axis, the front of the laser radar is the positive direction of the x axis, and the left direction of the laser radar is the positive direction of the y axis, where the x and y axes form a horizontal plane, and the Ransac algorithm is used to solve the laser radar point cloud set U i1 The plane equation of the plane where it is located, let the plane equation be aX+bY+cZ+d=0, and (a, b, c) are unit vectors, from U i1 Randomly select the coordinates of 4 laser points, solve the values of a, b, c, d, and use U i1 Substitute all laser points in the current aX+bY+cZ+d=0, and the statistical distance from this plane is less than the threshold D f The number of laser points n p ; Repeat m times, select n p The maximum a, b, c, d values are the best fitting plane parameters l(a, b, c, d), and the lidar point cloud cluster U i The plane parameters of the plane are l(a, b, c, d), then the normal vector corresponding to the plane is (a, b, c), then the rotation matrix corresponding to the plane is After transformation, the point cloud cluster U i Transform to be parallel to YOZ; record the laser point cloud cluster U i The mean coordinates of all lidar points in Then let M i1 =(R i1 , t i1 ), so that the laser point cloud cluster U i M i1 After the transformation, it coincides with the YOZ plane, and the laser point cloud cluster after the transformation is recorded as U i ′ , after the transformation matrix M i1 The transformed laser point cloud cluster U i ′ , apply translation and rotation, and record the parameters as (dy, dz, dθ), where dy represents the translation along the y-axis, dz represents the translation along the z-axis, and dθ represents the rotation around the x-axis. The rotation matrix corresponding to dθ is The displacement matrix is make through Transformed laser point cloud cluster U i ′ After changing position, it becomes U i ′’ , will U i ′’ The laser points in the square through-hole area and the non-through-hole area are projected onto the YOZ surface, and then the numbers of laser points in the square through-hole area and the non-through-hole area on the virtual calibration board are counted as num6 and num6 respectively. B , set the weights to weight6, weight B , weight6, weight B The ratio of If n is a natural number less than the total number of white squares in the chessboard, the objective function is as follows: Cost=num6*weight6+num B *weight B ; Used to determine the laser point cloud U after the transformation i ′’ , whether it can accurately coincide with the virtual calibration plate, and obtain the optimal translation and rotation parameters corresponding to the minimum objective function Cost value; the value range of the translation and rotation parameters is determined by the relative position of the calibration carrier and the laser radar. The translation distance generally does not exceed half of the length of the short side of the calibration carrier, and the rotation angle does not exceed 30° by placing the calibration carrier in advance. The grid search method is used to find the optimal translation and rotation parameters that minimize the Cost value within the value range, which is recorded as the transformation matrix M i2 ; The first remaining lidar point cloud set U i0 The distance from the laser point to the fitting plane l is less than or equal to the preset threshold Ds as the filtering condition, and the third filtering is performed to obtain the lidar point cloud cluster U i According to the fitting plane l, a transformation matrix M is obtained i1 , for the lidar point cloud cluster U projected on the calibration plate i After this transformation, it is moved to the YOZ plane in the laser coordinate system, and the position of the calibration plate after the movement is recorded as Q i , record the transformation to the point cloud cluster U at this position i The vertex coordinates of the through hole in the internal calibration plate are but A virtual calibration plate identical to the calibration carrier is constructed on the YOZ plane in the laser coordinate system. In this position, the lower left corner of the virtual calibration plate coincides with point O, and its two sides are located on the Y axis and the Z axis respectively. The corner point of the through hole on it is According to the preset matching error function Cost calculation, a transformation matrix M is obtained i2 , the transformation matrix M i2 Contains rotation around the x-axis and translation along the y-axis and z-axis, so that the point cloud cluster U i After the matrix M i2 After the transformation, it coincides with the virtual calibration plate, and then satisfies Through the obtained transformation matrix M i1 、M i2 Obtain the original coordinates of the corner points of the through hole in the laser coordinate system as well as S150, from E i Select an extrinsic parameter matrix E with the smallest reprojection error.

2. The laser radar and camera calibration method according to claim 1, wherein: The calibration carrier is a checkerboard tile with a total of multiple squares, and rectangular through holes with known side lengths of n are hollowed out at the locations where some white squares are located, and adjacent rectangular through holes are arranged in at least alternate rows and columns.

3. The laser radar and camera calibration method according to claim 1, wherein: In step S130, the coordinates of the corner points of the calibration plate at different positions in the image obtained in step S120 are obtained. Obtaining the camera intrinsic parameter matrix K and distortion parameter s includes the following steps: S131, collecting the coordinates of the black and white grid corner points of the calibration carrier in the image at different postures S132. Calculate the camera's intrinsic parameters according to Zhang's calibration method to obtain the camera's intrinsic parameter matrix K and distortion parameters s(k1, k2, p1, p2, k3); S133, record the three-dimensional coordinates of each corner point in the camera coordinate system as According to the pinhole model, the transformation from the camera three-dimensional space to the image plane is given by the formula (1) means, f is the focal length of the camera, u0, v0 is the difference between the projection of the camera optical center on the image plane and the image center, Represents the coordinates of the image point before dedistortion processing; S134, obtain S135. Considering the camera lens distortion, according to the camera distortion model, the camera distortion removal formula S136, combined with formula (1)(2)(3), the black and white grid corner points of the calibration plate in the camera's three-dimensional coordinate system The coordinates projected onto the image are Right now 4. The laser radar and camera calibration method according to claim 3, wherein: In step S150, i Selecting an external parameter matrix E with the minimum reprojection error includes the following steps: S151, let the external parameter matrix from the lidar coordinate system to the image coordinate system at each posture i be E i ,but S152, combining formulas (4) and (5), project the three-dimensional point in the laser radar coordinate system onto the image plane, which is recorded as The projection error is recorded as Then the external parameter matrix E is solved by iterative optimization based on the projection error i ; S153, the external parameter matrix E obtained under different postures i The corresponding reprojection error Sort by the external parameter matrix E with the smallest reprojection error. i As the optimal external parameter matrix E.

5. The laser radar and camera calibration method according to claim 4, wherein: The following steps are also included: S160. Obtain a first transformation matrix from the camera coordinate system to the image coordinate system based on the intrinsic parameter matrix K of the camera, obtain a second transformation matrix from the laser coordinate system to the camera coordinate system based on the extrinsic parameter matrix E of the camera, and obtain a third transformation matrix from the laser coordinate system to the image coordinate system through the first transformation matrix and the second transformation matrix.

6. A laser radar and camera calibration system, characterized in that: The method for calibrating a laser radar and a camera according to claim 1 comprises: The calibration carrier module provides a calibration carrier, wherein the surface of the calibration carrier includes black squares and white squares arranged in a grid and through holes for laser to pass through; The laser point cloud module fixes the laser radar and the camera to align with the calibration carrier, changes the calibration plate posture, collects laser point cloud data at different postures i, and obtains the intersection corners of the black and white grids on the calibration plate in the image Subscript j indicates the jth corner point; The internal parameter matrix module is used to calculate the corner coordinates of the calibration plate at different positions in the image obtained by the laser point cloud module. Obtain the camera intrinsic parameter matrix K and distortion parameter s; The external parameter matrix module is based on the image corner coordinates at each pose i And the vertex coordinates of the through hole in the laser coordinate system Correspondence, calculate the external parameter matrix E from the lidar coordinate system to the camera coordinate system i ; Under different calibration carrier poses i, the coordinates of the vertex of the calibration plate through hole in the lidar coordinate system The calculation steps are as follows: Each laser point All three-dimensional coordinate information is collected and a reflection intensity information The initial calibration carrier position is recorded as Q0. By estimating the position of the calibration plate in the laser coordinate system, the laser points other than those on the calibration plate are filtered for the first time using the preset coordinate threshold to obtain the first residual lidar point cloud set U. i0 ; The first remaining lidar point cloud set U i0 Calculate the average value of the three-dimensional coordinates of the laser points on the calibration plate, record it as P0 (x0, y0, z0), and set up a sphere with this as the center and W times the length of the short side of the calibration plate as the diameter. Perform a second filtering to remove the laser points outside the sphere and obtain the second residual lidar point cloud set U i1 , with U i1 The remaining laser point coordinates in the plane where the calibration carrier is located are fitted to the plane, and the plane parameters are recorded as l (a, b, c, d). The value range of W is 0.6 to 0.9; the laser coordinate system is vertically upward as the positive direction of the z axis, the front of the laser radar is the positive direction of the x axis, and the left direction of the laser radar is the positive direction of the y axis, where the x and y axes form a horizontal plane, and the Ransac algorithm is used to solve the laser radar point cloud set U i1 The plane equation of the plane where it is located, let the plane equation be aX+bY+cZ+d=0, and (a, b, c) are unit vectors, from U i1 Randomly select the coordinates of 4 laser points, solve the values of a, b, c, d, and use U i1 Substitute all laser points in the current aX+bY+cZ+d=0, and the statistical distance from this plane is less than the threshold D f The number of laser points n p ; Repeat m times, select n p The maximum a, b, c, d values are the best fitting plane parameters l(a, b, c, d), and the lidar point cloud cluster U i The plane parameters of the plane are l(a, b, c, d), then the normal vector corresponding to the plane is (a, b, c), then the rotation matrix corresponding to the plane is After transformation, the point cloud cluster U i Transform to be parallel to YOZ; record the laser point cloud cluster U i The mean coordinates of all lidar points in Then let M i1 =(R i1 , t i1 ), so that the laser point cloud cluster U i M i1 After the transformation, it coincides with the YOZ plane, and the laser point cloud cluster after the transformation is recorded as U i ′ , after the transformation matrix M i1 The transformed laser point cloud cluster U i ′ , apply translation and rotation, and record the parameters as (dy, dz, dθ), where dy represents the translation along the y-axis, dz represents the translation along the z-axis, and dθ represents the rotation around the x-axis. The rotation matrix corresponding to dθ is The displacement matrix is make through Transformed laser point cloud cluster U i ′ After changing position, it becomes U i ′’ , will U i ′’ The laser points in the square through-hole area and the non-through-hole area are projected onto the YOZ surface, and then the numbers of laser points in the square through-hole area and the non-through-hole area on the virtual calibration board are counted as num6 and num6 respectively. B , set the weights to weight6, weight B , weight6, weight B The ratio is set to If n is a natural number less than the total number of white squares in the chessboard, the following objective function is obtained: Cost=num6*weight6+num B *weight B ; Used to determine the laser point cloud U after the transformation i ′’ , whether it can accurately coincide with the virtual calibration plate, and obtain the optimal translation and rotation parameters corresponding to the minimum objective function Cost value; the value range of the translation and rotation parameters is determined by the relative position of the calibration carrier and the laser radar. The translation distance generally does not exceed half of the length of the short side of the calibration carrier, and the rotation angle does not exceed 30° by placing the calibration carrier in advance. The grid search method is used to find the optimal translation and rotation parameters that minimize the Cost value within the value range, which is recorded as the transformation matrix M i2 ; The first remaining lidar point cloud set U i0 The distance from the laser point to the fitting plane l is less than or equal to the preset threshold Ds as the filtering condition, and the third filtering is performed to obtain the lidar point cloud cluster U i According to the fitting plane l, a transformation matrix M is obtained i1 , for the lidar point cloud cluster U projected on the calibration plate i After this transformation, it is moved to the YOZ plane in the laser coordinate system, and the position of the calibration plate after the movement is recorded as Q i , record the transformation to the point cloud cluster U at this position i The vertex coordinates of the through hole in the internal calibration plate are but A virtual calibration plate identical to the calibration carrier is constructed on the YOZ plane in the laser coordinate system. In this position, the lower left corner of the virtual calibration plate coincides with point O, and its two sides are located on the Y axis and the Z axis respectively. The corner point of the through hole on it is According to the preset matching error function Cost calculation, a transformation matrix M is obtained i2 , the transformation matrix M i2 Contains rotation around the x-axis and translation along the y-axis and z-axis, so that the point cloud cluster U i After the matrix M i2 After the transformation, it coincides with the virtual calibration plate, and then satisfies Through the obtained transformation matrix M i1 、M i2 Obtain the original coordinates of the corner points of the through hole in the laser coordinate system as well as External parameter sorting module, from E i Select an extrinsic parameter matrix E with the smallest reprojection error.

7. A laser radar and camera calibration device, characterized in that: include: processor; a memory storing executable instructions for the processor; The processor is configured to execute the steps of the laser radar and camera calibration method of claim 1 by executing executable instructions.

8. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the laser radar and camera calibration method described in claim 1 are implemented.

Images