Camera and lidar calibration method

Abstract

This application discloses a camera and LiDAR calibration method. On one hand, the 2D pixel coordinates of the calibration board vertices are extracted from the image captured by the camera of a pre-set calibration board in the environment. By establishing a 2D world coordinate system and a 3D world coordinate system, the 2D world coordinates and 3D world coordinates of the calibration board vertices are obtained. Based on the 2D pixel coordinates, 2D world coordinates, and 3D world coordinates of the calibration board vertices, as well as the camera's internal parameters, the 3D camera coordinates of the calibration board vertices are calculated. On the other hand, the 3D point cloud coordinates of the calibration board vertices are extracted from the point cloud collected by the LiDAR of the pre-set calibration board in the environment. Then, based on the set of 3D camera coordinates and the set of 3D point cloud coordinates of the calibration board vertices, the calibration parameters between the camera and the LiDAR are solved. These calibration parameters include rotation parameters and translation parameters. Since no depth information is lost, the calibration accuracy of this method is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fusion between a camera and a laser radar, and particularly relates to a camera and laser radar calibration method. BACKGROUND

[0002] Calibration between a camera and a laser radar aims to obtain the transformation of coordinates of two sensors, that is, to convert point data from the laser radar and the camera into the same coordinate system, and then to fuse data from the two sensors to improve their perception, such as object detection, classification, tracking, and the like.

[0003] According to whether manual intervention is required in the calibration process, external parameter calibration between a camera and a laser radar sensor can be divided into two categories: a manual method and an automatic method. The manual method requires an engineer to manually specify the correspondence between laser radar point clouds and camera images according to one or more calibration objects, so as to impose geometric constraints between corresponding 3D points in the point cloud and pixels in the image, and estimate the external parameters between the camera and the laser radar sensor. The automatic method aims to eliminate manual intervention and automatically extract and match features from the calibration target observed by the two sensors. After the sensor calibration environment is built, the system can automatically calibrate the external parameters between the camera and the laser radar sensor.

[0004] The existing automatic calibration method between a camera and a laser radar usually maps 3D points in the point cloud obtained by the laser radar to an image, and solves the calibration parameters between the camera and the laser radar based on the mapped image and the image obtained by the camera. However, since the mapping of the 3D points in the point cloud obtained by the laser radar to the image is a conversion from 3D to 2D, the depth information is lost, and therefore, this method has a large error. SUMMARY

[0005] To solve the above technical problems, the embodiments of the present application provide a camera and laser radar calibration method to solve the calibration parameters between the camera and the laser radar based on the 3D camera coordinates and 3D point cloud coordinates of the vertexes of a calibration board, and improve the calibration accuracy.

[0006] To achieve the above object, the embodiments of the present application provide the following technical solutions:

[0007] A camera and laser radar calibration method comprises:

[0008] obtaining an image of a calibration board pre-set in an environment shot by a camera, and extracting 2D pixel coordinates of vertexes of the calibration board from the image;

[0009] establishing a 2D world coordinate system and a 3D world coordinate system based on physical parameters of the calibration board, and obtaining 2D world coordinates and 3D world coordinates of the vertexes of the calibration board.

[0010] calculating 3D camera coordinates of the vertices of the calibration board based on the 2D pixel coordinates, the 2D world coordinates and the 3D world coordinates of the vertices of the calibration board, and internal parameters of the camera, to obtain a set of 3D camera coordinates of the vertices of the calibration board, wherein the internal parameters of the camera are calibration parameters of mapping from a 3D camera coordinate system to a 2D pixel coordinate system;

[0011] obtaining point cloud data collected by the laser radar on the preset calibration board in the environment, and extracting 3D point cloud coordinates of the vertices of the calibration board from the point cloud data to obtain a set of 3D point cloud coordinates of the vertices of the calibration board;

[0012] solving the calibration parameters between the camera and the laser radar based on the set of 3D camera coordinates and the set of 3D point cloud coordinates of the vertices of the calibration board, wherein the calibration parameters between the camera and the laser radar include rotation parameters and translation parameters.

[0013] Optionally, before extracting the 2D pixel coordinates of the vertices of the calibration board from the image, the method further comprises:

[0014] correcting distortion of the image to obtain a corrected image.

[0015] Optionally, extracting the 2D pixel coordinates of the vertices of the calibration board from the image comprises:

[0016] calculating an inner corner point response value of each pixel in the image based on a gray value of the pixel, wherein the greater the inner corner point response value of a pixel is, the more likely the pixel is an inner corner point of the calibration board, and the inner corner point of the calibration board is a vertex of a black and white grid inside the calibration board, excluding vertices of the calibration board and vertices of the black and white grid on edges of the calibration board;

[0017] determining whether the inner corner point response value of each pixel in the image is greater than the inner corner point response value of other surrounding pixels, and if so, the pixel is an inner corner point candidate;

[0018] performing linear fitting on each of the inner corner point candidates, and determining an intersection of any two mutually perpendicular fitting lines as an inner corner point of the calibration board in the image;

[0019] obtaining the 2D pixel coordinates of the vertices of the calibration board in the image based on the determined inner corner points of the calibration board in the image.

[0020] Optionally, the 2D world coordinate system and the 3D world coordinate system are established based on physical parameters of the calibration board, and the 2D world coordinates and the 3D world coordinates of the vertices of the calibration board are obtained by:

[0021] A 2D world coordinate system is established with one vertex of the calibration board as the origin and two edges of the calibration board perpendicular to each other as the coordinate axes;

[0022] A 3D world coordinate system is established by adding a dimension to the 2D world coordinate system and assigning all values in the dimension as 0;

[0023] Based on the physical parameters of the calibration board, the 2D world coordinate system and the 3D world coordinate system, 2D world coordinates and 3D world coordinates of the vertices of the calibration board are obtained.

[0024] Optionally, based on the 2D pixel coordinates, the 2D world coordinates and the 3D world coordinates of the vertices of the calibration board, and the internal parameters of the camera, 3D camera coordinates of the vertices of the calibration board are calculated to obtain a 3D camera coordinate set of the vertices of the calibration board, including:

[0025] Based on the 2D pixel coordinates and the 2D world coordinates of the vertices of the calibration board, a corresponding relationship between the 2D pixel coordinates and the 2D world coordinates of the vertices of the calibration board is obtained;

[0026] Based on the 2D world coordinates and the 3D world coordinates of the vertices of the calibration board, calibration parameters of the 3D world coordinate system mapped to the 2D world coordinate system are obtained;

[0027] Based on the corresponding relationship between the 2D pixel coordinates and the 2D world coordinates of the vertices of the calibration board, the calibration parameters of the 3D world coordinate system mapped to the 2D world coordinate system, and the internal parameters of the camera, calibration parameters between the 3D world coordinate system and the 3D camera coordinate system are obtained;

[0028] According to the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system, the 3D world coordinates of the vertices of the calibration board are converted into 3D camera coordinates of the vertices of the calibration board to obtain a 3D camera coordinate set of the vertices of the calibration board.

[0029] Optionally, before extracting the 3D point cloud coordinates of the vertices of the calibration board from the point cloud data to obtain a 3D point cloud coordinate set of the vertices of the calibration board, the method further includes:

[0030] The point cloud data is format-converted to obtain converted point cloud data.

[0031] Optionally, extracting the 3D point cloud coordinates of the vertices of the calibration board from the point cloud data to obtain a 3D point cloud coordinate set of the vertices of the calibration board includes:

[0032] The region where the calibration board is located is determined in the point cloud data;

[0033] The plane of the calibration board is determined in the region where the calibration board is located.

[0034] extracting 3D point cloud coordinates of the calibration board vertices from the determined plane of the calibration board.

[0035] Optionally, the solving of the calibration parameters between the camera and the lidar based on the set of 3D camera coordinates of the calibration board vertices and the set of 3D point cloud coordinates comprises:

[0036] mapping the 3D camera coordinates of the calibration board vertices into a 3D point cloud coordinate system according to the predefined calibration parameters to obtain mapped 3D point cloud coordinates of the calibration board vertices;

[0037] calculating Euclidean distances between the mapped 3D point cloud coordinates of the calibration board vertices and actual 3D point cloud coordinates in the 3D point cloud coordinate system;

[0038] judging whether the Euclidean distances between the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system are less than a first threshold value, and if not, adjusting the calibration parameters between the camera and the lidar until the Euclidean distances between the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system are less than the first threshold value;

[0039] calculating edge directions and surface normal directions of the mapped calibration board based on the mapped 3D point cloud coordinates of the vertices of the calibration board, and calculating edge directions and surface normal directions of the calibration board in the point cloud based on the set of 3D point cloud coordinates of the calibration board vertices;

[0040] judging whether the edge directions of the mapped calibration board and the edge directions of the calibration board in the point cloud are parallel, and whether the surface normal directions of the mapped calibration board and the surface normal directions of the calibration board in the point cloud are parallel, and adjusting the calibration parameters between the camera and the lidar as long as one of them is not parallel until the edge directions of the mapped calibration board and the edge directions of the calibration board in the point cloud are parallel, and the surface normal directions of the mapped calibration board and the surface normal directions of the calibration board in the point cloud are parallel.

[0041] Optionally, the method further comprises:

[0042] mapping each 3D point in the point cloud data into an image according to the calibration parameters between the camera and the lidar, and taking distance information of each 3D point as distance information of a mapped pixel point of the 3D point in the image;

[0043] assigning corresponding colors to each mapped pixel point in the image according to its distance information to obtain a fused image, wherein the colors corresponding to different distance information are different;

[0044] output the fused image, so that a user judges whether the calibration parameters between the camera and the lidar are qualified based on a display effect of the fused image.

[0045] Optionally, the method further comprises:

[0046] mapping the 3D point cloud coordinates of the calibration board vertex into a 2D pixel coordinate system according to the calibration parameters between the camera and the lidar, to obtain the mapped 2D pixel coordinates of the calibration board vertex;

[0047] calculating the Euclidean distance between the mapped 2D pixel coordinates of the calibration board vertex and the actual 2D pixel coordinates in the 2D pixel coordinate system;

[0048] judging whether the Euclidean distance between the mapped 2D pixel coordinates of the calibration board vertex and the actual 2D pixel coordinates in the 2D pixel coordinate system is less than a second threshold value, and if so, the calibration parameters are qualified.

[0049] Compared with the prior art, the above technical solution has the following advantages:

[0050] The camera and lidar calibration method provided by the embodiments of the present application has the following advantages. First, the image of the calibration board pre-set in the environment and captured by the camera is obtained, and the 2D pixel coordinates of the calibration board vertex are extracted from the image. Since there is a lack of depth information, the internal parameters of the camera can only realize the conversion from the 3D camera coordinate system to the 2D pixel coordinate system, and cannot directly convert the 2D pixel coordinate system into the 3D camera coordinate system. Therefore, based on the physical parameters of the calibration board, the 2D world coordinate system and the 3D world coordinate system are established, the 2D world coordinates and the 3D world coordinates of the calibration board vertex are obtained, and then based on the 2D pixel coordinates, the 2D world coordinates and the 3D world coordinates of the calibration board vertex, and the internal parameters of the camera, the 3D camera coordinates of the calibration board vertex are calculated, to obtain the 3D camera coordinate set of the calibration board vertex. Second, the point cloud data of the calibration board pre-set in the environment and collected by the lidar is obtained, and the 3D point cloud coordinates of the calibration board vertex are extracted from the point cloud data, to obtain the 3D point cloud coordinate set of the calibration board vertex. Then, based on the 3D camera coordinate set and the 3D point cloud coordinate set of the calibration board vertex, the calibration parameters between the camera and the lidar are solved, which include the rotation parameters and the translation parameters. Since there is no loss of depth information, the calibration accuracy of the method is high. BRIEF DESCRIPTION OF DRAWINGS

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic flowchart illustrating a camera and lidar calibration method provided in an embodiment of this application.

[0053] Figure 2 The image taken by the camera of a pre-set calibration board in the environment before correction;

[0054] Figure 3 To Figure 2 The image shown is the image after distortion correction;

[0055] Figure 4 A schematic diagram of a chessboard grid marking;

[0056] Figure 5 A schematic diagram of the features of the vertices of the calibration board extracted from the image;

[0057] Figure 6 A schematic diagram showing the features of the vertices of the calibration board extracted from the point cloud;

[0058] Figure 7 This is a schematic flowchart illustrating another camera and lidar calibration method provided in an embodiment of this application.

[0059] Figure 8 This is a schematic flowchart illustrating another camera and lidar calibration method provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] This application provides a method for calibrating a camera and a lidar, such as... Figure 1 As shown, the method includes:

[0062] S100: Acquire an image of a calibration board in the environment captured by a camera, and extract the 2D pixel coordinates of the vertices of the calibration board from the image.

[0063] It should be noted that the calibration board can be rectangular, and the vertices of the calibration board are the four vertices of the rectangle. The surface of the calibration board includes a grid of black and white squares, i.e., the calibration board is a checkerboard calibration board.

[0064] It should be further noted that the image captured by the camera of the pre-set calibration board in the environment is in a 2D pixel coordinate system, and therefore the coordinates of the vertices of the calibration board extracted from the image are 2D pixel coordinates.

[0065] In the embodiments of the present application, the four vertices of the calibration board are used as explicit point features, which have strong significance and prior knowledge, and can also bring strong constraints, which is conducive to subsequent accurate feature matching, and the position of the calibration board placed in the environment is not limited, i.e., the calibration board can be placed at any position in the environment, and accurate features of the vertices of the calibration board can be extracted from the image of the calibration board, thereby making the robustness of the method better.

[0066] Specifically, in the robot operating system (ROS), a rosbag can be used to record data files, and the camera shooting of the pre-set calibration board in the environment can be recorded as a bag package with a camera topic, and then an image extraction module is used to extract the required image data.

[0067] S200: Based on the physical parameters of the calibration board, a 2D world coordinate system and a 3D world coordinate system are established to obtain 2D world coordinates and 3D world coordinates of the vertices of the calibration board.

[0068] S300: Based on the 2D pixel coordinates, the 2D world coordinates and the 3D world coordinates of the vertices of the calibration board, and the internal parameters of the camera, the 3D camera coordinates of the vertices of the calibration board are calculated to obtain a 3D camera coordinate set of the vertices of the calibration board, wherein the internal parameters of the camera are calibration parameters for mapping the 3D camera coordinate system to the 2D pixel coordinate system.

[0069] It should be noted that the 3D camera coordinate system is fixed on the camera entity, which is a coordinate system established with the camera optical center as the origin, the two axes of the image sensor as the Xc and Yc axes, and the optical axis as the Zc axis. The coordinates of the vertices of the pre-set calibration board in the environment in the 3D camera coordinate system are the 3D camera coordinates of the vertices of the calibration board.

[0070] The internal parameter of the camera is a calibration parameter of mapping the 3D camera coordinate system to the 2D pixel coordinate system, that is, a transformation relationship of mapping the 3D camera coordinate system to the 2D pixel coordinate system, but because of the lack of depth information, the internal parameter of the camera can only map the 3D camera coordinate system to the 2D pixel coordinate system, and cannot map the 2D pixel coordinate system to the 3D camera coordinate system, that is, the 2D pixel coordinates of the extracted vertex of the calibration board cannot be directly converted into the 3D camera coordinates.

[0071] Therefore, in the embodiment of the present application, in step S200, based on the physical parameters of the calibration board, the 2D world coordinate system and the 3D world coordinate system are established, and the 2D world coordinates and the 3D world coordinates of the vertices of the calibration board are obtained, wherein the coordinates of the vertices of the calibration board in the 2D world coordinate system are the 2D world coordinates thereof, and the coordinates of the vertices of the calibration board in the 3D world coordinate system are the 3D world coordinates thereof.

[0072] It can be understood that the physical parameters of the calibration board include the length and width of the calibration board and the length and width of the black and white grids in the calibration board and the like, and according to these physical parameters, as long as the origin and the coordinate axes are set, the 2D world coordinate system and the 3D world coordinate system can be established, so as to obtain the 2D world coordinates and the 3D world coordinates of the vertices of the calibration board.

[0073] At this time, the 2D pixel coordinates of the vertices of the calibration board and the 2D world coordinates thereof have a one-to-one correspondence, and the 3D world coordinates of the vertices of the calibration board and the 2D world coordinates thereof also have a one-to-one correspondence, and the calibration parameters of mapping the 3D camera coordinate system to the 2D pixel coordinate system are known, that is, the internal parameter of the camera, so that the 3D camera coordinates of the vertices of the calibration board can be calculated based on the 2D pixel coordinates, the 2D world coordinates and the 3D world coordinates of the vertices of the calibration board and the internal parameter of the camera, and the 3D camera coordinate set of the vertices of the calibration board is obtained.

[0074] S400: Obtain the point cloud data collected by the laser radar on the preset calibration board in the environment, extract the 3D point cloud coordinates of the vertices of the calibration board from the point cloud data, and obtain the 3D point cloud coordinate set of the vertices of the calibration board.

[0075] The point cloud is a set of 3D points in the laser radar coordinate system (i.e. the 3D point cloud coordinate system), and after obtaining the point cloud data collected by the laser radar on the preset calibration board in the environment, the 3D point cloud coordinates of the vertices of the calibration board extracted from the point cloud data are the coordinates of the vertices of the calibration board in the laser radar coordinate system (i.e. the 3D point cloud coordinate system), so as to obtain the 3D point cloud coordinate set of the vertices of the calibration board.

[0076] Specifically, in the Robot Operating System (ROS), a rosbag can be used to record a data file, and the laser radar scans the preset calibration board in the environment, which can be recorded as a bag package with a laser radar topic, and then a point cloud extraction module is used to extract the required point cloud data.

[0077] S500: Based on the 3D camera coordinate set of the calibration board vertex and the 3D point cloud coordinate set, the calibration parameters between the camera and the laser radar are solved, and the calibration parameters between the camera and the laser radar include rotation parameters and translation parameters.

[0078] It should be noted that the calibration parameters between the camera and the laser radar, i.e. the conversion relationship between the two sensors of the camera and the laser radar, include rotation parameters and translation parameters, wherein the translation parameters are usually represented by a translation vector, and as for the rotation parameters, there are various representation methods, such as Euler angles, rotation matrices or quaternions. Specifically, Euler angles are three angles introduced by Euler to describe the attitude of a rigid body, and are also the most easily understood and most easily visualized way of expression. The rotation matrix reflects the conversion relationship of the coordinates in one coordinate system to another coordinate system. The quaternion is composed of 1 real number and 3 complex numbers, which eliminates the problem of point lock of Euler angles. The representation method of the rotation parameter is not limited in the present application, and the rotation parameter in the calibration parameter can be converted into different formats according to the requirements.

[0079] As can be seen, compared with the existing automatic calibration method of camera and laser radar, the 3D points in the point cloud obtained by the laser radar need to be mapped to the image, thereby losing depth information, resulting in large calibration error. The camera and laser radar calibration method provided in the present application is based on the 2D pixel coordinates of the calibration board vertex extracted from the image, and by means of establishing a 2D world coordinate system and a 3D world coordinate system, the 2D world coordinates and the 3D world coordinates of the calibration board vertex are obtained, so as to calculate the 3D camera coordinates of the calibration board vertex based on the 2D pixel coordinates, the 2D world coordinates and the 3D world coordinates of the calibration board vertex, and the internal parameters of the camera, obtain the 3D camera coordinate set of the calibration board vertex, and then based on the 3D camera coordinate set of the calibration board vertex and the 3D point cloud coordinate set, the calibration parameters between the camera and the laser radar are solved. Since there is no loss of depth information, the calibration accuracy of the method is high.

[0080] And, the camera and laser radar calibration method provided by the embodiment of the application has strong significance and prior knowledge by taking the four vertices of the calibration board as explicit point features, can bring strong constraints, is conducive to realizing accurate feature matching, and does not limit the position of the calibration board in the environment, that is, the calibration board can be placed at any position in the environment, so that the robustness of the method is also better.

[0081] In actual application, after the image captured by the camera on the pre-set calibration board in the environment is acquired, since the generated image generally has distortion, including radial distortion and tangential distortion, which is not conducive to subsequent feature extraction from the image, therefore, optionally, in an embodiment of the application, before the 2D pixel coordinates of the vertices of the calibration board are extracted from the image in step S100, the method further includes:

[0082] correcting the image for distortion to obtain a corrected image.

[0083] That is, step S100 specifically includes:

[0084] S110: acquiring an image captured by the camera on the pre-set calibration board in the environment;

[0085] S120: correcting the image for distortion to obtain a corrected image;

[0086] S130: extracting the 2D pixel coordinates of the vertices of the calibration board from the corrected image.

[0087] Specifically, in step S120, the distortion of the image can be corrected by using the internal parameters of the camera to obtain a corrected image. Figure 2 An image captured by the camera on the pre-set calibration board 10 in the environment before correction is shown, Figure 3 An image after the image is corrected for distortion is shown.

[0088] Optionally, in an embodiment of the application, extracting the 2D pixel coordinates of the vertices of the calibration board from the image includes:

[0089] S131: calculating the inner corner point response value of each pixel in the image based on the gray value of each pixel in the image, the greater the inner corner point response value of a pixel, the more likely the pixel is an inner corner point of the calibration board, the inner corner point of the calibration board being the vertex of the black and white grid inside the calibration board, excluding the vertex of the calibration board and the vertex of the black and white grid on the edge of the calibration board.

[0090] Figure 4 A schematic diagram of a checkerboard calibration board is shown, such as Figure 4As shown, the calibration board surface includes a grid arrangement of black and white squares, and it can be understood that the inner corner points of the calibration board, i.e. the black and white grid vertices inside the calibration board, are the intersection points of two black squares and two white squares, and the pixel gray value transformation (second derivative) of the intersection points is larger.

[0091] Thus, in step S131, based on the gray value of each pixel in the image, specifically based on the second derivative of the gray value of each pixel in the image, the inner corner point response value of each pixel in the image is calculated, specifically the inner corner point response value of each pixel in the image can be calculated by using a corner point detection algorithm, wherein the inner corner point response value of a pixel represents the probability that the pixel is an inner corner point of the calibration board in the image, and the larger the inner corner point response value of a pixel is, the more likely the pixel is an inner corner point of the calibration board.

[0092] S132: Determine whether the inner corner point response value of each pixel in the image is greater than the inner corner point response value of other surrounding pixels, if so, the pixel is an inner corner point candidate.

[0093] It can be understood that if the inner corner point response value of a pixel in the image is greater than the inner corner point response value of other surrounding pixels, it means that the transformation (second derivative) of the gray value of the pixel is greater than the transformation (second derivative) of the gray value of other surrounding pixels, i.e. the pixel is an inner corner point candidate.

[0094] S133: Linear fitting is performed on each inner corner point candidate, and the intersection point of any mutually perpendicular fitting lines is determined as the inner corner point of the calibration board in the image.

[0095] It should be noted that the inner corner point candidates obtained in step S132 are discrete points, in order to improve the positioning accuracy and make the position of the determined inner corner point more accurate, in step S133, linear fitting is performed on each inner corner point candidate, specifically the linear least square estimation algorithm of sub-pixel accuracy can be used to perform linear fitting on each inner corner point candidate, so that the intersection point of any mutually perpendicular fitting lines is determined as the inner corner point of the calibration board in the image.

[0096] S134: Based on the determined inner corner points of the calibration board in the image, the 2D pixel coordinates of the calibration board vertex in the image are obtained.

[0097] Reference Figure 4 As shown, it can be understood that after determining the inner corner points of the calibration board in the image, i.e. all the black and white grids inside the calibration board are determined, since the length and width of each black and white grid in the calibration board are equal, the 2D pixel coordinates of the calibration board vertex in the image can be obtained, Figure 5 A feature diagram of the vertex of the calibration board extracted from the image is shown.

[0098] Optionally, in one embodiment of the present application, step S200, based on the physical parameters of the calibration board, a 2D world coordinate system and a 3D world coordinate system are established, and 2D world coordinates and 3D world coordinates of the vertices of the calibration board are obtained, including:

[0099] S210: A 2D world coordinate system is established with one vertex of the calibration board as the origin and two mutually perpendicular edges of the calibration board as the coordinate axes.

[0100] The present application does not limit which vertex of the calibration board is taken as the origin, which can be determined according to actual conditions. Specifically, for example, as shown in Figure 4 the left upper corner of the calibration board is taken as the origin O, and two mutually perpendicular edges of the calibration board are taken as the coordinate axes X-axis and Y-axis, and thus the 2D world coordinate system can be established.

[0101] S220: A dimension is added to the 2D world coordinate system, and the values on this dimension are all assigned as 0, and thus a 3D world coordinate system is established.

[0102] It should be noted that the above dimension is a dimension perpendicular to the surface of the calibration board. Specifically, for example, as shown in Figure 4 a dimension is added to the 2D world coordinate system, which is perpendicular to the surface of the calibration board, and the coordinates of the calibration board on this dimension are all assigned as 0, and thus the 3D world coordinate system can be established.

[0103] S230: Based on the physical parameters of the calibration board and the established 2D world coordinate system and 3D world coordinate system, 2D world coordinates and 3D world coordinates of the vertices of the calibration board are obtained.

[0104] The physical parameters of the calibration board include the length and width of the calibration board and the length and width of the black and white grids in the calibration board and the like. According to these physical parameters, 2D world coordinates and 3D world coordinates of the vertices of the calibration board can be obtained in the established 2D world coordinate system and 3D world coordinate system.

[0105] Optionally, in one embodiment of the present application, step S300, based on the 2D pixel coordinates, 2D world coordinates and 3D world coordinates of the vertices of the calibration board and the internal parameters of the camera, 3D camera coordinates of the vertices of the calibration board are calculated, and a 3D camera coordinate set of the vertices of the calibration board is obtained, including:

[0106] S310: Based on the 2D pixel coordinates and 2D world coordinates of the vertices of the calibration board, a corresponding relationship between the 2D pixel coordinates and 2D world coordinates of the vertices of the calibration board is obtained.

[0107] It can be understood that the 2D pixel coordinates of the calibration board vertex are obtained in step S100, and the 2D world coordinates of the calibration board vertex are obtained in step S200, so that a one-to-one correspondence between the 2D pixel coordinates and the 2D world coordinates of the calibration board vertex can be obtained.

[0108] S320: Obtain the calibration parameters of the 3D world coordinate system mapping to the 2D world coordinate system based on the 2D world coordinates and the 3D world coordinates of the calibration board vertex.

[0109] Similarly, the calibration parameters of the 3D world coordinate system mapping to the 2D world coordinate system are the conversion relationship of the 3D world coordinate system mapping to the 2D world coordinate system, similar to the calibration parameters of the 3D camera coordinate system mapping to the 2D pixel coordinate system (i.e. the internal parameters of the camera).

[0110] Since the 2D world coordinates and the 3D world coordinates of the calibration board vertex have been obtained, a one-to-one correspondence between the 2D world coordinates and the 3D world coordinates of the calibration board vertex can be obtained, and then the calibration parameters of the 3D world coordinate system mapping to the 2D world coordinate system can be obtained.

[0111] S330: Obtain the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system based on the correspondence between the 2D pixel coordinates and the 2D world coordinates of the calibration board vertex, the calibration parameters of the 3D world coordinate system mapping to the 2D world coordinate system, and the internal parameters of the camera.

[0112] In step S330, the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system are calculated based on the correspondence between the 2D pixel coordinates and the 2D world coordinates of the calibration board vertex, the calibration parameters of the 3D world coordinate system mapping to the 2D world coordinate system, and the internal parameters of the camera (i.e. the calibration parameters of the 3D camera coordinate system mapping to the 2D pixel coordinate system).

[0113] Specifically, assuming that the internal parameters of the camera (i.e. the calibration parameters of the 3D camera coordinate system mapping to the 2D pixel coordinate system) are K, then the coordinates (X_c) of a vertex (denoted as A vertex) of the calibration board in the 3D camera coordinate system mapping to the 2D pixel coordinate system are K*(X_c).

[0114] Similarly, assuming that the calibration parameters of the 3D world coordinate system mapping to the 2D world coordinate system are H, then the coordinates (X_w) of the A vertex of the calibration board in the 3D world coordinate system mapping to the 2D world coordinate system are H*(X_w).

[0115] And, since there is a one-to-one correspondence between the 2D pixel coordinates and the 2D world coordinates of the vertex of the calibration board, the coordinates K*(X_c) of the vertex of the calibration board A in the 2D pixel coordinate system have a one-to-one correspondence with the coordinates H*(X_w) of the vertex in the 2D world coordinate system, which can be expressed as:

[0116] K*(X_c) = H*(X_w) (1)

[0117] It can be understood that the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system include rotation parameters and translation parameters, wherein the rotation parameters can be represented by a rotation matrix R, and the translation parameters can be represented by a translation vector t, that is, the coordinates (X_w) of the vertex of the calibration board A in the 3D world coordinate system are converted to the coordinates (X_c) in the 3D camera coordinate system through the rotation matrix R and the translation vector t, then the relationship between the coordinates (X_c) of the vertex of the calibration board A in the 3D camera coordinate system and the coordinates (X_w) of the vertex in the 3D world coordinate system can be expressed as:

[0118] (X_c) = R *(X_w) + t (2)

[0119] Since the corresponding relationship between the coordinates K*(X_c) of the vertex of the calibration board A in the 2D pixel coordinate system and the coordinates H*(X_w) of the vertex in the 2D world coordinate system, the calibration parameters H of the 3D world coordinate system mapping to the 2D world coordinate system, and the internal parameters K of the camera are known, formula (1) can be transformed into:

[0120] (X_c) = (K^ -1 * H) *(X_w) (3)

[0121] Where K^ -1 is the inverse of K.

[0122] Comparing formula (2) and formula (3), further, (K^ -1 * H) can be decomposed into a rotation matrix R and a translation vector t, for example:

[0123] R = (K^-1 * H)[1:3, 1:3] (4)

[0124] t = (K^-1 * H)[1:3, 4] (5)

[0125] Where [1:3, 1:3] represents a 3*3 matrix, and [1:3, 4] represents the 4th column in all rows (1st row to 3rd row).

[0126] Through the above process, the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system can be obtained, including the rotation matrix R and the translation vector t.

[0127] It should be noted that the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system are 3D point to 3D point conversion relationship, so it can realize the mapping of the 3D world coordinate system to the 3D camera coordinate system, or the mapping of the 3D camera coordinate system to the 3D world coordinate system.

[0128] S340: According to the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system, the 3D world coordinates of the calibration board vertices are converted into the 3D camera coordinates of the calibration board vertices, and a 3D camera coordinate set of the calibration board vertices is obtained.

[0129] As known from the foregoing, the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system can realize the mapping of the 3D world coordinate system to the 3D camera coordinate system, and in the case of knowing the 3D world coordinates of the calibration board vertices, the 3D world coordinates of the calibration board vertices can be converted into 3D camera coordinates according to the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system, and a 3D camera coordinate set of the calibration board vertices is obtained.

[0130] Optionally, in an embodiment of the present application, before the step S400 of extracting the 3D point cloud coordinates of the calibration board vertices from the point cloud data to obtain a 3D point cloud coordinate set of the calibration board vertices, according to the subsequent requirement for the point cloud format, the method further comprises:

[0131] The point cloud data is format-converted to obtain converted point cloud data.

[0132] That is, the step S400 comprises:

[0133] S410: Obtain the point cloud data collected by the laser radar on the preset calibration board in the environment;

[0134] S420: Format-convert the point cloud data to obtain converted point cloud data;

[0135] S430: Extract the 3D point cloud coordinates of the calibration board vertices from the converted point cloud data to obtain a 3D point cloud coordinate set of the calibration board vertices.

[0136] Optionally, in an embodiment of the present application, the step of extracting the 3D point cloud coordinates of the calibration board vertices from the point cloud data to obtain a 3D point cloud coordinate set of the calibration board vertices comprises:

[0137] S431: Determine the region where the calibration board is located in the point cloud data;

[0138] S432: Determine the plane of the calibration board in the region where the calibration board is located.

[0139] S433: Extract the 3D point cloud coordinates of the calibration board vertices from the determined plane of the calibration board.

[0140] Specifically, in step S431, the possible spatial position of the calibration board in the point cloud is determined according to the range of 3D points in the point cloud graph. Then, in step S432, the plane representing the calibration board can be estimated in the point cloud at the corresponding position using the RANSAC (Random Sample Consensus) method. Next, in step S433, the 3D point cloud coordinates of the calibration board vertices are extracted from the determined plane of the calibration board.

[0141] RANSAC (Random Sample Consensus) is an iterative algorithm for finding the optimal parameters of a model in a dataset. It is suitable for data containing noise or outliers and can effectively fit various types of models. Using the RANSAC algorithm to calculate the plane in the point cloud is a typical application scenario, which specifically includes:

[0142] First, a model type is selected, such as a plane; second, a set of points is randomly selected from the point cloud, and these points are used to fit the model; then, the distances of the remaining points in the point cloud to the fitted model are calculated, and the points with distances less than a certain threshold are considered as inliers; after that, the model is refitted using the inliers, and the above process is repeated until the optimal model is found. Figure 6 A feature diagram of the four vertices of the calibration board extracted from the point cloud is shown.

[0143] Optionally, in an embodiment of the present application, in step S500, based on the set of 3D camera coordinates of the calibration board vertices and the set of 3D point cloud coordinates, the calibration parameters between the camera and the lidar are solved, including:

[0144] S510: Map the 3D camera coordinates of the calibration board vertices to the 3D point cloud coordinate system according to the predefined calibration parameters, to obtain the mapped 3D point cloud coordinates of the calibration board vertices.

[0145] The predefined calibration parameters can be a unit matrix.

[0146] S520: Calculate the Euclidean distance between the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system.

[0147] S530: determining whether the Euclidean distance between the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system is less than a first threshold value, and if not, adjusting the calibration parameters between the camera and the lidar until the Euclidean distance between the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system is less than the first threshold value.

[0148] It can be understood that the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system should be aligned, i.e. the Euclidean distance between the two is less than the first threshold value, and if the Euclidean distance between the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system is not less than the first threshold value, it means that the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system are not aligned, and the calibration parameters between the camera and the lidar need to be adjusted until the Euclidean distance between the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system is less than the first threshold value.

[0149] Specifically, the Iterative Closest Point (ICP) algorithm can be used to align the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system, so that the Euclidean distance between the mapped 3D point cloud coordinates of the calibration board vertices and the actual 3D point cloud coordinates in the 3D point cloud coordinate system is less than the first threshold value.

[0150] S540: based on the mapped 3D point cloud coordinates of each vertex of the calibration board, calculating the edge direction and surface normal direction of the mapped calibration board, and based on the set of 3D point cloud coordinates of the calibration board vertices, obtaining the edge direction and surface normal direction of the calibration board in the point cloud;

[0151] S550: determining whether the edge direction of the mapped calibration board and the edge direction of the calibration board in the point cloud are parallel, and whether the surface normal direction of the mapped calibration board and the surface normal direction of the calibration board in the point cloud are parallel, and as long as one of them is not parallel, adjusting the calibration parameters between the camera and the lidar until the edge direction of the mapped calibration board and the edge direction of the calibration board in the point cloud are parallel, and the surface normal direction of the mapped calibration board and the surface normal direction of the calibration board in the point cloud are parallel.

[0152] It can be understood that in the 3D point cloud coordinate system, the edge direction of the mapped calibration board and the edge direction of the calibration board in the point cloud should be parallel, and the surface normal direction of the mapped calibration board and the surface normal direction of the calibration board in the point cloud should be parallel. If the edge direction of the mapped calibration board and the edge direction of the calibration board in the point cloud are not parallel, and / or the surface normal direction of the mapped calibration board and the surface normal direction of the calibration board in the point cloud are not parallel, it indicates that the calibration parameters between the camera and the lidar are not qualified and need to be adjusted until the edge direction of the mapped calibration board and the edge direction of the calibration board in the point cloud are parallel, and the surface normal direction of the mapped calibration board and the surface normal direction of the calibration board in the point cloud are parallel.

[0153] Specifically, the Levenberg-Marquardt algorithm can be used to make the edge direction of the mapped calibration board and the edge direction of the calibration board in the point cloud parallel, and the surface normal direction of the mapped calibration board and the surface normal direction of the calibration board in the point cloud parallel.

[0154] Finally, the accurate calibration parameters (i.e., the transformation matrix) between the camera and the lidar are obtained.

[0155] In order to verify whether the calibration parameters between the camera and the lidar obtained above are qualified, the calibration parameters between the camera and the lidar need to be verified.

[0156] Optionally, in an embodiment of the present application, as shown in Figure 7 the method further comprises:

[0157] S610: mapping each 3D point in the point cloud data to an image according to the calibration parameters between the camera and the lidar, and taking the distance information of each 3D point as the distance information of the mapped pixel point of the 3D point in the image;

[0158] S620: assigning a corresponding color to each mapped pixel point in the image according to its distance information to obtain a fused image, wherein the colors corresponding to different distance information are different.

[0159] S630: outputting the fused image so that a user can judge whether the calibration parameters between the camera and the lidar are qualified based on the display effect of the fused image.

[0160] It can be understood that in the fused image, the colors of the mapped pixel points with the same distance should be the same. If the colors of the mapped pixel points with the same distance are the same in whole or within an acceptable probability range, it indicates that the calibration parameters between the camera and the lidar are qualified.

[0161] Optionally, in another embodiment of the present application, as shown in Figure 8 the method further comprises:

[0162] S710: mapping the 3D point cloud coordinates of the calibration board vertexes to 2D pixel coordinates according to the calibration parameters between the camera and the laser radar, to obtain the mapped 2D pixel coordinates of the calibration board vertexes;

[0163] S720: calculating the Euclidean distance between the mapped 2D pixel coordinates of the calibration board vertexes and the actual 2D pixel coordinates in the 2D pixel coordinate system;

[0164] S730: judging whether the Euclidean distance between the mapped 2D pixel coordinates of the calibration board vertexes and the actual 2D pixel coordinates in the 2D pixel coordinate system is less than the second threshold value, if yes, the calibration parameters between the camera and the laser radar are qualified.

[0165] It can be understood that the mapped 2D pixel coordinates of the calibration board vertexes and the actual 2D pixel coordinates in the 2D pixel coordinate system should be aligned, i.e. the Euclidean distance between the two is less than the second threshold value, if the Euclidean distance between the mapped 2D pixel coordinates of the calibration board vertexes and the actual 2D pixel coordinates in the 2D pixel coordinate system is less than the second threshold value, it means that the mapped 2D pixel coordinates of the calibration board vertexes and the actual 2D pixel coordinates in the 2D pixel coordinate system are aligned, and the calibration parameters between the camera and the laser radar are qualified.

[0166] In the present specification, each part is described in a way of combination of parallelism and progression, and each part mainly explains the difference from other parts, and the same or similar parts between each part can be referred to each other.

[0167] The above description of the disclosed embodiments, the features described in each embodiment in the present specification can be replaced or combined with each other, so that the person skilled in the art can realize or use the present application. Various modifications of the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A camera and lidar calibration method, characterized in that, The method comprises: acquiring an image of a preset calibration board in an environment captured by a camera, and extracting 2D pixel coordinates of vertices of the calibration board from the image; establishing a 2D world coordinate system and a 3D world coordinate system based on physical parameters of the calibration board, to obtain 2D world coordinates and 3D world coordinates of the vertices of the calibration board; calculating 3D camera coordinates of the vertices of the calibration board based on the 2D pixel coordinates, the 2D world coordinates and the 3D world coordinates of the vertices of the calibration board, and internal parameters of the camera, to obtain a 3D camera coordinate set of the vertices of the calibration board, wherein the internal parameters of the camera are calibration parameters for mapping a 3D camera coordinate system to a 2D pixel coordinate system; acquiring point cloud data of the preset calibration board in the environment collected by a laser radar, and extracting 3D point cloud coordinates of the vertices of the calibration board from the point cloud data, to obtain a 3D point cloud coordinate set of the vertices of the calibration board; solving calibration parameters between the camera and the laser radar based on the 3D camera coordinate set and the 3D point cloud coordinate set of the vertices of the calibration board, wherein the calibration parameters between the camera and the laser radar include rotation parameters and translation parameters; solving the calibration parameters between the camera and the laser radar based on the 3D camera coordinate set and the 3D point cloud coordinate set of the vertices of the calibration board comprises: mapping the 3D camera coordinates of the vertices of the calibration board into a 3D point cloud coordinate system according to predefined calibration parameters, to obtain mapped 3D point cloud coordinates of the vertices of the calibration board; calculating Euclidean distances between the mapped 3D point cloud coordinates of the vertices of the calibration board and actual 3D point cloud coordinates in the 3D point cloud coordinate system; judging whether the Euclidean distances between the mapped 3D point cloud coordinates of the vertices of the calibration board and the actual 3D point cloud coordinates in the 3D point cloud coordinate system are less than a first threshold value, and if not, adjusting the calibration parameters between the camera and the laser radar until the Euclidean distances between the mapped 3D point cloud coordinates of the vertices of the calibration board and the actual 3D point cloud coordinates in the 3D point cloud coordinate system are less than the first threshold value; calculating edge directions and surface normal directions of the calibration board after mapping based on the mapped 3D point cloud coordinates of the vertices of the calibration board, and calculating the edge directions and the surface normal directions of the calibration board in the point cloud based on the 3D point cloud coordinate set of the vertices of the calibration board; judging whether the edge directions of the calibration board after mapping and the edge directions of the calibration board in the point cloud are parallel, and whether the surface normal directions of the calibration board after mapping and the surface normal directions of the calibration board in the point cloud are parallel, and if either of them is not parallel, adjusting the calibration parameters between the camera and the laser radar until the edge directions of the calibration board after mapping and the edge directions of the calibration board in the point cloud are parallel, and the surface normal directions of the calibration board after mapping and the surface normal directions of the calibration board in the point cloud are parallel.

2. The camera and lidar calibration method of claim 1, wherein, Before the 2D pixel coordinates of the vertices of the calibration board are extracted from the image, the method further comprises: performing distortion correction on the image to obtain a corrected image.

3. The camera and lidar calibration method of claim 1, wherein, The method comprises the following steps: extracting 2D pixel coordinates of the calibration board vertex from the image comprises: calculating an inner corner point response value of each pixel in the image based on a gray value of each pixel in the image, the greater the inner corner point response value of a pixel, the more likely the pixel is an inner corner point of the calibration board, the inner corner point of the calibration board is a black and white grid vertex inside the calibration board, excluding the vertex of the calibration board and the black and white grid vertex on the edge of the calibration board; determining whether the inner corner point response value of each pixel in the image is greater than the inner corner point response value of other surrounding pixels, if yes, the pixel is an inner corner point candidate; linear fitting each inner corner point candidate, and determining an intersection of any two perpendicular fitting lines as an inner corner point of the calibration board in the image; 4. The camera and lidar calibration method of claim 1, wherein, based on the determined inner corner points of the calibration board in the image, obtaining 2D pixel coordinates of the calibration board vertex. Based on the physical parameters of the calibration board, a 2D world coordinate system and a 3D world coordinate system are established, and 2D world coordinates and 3D world coordinates of the calibration board vertex are obtained, which comprises the following steps: establishing a 2D world coordinate system with one vertex of the calibration board as the origin and two mutually perpendicular edges of the calibration board as the coordinate axes; adding a dimension to the 2D world coordinate system and assigning all values on this dimension as 0 to establish a 3D world coordinate system; 5. The camera and lidar calibration method of claim 1, wherein, based on the physical parameters of the calibration board, and the established 2D world coordinate system and 3D world coordinate system, obtaining 2D world coordinates and 3D world coordinates of the calibration board vertex. Based on the 2D pixel coordinates, 2D world coordinates and 3D world coordinates of the calibration board vertex, and the internal parameters of the camera, the 3D camera coordinates of the calibration board vertex are calculated, and the 3D camera coordinate set of the calibration board vertex is obtained, which comprises the following steps: based on the 2D pixel coordinates and 2D world coordinates of the calibration board vertex, obtaining the corresponding relationship between the 2D pixel coordinates and 2D world coordinates of the calibration board vertex; based on the 2D world coordinates and 3D world coordinates of the calibration board vertex, obtaining the calibration parameters of the 3D world coordinate system mapped to the 2D world coordinate system; based on the corresponding relationship between the 2D pixel coordinates and 2D world coordinates of the calibration board vertex, the calibration parameters of the 3D world coordinate system mapped to the 2D world coordinate system, and the internal parameters of the camera, obtaining the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system; 6. The camera and lidar calibration method of claim 1, wherein, according to the calibration parameters between the 3D world coordinate system and the 3D camera coordinate system, converting the 3D world coordinates of the calibration board vertex into the 3D camera coordinates of the calibration board vertex, and obtaining the 3D camera coordinate set of the calibration board vertex. Before extracting the 3D point cloud coordinates of the calibration board vertex from the point cloud data to obtain the 3D point cloud coordinate set of the calibration board vertex, the method further comprises:

7. The camera and lidar calibration method of claim 1, wherein, format conversion of the point cloud data to obtain converted point cloud data. extracting the 3D point cloud coordinates of the calibration board vertex from the point cloud data to obtain the 3D point cloud coordinate set of the calibration board vertex comprises: determining the region where the calibration board is located in the point cloud data; determining a plane of the calibration board in a region where the calibration board is located; extracting 3D point cloud coordinates of the calibration board vertex from the determined plane of the calibration board.

8. The camera and lidar calibration method of any one of claims 1-7, wherein, The method further comprises: mapping each 3D point in the point cloud data to an image according to the calibration parameters between the camera and the lidar, and taking distance information of each 3D point as distance information of a pixel point in the image to which the 3D point is mapped; assigning each mapped pixel point in the image a corresponding color according to its distance information, to obtain a fused image, wherein different colors correspond to different distance information; outputting the fused image, so that a user can determine whether the calibration parameters between the camera and the lidar are qualified based on a display effect of the fused image.

9. The camera and lidar calibration method of any one of claims 1-7, wherein, The method further comprises: mapping the 3D point cloud coordinates of the calibration board vertex to a 2D pixel coordinate system according to the calibration parameters between the camera and the lidar, to obtain mapped 2D pixel coordinates of the calibration board vertex; calculating Euclidean distances between the mapped 2D pixel coordinates of the calibration board vertex and actual 2D pixel coordinates in the 2D pixel coordinate system; determining whether the Euclidean distances between the mapped 2D pixel coordinates of the calibration board vertex and the actual 2D pixel coordinates in the 2D pixel coordinate system are less than a second threshold value, and if so, the calibration parameters are qualified.

Images