Method and device for joint calibration of lidar and camera

Abstract

The present application discloses a method and apparatus for joint calibration of a lidar and a camera. The method includes: calculating, based on image data obtained by the camera photographing a V-shaped calibration board, the plane normal vectors of the main surfaces, the first plane, and the second plane of the two boards in the V-shaped calibration board that face the camera in the coordinate system of the camera; determining, based on the single-line laser point cloud obtained by the lidar scanning the V-shaped calibration board, the coordinates of a first intersection point, a second intersection point, and a third intersection point in the coordinate system of the lidar; and determining the external parameters between the camera and the lidar based on the plane normal vectors of the main surfaces, the first plane, and the second plane of the two boards, and the coordinates of the first intersection point, the second intersection point, and the third intersection point. The above solution can improve the accuracy of the calibration result.

Description

Technical Field

[0001] The present application relates to the technical field of parameter calibration, and particularly to a method and device for jointly calibrating a lidar and a camera. Background Art

[0002] The unmanned factory has become an important way to realize Industry 4.0, which also puts forward higher requirements for AGVs. A single sensor can no longer adapt to the complex factory environment, and the multi-sensor information fusion has gradually become the mainstream trend. Whether the multi-sensor fusion is accurate depends to a large extent on whether the external parameters calculation between sensors is accurate. Therefore, the multi-sensor calibration is crucial. However, the results obtained by the current calibration methods have large errors. Summary of the Invention

[0003] The main technical problem to be solved by the present application is to provide a method and device for jointly calibrating a lidar and a camera, which can improve the accuracy of the calibration result.

[0004] To solve the above technical problem, a first aspect of the present application provides a method for jointly calibrating a lidar and a camera, including: calculating, based on the image data obtained by the camera photographing a V-shaped calibration board, the plane normal vectors of the main surfaces, the first plane, and the second plane of the two boards in the V-shaped calibration board facing the camera in the coordinate system of the camera, where the first plane is the plane formed by the first edge line and the first preset point, the second plane is the plane formed by the second edge line and the second preset point, the first edge line and the second edge line are the edge lines opposite to the intersection line on the main surfaces of the two boards respectively, the intersection line is the line where the main surfaces of the two boards intersect, and both the first preset point and the second preset point are not in the plane where the main surfaces of the two boards are located;

[0005] Determining the coordinates of the first intersection point, the second intersection point, and the third intersection point in the coordinate system of the radar based on the single-line laser point cloud obtained by the radar scanning the V-shaped calibration board, where the first intersection point is the intersection point of the single-line laser and the first edge line, the second intersection point is the intersection point of the single-line laser and the second edge line, and the third intersection point is the intersection point of the single-line laser and the intersection line;

[0006] Determining the external parameters between the camera and the radar based on the plane normal vectors of the main surfaces, the first plane, and the second plane of the two boards, and the coordinates of the first intersection point, the second intersection point, and the third intersection point.

[0007] Wherein, determining the coordinates of the first intersection point, the second intersection point, and the third intersection point in the coordinate system of the radar based on the single-line laser point cloud obtained by the radar scanning the V-shaped calibration board includes:

[0008] Determining the laser line segment point sets on the main surfaces of the two boards based on the single-line laser point cloud;

[0009] Based on the set of laser line segment points on the main surfaces of two boards, determine the first intersection point coordinates and the second intersection point coordinates in the coordinate system of the radar, and determine the equations of the laser line segments on the main surfaces of the two boards respectively;

[0010] Calculate the third intersection point coordinates through the equations of the laser line segments on the two boards.

[0011] Among them, determining the set of laser line segment points on the main surfaces of two boards based on single-line laser point cloud data includes:

[0012] Connect the first point and the last point in the single-line laser point cloud to obtain an initial straight line;

[0013] When the maximum value among the distances from all laser points between the starting point and the ending point of the current straight line to the current straight line exceeds the first distance threshold, take the point corresponding to the maximum value as the break point, and connect the starting point and the ending point of the current straight line with the break point respectively to obtain two sub-line segments, where the current straight line is initially the initial straight line;

[0014] Take each of the two sub-line segments as the current straight line respectively, and execute the operation that when the maximum value among the distances from all laser points between the starting point and the ending point of the current straight line to the current straight line exceeds the first distance threshold, take the point corresponding to the maximum value as the break point, and connect the starting point and the ending point of the current straight line with the break point respectively to obtain two sub-line segments, until no new break points appear in each sub-line segment, so as to obtain a set of point cloud line segments;

[0015] Select two point cloud line segments from the set of point cloud line segments as the set of laser line segment points on the main surfaces of the two boards, where the distance between the center point of the two selected point cloud line segments and the origin of the radar coordinate system is less than the distance between the center point of the unselected point cloud line segments in the set of point cloud line segments and the origin of the radar coordinate system.

[0016] Among them, based on the set of laser line segment points on the main surfaces of two boards, determining the first intersection point coordinates and the second intersection point coordinates in the coordinate system of the radar, and determining the equations of the laser line segments on the main surfaces of the two boards respectively includes:

[0017] When, along the laser scanning direction, the board to which the first edge line belongs is downstream of the board to which the second edge line belongs, take the point cloud line segment with the larger center point Y coordinate among the two selected point cloud line segments as the laser line segment on the main surface of the board to which the first edge line belongs, take the coordinates of the last scanned point in the point cloud line segment with the larger center point Y coordinate as the first intersection point coordinates, take the point cloud line segment with the smaller center point Y coordinate among the two selected point cloud line segments as the laser line segment on the main surface of the board to which the second edge line belongs, and take the coordinates of the first scanned point in the point cloud line segment with the smaller center point Y coordinate as the second intersection point coordinates.

[0018] Among them, based on the image data obtained by photographing the V-shaped calibration plate with a camera, the plane normal vectors of the main surfaces, the first plane, and the second plane of the two plates in the V-shaped calibration plate facing the camera in the coordinate system of the camera are calculated, including:

[0019] Based on the image data, the plane point clouds and plane normal vectors of the main surfaces of the two plates are calculated;

[0020] Based on the plane point clouds of the main surfaces of the two plates, the direction vectors of the first edge line and the second edge line are determined;

[0021] The plane normal vector of the first plane is calculated using the direction vector of the first edge line and the coordinates of the first preset point, and the plane normal vector of the second plane is calculated using the direction vector of the second edge line and the coordinates of the second preset point.

[0022] Among them, based on the image data, the plane point clouds and plane normal vectors of the main surfaces of the two plates in the V-shaped calibration plate facing the camera are calculated, including:

[0023] Extract the plane point clouds of the three planes from the image data;

[0024] Determine the plane point clouds and plane normal vectors of the main surfaces of the two plates from the extracted plane point clouds of the three planes.

[0025] Among them, determining the plane point clouds and plane normal vectors of the main surfaces of the two plates from the extracted plane point clouds of the three planes includes:

[0026] Calculate the inner product of the normal vectors of every two planes among the three planes;

[0027] Based on the inner product, determine the ground plane and its normal vector from the three planes;

[0028] Calculate the plane vector of the third plane formed by the third preset point in the plane point cloud of the ground plane and the intersection line of the two planes other than the ground plane, and calculate the first vector, where the first vector is the vector between the non-intersection line points in the plane point cloud of one of the two planes and the non-intersection line points in the plane point cloud of the other plane of the two planes;

[0029] Based on the inner product of the first vector and the plane vector of the third plane, confirm which main surface of the two plates each plane corresponds to, so as to determine the plane point clouds and plane normal vectors of the main surfaces of the two plates.

[0030] Among them, determining the ground plane and its normal vector from the three planes based on the inner product includes:

[0031] Based on the inner product and the placement method of the V-shaped calibration plate, determine the ground plane from the three planes;

[0032] Calculate the second vector, which is the vector between the fourth preset point and the points in the plane point cloud of the ground plane. The fourth preset point is a point in the plane point cloud of a plane other than the ground plane among the three planes;

[0033] Based on the positive or negative attribute of the inner product of the plane normal vector corresponding to the ground plane and the second vector, and the plane normal vector corresponding to the ground plane, determine the plane normal vector of the ground plane.

[0034] Among them, the fourth preset point is the center point in the plane point cloud of a plane other than the ground plane among the three planes; the second vector is the vector between the fourth preset point and the center point in the plane point cloud of the ground plane.

[0035] Among them, extract the plane point clouds of the three planes from the image data, including:

[0036] Use the method of determining a plane by three points to extract the plane point clouds of the three planes and their corresponding plane normal vectors from the point cloud data one by one.

[0037] Among them, using the method of determining a plane by three points to extract the plane point clouds of the three planes and their corresponding plane normal vectors from the point cloud data one by one includes:

[0038] Adopt the ransac algorithm combined with the method of determining a plane by three points to roughly extract the plane, and obtain the rough extraction plane equation and the rough extraction plane point cloud of the plane to be extracted among the three planes;

[0039] Taking the coefficients of the rough extraction plane equation as the initial values, use the least squares method to minimize the distance from the points to the plane to be extracted, and construct a residual equation;

[0040] Use the residual equation for fitting to obtain the plane equation and the plane normal vector corresponding to the plane to be extracted;

[0041] Based on the plane equation corresponding to the plane to be extracted and the rough extraction plane point cloud, determine the plane point cloud of the plane to be extracted.

[0042] Among them, based on the plane point clouds of the main surfaces of the two boards respectively, determining the direction vectors of the first edge line and the second edge line includes:

[0043] Based on the direction vector of the intersection line and the intersection coordinates of the three planes, determine the maximum and minimum values of the distances from all points in the plane point cloud of the main surface of the board to which the first edge line or the second edge line belongs to the preset intersection line. The preset intersection line is the intersection line of the main surface of the board to which the first edge line or the second edge line belongs and the ground plane;

[0044] Determine at least one point with the largest distance from the intersection line among all points within the respective distance ranges between the minimum value and the maximum value corresponding to the first edge line or the second edge line;

[0045] Summarize at least one point in all distance ranges between the minimum value and the maximum value corresponding to the first edge line or the second edge line to obtain a point set of the first edge line or the second edge line;

[0046] Calculate the direction vector of the first edge line or the second edge line based on the point set of the first edge line or the second edge line.

[0047] Among them, based on the plane normal vectors of the main surfaces of the two boards, the first plane and the second plane, as well as the first intersection coordinate, the second intersection coordinate, and the third intersection coordinate, the external parameters between the camera and the radar are determined, including:

[0048] Construct a first point-plane constraint equation by using the first intersection coordinate and the normal vector of the first plane; construct a second point-plane constraint equation by using the first intersection coordinate and the plane normal vector of the main surface of the board to which the first edge line belongs, construct a third point-plane constraint equation by using the second intersection coordinate and the normal vector of the second plane, construct a fourth point-plane constraint equation by using the second intersection coordinate and the plane normal vector of the main surface of the board to which the second edge line belongs, and construct a fifth point-plane constraint equation and a sixth point-plane constraint equation by using the third intersection coordinate and the plane normal vectors of the main surfaces of the two boards respectively;

[0049] Calculate the external parameters based on the first point-plane constraint equation, the second point-plane constraint equation, the third point-plane constraint equation, the fourth point-plane constraint equation, the fifth point-plane constraint equation, and the sixth point-plane constraint equation.

[0050] To solve the above technical problems, a second aspect of the present application provides an electronic device, which includes a processor and a memory. The memory is used to store a computer program, and the processor is used to execute the computer program to implement the above method.

[0051] To solve the above technical problems, a third aspect of the present application provides a computer-readable storage medium, which stores a program and / or instructions, and the program and / or instructions are used to be executed to implement the above method.

[0052] The above solution can determine the intersection coordinates of the single-line laser with the first edge line, the second edge line, and the intersection line in the coordinate system of the radar based on the single-line laser point cloud data obtained by the lidar scanning the V-shaped calibration board, so as to use the intersection coordinates of the single-line laser with the first edge line, the second edge line, and the intersection line in the coordinate system of the radar and the plane normal vectors of the main surfaces of the two boards facing the camera, the first plane, and the second plane in the coordinate system of the camera in the V-shaped calibration board to construct multiple point-plane constraint equations, and under the constraints of the constructed point-plane constraint equations, optimize and calculate the accurate values of the structural parameters between the lidar and the camera, so that the calibrated external parameter results are relatively accurate. Description of the Drawings

[0053] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings. Among them:

[0054] Figure 1 is a schematic flowchart of an implementation manner of the method for joint calibration of a lidar and a camera in the present application;

[0055] Figure 2 is a schematic structural diagram of a V-shaped calibration board in the method for joint calibration of a lidar and a camera in the present application;

[0056] Figure 3 is a schematic diagram of the placement positions of the camera and the lidar in the method for joint calibration of a lidar and a camera in the present application;

[0057] Figure 4 is a schematic diagram of the positions of points, lines, and planes in the method for joint calibration of a lidar and a camera in the present application;

[0058] Figure 5 is a schematic diagram of the normal vector directions of the main surfaces corresponding to the two plates in the V-shaped calibration board in the method for joint calibration of a lidar and a camera in the present application;

[0059] Figure 6 is a schematic diagram of the scanning line segments of the lidar on the main surfaces corresponding to the two plates in the V-shaped calibration board in the method for joint calibration of a lidar and a camera in the present application;

[0060] Figure 7 is a schematic diagram of the constraint information for constructing a constraint equation in the method for joint calibration of a lidar and a camera in the present application;

[0061] Figure 8 is a schematic structural diagram of an implementation manner of an electronic device in the present application;

[0062] Figure 9 is a schematic structural diagram of an implementation manner of a computer-readable storage medium in the present application. Specific implementation manners

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0064] The terms "system" and "network" are often used interchangeably in this document. The term " / or" in this document is merely a description of the relationship between associated objects, indicating that there can be three relationships. For example, A / or B can represent three situations: A exists alone, both A and B exist simultaneously, and B exists alone. Additionally, the character " / " in this document generally indicates that the associated objects before and after are in an "or" relationship. Furthermore, "plurality" in this document means two or more than two.

[0065] Please refer to Figure 1 , Figure 1 is a schematic flowchart of an embodiment of the method for jointly calibrating a lidar and a camera in this application. The method includes:

[0066] S101: Based on the image data obtained by the camera photographing the V-shaped calibration plate, calculate the plane normal vectors of the two planes, the first plane and the second plane, of the two plates in the V-shaped calibration plate that face the camera in the coordinate system of the camera.

[0067] Among them, the V-shaped calibration plate can be composed of two plates (such as Figure 2 the A plate and the B plate shown). Specifically, as Figure 3 shown, the two plates can be placed at an angle of α, and one side of the two plates is arranged to fit together. In this way, the V-shaped calibration plate of this application is formed by combining the two plates. Among them, α can be set according to the actual situation and is not limited here. For example, the range of α can be [10°, 170°]. More preferably, 30° ≤ α ≤ 75° or 105° ≤ α ≤ 150°. Most preferably, α = 135°.

[0068] Furthermore, for the convenience of placing the V-shaped calibration plate, the two plates in the V-shaped calibration plate can be connected by a hinge, so that the two plates can be opened and closed to any angle, which is convenient for cooperating with the external parameter calibration between the lidar and the camera and is also convenient for carrying.

[0069] In addition, the two plates in the V-shaped calibration plate can both be right-angled V-shaped calibration plates. And the right-angled sides of the two plates can be mutually attached and connected by hinges at both ends. To improve the calibration efficiency, the longer right-angled plates of the two plates can be mutually attached and fixed. In other embodiments, the shapes of the two plates in the V-shaped calibration plate are not limited to right-angled V-shaped calibration plates. For example, they can also be quadrilateral plates such as rectangular plates.

[0070] In step S101, the image data obtained by the camera photographing the V-shaped calibration plate can be obtained first, and then based on the obtained image data, the plane normal vectors of the main surfaces, the first plane and the second plane, of the two plates in the V-shaped calibration plate that face the camera are calculated.

[0071] To obtain better image data, as Figure 3As shown, the camera can be oriented to photograph the surface with the largest area of the V-shaped calibration plate, that is, the camera can photograph the surfaces with the largest area of the two plates of the V-shaped calibration plate respectively. For example, the camera can be placed at Figure 3 points 1, 2, 3, 4, or 5 as shown, and the orientation of the camera can be adjusted to the arrow direction, and then the camera is controlled to photograph to obtain image data including the V-shaped calibration plate.

[0072] After obtaining the image data of the V-shaped calibration plate photographed by the camera, the plane normal vector of the required plane can be calculated based on the obtained image data.

[0073] Among them, the plane point clouds and plane normal vectors of the main surfaces of the two plates in the V-shaped calibration plate facing the camera respectively (for example Figure 2 the surface with the largest area of plate A facing the camera and the surface with the largest area of plate B facing the camera as shown) can be calculated based on the obtained image data; then the direction vectors of the first edge line (such as Figure 4 line QS as shown) and the second edge line (such as Figure 4 line QR as shown) can be determined based on the plane point clouds of the main surfaces of the two plates facing the camera respectively, where the first edge line and the second edge line are the edge lines on the main surfaces of the two plates respectively opposite to the intersection line (such as Figure 4 line QP cross ), and the intersection line is the intersection line of the main surfaces of the two plates facing the camera; the plane normal vector of the first plane is calculated using the direction vector of the first edge line and the coordinates of the first preset point, where the first plane is the plane formed by the first edge line and the first preset point; the plane normal vector of the second plane is calculated using the direction vector of the second edge line and the coordinates of the second preset point, where the second plane is the plane formed by the second edge line and the second preset point. And neither the first preset point nor the second preset point is on the main surfaces of the two plates facing the camera. In addition, the first preset point and the second preset point can be the same or different points. Preferably, the first preset point and the second preset point are the coordinate origin of the camera coordinate system.

[0074] Among them, the main surface of each plate facing the camera can be understood as: the surface with the largest area among the surfaces of each plate facing the camera. Among them, the image data can be presented in the form of point cloud data, and the process of calculating the plane point clouds and plane normal vectors of the main surfaces of the two plates in the V-shaped calibration plate facing the camera respectively based on the obtained image data can include: extracting the plane point clouds of three planes from the image data; determining the plane point clouds and plane normal vectors of the main surfaces of the two plates respectively from the three extracted plane point clouds.

[0075] Among them, the process of extracting the plane point clouds of three planes from the image data can be as follows: Use the method of determining a plane with three points to extract the plane point cloud of the first plane from the image data and determine the plane normal vector corresponding to this plane; Filter out the plane point cloud of the first plane from the image data to obtain the first point cloud data; Use the method of determining a plane with three points to extract the plane point cloud of the second plane from the first point cloud data and determine the plane normal vector corresponding to the second plane; Filter out the plane point cloud of the second plane from the first point cloud data to obtain the second point cloud data; Use the method of determining a plane with three points to extract the plane point cloud of the third plane from the second point cloud data and determine the plane normal vector corresponding to the third plane.

[0076] Specifically, the process of extracting the plane point cloud and the plane normal vector from the point cloud data by using the method of determining a plane with three points can include: Coarsely extract the plane by using the RANSAC algorithm combined with the method of determining a plane with three points to obtain the coarsely extracted plane equation and the coarsely extracted plane point cloud of the plane to be extracted; Use the coefficients of the coarsely extracted plane equation of the plane to be extracted as the initial values, and use the least squares method to minimize the distance from the points to the plane to construct a residual equation; Use the residual equation for fitting to obtain the plane equation and the plane normal vector corresponding to the plane to be extracted; Determine the plane point cloud of the plane to be extracted based on the plane equation obtained by fitting and the coarsely extracted plane point cloud.

[0077] Exemplarily, the process of extracting the plane point clouds of three planes from the image data can be as follows:

[0078] 1>. Point cloud filtering: Filter out the invalid data in C k ;

[0079] 2>. Coarse plane extraction: Coarsely extract the plane from C k by using the RANSAC algorithm combined with the method of determining a plane with three points to obtain the plane equation A0X + B0Y + C0Z + D0 = 0 and the plane point cloud Cloud0;

[0080] 3>. Plane fitting: Use the coefficients of the plane equation obtained in 2> as the initial values, and use the least squares method to minimize the distance from the points to the plane to construct the following residual equation

[0081]

[0082] Solving gives the plane equation A1X + B1Y + C1Z + D1 = 0, then the plane normal vector is [A1, B1, C1] T , and normalize this vector to obtain the final plane vector Calculate the distance from each point in Cloud0 to this plane and sort them, and delete the 10% of the points with the largest distance to finally obtain the plane point cloud Cloud1;

[0083] 4> From Ck Filter out the planar point cloud Cloud1, and repeat steps 2> and 3> to obtain the planar normal vector and the planar point cloud Cloud2;

[0084] 5> Continue to filter out the planar point cloud Cloud2 from C k and repeat steps 2> and 3> to obtain the plane equation and the planar point cloud Cloud3.

[0085] After extracting the planar point clouds of the three planes from the image data based on the above steps, the planar point clouds and planar normal vectors of the main surfaces of the two plates can be determined from the three extracted planar point clouds.

[0086] The steps of determining the planar point clouds and planar normal vectors of the main surfaces of the two plates from the three extracted planar point clouds may include: calculating the inner product of the normal vectors of every two planes among the three planes, and determining the ground plane and its normal vector from the three planes based on the inner product; calculating the planar vector of the third plane formed by the third preset point in the planar point cloud of the ground plane and the intersection line of the two planes other than the ground plane, and calculating the first vector, where the first vector is the vector from the non-intersection line point in the planar point cloud of one plane among the two planes to the non-intersection line point in the planar point cloud of the other plane among the two planes; confirming which main surface of the two plates each plane corresponds to based on the inner product of the first vector and the planar vector of the third plane.

[0087] Because it is necessary to determine the ground plane and its normal vector from the three planes based on the inner product, there are corresponding requirements for the placement of the V-shaped calibration plate. Preferably, the main surfaces of the two plates of the V-shaped calibration plate are substantially perpendicular to the ground plane, and the main surfaces of the two plates of the V-shaped calibration plate are significantly not perpendicular. For example, the difference between the angle between the main surfaces of the two plates of the V-shaped calibration plate and the ground plane and 90° is less than the threshold, and the difference between the angle between the main surfaces of the two plates of the V-shaped calibration plate and 90° is greater than the threshold. In this way, the inner product of the ground normal vector and the normal vector of the main surface of plate A is approximately equal to 0, and the inner product of the ground normal vector and the normal vector of the main surface of plate B is also approximately equal to 0, that is And there is a certain gap between the inner product of the normal vector of the main surface of plate A and the normal vector of the main surface of plate B and 0. In this way, the ground plane can be determined from the three planes through the three inner products. Among them, the threshold can be set according to the actual situation and is not limited here. For example, it can be 5°.

[0088] After determining the ground plane, the plane normal vector of the ground plane can be determined. Among them, a second vector can be calculated. The second vector is the vector from the fifth preset point in the plane point cloud of the ground plane to the fourth preset point. The fourth preset point is a point in the plane point cloud of a plane other than the ground plane. Based on the positive or negative property of the inner product of the plane normal vector corresponding to the ground plane and the second vector, and the plane normal vector corresponding to the ground plane, the plane normal vector of the ground plane is determined. Specifically, if the positive or negative property of the inner product of the plane normal vector corresponding to the ground plane and the second vector is negative, that is, the inner product of the plane normal vector corresponding to the ground plane and the second vector is a negative number, then the plane normal vector of the ground plane is the opposite of the plane normal vector corresponding to the ground plane; if the positive or negative property of the inner product of the plane normal vector corresponding to the ground plane and the second vector is positive, that is, the inner product of the plane normal vector corresponding to the ground plane and the second vector is a positive number, then the plane normal vector of the ground plane is equal to the plane normal vector corresponding to the ground plane.

[0089] Exemplarily, to improve the accuracy of the calculated plane normal vector of the ground plane and thus improve the calculation accuracy of the plane normal vector of the ground plane, the center point of the plane point cloud of a plane other than the ground plane can be used as the fourth preset point, and the center point of the plane point cloud of the ground plane can be used as the fifth preset point. Of course, this is not limited to this.

[0090] In addition, the calculation process of the plane normal vector of the third plane can be as follows: calculate the intersection coordinates of the three planes; calculate a third vector, where the third vector is the vector between the third preset point and the intersection of the three planes; cross-multiply the third vector and the normal vector of the ground plane to obtain the plane normal vector of the third plane. Among them, the third preset point can be the center point of the plane point cloud of the ground plane, or of course any point in the plane point cloud of the ground plane.

[0091] After calculating the plane normal vector of the third plane, based on the positive or negative property of the inner product of the plane normal vector of the third plane and the first vector, it can be confirmed which main surface of the plate each of the two planes corresponds to. Specifically, if the positive or negative property of the inner product of the plane normal vector of the third plane and the first vector is negative, then one of the two planes is the main surface of the plate on the right side among the two plates, and the other plane of the two planes is the main surface of the plate on the left side among the two plates; if the positive or negative property of the inner product of the plane normal vector of the third plane and the first vector is positive, then one of the two planes is the main surface of the plate on the left side among the two plates, and the other plane of the two planes is the main surface of the plate on the right side among the two plates. In this way, the plane point clouds and plane normal vectors of the main surfaces of the two plates in the V-shaped calibration plate facing the camera are determined.

[0092] Then, the direction vectors of the first edge line and the second edge line can be determined based on the plane point clouds of the main surfaces of the two plates facing the camera.

[0093] Among them, the first edge line can be the edge line on the main surface of the left plate opposite to the intersection line. In this way, the direction vector of the intersection line can be calculated first; then, based on the direction vector of the intersection line and the intersection coordinates of the three planes, the maximum and minimum values of the distances from all points in the plane point cloud of the main surface of the left plate to the intersection line of the main surface of the left plate and the ground plane (for the convenience of description, the intersection line of the main surface of the left plate and the ground plane can be called the first intersection line) are determined; at least one point with the maximum distance from the intersection line among all points within each distance range between the minimum and maximum values of the distances from the plane point cloud of the main surface of the left plate to the first intersection line is determined; at least one point in all ranges between the minimum and maximum values is aggregated to obtain the point set of the first edge line; the direction vector of the first edge line is calculated based on the point set of the first edge line. And calculating the direction vector of the first edge line based on the point set of the first edge line can include: using the least squares method and determining the equation of the first edge line based on the point set of the first edge line, and thus obtaining the direction vector of the first edge line. Specifically, after calculating the point set of the first edge line, a residual equation of the distance from a point to the first edge line can be constructed by methods such as the least squares method, and then the residual equation is iterated successively until convergence to calculate the equation of the first edge line, thereby obtaining the direction vector of the first edge line.

[0094] Among them, the intersection line of the main surface of the left plate and the ground plane can be perpendicular to the intersection line, that is, the lower right corner of the main surface of the left plate is approximately a right angle. In this way, the direction vector of the intersection line can be used as the normal vector of the intersection line of the main surface of the left plate and the ground plane. In this way, when calculating the distances from each point in the plane point cloud of the main surface of the left plate to the first intersection line, the vectors from each point in the plane point cloud of the main surface of the left plate to the intersection point of the three planes can be calculated first, and then the vectors from each point in the plane point cloud of the main surface of the left plate to the intersection point of the three planes and the direction vector of the intersection line are substituted into the formula for the distance from a point to a line of a vector to calculate the distances from each point in the plane point cloud of the main surface of the left plate to the first intersection line.

[0095] In addition, in order to reduce the calculation error of the direction vector of the first edge line, the distance range with relatively large noise between the minimum value and the maximum value can be filtered out first, and then at least one point with the largest distance from the intersection line among all points in each distance range within the filtered range of the distances from the plane points of the main surface of the plate on the left to the first intersection line is determined, and then the point set of the relatively accurate first edge line can be obtained by summarization. Among them, since the point clouds of the uppermost and lowermost surfaces of the V-shaped calibration plate have relatively large noise, the distance range with relatively large noise can be set as [minimum value, minimum value + first preset value] ∪ [maximum value - second preset value, maximum value]. The first preset value and the second preset value can be set according to the actual situation and are both positive numbers, which are not limited here. For example, both can be set to 0.1. Moreover, the widths of all distance ranges can be equal. For example, they are all 2 cm or 3 cm. Of course, in other embodiments, the widths of the distance ranges may not be equal.

[0096] Correspondingly, the second edge line can be the edge line opposite to the intersection line on the main surface of the plate on the right, and the direction vector of the second edge line can be calculated by the same method as the first edge line. Specifically, the direction vector of the intersection line can be calculated first; then, based on the direction vector of the intersection line and the intersection coordinates of the three planes, the maximum value and the minimum value of the distances from all points in the plane points of the main surface of the plate on the right to the intersection line of the main surface of the plate on the right and the ground plane (for the convenience of description, the intersection line of the main surface of the plate on the left and the ground plane can be called the second intersection line) are determined; at least one point with the largest distance from the intersection line among all points in each distance range within the range of the distances from the plane points of the main surface of the plate on the right to the second intersection line between the minimum value and the maximum value is determined; at least one point in all ranges between the minimum value and the maximum value is summarized to obtain the point set of the second edge line; and the direction vector of the second edge line is calculated based on the point set of the second edge line.

[0097] After determining the direction vectors of the first edge line and the second edge line based on the plane point clouds of the main surfaces of the two plates facing the camera respectively, the normal vectors of the first plane and the second plane can be calculated by using the direction vectors of the first edge line and the second edge line respectively.

[0098] Specifically, the coordinates of any point on the first edge line (for example, the coordinates of a point that can be called point H) can be determined based on the direction vector of the first edge line first, the first direction vector is calculated by using the coordinates of the selected point on the first edge line and the coordinates of the first preset point, and then the first direction vector and the direction vector of the first edge line are cross-multiplied to obtain the normal vector of the first plane.

[0099] Accordingly, the coordinates of any point on the second edge line (e.g., the coordinates of a point that can be called point K) can be determined based on the direction vector of the second edge line first. The second direction vector is calculated using the coordinates of the selected point on the second edge line and the coordinates of the second preset point. Then, the cross product of the second direction vector and the direction vector of the second edge line is used to obtain the normal vector of the second plane.

[0100] Among them, the first preset point and the second preset point can be the same point or different points. To improve the feasibility of the method and accurately select the first preset point and the second preset point that are not on the main surfaces of the two plates facing the camera, the origin of the camera coordinates can be used as the first preset point and the second preset point.

[0101] S102: Based on the single-line laser point cloud data obtained by lidar scanning the V-shaped calibration plate, determine the intersection coordinates of the single-line laser with the first edge line, the second edge line, and the intersection line in the lidar coordinate system.

[0102] Based on the single-line laser point cloud data obtained by lidar scanning the V-shaped calibration plate, the intersection coordinates of the single-line laser with the first edge line, the second edge line, and the intersection line in the lidar coordinate system can be determined, so as to use the intersection coordinates of the single-line laser with the first edge line, the second edge line, and the intersection line in the lidar coordinate system and the plane normal vectors of the main surfaces of the two plates facing the camera, the first plane, and the second plane in the camera coordinate system of the V-shaped calibration plate to construct multiple point-plane constraint equations, and under the constraints of the constructed point-plane constraint equations, optimize the calculation to obtain the accurate values of the structural parameters between the lidar and the camera.

[0103] In step S102, the laser line segment point sets on the main surfaces of the two plates can be determined based on the single-line laser point cloud data; based on the laser line segment point sets on the main surfaces of the two plates, the intersection coordinates of the single-line laser with the first edge line and the second edge line in the lidar coordinate system are determined, and the equations of the laser line segments on the main surfaces of the two plates are determined; the intersection coordinates of the two laser line segments are calculated using the equations of the laser line segments on the main surfaces of the two plates, that is, the intersection coordinates of the single-line laser with the intersection line in the lidar coordinate system are calculated.

[0104] In one implementation, a first distance threshold can be set; connect the first point and the last point in the laser point cloud to obtain an initial straight line; when the maximum value among the distances from all laser points between the first point and the last point of the current straight line to the current straight line exceeds the first distance threshold, use the point corresponding to the maximum value as a break point, and connect the starting point and the ending point of the current straight line to the break point respectively to obtain two sub-line segments, where the current straight line is initially the initial straight line; use each of the two sub-line segments as the current straight line respectively, and execute the step of "when the maximum value among the distances from all laser points between the first point and the last point of the current straight line to the current straight line exceeds the first distance threshold, use the point corresponding to the maximum value as a break point, and connect the break point to the starting point and the ending point of the current straight line respectively to obtain two sub-line segments" to confirm whether there is a break point between the first point and the last point of each sub-line segment, and when there is a break point between the first point and the last point of a sub-line segment, obtain the line segments connected by the first point and the last point of the sub-line segment to their corresponding break points respectively. Repeat the above operations until no new break points appear in each sub-line segment, and finally obtain a set of point cloud line segments; calculate the central point coordinates of each line segment in the set of point cloud line segments; use the two point cloud line segments with the smallest central point X coordinates in the set of point cloud line segments as the laser point cloud line segments on the main surfaces of the two plates respectively. Among them, the first distance threshold can be set according to the actual situation (such as the size of the V-shaped calibration plate), which is not limited here. For example, it can be set to 5 cm.

[0105] Next, based on the magnitude relationship of the central point Y coordinates of the two point cloud line segments and the radar laser scanning direction, the intersection coordinates of the single-line laser with the first edge line and the second edge line in the radar coordinate system can be determined. Specifically, if the laser scanning direction is from left to right, the point cloud line segment with the larger central point Y coordinate can be used as the laser point cloud line segment on the main surface of the plate on the right side, and the coordinate of the last scanned point in the point cloud line segment with the larger central point Y coordinate can be used as the intersection coordinate of the single-line laser with the second edge line in the radar coordinate system, and the coordinate of the first scanned point in the point cloud line segment with the smaller central point Y coordinate can be used as the intersection coordinate of the single-line laser with the first edge line in the radar coordinate system. Another example, if the laser scanning direction is from right to left, the point cloud line segment with the smaller central point Y coordinate can be used as the laser point cloud line segment on the main surface of the plate on the right side, and the coordinate of the first scanned point in the point cloud line segment with the smaller central point Y coordinate can be used as the intersection coordinate of the single-line laser with the second edge line in the radar coordinate system, and the coordinate of the last scanned point in the point cloud line segment with the larger central point Y coordinate can be used as the intersection coordinate of the single-line laser with the first edge line in the radar coordinate system.

[0106] Of course, it is not limited to determining the intersection coordinates of the single-line laser with the first edge line and the second edge line in the coordinate system of the radar through the above method. For example, two sets of lidar point cloud data in the calibration scene with and without the V-shaped calibration plate placed can also be extracted under the same radar scanning conditions; since there will be relatively large changes in the point cloud values at the corresponding angles of the calibration plate before and after the target is placed, the lidar point cloud data on the calibration plate can be extracted by setting a second distance threshold; then, the point cloud data between the maximum value point and the first scanning point in the lidar point cloud data on the calibration plate is used as the point cloud data on one main surface of the plate, and the point cloud data between the maximum value point and the last scanning point in the lidar point cloud data on the calibration plate is used as the point cloud data on the other main surface of the plate. Next, similarly, based on the size of the Y coordinates of the center points of the two point cloud line segments and the radar laser scanning direction, the intersection coordinates of the single-line laser with the first edge line and the second edge line in the coordinate system of the radar can be determined.

[0107] S103: Determine the external parameters between the camera and the lidar based on the plane normal vectors of the main surfaces of the two plates, the first plane and the second plane, as well as the first intersection coordinate, the second intersection coordinate, and the third intersection coordinate.

[0108] The external parameters between the camera and the lidar can be calculated based on the plane normal vectors of the main surfaces of the two plates, the first plane and the second plane in the camera coordinate system calculated in step S101, and the first intersection coordinate, the second intersection coordinate, and the third intersection coordinate in the lidar coordinate system calculated in step S102.

[0109] Optionally, a first point-plane constraint equation can be constructed using the first intersection coordinate and the normal vector of the first plane, a second point-plane constraint equation can be constructed using the first intersection coordinate and the plane normal vector of the main surface of the plate to which the first edge line belongs, a third point-plane constraint equation can be constructed using the second intersection coordinate and the normal vector of the second plane, a fourth point-plane constraint equation can be constructed using the second intersection coordinate and the plane normal vector of the main surface of the plate to which the second edge line belongs, a fifth point-plane constraint equation and a sixth point-plane constraint equation can be constructed using the third intersection coordinate and the plane normal vectors of the main surfaces of the two plates respectively, and the external parameters between the camera and the lidar can be calculated based on the first point-plane constraint equation, the second point-plane constraint equation, the third point-plane constraint equation, the fourth point-plane constraint equation, the fifth point-plane constraint equation, and the sixth point-plane constraint equation.

[0110] Optionally, under the constraints of the above constraint equations, a non-linear optimization method such as the LM (Levenberg-Marquardt) algorithm can be used to optimize the initial value of the external parameters to obtain the final external parameters between the lidar and the camera. In the iterative optimization process, specifically, the residuals can be calculated using the above 6 point-plane constraint equations, and the optimal solution of the external parameters can be continuously iteratively optimized based on the residuals.

[0111] Among them, the external parameters between the camera and the lidar may include a rotation matrix and an offset vector

[0112] In this embodiment, based on the single-line laser point cloud data obtained by the lidar scanning the V-shaped calibration plate, the intersection coordinates of the single-line laser with the first edge line, the second edge line, and the intersection line in the lidar coordinate system can be determined, so as to use the intersection coordinates of the single-line laser with the first edge line, the second edge line, and the intersection line in the lidar coordinate system and the plane normal vectors of the main surfaces, the first plane, and the second plane of the two plates of the V-shaped calibration plate facing the camera in the camera coordinate system to construct multiple point-plane constraint equations, and under the constraints of the constructed point-plane constraint equations, optimize the calculation to obtain the accurate values of the structural parameters between the lidar and the camera, so that the calibrated external parameter results are relatively accurate.

[0113] The following is to better illustrate the method for jointly calibrating the lidar and the camera of the present application. The following specific embodiments of jointly calibrating the lidar and the camera are provided for exemplary illustration:

[0114] Preparation process:

[0115] 1. Stand the V-shaped calibration plate on the ground and adjust the opening angle of the two plates of the V-shaped calibration plate to about 135°;

[0116] 2. Place the AGV at each point on the dotted line, adjust the head direction of the AGV to the arrow direction, collect laser data and TOF data, and finally obtain the laser point cloud set S = {S1, S2,..., S5} and the TOF point cloud set C = {C1, C2,..., C5}.

[0117] Calibration process:

[0118] 1. For each pair of laser point cloud S k ∈S and TOF point cloud C k ∈C, perform the following processing

[0119] a. Extract three planes in C k :

[0120] Ground normal vector and point cloud Cloud g , plane A normal vector and point cloud Cloud A , and plane B normal vector and point cloud Cloud B .

[0121] 1> Point cloud filtering: Filter the invalid data in C k ;

[0122] 2 > Coarse extraction of the plane: For C k Coarsely extract the plane by using the RANSAC algorithm combined with the method of determining a plane by three points, obtaining the plane equation A0X + B0Y + C0Z + D0 = 0 and the plane point cloud Cloud0;

[0123] 3 > Fitting the plane: Use the coefficients of the plane equation obtained in 2> as the initial values, and adopt the least - squares method to minimize the distance from points to the plane, constructing the following residual equation

[0124]

[0125] Solving it gives the plane equation A1X + B1Y + C1Z + D1 = 0, then the plane normal vector is [A1, B1, C1] T , and normalize this plane normal vector to obtain the final plane vector Calculate the distance from each point in Cloud0 to this plane and sort them, delete the 10% of the points with the largest distance, and finally obtain the plane point cloud Cloud1;

[0126] 4 > Filter out the plane point cloud Cloud1 from C k , and repeat the steps of 2> and 3> to obtain the plane normal vector and the plane point cloud Cloud2;

[0127] 5 > Continue to filter out the plane point cloud Cloud2 from C k , and repeat the steps of 2> and 3> to obtain the plane equation and the plane point cloud Cloud3.

[0128] b. Determine the ground, plane A, and plane B from the above three planes

[0129] Assume the finally obtained ground normal vector and the point cloud Cloud g , the plane A normal vector

[0130] and the point cloud Cloud A as well as the plane B normal vector and the point cloud Cloud B ;

[0131] 1 > Determine the ground normal vector and Cloud g

[0132] Calculate the inner products between every two of the three plane normal vectors

[0133] ​Since plane A and plane B are almost perpendicular to the ground (in practice, they will not be absolutely perpendicular and there will definitely be some errors), the inner product of the ground normal vector and the normal vector of plane A or B is approximately equal to 0, that is And as Figure 5 shown, there are two cases: or

[0134] Let

[0135] 1) When -0.05 < val1 < 0.05 and at this time

[0136] 2) When and -0.05 < val2 < 0.05

[0137] After determining which vector is it is also necessary to ensure that its direction points above the ground;

[0138] Calculate the center points P g of Cloud A and Cloud B respectively, where meang P meanA and P meanB , among which,

[0139] Calculate the vector from point P meang to point P meanA

[0140] Find and inner product If is less than 0, then

[0141] So far, the ground normal vector has been determined, and Cloud g can also be determined according to the correspondence between the vector and the point cloud. However, plane A and plane B have not been determined yet. It may be like the above, or the normal vectors may be swapped. Temporarily, their correspondence is as in 1) and 2).

[0142] 2> Determine the normal vector of plane A and the normal vector of plane B

[0143] Use the formula to solve the intersection line vector of plane A and plane B; ​

[0144] Use the formula to solve for the intersection points of the ground, plane A, and plane B;

[0145] where,

[0146] Calculate the point P meang to the point P cross vector

[0147] Then the plane vector is

[0148] Calculate the point P meanA to the point P meanB vector Also find and inner product

[0149] If the inner product is less than 0, then and are swapped, and similarly the point clouds Cloud A and Cloud B are also swapped. At this point, the normal vectors and point clouds of the two planes have been obtained.

[0150] c. Find the direction vectors of the two hypotenuses (i.e., the first edge line and the second edge line) of the V-shaped calibration plate and

[0151] First, process the point cloud data Cloud of plane A A :

[0152] 1> Calculate the distance dist1 from each point P in Cloud A to the line P A ∈ Cloud A S through the formula cross ;

[0153] 2> Sort all the points in Cloud A from smallest to largest according to dist1, and obtain the maximum value dist 1max and the minimum value dist 1min ;

[0154] 3> Use the sliding window method to sample the edge points of the V-shaped calibration plate from bottom to top along the direction;

[0155] 1) The sliding window size is 2 cm;

[0156] 2) The sliding window moves along the in the direction from dist1 = dist 1min +0.1, and ending at dist1 = dist 1max -0.1, because the point clouds at the bottom and top have relatively large noise;

[0157] 3) Assume that the height corresponding to the bottom edge of the current sliding window is dist1′, then the top edge is dist1′ + 0.02. This sliding window will filter out the points where dist1′ <= dist1 < dist1′ + 0.02 to obtain the point cloud set S slip .

[0158] 4) Through the formula dist2 = (P slip -P cross ) T v cross->S , calculate the perpendicular distance dist2 from P slip ∈S slip to the line QP cross . Select the two points with the largest distances and add them to the set S margin of the edge points of plane A;

[0159] 5) Repeat the above operations until the sliding window finishes sliding to obtain the final set S margin of edge points.

[0160] 3> Calculate the hypotenuse QS, use the least squares method to minimize the distance from points to the line, and construct the following residual equation Based on the residual equation, calculate the equation a0x + b0y + c0 = 0 of QS, and the direction vector of QS is

[0161] Process Cloud B in the same way to obtain the direction vector of QR

[0162] 1> Calculate the normal vectors of plane OSQ and plane ORQ through the following formula and

[0163]

[0164]

[0165] d. Extract laser points

[0166] 1> Select the laser points within the laser scanning angles of ±45° from S k , and filter out the invalid points and the points that are too close.

[0167] 2> Set the distance threshold th. Connect the first point and the last point of the laser point cloud to obtain a straight line. Calculate the maximum value of the distances from the remaining points to this straight line. If this maximum value exceeds th, then the two boundary points and this inflection point form straight lines respectively, and repeat the above operations on these straight lines until no new inflection points appear. Finally, obtain a set of point cloud line segments L = {L1, L2,..., L n}, where L k = {p1, p2,..., p m} ∈ L and is an ordered point set.

[0168] 3> Calculate the center point of each line segment through the formula .

[0169] 4> Select two laser point cloud line segments hitting the V-shaped calibration plate; select the two points with the smallest x coordinates in the above center point set , and and Then the corresponding laser line segments are L q and L r . Assume that the laser line segment hitting the A plate is L A , and the laser line segment hitting the B plate is L B . If:

[0170] 1) 's y coordinate is greater than 's y coordinate, then L A = L q , L B = L r ;

[0171] 2) 's y coordinate is less than 's y coordinate, then L A = L r , L B = L q .

[0172] 5> Calculate the coordinates of points M and N shown in Figure 6 ;

[0173] Based on the above L A = {p A1 , p A2 ,..., p An} and L B = {p B1 , p B2 ,..., p Bn}, the point M where the laser hits the A plate is p M = p An , and the point N where the laser hits the B plate is p N = pB1 .

[0174] 6> Calculate Figure 6 the coordinates of point P shown;

[0175] Fit the straight lines MP and NP, minimize the distance from the points to the straight lines by the least squares method, construct the equation shown in formula (10), and calculate to obtain the MP equation a1x + b1y + c1 = 0 and the NP equation a2x + b2y + c2 = 0. Then, the calculation formula for the coordinates of the intersection point can be

[0176] where,

[0177] 2. Calculate the external parameter transformation matrix from the laser to the TOF camera (i.e., the external parameters between the camera and the radar)

[0178] Assume that the external parameter conversion rotation matrix from the laser to the TOF coordinate system is The offset vector is Then, the above normal vector and the laser points satisfy the following constraints:

[0179] 1) The laser point M is on the plane OSQ and satisfies the constraint

[0180]

[0181] 2) The laser point N is on the plane ORQ and satisfies the constraint

[0182]

[0183] 3) The laser points M and P are on the plane P cross SQ and satisfy the constraint

[0184]

[0185]

[0186] 4) The laser points N and P are on the plane P cross RQ and satisfy the constraint

[0187]

[0188]

[0189] The data of the AGV at each position can provide a total of 6 constraints of 1), 2), 3), and 4) and uniquely determine an external parameter transformation matrix. All the constraints provided by 5 pairs of laser point clouds and TOF point clouds construct the following constraint equations:

[0190]

[0191]

[0192] The above formula can be optimized by using the LM (Levenberg-Marquardt) algorithm to obtain and

[0193] According to [roll, pitch, yaw] can be obtained T .

[0194] So far, the 6-DOF external parameters [roll pitch yaw x y z] have all been obtained.

[0195] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of an embodiment of the electronic device 20 of the present application. The electronic device 20 of the present application includes a processor 22, and the processor 22 is used to execute instructions to implement the method of any one of the above embodiments of the present application and the method provided by any non-conflicting combination.

[0196] The processor 22 can also be called a CPU (Central Processing Unit, central processing unit). The processor 22 may be an integrated circuit chip with signal processing capabilities. The processor 22 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor, or the processor 22 may also be any conventional processor, etc.

[0197] The electronic device 20 may further include a memory 21 for storing instructions and data required for the operation of the processor 22.

[0198] Please refer to Figure 9 , Figure 9This is a schematic structural diagram of a computer-readable storage medium in an embodiment of the present application. The computer-readable storage medium 30 of the embodiment of the present application stores instruction / program data 31, and when the instruction / program data 31 is executed, it implements the method provided by any one of the above-mentioned methods of the present application and any non-conflicting combination. Among them, the instruction / program data 31 can form a program file and be stored in the above storage medium 30 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor can execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium 30 includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical discs, or devices such as computers, servers, mobile phones, and tablets.

[0199] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in an electrical, mechanical, or other form.

[0200] In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0201] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0202] The above are only the embodiments of the present application, and do not thus limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall similarly be included within the patent protection scope of the present application.

Claims

1. A method for joint calibration of a lidar and a camera, characterized in that, The method includes: Based on the image data obtained by the camera photographing the V-shaped calibration plate, calculate the plane normal vectors of the main surfaces, the first plane, and the second plane of the two plates in the V-shaped calibration plate facing the camera in the coordinate system of the camera, where the first plane is the plane formed by the first edge line and the first preset point, the second plane is the plane formed by the second edge line and the second preset point, the first edge line and the second edge line are respectively the edge lines opposite to the intersection line on the main surfaces of the two plates, the intersection line is the line where the main surfaces of the two plates intersect, and neither the first preset point nor the second preset point is in the plane where the main surfaces of the two plates are located; Based on the single-line laser point cloud obtained by the radar scanning the V-shaped calibration plate, determine the coordinates of the first intersection point, the second intersection point, and the third intersection point in the coordinate system of the radar, where the first intersection point is the intersection point of the single-line laser and the first edge line, the second intersection point is the intersection point of the single-line laser and the second edge line, and the third intersection point is the intersection point of the single-line laser and the intersection line; Based on the plane normal vectors of the main surfaces, the first plane, and the second plane of the two plates, as well as the coordinates of the first intersection point, the second intersection point, and the third intersection point, determine the external parameters between the camera and the radar.

2. The calibration method according to claim 1, characterized in that, The determining the coordinates of the first intersection point, the second intersection point, and the third intersection point in the coordinate system of the radar based on the single-line laser point cloud obtained by the radar scanning the V-shaped calibration plate includes: Based on the single-line laser point cloud, determine the set of laser segment points on the main surfaces of the two plates; Based on the set of laser segment points on the main surfaces of the two plates, determine the coordinates of the first intersection point and the second intersection point in the coordinate system of the radar, and determine the equations of the laser segments on the main surfaces of the two plates respectively; Calculate the coordinates of the third intersection point through the equations of the laser segments on the two plates.

3. The calibration method according to claim 2, wherein The determining the set of laser segment points on the main surfaces of the two plates based on the single-line laser point cloud includes: Connect the first point and the last point in the single-line laser point cloud to obtain an initial straight line; In the case where the maximum value among the distances from all laser points between the starting point and the ending point of the current straight line to the current straight line exceeds the first distance threshold, take the point corresponding to the maximum value as the break point, and connect the starting point and the ending point of the current straight line to the break point respectively to obtain two sub-segments, where the current straight line is initially the initial straight line; Take each of the two sub-segments as the current straight line, and execute the step that in the case where the maximum value among the distances from all laser points between the starting point and the ending point of the current straight line to the current straight line exceeds the first distance threshold, take the point corresponding to the maximum value as the break point, and connect the starting point and the ending point of the current straight line to the break point respectively to obtain two sub-segments, until no new break point appears in each sub-segment, so as to obtain a set of point cloud segments; Select two point cloud line segments from the set of point cloud line segments as the laser line segment point sets on the main surfaces of the two boards, where the distances from the center points of the two selected point cloud line segments to the origin of the radar coordinate system are less than the distances from the center points of the unselected point cloud line segments in the set of point cloud line segments to the origin of the radar coordinate system.

4. The calibration method according to claim 2 or 3, characterized in that, Based on the laser line segment point sets on the main surfaces of the two boards, determine the first intersection point coordinates and the second intersection point coordinates in the coordinate system of the radar, and determine the equations of the laser line segments on the main surfaces of the two boards respectively, including: When, along the laser scanning direction, the board to which the first edge line belongs is downstream of the board to which the second edge line belongs, use the point cloud line segment with the larger Y coordinate of the center point among the two selected point cloud line segments as the laser line segment on the main surface of the board to which the first edge line belongs, use the coordinates of the last scanned point in the point cloud line segment with the larger Y coordinate of the center point as the first intersection point coordinates, use the point cloud line segment with the smaller Y coordinate of the center point among the two selected point cloud line segments as the laser line segment on the main surface of the board to which the second edge line belongs, and use the coordinates of the first scanned point in the point cloud line segment with the smaller Y coordinate of the center point as the second intersection point coordinates.

5. The calibration method according to claim 1, wherein Based on the image data obtained by the camera photographing the V-shaped calibration board, calculate the plane normal vectors of the main surfaces, the first plane, and the second plane of the two boards respectively facing the camera in the coordinate system of the camera, including: Based on the image data, calculate the plane point clouds and plane normal vectors of the main surfaces of the two boards respectively; Based on the plane point clouds of the main surfaces of the two boards respectively, determine the direction vectors of the first edge line and the second edge line; Calculate the plane normal vector of the first plane by using the direction vector of the first edge line and the coordinates of the first preset point, and calculate the plane normal vector of the second plane by using the direction vector of the second edge line and the coordinates of the second preset point.

6. The calibration method according to claim 5, wherein, The calculation of the plane point clouds and plane normal vectors of the main surfaces of the two boards respectively based on the image data includes: Extract the plane point clouds of the three planes from the image data; Determine the plane point clouds and plane normal vectors of the main surfaces of the two boards respectively from the extracted plane point clouds of the three planes.

7. The calibration method according to claim 6, wherein The determination of the plane point clouds and plane normal vectors of the main surfaces of the two boards respectively from the extracted plane point clouds of the three planes includes: Calculate the inner product of the normal vectors of every two planes among the three planes; Based on the inner product, determine the ground plane and its normal vector from the three planes; Calculate the plane vector of the third plane formed by the third preset point in the plane point cloud of the ground plane and the intersection line of the two planes other than the ground plane, and calculate the first vector, where the first vector is the vector between the non-intersection line points in the plane point cloud of one of the two planes and the non-intersection line points in the plane point cloud of the other plane among the two planes. Based on the inner product of the plane vectors of the first vector and the third plane, confirm which main surface of each board the two planes correspond to respectively, so as to determine the plane point cloud and the plane normal vector of the main surface of each of the two boards.

8. The calibration method according to claim 7, characterized in that, Determining the ground plane and its normal vector from the three planes based on the inner product includes: Determining the ground plane from the three planes based on the inner product and the placement mode of the V-shaped calibration board; Calculating a second vector, where the second vector is the vector between the fourth preset point and a point in the plane point cloud of the ground plane, and the fourth preset point is a point in the plane point cloud of a plane other than the ground plane among the three planes; Based on the positive and negative attribute of the inner product of the plane normal vector corresponding to the ground plane and the second vector, and the plane normal vector corresponding to the ground plane, determine the plane normal vector of the ground plane.

9. The calibration method according to claim 8, characterized in that, The fourth preset point is the center point in the plane point cloud of a plane other than the ground plane among the three planes; the second vector is the vector between the fourth preset point and the center point in the plane point cloud of the ground plane.

10. The calibration method according to claim 6, characterized in that Extracting the plane point clouds of the three planes from the image data includes: Using the method of determining a plane by three points to extract the plane point clouds of the three planes and their corresponding plane normal vectors one by one from the point cloud data.

11. The calibration method according to claim 10, wherein, The method of using the method of determining a plane by three points to extract the plane point clouds of the three planes and their corresponding plane normal vectors one by one from the point cloud data includes: Coarsely extracting the plane by using the ransac algorithm combined with the method of determining a plane by three points to obtain the coarsely extracted plane equation and the coarsely extracted plane point cloud of the plane to be extracted among the three planes; Taking the coefficients of the coarsely extracted plane equation as the initial values, using the least squares method to minimize the distance from the points to the plane to be extracted, and constructing a residual equation; Performing fitting by using the residual equation to obtain the plane equation and the plane normal vector corresponding to the plane to be extracted; Based on the plane equation corresponding to the plane to be extracted and the coarsely extracted plane point cloud, determine the plane point cloud of the plane to be extracted.

12. The calibration method according to claim 6, wherein, Based on the plane point clouds of the main surfaces of the two boards respectively, determining the direction vector of the first edge line and the direction vector of the second edge line includes: Based on the direction vector of the intersection line and the intersection coordinates of the three planes, determine the maximum value and the minimum value of the distances from all points in the plane point cloud of the main surface of the board to which the first edge line or the second edge line belongs to a preset intersection line, where the preset intersection line is the intersection line of the main surface of the board to which the first edge line or the second edge line belongs and the ground plane among the three planes; Determine at least one point with the maximum distance from the intersection line among all points within each distance range between the minimum value and the maximum value corresponding to the first edge line or the second edge line; Summarize at least one point in all distance ranges between the minimum value and the maximum value corresponding to the first edge line or the second edge line to obtain the point set of the first edge line or the second edge line; Calculate the direction vector of the first edge line or the second edge line based on the point set of the first edge line or the second edge line.

13. The calibration method according to claim 1, characterized in that, Determining the external parameters between the camera and the radar based on the plane normal vectors of the main surfaces, the first plane, and the second plane of the two boards, as well as the first intersection point coordinates, the second intersection point coordinates, and the third intersection point coordinates, includes: Constructing a first point-plane constraint equation using the first intersection point coordinates and the normal vector of the first plane; constructing a second point-plane constraint equation using the first intersection point coordinates and the plane normal vector of the main surface of the board to which the first edge line belongs, constructing a third point-plane constraint equation using the second intersection point coordinates and the normal vector of the second plane, constructing a fourth point-plane constraint equation using the second intersection point coordinates and the plane normal vector of the main surface of the board to which the second edge line belongs, constructing a fifth point-plane constraint equation and a sixth point-plane constraint equation using the third intersection point coordinates and the plane normal vectors of the main surfaces of the two boards respectively; Calculating the external parameters based on the first point-plane constraint equation, the second point-plane constraint equation, the third point-plane constraint equation, the fourth point-plane constraint equation, the fifth point-plane constraint equation, and the sixth point-plane constraint equation.

14. An electronic device, characterized in that, The electronic device includes a processor and a memory. The memory is used to store a computer program, and the processor is used to execute the computer program to implement the method according to any one of claims 1-13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program and / or instructions, and the program and / or instructions are used to be executed to implement the method according to any one of claims 1-13.

Images