Calibration method for external parameters of camera and laser radar and related device

Abstract

The invention discloses an external parameter calibration method for a camera and a laser radar and a related device, the camera corresponds to a camera calibration board, the laser radar corresponds to a laser calibration board, and the method comprises the following steps: obtaining coordinates of an original point of the laser calibration board in a camera calibration board coordinate system; wherein the laser calibration plate and the camera calibration plate are fixed on the reference surface on the same side in the two intersected reference surfaces, and the two intersected reference surfaces correspond to an intersecting line; camera feature points are obtained based on the coordinates of the original point of the laser calibration plate in the camera calibration plate coordinate system and the camera calibration plate and the camera coordinate system corresponding to the camera; obtaining laser feature points based on the intersection line and the first point cloud data corresponding to the laser calibration plate; and obtaining relative pose parameters of the camera and the laser radar based on the camera feature points and the laser feature points. According to the scheme, the efficiency and precision of external parameter calibration can be improved.

Description

Technical Field

[0001] This application relates to the technical field of sensor calibration, and in particular, to an external parameter calibration method for a camera and a lidar, and related devices. Background Art

[0002] In mobile robot applications, it is usually necessary to use a camera and a lidar jointly to detect obstacles. However, the coordinate systems of the camera and the lidar are separate. Therefore, it is necessary to calibrate the external parameters of the camera and the lidar, that is, to calibrate the translation parameters of the x-axis, y-axis, and z-axis, as well as the rotation parameters of the roll angle, pitch angle, and yaw angle, so that the detection data of the camera and the lidar can be converted to the same coordinate system. However, the existing lidar-camera calibration methods currently have cumbersome scene arrangements and rely on manual annotation, and the calibration efficiency and accuracy are not high. In view of this, how to improve the efficiency and accuracy of external parameter calibration has become an urgent problem to be solved. Summary of the Invention

[0003] The main technical problem to be solved by this application is to provide an external parameter calibration method for a camera and a lidar, and related devices, which can improve the efficiency and accuracy of external parameter calibration.

[0004] To solve the above technical problem, in the first aspect of this application, an external parameter calibration method for a camera and a lidar is provided. The camera corresponds to a camera calibration board, and the lidar corresponds to a lidar calibration board. The method includes: obtaining the coordinates of the origin of the lidar calibration board in the camera calibration board coordinate system; wherein, the lidar calibration board and the camera calibration board are fixed on the reference surface on the same side of two intersecting reference surfaces, and the two intersecting reference surfaces correspond to an intersection line; based on the coordinates of the origin of the lidar calibration board in the camera calibration board coordinate system, the camera calibration board, and the camera coordinate system corresponding to the camera, obtaining camera feature points; based on the intersection line and the first point cloud data corresponding to the lidar calibration board, obtaining lidar feature points; based on the camera feature points and the lidar feature points, obtaining the relative pose parameters of the camera and the lidar.

[0005] To solve the above technical problem, in the second aspect of this application, an electronic device is provided, including a memory and a processor coupled to each other. Program instructions are stored in the memory, and the processor is configured to execute the program instructions to implement the method described in the first aspect above.

[0006] To solve the above technical problem, in the third aspect of this application, a computer-readable storage medium is provided, storing program instructions that can be run by a processor, and the program instructions are used to implement the method described in the first aspect above.

[0007] In the above solution, the laser calibration board and the camera calibration board are fixed on the reference surfaces on the same side of two intersecting reference planes, and the two intersecting reference planes correspond to an intersection line. The coordinates of the origin of the laser calibration board in the camera calibration board coordinate system corresponding to the camera calibration board are obtained. Based on the obtained coordinates of the origin of the laser calibration board in the camera calibration board coordinate system, the camera calibration board and the camera coordinate system corresponding to the camera, camera feature points are obtained. Among them, the camera feature points are represented as the coordinates of the origin of the laser calibration board in the camera coordinate system. Based on the intersection line of the two intersecting reference planes and the first point cloud data corresponding to the laser calibration board, laser feature points are obtained. Among them, the laser feature points are represented as the coordinates of the origin of the laser calibration board in the laser coordinate system. Based on the corresponding camera feature points and laser feature points, the relative pose parameters between the camera and the lidar are obtained. By adding the constraint of the intersection line of the reference plane in the process of extracting the laser feature points, the accuracy of the laser feature point extraction is improved, and the scene layout is simple. Only ordinary reference planes and calibration boards are required to implement the calibration, thereby improving the efficiency and accuracy of the external parameter calibration. Description of the Drawings

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. Among them:

[0009] Figure 1 It is a schematic flowchart of an implementation manner of the external parameter calibration method for the camera and the lidar of the present application;

[0010] Figure 2 It is a schematic diagram of the vehicle body model of the camera and the lidar of the present application;

[0011] Figure 3 It is a schematic diagram of the calibration scene of the camera and the lidar of the present application;

[0012] Figure 4 It is a schematic flowchart of another implementation manner of the external parameter calibration method for the camera and the lidar of the present application;

[0013] Figure 5 It is a schematic diagram of the lidar observing the laser calibration board at a certain pitch angle of the present application;

[0014] Figure 6 It is a schematic structural diagram of an implementation manner of the electronic device of the present application;

[0015] Figure 7 It is a schematic structural diagram of an implementation manner of the computer-readable storage medium of the present application. Detailed Implementation Manner

[0016] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments, and adaptive combinations can be made between different embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0017] The terms "system" and "network" are often used interchangeably herein. The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after. In addition, "multiple" in this article means two or more than two.

[0018] Please refer to Figures 1 - 3 , Figure 1 which is a schematic flowchart of an implementation manner of the external parameter calibration method for the camera and lidar of the present application. Figure 2 which is a schematic diagram of the vehicle body model of the camera and lidar of the present application. Figure 3 which is a schematic diagram of the calibration scene of the camera and lidar of the present application. Among them, the camera corresponds to a camera calibration board, and the lidar corresponds to a laser calibration board. The method includes:

[0019] S101: Obtain the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system.

[0020] Specifically, the laser calibration board and the camera calibration board are fixed on the reference surface on the same side of the two intersecting reference surfaces, and the two intersecting reference surfaces correspond to an intersection line, and the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system corresponding to the camera calibration board are obtained.

[0021] It should be noted that the black area is the laser calibration board, the gray area partially overlapping with the black area is the reflective strip, and the other gray area is the camera calibration board. When collecting data, the lidar observes the laser calibration board, and the camera observes the camera calibration board.

[0022] Optionally, the camera calibration board can be a checkerboard or any other target for camera calibration such as an apriltag QR code. The present application does not make any limitations in this regard.

[0023] Optionally, the reference surface can be an L-shaped wall surface or other intersecting planes. The present application does not make any limitations in this regard.

[0024] In one application mode, the laser calibration board and the camera calibration board are pasted at fixed positions on the same side reference surface by pre-drawing lines, so as to obtain the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system.

[0025] In another application mode, after the laser calibration board and the camera calibration board are pasted at different positions on the same side reference surface, the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system are obtained by measurement.

[0026] S102: Based on the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system, the camera calibration board, and the camera coordinate system corresponding to the camera, camera feature points are obtained.

[0027] Specifically, based on the obtained coordinates of the origin of the laser calibration board in the camera calibration board coordinate system, the camera calibration board, and the camera coordinate system corresponding to the camera, camera feature points are obtained, where the camera feature points are represented as the coordinates of the origin of the laser calibration board in the camera coordinate system.

[0028] In one application mode, after using the camera to collect an image of the camera calibration board, the pose of the feature points in the camera calibration board is recognized to calculate the pose transformation matrix from the camera coordinate system to the camera calibration board coordinate system. Then, based on the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system and the pose transformation matrix, the coordinates of the origin of the laser calibration board in the camera coordinate system are obtained and used as camera feature points.

[0029] In another application mode, after using the camera to collect an image of the camera calibration board, the collected image of the camera calibration board is de-distorted according to the pre-calibrated internal parameters and distortion coefficients of the camera. Then, the pose of the feature points in the camera calibration board is recognized to calculate the pose transformation matrix from the camera coordinate system to the camera calibration board coordinate system. Next, based on the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system and the pose transformation matrix, the coordinates of the origin of the laser calibration board in the camera coordinate system are obtained and used as camera feature points.

[0030] In some application scenarios, the feature points in the camera calibration board can be apriltags. An apriltag is a common type of two-dimensional code and is widely used in the industrial field, commonly found in object detection, object localization, and object recognition.

[0031] In some application scenarios, the feature points in the camera calibration board can also be the corner points of a checkerboard.

[0032] It should be noted that the internal parameter calibration of the camera and the pose recognition of the feature points are prior arts and will not be elaborated here.

[0033] S103: Based on the intersection line and the first point cloud data corresponding to the laser calibration board, laser feature points are obtained.

[0034] Specifically, based on the intersection line of two intersecting reference planes and the first point cloud data corresponding to the laser calibration board, laser feature points are obtained, where the laser feature points are represented as the coordinates of the origin of the laser calibration board in the laser coordinate system.

[0035] In one application mode, the first point cloud data corresponding to the laser calibration board is fitted by intersection line constraints, and the laser feature points are obtained by solving through nonlinear optimization.

[0036] In one application scenario, the edge points and vertex points in the first point cloud data corresponding to the laser calibration board are obtained, where the edge points are the points on the four sides of the laser calibration board. The edge points and vertex points are fitted by intersection line constraints, and the laser feature points are obtained by solving through nonlinear optimization.

[0037] In another application mode, after the first point cloud data corresponding to the laser calibration board is obtained, the first point cloud data is filtered, the filtered first point cloud data is fitted by intersection line constraints, and the laser feature points are obtained by optimizing and solving through a trained neural network model.

[0038] S104: Based on the camera feature points and the laser feature points, the relative pose parameters between the camera and the lidar are obtained.

[0039] Specifically, based on the corresponding camera feature points and laser feature points, the relative pose parameters between the camera and the lidar are obtained.

[0040] It can be understood that since the camera feature points are represented as the coordinates of the origin of the laser calibration board in the camera coordinate system, and the laser feature points are represented as the coordinates of the origin of the laser calibration board in the laser coordinate system, therefore, the pose transformation parameters between the laser coordinate system and the camera coordinate system can be obtained through multiple groups of corresponding camera feature points and laser feature points, and thus the relative pose parameters between the camera and the lidar can be obtained by using the obtained pose transformation parameters.

[0041] In the above solution, the laser calibration board and the camera calibration board are fixed on the reference surface on the same side of two intersecting reference surfaces, and the two intersecting reference surfaces correspond to an intersection line. Obtain the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system corresponding to the camera calibration board. Based on the obtained coordinates of the origin of the laser calibration board in the camera calibration board coordinate system and the camera calibration board and the camera coordinate system corresponding to the camera, obtain camera feature points. Among them, the camera feature points are represented as the coordinates of the origin of the laser calibration board in the camera coordinate system. Based on the intersection line of the two intersecting reference surfaces and the first point cloud data corresponding to the laser calibration board, obtain laser feature points. Among them, the laser feature points are represented as the coordinates of the origin of the laser calibration board in the laser coordinate system. Based on the corresponding camera feature points and laser feature points, obtain the relative pose parameters between the camera and the lidar. By adding the constraint of the intersection line of the reference surface in the process of extracting the laser feature points, the accuracy of the laser feature point extraction is improved, and the scene layout is simple. Only ordinary reference surfaces and calibration boards are required to implement the calibration, thereby improving the efficiency and accuracy of the external parameter calibration.

[0042] In one embodiment, please refer to Figure 4 , Figure 4 which is a schematic flowchart of another embodiment of the method for calibrating the external parameters of the camera and the lidar in this application. The method includes:

[0043] S401: Obtain the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system.

[0044] Specifically, obtain the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system corresponding to the camera calibration board.

[0045] In an application scenario, the laser calibration board and the camera calibration board are pasted on the same side wall of the L-shaped wall. By drawing lines on the wall in advance, ensure that the y-axis of the laser calibration board coordinate system O BL is parallel to the y-axis of the camera calibration board coordinate system O BC , so that the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system can be calculated:

[0046]

[0047] S402: Use the camera to collect the reference image of the camera calibration board.

[0048] Specifically, use the camera to photograph the camera calibration board to obtain the reference image.

[0049] S403: Based on the reference image, obtain the pose transformation matrix from the camera coordinate system corresponding to the camera to the camera calibration board coordinate system.

[0050] Specifically, perform a distortion removal operation on the acquired reference image according to the pre-calibrated internal parameters and distortion coefficients of the camera, and then calculate the pose of the feature points on the camera calibration board in the reference image to obtain the camera coordinate system O C to the pose transformation matrix of the camera calibration board coordinate system O BC

[0051] S404: Based on the pose transformation matrix and the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system, obtain the coordinates of the origin of the laser calibration board in the corresponding camera coordinate system of the camera, and use the coordinates of the origin of the laser calibration board in the corresponding camera coordinate system of the camera as the camera feature point.

[0052] Specifically, based on the pose transformation matrix and the coordinates of the origin of the laser calibration board in the camera calibration board coordinate system, the coordinates of the origin of the laser calibration board in the camera coordinate system are obtained as:

[0053]

[0054] where p C is the coordinate of the origin p of the laser calibration board in the camera coordinate system O C under, p BC is the coordinate of the origin p of the laser calibration board in the camera calibration board coordinate system O BC under, is the rotation matrix component of the pose transformation from the camera coordinate system to the camera calibration board coordinate system, is the translation vector component of the pose transformation from the camera coordinate system to the camera calibration board coordinate system, and use p C as the camera feature point, that is, it is equivalent to the camera directly observing the origin of the laser calibration board, and the camera feature point is calculated through the camera with pre-calibrated internal parameters and distortion coefficients to improve the accuracy of subsequent external parameter calibration using the camera feature point.

[0055] S405: Based on the intersection line and the first point cloud data corresponding to the laser calibration board, obtain the laser feature point.

[0056] Specifically, based on the intersection line of the two intersecting reference planes and the first point cloud data corresponding to the laser calibration board, obtain the laser feature point.

[0057] In one embodiment, before step S405, it includes: obtaining the second point cloud data corresponding to the two intersecting reference planes, and based on the second point cloud data, obtaining the first plane equation and the second plane equation; based on the first plane equation and the second plane equation, obtaining the intersection line equation corresponding to the intersection line.

[0058] ​Specifically, the second point cloud data corresponding to the two intersecting reference surfaces are obtained, and the first plane equation and the second plane equation corresponding to the two reference surfaces in the second point cloud data are simultaneously extracted using the Ransac method:

[0059] π 1 :A 1 X+B 1 Y+C 1 Z+D 1 =0(3)

[0060] π 2 :A 2 X+B 2 Y+C 2 Z+D 2 =0(4)

[0061] In one implementation scenario, the steps of obtaining the intersection line equation corresponding to the intersection line based on the first plane equation and the second plane equation specifically include: obtaining a reference point on the intersection line based on the first plane equation and the second plane equation; obtaining the first unit normal vector of the first plane equation, and obtaining the second unit normal vector of the second plane equation; obtaining the direction vector of the intersection line based on the first unit normal vector and the second unit normal vector; obtaining the intersection line equation based on the reference point and the direction vector.

[0062] Specifically, firstly transform the first plane equation π 1 and the second plane equation π 2 Combine and set z = 0 to get the reference point p on the intersection line, and then get the first plane equation π 1 The first unit normal vector n 1 And get the second plane equation π 2 The second unit normal vector n 2 , then according to the first unit normal vector n 1 and the second unit normal vector n 2 , the direction vector of the intersection line is: l = n 1 ×n 2 Finally, based on the reference point p and the direction vector l, the intersection line equation is obtained: L = (l, p). The laser feature points are constrained and fitted through the intersection line to further improve the accuracy of laser feature point extraction.

[0063] Further, step S405 specifically includes: obtaining a laser point cloud obtained when the laser radar observes the laser calibration plate and filtering the laser point cloud to obtain first point cloud data; and obtaining laser feature points based on the intersection line equation and the first point cloud data.

[0064] Specifically, when the lidar observes the laser calibration board, it returns a laser point cloud. By filtering the laser point cloud, the first point cloud data corresponding to the laser calibration board is obtained. Based on the intersection line equation L and the first point cloud data, laser feature points are obtained. By filtering the point cloud data, noise can be removed, the quality of the first point cloud data can be improved, and the accuracy of laser feature point extraction can be increased, thereby improving the accuracy of subsequent extrinsic parameter calibration using the laser feature points.

[0065] In an implementation scenario, please refer to Figure 5 , Figure 5 which is a schematic diagram of the lidar of the present application observing the laser calibration board at a certain pitch angle. Since there are differences in the laser point clouds returned when the laser beam hits the reflective strip (gray area) and the laser calibration board (black area), it is necessary to filter the laser point cloud returned when the lidar observes the laser calibration board to obtain the first point cloud data corresponding to the laser calibration board.

[0066] In an application scenario, the steps of obtaining the laser point cloud obtained when the lidar observes the laser calibration board and filtering the laser point cloud to obtain the first point cloud data specifically include: obtaining a preset initial intensity threshold and a convergence threshold; based on the initial intensity threshold and the laser point cloud, obtaining a first point cloud set and a second point cloud set; based on the first point cloud set and the second point cloud set, updating the initial intensity threshold and obtaining the update change amount of the initial intensity threshold; in response to the update change amount being less than the convergence threshold, obtaining the reflection intensity threshold; based on the reflection intensity threshold, filtering the laser point cloud to obtain the first point cloud data.

[0067] Specifically, there are significant differences in the reflection intensities of the laser point clouds returned when the laser beam of the lidar hits the reflective strip (gray area) and the laser calibration board (black area). Therefore, by setting a reasonable reflection intensity threshold, the point cloud in the reflective strip area can be filtered out, and the first point cloud data corresponding to the laser calibration board can be extracted. First, obtain the preset initial intensity threshold and the convergence threshold. Divide the laser point cloud by the set initial intensity threshold to obtain two point cloud sets, namely the first point cloud set and the second point cloud set. Then, update the initial intensity threshold according to the divided first point cloud set and second point cloud set, and at the same time obtain the update change amount of the initial intensity threshold. When the update change amount is less than the convergence threshold, obtain the reflection intensity threshold. Finally, filter the laser point cloud according to the obtained reflection intensity threshold to obtain the first point cloud data corresponding to the laser calibration board. Through iterative calculation, the reflection intensity threshold can be dynamically adjusted to more accurately remove noise and outliers.

[0068] In an application scenario, the steps of updating the initial intensity threshold based on the first point cloud set and the second point cloud set and obtaining the update change amount of the initial intensity threshold specifically include: obtaining the first average reflection intensity based on the first point cloud set, and obtaining the second average reflection intensity based on the second point cloud set; updating the initial intensity threshold based on the first average reflection intensity and the second average reflection intensity and obtaining the update change amount of the initial intensity threshold.

[0069] Specifically, calculate the average reflection intensity of the two sets of point cloud sets respectively to obtain the first average reflection intensity and the second average reflection intensity, and update the initial reflection intensity threshold based on the first average reflection intensity and the second average reflection intensity and obtain the update change amount of the initial reflection intensity threshold.

[0070] In a specific application scenario, the preset initial intensity threshold T 0 is the median of the reflection intensity of the entire point cloud. Through T 0 perform point cloud segmentation, and the two sets of point cloud sets obtained are:

[0071] G 1 ={p∈cloud|p.intensity<T 0}(5)

[0072] G 2 ={p∈cloud|p.intensity>T 0}(6)

[0073] Furthermore, calculate the average reflection intensity of the two sets of point cloud sets respectively to obtain the first average reflection intensity u 1 and the second average reflection intensity u 2 , and update the initial reflection intensity threshold to:

[0074]

[0075] Repeat the above steps repeatedly until the update change amount of the initial reflection intensity threshold is less than the convergence threshold ΔT to obtain the final reflection intensity threshold. Filter out the laser point cloud in the reflective strip area through this reflection intensity threshold to obtain the first point cloud data corresponding to the laser calibration plate. The process of iteratively calculating the reflection intensity threshold can be automated, reducing the need for manual intervention, thereby not only improving the processing efficiency but also reducing the possibility of human errors.

[0076] In an implementation scenario, the steps of obtaining laser feature points based on the intersection line equation and the first point cloud data specifically include: obtaining the first fitting distance from the edge points of the laser calibration board in the first point cloud data to the corresponding edge on the laser calibration board; obtaining the second fitting distance from the vertex points of the laser calibration board in the first point cloud data to the intersection line, and the actual distance from the vertex points of the laser calibration board to the intersection line; based on the first fitting distance, the second fitting distance, and the actual distance, obtaining the coordinates of the origin of the laser calibration board in the laser coordinate system corresponding to the lidar, and using the coordinates of the origin of the laser calibration board in the laser coordinate system corresponding to the lidar as the laser feature points.

[0077] Specifically, please refer to Figure 5 , when a circle of laser beams scanned by the lidar at a certain height intersects with the laser calibration board, it will intersect with the internal laser calibration board and the external reflective strips at a total of 4 points. Take the middle two points as the two edge points of the laser calibration board. After classifying all edge points to the corresponding edges using the geometric features of the laser calibration board, the 4 vertex points and 4 edges of the laser calibration board are obtained by least-squares fitting. Obtain the first fitting distance from each edge point of the laser calibration board to the corresponding edge on the laser calibration board, obtain the second fitting distance from the vertex points of the laser calibration board to the intersection line and the actual distance from the vertex points of the laser calibration board to the intersection line, and based on the first fitting distance, the second fitting distance, and the actual distance, obtain the coordinates of the origin of the laser calibration board in the laser coordinate system corresponding to the lidar, and use it as the laser feature point. Through the fitting constraint of the reference plane intersection line on the laser feature point, the extraction accuracy of the laser feature point is improved, and there is no need for manual intervention, no need to customize complex calibration tooling, and there can be completely no co-visible area between the lidar and the camera, thereby improving the efficiency and accuracy of the external parameter calibration.

[0078] In an application scenario, the steps of obtaining the coordinates of the origin of the laser calibration board in the laser coordinate system corresponding to the lidar based on the first fitting distance, the second fitting distance, and the actual distance, and using the coordinates of the origin of the laser calibration board in the laser coordinate system corresponding to the lidar as the laser feature points specifically include: determining the first distance residual based on multiple first fitting distances; determining the second distance residual based on multiple second fitting distances and the corresponding actual distances; based on the first distance residual and the second distance residual, obtaining the coordinates of the origin of the laser calibration board in the laser coordinate system corresponding to the lidar, and using the coordinates of the origin of the laser calibration board in the laser coordinate system corresponding to the lidar as the laser feature points.

[0079] Specifically, based on the first fitting distance from each edge point to the edge on the laser calibration board corresponding to this edge point, determine the first distance residual from each edge point to the edge on the laser calibration board corresponding to this edge point. Based on the second fitting distance from each vertex of the laser calibration board to the intersection line and the corresponding actual distance, determine the second distance residual from each vertex of the laser calibration board to the intersection line. Based on the first distance residual and the second distance residual, obtain the coordinates of the origin of the laser calibration board in the laser coordinate system corresponding to the lidar, and use it as the laser feature point.

[0080] In a specific application scenario, use nonlinear optimization to solve for the coordinates of the origin of the laser calibration board in the laser coordinate system corresponding to the lidar. The objective function of the optimization problem is expressed as:

[0081]

[0082] where \(i = 0, 1, 2, 3\), \(v\) 0 is the origin of the laser calibration board ( Figure 5 the topmost vertex of the laser calibration board in 0 ) is \(v\) 0 , \(v\) 1 , \(v\) 2 , \(v\) 3 are the vertices of the laser calibration board (in counterclockwise order), \(E\) i is the edge \(v\) i \(v\) i+1 (it is stipulated that \(v\) 4 = \(v\) 0 ), \(L\) i is the parametric equation of the fitting line corresponding to the edge \(E\) i (which can be written as a function of \(v\) 0 , \(L\) 0 ), \(distance(p, L\) i (\(v\) 0 , \(L\) 0 )) is the first fitting distance from the edge point \(p\) to the edge \(E\) i corresponding to this edge point \(p\), \(L\) is the equation of the intersection line, \(distance(v\) i , \(L)\) is the second fitting distance from the vertex \(v\) i to the intersection line \(L\), \(d\) i is the actual distance from the vertex \(v\) i to the intersection line \(L\), \(\lambda\) is the regularization parameter, is the first distance residual from the edge point \(p\) to the edge \(E\) i corresponding to this edge point \(p\), \(\lambda\sum\) i \(\vert\vert d\) i - \(distance(v\) i , \(L)\vert\vert\) is the second distance residual from the vertex \(v\) iThe second distance residual to the intersection line L. Thus, the laser feature points are obtained as follows:

[0083]

[0084] Wherein, is the solution to the above optimization problem, and p L is the origin v of the laser calibration plate 0 in the laser coordinate system O L under the coordinates.

[0085] S406: Based on the camera feature points and the laser feature points, obtain the relative pose parameters of the camera and the lidar.

[0086] Specifically, based on multiple groups of corresponding camera feature points and laser feature points, obtain the relative pose parameters of the camera and the lidar.

[0087] In an application scenario, using the 3D-3D ICP algorithm, the relative pose from the camera to the lidar is fitted by the bundle adjustment method. The initial value of the optimization variable is set to the coupling of the external parameters of the respective structures of the lidar and the camera. The objective function of the optimization problem is expressed as:

[0088]

[0089] Wherein, i is the subscript corresponding to different groups of data, and p i L is the laser feature point coordinate corresponding to the i-th group of data, and p i C is the camera feature point coordinate corresponding to the i-th group of data, is the rotation matrix component of the pose transformation from the laser coordinate system O L to the camera coordinate system O C component, is the translation vector component of the pose transformation from the laser coordinate system O L to the camera coordinate system O C component. Thus, there is:

[0090]

[0091]

[0092]

[0093]

[0094] Among them, x is the x - component in the relative pose parameters between the camera and the lidar, y is the y - component in the relative pose parameters between the camera and the lidar, z is the z - component in the relative pose parameters between the camera and the lidar, roll is the roll angle in the relative pose parameters between the camera and the lidar, pitch is the pitch angle in the relative pose parameters between the camera and the lidar, yaw is the yaw angle in the relative pose parameters between the camera and the lidar, and (12), (13), (14) are the formulas for inverse - solving Euler angles from the rotation matrix. Since a set of points provides 3 constraint relations and there are a total of 6 degrees of freedom, at least two sets of data of corresponding camera feature points and lidar feature points are required to solve all the variables x, y, z, roll, pitch, yaw in the relative pose parameters between the camera and the lidar.

[0095] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of an embodiment of the electronic device of the present application. The electronic device 60 includes a mutually - coupled memory 600 and a processor 602. Among them, the memory 600 stores program data (not shown in the figure), and the processor 602 calls the program data to implement the method in any of the above - mentioned embodiments. For the description of related content, please refer to the detailed description of the above - mentioned method embodiments and will not be repeated here. Specifically, the electronic device 60 includes, but is not limited to: desktop computers, laptop computers, tablet computers, servers, etc., which are not limited here. In addition, the processor 602 can also be called a CPU (Center Processing Unit, central processing unit). The processor 602 may be an integrated circuit chip with signal - processing capabilities. The processor 602 can also be a general - purpose processor, a digital signal processor (DSP), an application - specific integrated circuit (ASIC), a field - programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general - purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. Additionally, the processor 602 can be implemented jointly by integrated circuit chips.

[0096] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an embodiment of the computer - readable storage medium of the present application. The computer - readable storage medium 70 stores program data 700, and when the program data 700 is executed by the processor, it implements the method in any of the above - mentioned embodiments. For the description of related content, please refer to the detailed description of the above - mentioned method embodiments and will not be repeated here.

[0097] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, in each embodiment of the present application, the functional units can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0100] The above is only the embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A method for calibrating external parameters of a camera and a laser radar, characterized in that: The camera has a camera calibration plate corresponding to it, and the laser radar has a laser calibration plate corresponding to it. The method includes: Obtaining the coordinates of the origin of the laser calibration plate in the camera calibration plate coordinate system; wherein the laser calibration plate and the camera calibration plate are fixed on a reference surface on the same side of two intersecting reference surfaces, and the two intersecting reference surfaces have an intersection line; Obtaining camera feature points based on the coordinates of the origin of the laser calibration plate in the camera calibration plate coordinate system and the camera coordinate system corresponding to the camera calibration plate and the camera; Obtaining laser feature points based on the intersection line and the first point cloud data corresponding to the laser calibration plate; Based on the camera feature points and the laser feature points, relative posture parameters of the camera and the laser radar are obtained.

2. The method according to claim 1, characterized in that The obtaining of the camera feature points based on the coordinates of the origin of the laser calibration plate in the camera calibration plate coordinate system and the camera coordinate system corresponding to the camera calibration plate and the camera comprises: Using the camera to collect a reference image of the camera calibration plate; Based on the reference image, a pose transformation matrix from a camera coordinate system corresponding to the camera to a coordinate system of the camera calibration plate is obtained; Based on the pose transformation matrix and the coordinates of the origin of the laser calibration plate in the camera calibration plate coordinate system, the coordinates of the origin of the laser calibration plate in the camera coordinate system corresponding to the camera are obtained, and the coordinates of the origin of the laser calibration plate in the camera coordinate system corresponding to the camera are used as the camera feature points.

3. The method according to claim 1, characterized in that Before obtaining the laser feature point based on the intersection line and the first point cloud data corresponding to the laser calibration plate, the method includes: Acquire second point cloud data corresponding to the two intersecting reference surfaces, and obtain a first plane equation and a second plane equation based on the second point cloud data; Based on the first plane equation and the second plane equation, obtaining an intersection line equation corresponding to the intersection line; The step of obtaining laser feature points based on the intersection line and the first point cloud data corresponding to the laser calibration plate includes: Acquire a laser point cloud obtained when the laser radar observes the laser calibration plate and filter the laser point cloud to obtain the first point cloud data; The laser feature point is obtained based on the intersection line equation and the first point cloud data.

4. The method according to claim 3, characterized in that: The step of obtaining a laser point cloud obtained when the laser radar observes the laser calibration plate and filtering the laser point cloud to obtain the first point cloud data includes: Obtaining the preset intensity threshold initial value and convergence threshold; Based on the intensity threshold initial value and the laser point cloud, obtaining a first point cloud set and a second point cloud set; Based on the first point cloud set and the second point cloud set, updating the initial value of the intensity threshold and obtaining an updated change amount of the initial value of the intensity threshold; In response to the update change amount being less than the convergence threshold, obtaining a reflection intensity threshold; The laser point cloud is filtered based on the reflection intensity threshold to obtain the first point cloud data.

5. The method according to claim 4, characterized in that The updating of the initial value of the intensity threshold and obtaining an updated change amount of the initial value of the intensity threshold based on the first point cloud set and the second point cloud set includes: Based on the first point cloud set, obtain a first average reflection intensity, and based on the second point cloud set, obtain a second average reflection intensity; Based on the first average reflection intensity and the second average reflection intensity, the initial value of the intensity threshold is updated and an updated change amount of the initial value of the intensity threshold is obtained.

6. The method according to claim 3, characterized in that The obtaining, based on the first plane equation and the second plane equation, an intersection line equation corresponding to the intersection line comprises: Based on the first plane equation and the second plane equation, obtaining a reference point on the intersection line; Obtaining a first unit normal vector of the first plane equation, and obtaining a second unit normal vector of the second plane equation; Obtaining a direction vector of the intersection line based on the first unit normal vector and the second unit normal vector; The intersection line equation is obtained based on the reference point and the direction vector.

7. The method according to claim 3, characterized in that The obtaining of the laser feature point based on the intersection line equation and the first point cloud data comprises: Acquire a first fitting distance from an edge point of the laser calibration plate in the first point cloud data to a side of the laser calibration plate corresponding to the edge point; Acquire a second fitting distance from the vertex of the laser marking plate to the intersection line in the first point cloud data, and an actual distance from the vertex of the laser marking plate to the intersection line; Based on the first fitting distance, the second fitting distance and the actual distance, the coordinates of the origin of the laser calibration plate in the laser coordinate system corresponding to the laser radar are obtained, and the coordinates of the origin of the laser calibration plate in the laser coordinate system corresponding to the laser radar are used as the laser feature point.

8. The method according to claim 7, characterized in that The method of obtaining the coordinates of the origin of the laser calibration plate in the laser coordinate system corresponding to the laser radar based on the first fitting distance, the second fitting distance and the actual distance, and taking the coordinates of the origin of the laser calibration plate in the laser coordinate system corresponding to the laser radar as the laser feature point includes: determining a first distance residual based on a plurality of the first fitting distances; determining a second distance residual based on a plurality of the second fitted distances and the corresponding actual distances; Based on the first distance residual and the second distance residual, obtaining the coordinates of the origin of the laser calibration plate in the laser coordinate system corresponding to the laser radar; The coordinates of the origin of the laser calibration plate in the laser coordinate system corresponding to the laser radar are used as the laser feature points.

9. An electronic device, characterized in that: The method comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the method described in any one of claims 1 to 8.

Images