A method for calibrating extrinsic parameters between a camera and a lidar based on the vanishing point method.

Abstract

This invention discloses a method for extrinsic parameter calibration between a camera and a lidar based on the vanishing point method, comprising the following steps: S1, acquiring an optical image of a target reference object using a camera, and acquiring laser point cloud data of the target reference object using a lidar; S2, determining the principal point and focal length of the camera based on the optical image; S3, processing the laser point cloud data based on the determined principal point and focal length of the camera to obtain a lidar projection image; S4, calibrating the extrinsic parameters between the camera and the lidar using the vanishing points of the optical image and the lidar projection image, and obtaining the extrinsic parameter calibration results; S5, further optimizing the camera's principal point, focal length, and extrinsic parameter calibration results using an iterative optimization method to obtain optimized extrinsic parameter calibration results. This invention utilizes vanishing points to solve the extrinsic parameter calibration problem between a camera and a lidar in scenarios without a calibration board, improving the efficiency and accuracy of extrinsic parameter calibration between lidar and camera.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method for calibrating extrinsic parameters between a camera and a lidar based on the vanishing point method. Background Technology

[0002] In sensor data fusion algorithms, cameras and LiDAR are often combined. Cameras possess rich environmental information, including color, texture, and semantic information; LiDAR provides accurate 3D ranging information and has been widely used in surveying and obstacle detection industries. The fusion of their data not only yields stable and accurate navigation and positioning results but also possesses powerful environmental information perception capabilities, making it suitable for autonomous driving. Currently, with the advent of high-resolution LiDAR, its data resolution, approaching or even exceeding that of images, makes data-level fusion algorithms between the two sensors possible. However, achieving this requires high-precision extrinsic parameter calibration results between the camera and LiDAR.

[0003] Therefore, how to provide a widely applicable, stable, reliable, and highly accurate method for calibrating the extrinsic parameters between cameras and lidar is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a method for extrinsic parameter calibration between a camera and a lidar based on the vanishing point method, which can effectively improve the accuracy of the extrinsic parameter calibration results between the camera and the lidar.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for calibrating extrinsic parameters between a camera and a lidar based on the vanishing point method includes the following steps:

[0007] S1. Use a camera to acquire optical images of the target reference object, and use a lidar to acquire laser point cloud data of the target reference object;

[0008] S2. Based on the optical image, determine the principal point and focal length of the camera;

[0009] S3. Based on the solved principal point and focal length of the camera, process the laser point cloud data to obtain the laser radar projection image;

[0010] S4. Using the vanishing points of the optical image and the laser radar projection image, calibrate the extrinsic parameters between the camera and the laser radar, and obtain the extrinsic parameter calibration results.

[0011] Furthermore, step S4 further includes:

[0012] S5. Further optimize the principal point, focal length, and extrinsic parameter calibration results of the camera using an iterative optimization method to obtain optimized extrinsic parameter calibration results.

[0013] Furthermore, step S2 specifically includes:

[0014] S21. Extract feature lines from the optical image, select the required feature lines and calculate their corresponding vanishing points;

[0015] S22. Use the calculated orthogonal vanishing points to solve for the principal point and focal length of the camera.

[0016] Furthermore, the method for solving the principal point and focal length of the camera in step S22 is as follows:

[0017] The principal point of the camera is represented by p. i =[m0 n0 1] T It satisfies the following conditions:

[0018]

[0019] In the formula, p i m0 and m1 are the coordinate parameters of the camera principal point; U and C are the matrix of parameters related to the principal point. C = [-c1 -c2 … -c n ] T , where a j b j With c j Here are the relevant parameters for the j-th main line, and the mathematical expression for the j-th main line is a. j m0+b j n0+c j =0,j=1,2,…,n;

[0020] Specifically:

[0021]

[0022]

[0023]

[0024] In the formula, and To find the coordinates of the two sets of orthogonal vanishing points corresponding to the j-th principal line, where one set of orthogonal vanishing points is... and Another set of orthogonal vanishing points is and

[0025] The focal length of the camera is expressed as f = [fx f y It satisfies the following conditions:

[0026]

[0027] In the formula, f x f y This refers to the camera's focal length parameter.

[0028] Furthermore, step S3 specifically includes:

[0029] S31. Based on the solved principal point and focal length of the camera, the laser point cloud data is projected in two dimensions using a projective geometric pinhole model.

[0030] S32. Redundant points are removed from the projected two-dimensional image points using a two-dimensional rendering method in a three-dimensional scene to obtain the LiDAR projected image.

[0031] Furthermore, step S4 includes:

[0032] S41. Extract feature lines from the laser radar projection image, select a set of orthogonal parallel lines corresponding to the optical image and calculate their orthogonal vanishing points;

[0033] S42. Solve for the extrinsic parameters between the camera and the lidar based on the calculated orthogonal vanishing points;

[0034] S43. Following steps S41 and S42, solve for multiple external parameters and perform a weighted average of the multiple external parameters to obtain the external parameter calibration result after eliminating errors.

[0035] Furthermore, step S42 includes:

[0036] The rotation matrix between the camera and the lidar is:

[0037]

[0038] In the formula, c is the camera coordinate system, l is the laser coordinate system, and w is the world coordinate system; This is the rotation matrix from the camera coordinate system to the lidar coordinate system; The rotation matrix from the world coordinate system to the lidar coordinate system; Let be the rotation matrix from the camera coordinate system to the world coordinate system;

[0039] The rotation matrix from the world coordinate system to the camera coordinate system is:

[0040]

[0041] In the formula, r1, r2, and r3 are A column vector that satisfies the following condition:

[0042]

[0043]

[0044] r3 = r1 × r2

[0045] In the formula, and To find a set of orthogonal vanishing points corresponding to the principal line, where, K c The intrinsic parameter of the camera is expressed as:

[0046]

[0047] The position vector between the camera and the lidar is:

[0048]

[0049] In the formula, The position vector from the camera coordinate system to the lidar coordinate system; The position vector from the world coordinate system to the lidar coordinate system; The position vector from the camera coordinate system to the world coordinate system; and It can be obtained directly using the sine and cosine theorems.

[0050] Furthermore, step S5 includes:

[0051] S51. Update the lidar projection image according to the external parameter calibration result, and use the updated lidar projection image to solve the principal point and focal length of the camera in reverse, so as to obtain the optimized principal point and focal length of the camera.

[0052] S52. Update the LiDAR projection image again using the optimized principal point and focal length of the camera, and use the vanishing point method to solve the extrinsic parameters between the camera and the LiDAR.

[0053] S53. Repeat steps S51 and S52 to finally obtain the optimized extrinsic parameter calibration results.

[0054] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method for extrinsic parameter calibration between a camera and a lidar based on the vanishing point method. It uses projection and rendering technology to realize the data layer connection between three-dimensional laser point cloud data and two-dimensional optical images. At the same time, it uses the vanishing point method in projective geometry to realize rapid extrinsic parameter calibration between the laser and the camera. This not only solves the problem of extrinsic parameter calibration between the camera and the lidar in the absence of a calibration plate, but also effectively improves the speed and accuracy of the extrinsic parameter calibration results between the camera and the lidar. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the extrinsic parameter calibration method between a camera and a lidar based on the vanishing point method provided by the present invention.

[0057] Figure 2 This is a schematic diagram illustrating the process of extrinsic parameter calibration between a camera and a lidar provided by the present invention.

[0058] Figure 3 This is a schematic diagram of the process for optimizing the external parameter calibration results using an iterative method, as provided by the present invention.

[0059] Figure 4 This is a schematic diagram of the geometric structure for solving the position vector between the camera and the lidar, provided by the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] This invention discloses a method for calibrating extrinsic parameters between a camera and a lidar based on the vanishing point method. (See attached diagram.) Figure 1 This includes the following steps:

[0062] S1. Use a camera to acquire optical images of the target reference object, and use a lidar to acquire laser point cloud data of the target reference object.

[0063] The lidar and camera, whose extrinsic parameters need to be calibrated, are fixedly placed in the same scene containing geometric objects. These geometric objects are used as target references. The positions of the lidar and camera do not change during the process. The camera is used to acquire an optical image of the target reference. The lidar is used to acquire the laser point cloud data of the target reference for subsequent generation of laser point cloud projection images. The acquisition time is 1 to 2 minutes to ensure that the superposition of multiple frames of point cloud data has a high density and improves the quality of the laser point cloud projection image.

[0064] It is worth noting that the target reference should have rectangular features. At the same time, the camera and lidar that need to be calibrated do not need to have a large common field of view, but they need to have the same orthogonal parallel lines within the field of view to facilitate obtaining a more accurate orthogonal vanishing point.

[0065] S2. Based on the optical image, determine the principal point and focal length of the camera. The specific calculation process is as follows:

[0066] S21. Extract feature lines from the optical image, select the required feature lines and calculate their corresponding vanishing points.

[0067] First, feature lines are extracted from the optical images captured by the camera using LSD (Line Segment Detector), including the following steps:

[0068] (1) Gaussian downsampling is performed on the optical images captured by the camera;

[0069] (2) Solve the gray-level gradient of the pixels in the optical image to obtain the gradient value and gradient direction of each point, denoted as g, and take the normal vector direction of the gradient as the pixel direction h.

[0070] (3) Obtain the candidate straight line domain through the pixel direction h, and solve for the minimum bounding rectangle of the region and the main direction H of the bounding rectangle;

[0071] (4) If the smallest bounding rectangle is long and thin, it is likely that the feature line to be detected is long. Then, the difference between the direction h of each pixel point in the rectangle and the main direction H of the rectangle is judged, and the line detection is performed.

[0072] Secondly, select two sets of orthogonal parallel lines in the same plane from the feature lines extracted by the LSD method and calculate their corresponding vanishing points.

[0073] It is worth noting that when manually selecting orthogonal parallel lines, you should choose lines that are long, have small pixel errors, and have clear edges.

[0074] S22. Use the calculated orthogonal vanishing points to solve for the principal point and focal length of the camera.

[0075] Specifically, the principal point of the camera is represented as p.i =[m0 n0 1] T It satisfies the following conditions:

[0076]

[0077] In the formula, p i m0 and m1 are the coordinate parameters of the camera principal point; U and C are the matrix of parameters related to the principal point. C = [-c1-c2…-c n ] T , where a j b j With c j Here are the relevant parameters for the j-th main line, and the mathematical expression for the j-th main line is a. j m0+b j n0+c j =0,j=1,2,…,n;

[0078] Further:

[0079]

[0080]

[0081]

[0082] In the formula, and To find the coordinates of the two sets of orthogonal vanishing points corresponding to the j-th principal line, where one set of orthogonal vanishing points is... and Another set of orthogonal vanishing points is and

[0083] The focal length of a camera is expressed as f = [f x f y It satisfies the following conditions:

[0084]

[0085] In the formula, f x f y This refers to the camera's focal length parameter.

[0086] S3. Based on the solved principal point and focal length of the camera, process the laser point cloud data to obtain the lidar projection image. Specific steps include:

[0087] S31. Based on the solved principal point and focal length of the camera, perform two-dimensional projection of the laser point cloud data using a projective geometric pinhole model.

[0088] The projected two-dimensional plane points are:

[0089]

[0090] In the formula, [mn 1] T For the projected two-dimensional plane points; [X l Y l Z l 1] T R represents the three-dimensional laser point before projection. init With t init These are arbitrary initial values ​​for the extrinsic parameters, typically t init =[0 0 0].

[0091] By traversing and projecting all the collected laser point cloud data, the mapping from three-dimensional points in space to two-dimensional points on a plane can be realized.

[0092] S32. Redundant points are removed from the projected two-dimensional image points using a two-dimensional rendering method in a three-dimensional scene to obtain the LiDAR projected image.

[0093] After mapping, the Z-Buffer rendering method in a 3D scene is used to cull and filter existing 2D projected points. Specifically, in the same 2D coordinate system, points with greater depth are occluded by points with less depth; the points with greater depth are removed to reduce computational complexity. The depth and reflection intensity of the remaining 2D points are used to generate grayscale values ​​for their corresponding 2D locations. The specific expression is as follows:

[0094]

[0095] In the formula, w depth The weighting is based on depth; d is the depth of the laser point, and I is the intensity of the laser point; d max I represents the maximum depth in the laser point cloud. max The maximum intensity in the laser point cloud.

[0096] Fill the image with grayscale to obtain the lidar projection image to be further processed.

[0097] S4. Use the vanishing points of the optical image and the lidar projection image to calibrate the external parameters between the camera and the lidar.

[0098] For details, please see the appendix. Figure 2 As shown, step S4 includes:

[0099] S41. Extract feature lines from the lidar projection image, select a set of orthogonal parallel lines corresponding to the optical image, and calculate their orthogonal vanishing points.

[0100] S42. Solve for the extrinsic parameters between the camera and the lidar based on the calculated orthogonal vanishing points.

[0101] Furthermore, the rotation matrix between the camera and the lidar is as follows:

[0102]

[0103] In the formula, c is the camera coordinate system; l is the lidar coordinate system; w is the world coordinate system, which is the coordinate system formed by the line of the selected vanishing point in reality; This is the rotation matrix from the camera coordinate system to the lidar coordinate system; This is the rotation matrix from the world coordinate system to the lidar coordinate system; This is the rotation matrix from the camera coordinate system to the world coordinate system;

[0104] To find a solution The reverse For example, the rotation matrix from the world coordinate system to the camera coordinate system is:

[0105]

[0106] In the formula, r1, r2, and r3 are A column vector that satisfies the following condition:

[0107]

[0108]

[0109] r3 = r1 × r2

[0110] In the formula, and To find a set of orthogonal vanishing points corresponding to the principal line, where, K c The intrinsic parameters of the camera used for projection are expressed as follows:

[0111]

[0112] The position vector between the camera and the lidar is:

[0113]

[0114] In the formula, This is the position vector from the camera coordinate system to the lidar coordinate system; This is the position vector from the world coordinate system to the lidar coordinate system; This is the position vector from the camera coordinate system to the world coordinate system; and It can be obtained directly using the sine and cosine theorems.

[0115] To solve for the position vector from the camera coordinate system to the world coordinate system For example, see the appendix. Figure 3 The geometric relationship between the camera coordinate system and the world coordinate system is as follows:

[0116]

[0117] In the formula, O c O is the origin of the camera coordinate system. w O is the origin of the world coordinate system; o is the projection point of the world coordinate system onto the image plane; B is the selected point passing through O. w The other endpoint of the straight line.

[0118] According to the law of sines, we can obtain:

[0119]

[0120] In the formula, p i The main point of the camera; For O w The vanishing point in direction B; b is the projection point of point B onto the image plane.

[0121] According to the Law of Cosines, we can obtain:

[0122]

[0123] S43. Following steps S41 and S42, solve for multiple external parameters and perform a weighted average of the multiple external parameters to eliminate the error in the straight line extraction during the solution of a single external parameter, and obtain the external parameter calibration result after eliminating the error.

[0124] Furthermore, the weights used in the weighted average are defined by the similarity of the triangles and the slope of the vanishing line:

[0125]

[0126]

[0127]

[0128]

[0129] In the formula, w1 and w2 are the triangle similarity weights, and w3 is the slope weight; O l O is the origin of the coordinate system for the LiDAR projected image; c The origin of the coordinate system for the optical image; and These are a set of orthogonal vanishing points in the lidar coordinate system. and A set of orthogonal vanishing points in the camera coordinate system; and A straight line formed by points; The slope of the vanishing line in the lidar projection image; The slope of the vanishing line in the optical image.

[0130] S5. Further optimize the camera principal point, focal length, and extrinsic parameter calibration results using iterative optimization methods to obtain optimized extrinsic parameter calibration results.

[0131] For details, please see the appendix. Figure 4 As shown, step S5 includes:

[0132] S51. Update the LiDAR projection image based on the external parameter calibration results, and use the updated LiDAR projection image to solve for the principal point and focal length of the camera in reverse, so as to obtain the optimized principal point and focal length of the camera.

[0133] S52. Update the LiDAR projection image again using the optimized principal point and focal length of the camera, and use the vanishing point method to solve for the extrinsic parameters between the camera and the LiDAR.

[0134] S53. Repeat steps S51 and S52 to finally obtain the optimized extrinsic parameter calibration results.

[0135] By executing the above process sequentially, we can obtain high-precision extrinsic parameter calibration results between the camera and the lidar based on the vanishing point method.

[0136] It is worth noting that if the extrinsic parameters between multiple sets of cameras and lidar are to be calibrated simultaneously, it is only necessary to collect data of the same target reference object using all camera and lidar equipment groups in step S1. Similarly, all camera and lidar equipment groups that need to have their extrinsic parameters calibrated need to have the same orthogonal parallel straight lines within their field of view. In step S4, the corresponding orthogonal parallel lines are selected to solve the extrinsic parameters between all camera and lidar equipment groups simultaneously.

[0137] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0138] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for calibrating extrinsic parameters between a camera and a lidar based on vanishing point method, characterized in that, Includes the following steps: S1. Use a camera to acquire optical images of the target reference object, and use a lidar to acquire laser point cloud data of the target reference object; S2. Based on the optical image, determine the principal point and focal length of the camera; S3. Based on the solved principal point and focal length of the camera, process the laser point cloud data to obtain the laser radar projection image; S4. Using the vanishing points of the optical image and the laser radar projection image, calibrate the extrinsic parameters between the camera and the laser radar, and obtain the extrinsic parameter calibration results; Step S4 includes: S41. Extract feature lines from the laser radar projection image, select a set of orthogonal parallel lines corresponding to the optical image and calculate their orthogonal vanishing points; S42. Solve for the extrinsic parameters between the camera and the lidar based on the calculated orthogonal vanishing points; S43. Following steps S41 and S42, solve for multiple external parameters and perform a weighted average of the multiple external parameters to obtain the external parameter calibration result after eliminating errors. Step S42 includes: The rotation matrix between the camera and the lidar is: ； In the formula, c represents the camera coordinate system. For laser coordinate system, Using the world coordinate system; This is the rotation matrix from the camera coordinate system to the lidar coordinate system; The rotation matrix from the world coordinate system to the lidar coordinate system; Let be the rotation matrix from the camera coordinate system to the world coordinate system; The rotation matrix from the world coordinate system to the camera coordinate system is: ； In the formula, , and for A column vector that satisfies the following condition: ； ； ； In the formula, and To find a set of orthogonal vanishing points corresponding to the principal line, where, , ; The intrinsic parameter of the camera is expressed as: ； The position vector between the camera and the lidar is: ； In the formula, The position vector from the camera coordinate system to the lidar coordinate system; The position vector from the world coordinate system to the lidar coordinate system; The position vector from the camera coordinate system to the world coordinate system; and It can be obtained directly using the sine and cosine theorems.

2. The method for calibrating extrinsic parameters between a camera and a lidar based on the vanishing point method according to claim 1, characterized in that, The step S4 is followed by: S5. Further optimize the principal point, focal length, and extrinsic parameter calibration results of the camera using an iterative optimization method to obtain optimized extrinsic parameter calibration results.

3. The method for calibrating extrinsic parameters between a camera and a lidar based on the vanishing point method according to claim 1, characterized in that, The specific content of step S2 includes: S21. Extract feature lines from the optical image, select the required feature lines and calculate their corresponding vanishing points; S22. Use the calculated orthogonal vanishing points to solve for the principal point and focal length of the camera.

4. The method for calibrating extrinsic parameters between a camera and a lidar based on the vanishing point method according to claim 3, characterized in that, The method for solving the principal point and focal length of the camera in step S22 is as follows: The principal point of the camera is represented as: It satisfies the following conditions: ； In the formula, As the camera's main point, and These are the coordinate parameters of the camera's principal point; and The main line related parameter matrix, , ,in, , and Here are the relevant parameters for the j-th main line, and the mathematical expression for the j-th main line is: ; Specifically: ； ； ； In the formula, , , , , , , and To solve the first The coordinates of the two sets of orthogonal vanishing points correspond to the main line. One set of orthogonal vanishing points is... and Another set of orthogonal vanishing points is and ; The focal length of the camera is expressed as It satisfies the following conditions: ； In the formula, , This refers to the camera's focal length parameter.

5. The method for calibrating extrinsic parameters between a camera and a lidar based on the vanishing point method according to claim 1, characterized in that, Step S3 specifically includes: S31. Based on the solved principal point and focal length of the camera, the laser point cloud data is projected in two dimensions using a projective geometric pinhole model. S32. Redundant points are removed from the projected two-dimensional image points using a two-dimensional rendering method in a three-dimensional scene to obtain the LiDAR projected image.

6. The method for calibrating extrinsic parameters between a camera and a lidar based on the vanishing point method according to claim 2, characterized in that, Step S5 includes: S51. Update the lidar projection image according to the external parameter calibration result, and use the updated lidar projection image to solve the principal point and focal length of the camera in reverse, so as to obtain the optimized principal point and focal length of the camera. S52. Update the LiDAR projection image again using the optimized principal point and focal length of the camera, and use the vanishing point method to solve the extrinsic parameters between the camera and the LiDAR. S53. Repeat steps S51 and S52 to finally obtain the optimized extrinsic parameter calibration results.

Images