Self-calibration method, device and storage medium for laser radar and camera

Abstract

The present invention relates to the field of computer vision technology, and to a self-calibration method, device and storage medium for laser radar and camera. The method has the following steps: S1, using laser radar and camera to collect data for the same scene, the laser radar is used to obtain depth image and intensity image, and the camera is used to obtain color image; S2, based on ORB feature detection, obtaining matching point pairs in intensity image and color image; S3, obtaining three-dimensional coordinates corresponding to matching points from the depth image, obtaining image coordinates corresponding to matching points from the color image, and establishing coordinate point pairs; S4, constructing a coordinate projection formula between the depth image and the color image, solving the coordinate projection formula through the coordinate point pairs, and obtaining the conversion matrix between the laser radar and the camera. The device and storage medium are both used to implement the above-mentioned self-calibration method. The present invention can better realize the autonomous alignment calibration between the laser radar coordinate system and the camera coordinate system.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a self-calibration method, device and storage medium for laser radar and camera. Background Art

[0002] LiDAR and visible light cameras are two commonly used imaging sensors in computer vision. LiDAR can obtain the three-dimensional position information of the environment, and visible light cameras can obtain the color information of the environment. In order to obtain more perceptual information such as the color point cloud of the target scene, the existing technology uses both LiDAR and cameras to detect the target scene. In order to achieve the fusion of LiDAR and camera data, it is necessary to calibrate the projection relationship between LiDAR and camera in the same system.

[0003] Currently, the calibration between LiDAR and cameras requires the use of specific known objects such as calibration plates and building right angles, making self-calibration difficult to achieve. Summary of the invention

[0004] The present invention provides a self-calibration method for a laser radar and a camera, which can overcome certain defects of the prior art.

[0005] A self-calibration method for a laser radar and a camera according to the present invention comprises the following steps:

[0006] S1. Use LiDAR and camera to collect data of the same scene. LiDAR is used to obtain depth image I d and intensity image I in , the camera is used to obtain the color image I rgb ;

[0007] S2. Obtain intensity image I based on ORB feature detection in With color image I rgb The matching points in is the intensity image I in ORB feature points in For color image I rgb ORB feature points in;

[0008] S3. Self-depth image I d Get and match points The corresponding three-dimensional coordinates (x k ,y k , z k ), from color image I rgb Get and match points The corresponding image coordinates (u k , v k), establish the coordinate point pair (x k ,y k , z k ,u k , v k );

[0009] S4. Constructing depth image I d With color image I rgb The coordinate projection formula between the coordinate points (x k ,y k , z k ,u k , v k ) Solve the coordinate projection formula to obtain the transformation matrix R between the lidar and the camera.

[0010] Preferably, step S2 specifically includes the following steps:

[0011] S21. Obtaining intensity image I i The first ORB feature point set P ref And the corresponding first ORB feature point vector set is the mth first ORB feature point, is the corresponding first ORB feature point vector;

[0012] S22, obtaining color image I rgb The second ORB feature point set P rgb And the corresponding second ORB feature point vector set Q rgb , is the nth second ORB feature point, is the corresponding second ORB feature point vector;

[0013] S23, the first ORB feature point vector set Q ref Each first ORB feature point in Sequentially with the second ORB feature point vector set Q rgb The second ORB feature point in Perform similarity matching; then obtain a set of matching point pairs in,

[0014] Preferably, in step S23, for the kth matching point pair and matching point and matching points The image coordinates are similar, and the corresponding first ORB feature point vector and the second ORB feature point vector resemblance.

[0015] Preferably, when performing similarity matching,

[0016] For any first ORB feature point Calculate the difference between them and the second ORB feature point set P in sequence rgb The second ORB feature point that is not matched in The Euler distance d of the pixel coordinates ref , and calculate the corresponding first ORB feature point vector in turn and the second ORB feature point vector set Q rgb The second ORB feature point that is not matched in The corresponding second ORB feature point vector The Hamming distance d rgb ;

[0017] If the second ORB feature point set P rgb There is any first ORB feature point The Euler distance d ref Less than the set Euler distance threshold D ref And the Hamming distance d rgb Less than the set Hamming distance threshold D rgb When the corresponding point of the first ORB feature point is found, the match is successful; when the match is successful, any first ORB feature point The corresponding point is regarded as a matching point pair.

[0018] Preferably, when there are multiple corresponding points, the Euler distance d ref The Hamming distance d rgb The point with the smallest sum is taken as any first ORB feature point matching point.

[0019] As a preferred embodiment, in step S3, obtain the matching point pair set P match The corresponding coordinate point pair set P c , P c ={(x k ,y k , z k ,u k , v k )|k∈N +}; In step S4, when solving the coordinate projection formula, the coordinate point pair set P c Select a sufficient number of coordinate points and substitute them into the coordinate projection formula.

[0020] Preferably, in step S4,

[0021] The constructed transformation matrix R is,

[0022]

[0023] The coordinate projection formula constructed is:

[0024]

[0025] Preferably, in step S4, the conversion matrix R is obtained based on the least squares method.

[0026] In addition, the present invention also provides a self-calibration device for a laser radar and a camera, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of any one of the above-mentioned self-calibration methods are implemented.

[0027] In addition, the present invention also provides a computer-readable medium on which a computer program is stored, and when the computer program is executed, the steps of any one of the above-mentioned self-calibration methods are implemented.

[0028] The present invention has the following beneficial effects:

[0029] 1. The introduced intensity image I in Capable of working with color images rgb To maintain basically the same perception of feature points in the same scene, we use the intensity image I in With color image I rgb By matching the feature points in (it can be understood that the two matching feature points correspond to the same point in the actual scene), the coordinates of a certain point in the actual scene in the laser radar and the camera can be obtained respectively, so the conversion matrix R can be obtained better. Through the above, the self-calibration between the laser radar and the camera can be realized without using a dedicated calibration target as a calibration object, thereby greatly simplifying the calibration process and improving the calibration efficiency.

[0030] 2. Matching points by constraints and matching points The image coordinates are similar, which can better ensure the matching point pair and have similar spatial positions; by constraining the first ORB feature point vector and the second ORB feature point vector Similar, it can better ensure the matching point pair and have similar reflection characteristics;

[0031] 3. The introduction of Euler distance and Hamming distance achieves better matching of point pairs and Improvement of matching accuracy and speed;

[0032] 4. Through the method of the present invention, when the laser radar and the camera are used in conjunction, autonomous alignment and calibration between the laser radar coordinate system and the camera coordinate system can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the process of the self-calibration method in Example 1. DETAILED DESCRIPTION

[0034] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0035] Example 1

[0036] Combination Figure 1 As shown, this embodiment provides a self-calibration method for laser radar and camera. It has the following steps:

[0037] S1. Use LiDAR and camera to collect data of the same scene. LiDAR is used to obtain depth image I d and intensity image I in , the camera is used to obtain the color image I rgb ;

[0038] S2. Obtain intensity image I based on ORB feature detection in With color image I rgb The matching points in is the intensity image I in ORB feature points in For color image I rgb ORB feature points in;

[0039] S3. Self-depth image I d Get and match points The corresponding three-dimensional coordinates (x k ,y k , z k ), from color image I rgb Get and match points The corresponding image coordinates (u k , v k ), establish the coordinate point pair (x k ,y k , z k ,u k , v k );

[0040] S4. Constructing depth image I d With color image I rgbThe coordinate projection formula between the coordinate points (x k ,y k , z k ,u k , v k ) Solve the coordinate projection formula to obtain the transformation matrix R between the lidar and the camera.

[0041] In this embodiment, the introduced intensity image I in Capable of working with color images rgb To maintain basically the same perception of feature points in the same scene, we use the intensity image I in With color image I rgb By matching the feature points in (it can be understood that the two matching feature points correspond to the same point in the actual scene), the coordinates of a certain point in the actual scene in the laser radar and the camera can be obtained respectively, so the conversion matrix R can be obtained better. Through the above, the self-calibration between the laser radar and the camera can be realized without using a dedicated calibration target as a calibration object, thus greatly simplifying the calibration process and improving the calibration efficiency.

[0042] When the laser radar is scanning, it can project the acquired ranging data (ie, depth data) and reflection intensity data into a depth image I according to the scanning rows and columns. d and intensity image I in , that is, the depth image I d and intensity image I in In fact, it is obtained by constructing two kinds of information obtained by the laser radar scanning the same position, so the depth image I d and intensity image I in Pixels at the same position in the image correspond to the same point in space.

[0043] It can be understood that the intensity image I in It represents the reflection intensity at a certain position of the target scene, while the color image I rgb It represents the R, G, and B scores at a certain position of the target scene, which are essentially the reflection data of the wave; therefore, it can be better assumed that a certain position in the same scene can have similar radar wave reflection characteristics and visible light reflection characteristics; that is, the intensity image I in With color image I rgb There will be overlapping feature points, and the overlapping feature points correspond to the same position in the target scene. Therefore, through step S2, the same position in the target scene can be obtained in the intensity image I in With color image I rgb The corresponding points in

[0044] In addition, after the laser radar obtains the ranging data (i.e., depth data) and reflection intensity data, it can obtain the depth value and reflection intensity value at any pixel coordinate according to the projection of the scanned rows and columns. Among them, the pixel coordinate value is the number of scanned rows and columns. By converting the pixel coordinate value and the depth value in a conventional conversion method, the depth image I representing the three-dimensional coordinates at each pixel coordinate can be obtained. d That is, the depth image I in this embodiment d What is represented is the three-dimensional spatial coordinates of the target scene in the coordinate system of the laser radar; it can be understood that for the laser radar, it can obtain the three-dimensional spatial coordinates with the geometric center of the laser radar as the origin based on the ranging data (ie, depth data). This conversion is an existing conventional technology and will not be described in detail in this embodiment.

[0045] Therefore, after obtaining the matching point pair After that, the same position in the target scene can be obtained in the depth image I d The coordinate value (x k ,y k , z k ) and in the color image I rgb The coordinate values ​​in (u k , v k ), that is, the coordinate point pair (x k ,y k , z k ,u k , v k ).

[0046] Therefore, through step S4, the conversion matrix R between the laser radar and the camera can be obtained more preferably.

[0047] Step S2 of this embodiment specifically includes the following steps:

[0048] S21. Obtaining intensity image I i The first ORB feature point set P ref And the corresponding first ORB feature point vector set Q ref , is the mth first ORB feature point, is the corresponding first ORB feature point vector;

[0049] S22, obtaining color image I rgb The second ORB feature point set P rgb And the corresponding second ORB feature point vector set Q rgb , is the nth second ORB feature point, is the corresponding second ORB feature point vector;

[0050] S23, the first ORB feature point vector set Q ref Each first ORB feature point in Sequentially with the second ORB feature point vector set Q rgb The second ORB feature point in Perform similarity matching; then obtain the matching point pair set P match , in,

[0051] Through the above, the matching point pairs can be better obtained.

[0052] Among them, ORB feature detection is a known method, so it is not described in detail in this embodiment.

[0053] In step S23 of this embodiment, for the kth matching point pair and matching point and matching points The image coordinates are similar, and the corresponding first ORB feature point vector and the second ORB feature point vector Therefore, it is possible to achieve better matching point pair and to obtain.

[0054] Among them, by constraining the matching points and matching points The image coordinates are similar, which can better ensure the matching point pair and have similar spatial positions; by constraining the first ORB feature point vector and the second ORB feature point vector Similar, it can better ensure the matching point pair and have similar reflection characteristics.

[0055] Among them, when performing similarity matching,

[0056] For any first ORB feature point Calculate the difference between them and the second ORB feature point set P in sequence rgb The second ORB feature point that is not matched in The Euler distance d of the pixel coordinates ref , and calculate the corresponding first ORB feature point vector in turn and the second ORB feature point vector set Q rgbThe second ORB feature point that is not matched in The corresponding second ORB feature point vector The Hamming distance d rgb ;

[0057] If the second ORB feature point set P rgb There is any first ORB feature point The Euler distance d ref Less than the set Euler distance threshold D ref And the Hamming distance d rgb Less than the set Hamming distance threshold D rgb When the corresponding point of the first ORB feature point is found, the match is successful; when the match is successful, any first ORB feature point The corresponding point is regarded as a matching point pair.

[0058] Through the above, similar matching point pairs can be better achieved and to obtain.

[0059] It is understandable that when the match is not successful, the next first ORB feature point to make a match.

[0060] In this embodiment, when there are multiple corresponding points, the Euler distance d ref The Hamming distance d rgb The point with the smallest sum is taken as any first ORB feature point Therefore, the matching point pair can be better achieved. and to obtain.

[0061] In this embodiment, in step S3, the matching point pair set P is obtained. match The corresponding coordinate point pair set P c , P c ={(x k ,y k , z k ,u k , v k )|k∈N +}; In step S4, when solving the coordinate projection formula, the coordinate point pair set P c Select a sufficient number of coordinate points and substitute them into the coordinate projection formula. This can better achieve the solution of the transformation matrix R.

[0062] In step S4 of this embodiment,

[0063] The constructed transformation matrix R is,

[0064]

[0065] The coordinate projection formula constructed is:

[0066]

[0067] It can be understood that (u, v) in the above formula is the color image I rgb The coordinates of any pixel point in the above formula, (x, y, z) is the depth image I d The three-dimensional coordinates of any point in .

[0068] in, is the homogeneous representation of the pixel coordinates (u, v), is a homogeneous representation of the three-dimensional coordinates (x, y, z).

[0069] Among them, R 11 -R 34 is the parameter to be determined in the transformation matrix R.

[0070] When solving the coordinate projection formula, the coordinate projection formula can be rewritten as,

[0071]

[0072] Let A = [xyz 1], The following constraint relationship can be established:

[0073]

[0074] By selecting a set of coordinate point pairs P c By inserting a sufficient number of coordinate points in the 11 -R 34 , so the conversion matrix R can be obtained better.

[0075] Of course, in this embodiment, the conversion matrix R can also be obtained based on, for example, the least square method. Therefore, it is better to obtain R based on a linear fitting method. 11 -R 34 The numerical value of .

[0076] Example 2

[0077] Based on the method of Example 1, this embodiment provides a self-calibration device for a laser radar and a camera, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the self-calibration method in Example 1 are implemented.

[0078] Example 3

[0079] Based on the method of Example 1, this embodiment provides a computer-readable medium on which a computer program is stored. When the computer program is executed, the steps of the self-calibration method in Example 1 are implemented.

[0080] It is easy to understand that those skilled in the art can combine, split, reorganize, etc. the embodiments of the present application to obtain other embodiments based on one or several embodiments provided in the present application, and these embodiments do not exceed the protection scope of the present application.

[0081] The present invention and its implementation methods are described schematically above, and the description is not restrictive. The embodiments shown in the embodiments are only part of the implementation methods of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by the embodiments and designs a structure and an implementation method similar to the technical solution without creativity without departing from the purpose of the invention, they should all fall within the protection scope of the present invention.

Claims

1. A self-calibration method for a laser radar and a camera, comprising the following steps: S1. Use LiDAR and camera to collect data of the same scene. LiDAR is used to obtain depth image I d and intensity image I in , the camera is used to obtain the color image I rgb ; S2. Obtain intensity image I based on ORB feature detection in With color image I rgb The matching points in is the intensity image I in ORB feature points in For color image I rgb ORB feature points in; S3. Self-depth image I d Get and match points The corresponding three-dimensional coordinates (x k ,y k , z k ), from the color image I rgb Get and match points The corresponding image coordinates (u k , v k ), establish the coordinate point pair (x k ,y k , z k ,u k , v k ); S4. Constructing depth image I d With color image I rgb The coordinate projection formula between the coordinate points (x k ,y k , z k ,u k , v k ) Solve the coordinate projection formula to obtain the transformation matrix R between the lidar and the camera.

2. The self-calibration method for laser radar and camera according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21. Obtaining intensity image I in The first ORB feature point set P ref And the corresponding first ORB feature point vector set Q ref , is the mth first ORB feature point, is the corresponding first ORB feature point vector; S22, obtaining color image I rgb The second ORB feature point set P rgb And the corresponding second ORB feature point vector set Q rgb , is the nth second ORB feature point, is the corresponding second ORB feature point vector; S23, the first ORB feature point vector set Q ref Each first ORB feature point in Sequentially with the second ORB feature point vector set Q rgb The second ORB feature point in Perform similarity matching; then obtain the matching point pair set P match , in, 3. The self-calibration method for laser radar and camera according to claim 2, characterized in that: In step S23, for the kth matching point pair and matching point and matching points The image coordinates are similar, and the corresponding first ORB feature point vector and the second ORB feature point vector resemblance.

4. The self-calibration method for laser radar and camera according to claim 3, characterized in that: When performing similarity matching, For any first ORB feature point Calculate the difference between them and the second ORB feature point set P in sequence rgb The second ORB feature point that is not matched in The Euler distance d of the pixel coordinates ref , and calculate the corresponding first ORB feature point vector in turn and the second ORB feature point vector set Q rgb The second ORB feature point that is not matched in The corresponding second ORB feature point vector The Hamming distance d rgb ; If the second ORB feature point set P rgb There is any first ORB feature point The Euler distance d ref Less than the set Euler distance threshold D ref And the Hamming distance d rgb Less than the set Hamming distance threshold D rgb When the corresponding point of the first ORB feature point is found, the match is successful; when the match is successful, any first ORB feature point The corresponding point is regarded as a matching point pair.

5. The self-calibration method for laser radar and camera according to claim 4, characterized in that: When there are multiple corresponding points, the Euler distance d ref The Hamming distance d rgb The point with the smallest sum is taken as any first ORB feature point matching point.

6. The self-calibration method for laser radar and camera according to claim 5, characterized in that: In step S3, obtain the matching point pair set P match The corresponding coordinate point pair set P c , P c ={(x k ,y k , z k ,u k , v k )|k∈N + }; In step S4, when solving the coordinate projection formula, the coordinate point pair set P c Select a sufficient number of coordinate points and substitute them into the coordinate projection formula.

7. The self-calibration method for laser radar and camera according to claim 6, characterized in that: In step S4, The constructed transformation matrix R is, The coordinate projection formula constructed is:

8. The self-calibration method for laser radar and camera according to claim 7, characterized in that: In step S4, the conversion matrix R is obtained based on the least squares method.

9. A self-calibration device for a laser radar and a camera, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the self-calibration method according to any one of claims 1 to 8 are implemented.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed, the steps of the self-calibration method according to any one of claims 1 to 8 are implemented.

Images