Camera and laser radar target-object-free calibration method and device based on linear feature matching

Abstract

The invention provides a camera and laser radar target-object-free calibration method and device based on linear feature matching, and the method comprises the steps: collecting scene data containing edge linear features, and extracting edge straight lines in a camera image; extracting three types of edge feature straight lines from the laser radar point cloud: dividing voxels for the laser point cloud and fitting planes, extracting an intersecting line of adjacent planes as a continuous edge straight line, projecting a depth value of the laser point cloud into a grey-scale map and extracting an edge straight line, and projecting an intensity value of the laser point cloud into a grey-scale map and extracting an edge straight line; constructing a k-dimensional tree for camera images and points on three types of laser edge straight lines, and generating matching point pairs through nearest neighbor search; calculating the included angle of the straight lines where the matching point pairs are located and eliminating mismatching point pairs; carrying out classified statistics on the matching point pairs according to the slope of the straight line and carrying out weighted compensation on the point pairs on the straight lines in different directions; and constructing a residual function based on the weighted matching point pairs and obtaining an optimal extrinsic parameter conversion matrix. The method has the characteristics of high environmental adaptability and high generalization.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of multi-sensor fusion technology, and in particular to a method and device for non-target calibration of a camera and a lidar based on straight line feature matching. Background Art

[0002] In the field of multi-sensor fusion, the combination of LiDAR and cameras is widely used, with their complementary advantages: LiDAR has millimeter-level measurement accuracy and strong anti-interference capabilities, but it lacks color and texture information; cameras can obtain rich visual information, but have shortcomings in ranging accuracy and environmental adaptability. To achieve the fusion of the two data, calibration is required to determine the external parameter conversion relationship between the LiDAR coordinate system and the camera coordinate system. Existing target-based calibration methods rely on manual calibration plates, which are limited by the deployment position, posture, and extreme lighting environments of the calibration plates, and lack versatility, environmental adaptability, and flexibility. In contrast, target-free calibration methods do not require specific auxiliary tools and directly use environmental features in natural scenes (such as edges, straight lines, etc.) to estimate external parameters. They have strong environmental adaptability and flexibility, and are more in line with the calibration needs of actual complex scenes. However, existing target-free calibration methods still suffer from low matching accuracy and poor robustness in complex environments, making them difficult to meet the needs of actual applications. Therefore, developing a camera and lidar non-target calibration method and equipment based on straight line feature matching can effectively overcome the defects in the above-mentioned related technologies and has important engineering value and application prospects. Summary of the Invention

[0003] In response to the above-mentioned problems existing in the prior art, an embodiment of the present invention provides a method and device for non-target calibration of a camera and a lidar based on straight line feature matching.

[0004] In a first aspect, an embodiment of the present invention provides a method for calibration of cameras and lidars without targets based on straight line feature matching, comprising: controlling equipment equipped with cameras and lidars to collect scene data containing edge straight line features, and extracting edge straight lines in the camera RGB image; extracting three types of edge feature straight lines from the lidar point cloud: dividing the laser point cloud into voxels and fitting planes, extracting adjacent plane intersections as continuous edge straight lines, projecting the laser point cloud depth values ​​into a grayscale image, extracting edge straight lines in the depth projection image, and projecting the laser point cloud intensity values ​​into a grayscale image, and extracting edge straight lines in the intensity projection image; constructing a k-dimensional tree for the camera image and the points on the three types of laser edge straight lines, and generating matching point pairs through nearest neighbor search; calculating the angle between the straight lines on which the matching point pairs are located, and eliminating mismatched point pairs whose angles exceed a predetermined threshold; classifying and counting the successfully matched point pairs according to the slope of the straight lines on which they are located, and performing weighted compensation on the point pairs on straight lines in different directions according to the slope distribution; constructing a residual function based on the weighted matching point pairs, and solving the optimal external parameter conversion matrix of the camera and lidar through nonlinear optimization.

[0005] Based on the contents of the above method embodiments, the embodiments of the present invention provide a camera and lidar non-target calibration method based on straight line feature matching, in which the laser point cloud is divided into voxels and planes are fitted, including: the voxel size of the indoor scene is set to 0.5 meters, the voxel size of the outdoor scene is set to 1 meter, the planes within the voxels are fitted by the RANSAC algorithm, and plane pairs with intersection angles in the range of 30 degrees to 150 degrees are retained for extracting continuous edge lines.

[0006] Based on the contents of the above method embodiments, the camera and laser radar non-target calibration method based on straight line feature matching provided in the embodiments of the present invention, wherein a k-dimensional tree is constructed for the camera image and the points on the three types of laser edge lines, and matching point pairs are generated through nearest neighbor search, including: constructing a k-dimensional tree for the pixel points on the edge lines of the camera image, searching for the nearest neighbors of the pixel points of the three types of laser edge lines respectively, and generating a point pair matching relationship between the camera and the laser.

[0007] Based on the contents of the above method embodiments, the camera and lidar non-target calibration method based on straight line feature matching provided in the embodiments of the present invention calculates the angle between the straight lines where the matching point pairs are located, and eliminates the mismatched point pairs whose angles exceed a predetermined threshold, including: the predetermined threshold is 10 degrees, when the angle between the straight lines where the matching point pairs are located is less than the predetermined threshold, it is determined to be a valid match, otherwise the matching point pairs are eliminated.

[0008] Based on the contents of the above method embodiments, the camera and lidar target-free calibration method based on straight line feature matching provided in the embodiments of the present invention, the successfully matched point pairs are classified and counted according to the slope of the straight line, and weighted compensation is performed on point pairs in different directions according to the slope distribution, including: calculating the angle θ between the straight line and the X-axis of the camera pixel coordinate system, if θ is less than 30°, it is divided into a horizontal straight line, if θ is greater than 60°, it is divided into a vertical straight line, if 30°≤θ≤60°, it is divided into an inclined straight line, and the number of point pairs of horizontal and vertical straight lines is counted for weighted compensation.

[0009] Based on the contents of the above method embodiments, the camera and lidar target-free calibration method based on straight line feature matching provided in the embodiments of the present invention constructs a residual function based on weighted matching point pairs, including: point cloud continuous edge straight line residuals, depth projection map straight line residuals and intensity projection map straight line residuals. During weighted compensation, horizontal or vertical point pairs are given weights inversely proportional to the number, and the weights of inclined direction point pairs are set to 1.

[0010] Based on the contents of the above method embodiments, the camera and lidar non-target calibration method based on straight line feature matching provided in the embodiments of the present invention, wherein the optimal extrinsic parameter conversion matrix of the camera and lidar is solved through nonlinear optimization, includes: using the Ceres optimization library to solve the least squares problem, the objective function is the sum of three types of residuals, and the optimal extrinsic parameter conversion matrix is ​​obtained through iterative optimization.

[0011] In a second aspect, an embodiment of the present invention provides a camera and laser radar non-target calibration device based on straight line feature matching, comprising: a first main module for controlling equipment equipped with a camera and a laser radar to collect scene data containing edge straight line features and extract edge straight lines in the camera's RGB image; a second main module for extracting three types of edge feature straight lines from the laser radar point cloud: dividing the laser point cloud into voxels and fitting a plane, extracting the intersection of adjacent planes as continuous edge straight lines, projecting the laser point cloud depth value into a grayscale image, extracting edge straight lines in the depth projection image, and projecting the laser point cloud intensity value into a grayscale image. Extract edge lines in the intensity projection image; the third main module is used to construct a k-dimensional tree for the points on the camera image and the three types of laser edge lines, and generate matching point pairs through nearest neighbor search; the fourth main module is used to calculate the angle between the lines where the matching point pairs are located, and eliminate the mismatched point pairs whose angles exceed the predetermined threshold; the fifth main module is used to classify and count the successfully matched point pairs according to the slope of the line where they are located, and perform weighted compensation on point pairs in different directions according to the slope distribution; the sixth main module is used to construct a residual function based on the weighted matching point pairs, and solve the optimal extrinsic parameter conversion matrix of the camera and lidar through nonlinear optimization.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0013] At least one processor, at least one memory and a communication interface; wherein,

[0014] The processor, memory and communication interface communicate with each other;

[0015] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the camera and lidar target-free calibration method based on straight line feature matching provided by any one of the various implementation methods of the first aspect.

[0016] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute the camera and lidar non-target calibration method based on straight line feature matching provided by any one of the various implementation methods of the first aspect.

[0017] The object-free camera and lidar calibration method and device based on line feature matching, provided by embodiments of the present invention, extracts laser edge feature lines from multiple sources and matches them with camera image lines, improving data matching richness. Matching point pairs are screened using angle thresholds to eliminate false matches and ensure matching accuracy. Classification and weighting based on line slopes compensate for small numbers of point pairs, enhancing the balance of feature optimization in different directions. Finally, through nonlinear optimization, calibration robustness and accuracy are significantly improved. The method eliminates the need for manual calibration objects, exhibits strong environmental adaptability, and exhibits strong generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic flow chart of a method for calibration of a camera and a lidar without an object based on line feature matching provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the structure of a camera and lidar non-target calibration device based on line feature matching provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention;

[0022] Figure 4 Schematic diagram of the intermediate image effect during the process of the camera and lidar target-free calibration method based on straight line feature matching provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention. If there are step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0024] The embodiment of the present invention provides a method for calibration of camera and laser radar without target based on line feature matching. Figure 1 The method includes: controlling equipment equipped with a camera and a laser radar to collect scene data containing edge line features, extracting edge lines in the camera's RGB image; extracting three types of edge feature lines from the laser radar point cloud: dividing the laser point cloud into voxels and fitting a plane, extracting adjacent plane intersections as continuous edge lines, projecting the laser point cloud depth value into a grayscale image, extracting edge lines in the depth projection image, projecting the laser point cloud intensity value into a grayscale image, and extracting edge lines in the intensity projection image; constructing a k-dimensional tree for the camera image and the points on the three types of laser edge lines, generating matching point pairs through nearest neighbor search; calculating the angle between the matching point pairs and the lines on which they are located, and eliminating mismatched point pairs whose angles exceed a predetermined threshold; classifying and counting the successfully matched point pairs according to the slope of the lines on which they are located, and performing weighted compensation on point pairs in different directions according to the slope distribution; constructing a residual function based on the weighted matching point pairs, and solving the optimal extrinsic parameter conversion matrix of the camera and the laser radar through nonlinear optimization.

[0025] Specifically, a camera and a laser radar are used to sample and collect image information of indoor and outdoor scenes with edge features. During the information collection process, the camera collects RGB image information and the laser radar collects laser point cloud information, where the point cloud information includes the three-dimensional coordinate information of the laser point, laser reflection intensity information, etc. The line extraction algorithm is used to extract the straight line in the image from the camera's RGB image, including but not limited to FastLineDetector, etc. The point on the extracted edge line is represented by p r ={p r1 ,pr2 ,...,p rm The slope of the straight line is recorded as: k r ={k r1 ,k r2 ,...,k rm}, where m is the number of points on the line.

[0026] Based on the content of the above method embodiment, as an optional embodiment, the camera and lidar non-target calibration method based on straight line feature matching provided in the embodiment of the present invention, wherein the laser point cloud is divided into voxels and the planes are fitted, including: the voxel size of the indoor scene is set to 0.5 meters, the voxel size of the outdoor scene is set to 1 meter, the planes within the voxels are fitted by the RANSAC algorithm, and plane pairs with intersection angles in the range of 30 degrees to 150 degrees are retained for extracting continuous edge lines.

[0027] Specifically, the point cloud is first divided into voxels of a given size (for example, 1m for outdoor scenes and 0.5m for indoor scenes). For each voxel, the RANSAC (Random Sample Consensus) algorithm is used to fit and extract the plane within the voxel. Then, the plane pairs that intersect within a certain range (the range in this paper is [30 degrees, 150 degrees]) are retained, and the plane intersection line (i.e., continuous edge) is solved. All 3D points on the 3D intersection line are projected into the 2D image coordinate system using the projection model, and the 2D projection point is represented as p p ={p p1 ,...,p pi ,...,p pn}, the slope of the line where the point is located is: k p ={k p1 ,...,k pi ,...,k pn}, where n is the number of points on the line.

[0028] The laser point is recorded as:

[0029]

[0030] All laser points are projected into the image coordinate system, and the pixel value is the depth value of the laser point. The depth value is recorded as:

[0031]

[0032] The depth value is then normalized:

[0033]

[0034] Among them, G max is the maximum value of grayscale image pixels. iis the pixel value at the normalized pixel point (u, v). The image is converted into a grayscale image. The image edge line is extracted using a line extraction algorithm such as FastLineDetector. The point on the extracted edge line is represented by p. d ={p d1 ,...,p di ,...,p dl}, the slope of the line where the point is located is recorded as: k d ={k d1 ,...,k di ,...,k dl}, where l is the number of points on the line.

[0035] The intensity value of the laser point I i It can be obtained directly from laser measurement data, normalizing the intensity value to a grayscale image:

[0036]

[0037] Among them, I max is the maximum value of the grayscale image pixel. i is the pixel value at the normalized pixel point (u, v). The image is converted into a grayscale image. A line extraction algorithm such as FastLineDetector is used to extract the line from the grayscale image. The points on the extracted line are represented as p I ={p I1 ,...,p Ii ,...,p Io}, the slope of the line where the point is located is: k I ={k I1 ,...,k Ii ,...,k Io}. Where o is the number of points on the line.

[0038] Based on the content of the above method embodiment, as an optional embodiment, the camera and laser radar non-target calibration method based on straight line feature matching provided in the embodiment of the present invention, wherein a k-dimensional tree is constructed for the camera image and the points on the three types of laser edge lines, and matching point pairs are generated through nearest neighbor search, including: constructing a k-dimensional tree for the pixel points on the edge lines of the camera image, searching for the nearest neighbors of the pixel points of the three types of laser edge lines respectively, and generating a point pair matching relationship between the camera and the laser.

[0039] Specifically, a k-dimensional tree is constructed for the points on the straight line detected by the camera image, and p is searched in the k-dimensional tree respectively. p 、p d 、p I The nearest neighbor of each point in the point set is point p on the camera image ri 、p rj 、prk Point p pi 、p di 、p Ii The nearest neighbor, p ri 、p rj 、p rk The slope of the line where the point lies is denoted as k ri 、k rj 、k rk .

[0040] Based on the content of the above method embodiment, as an optional embodiment, the camera and lidar non-target calibration method based on straight line feature matching provided in the embodiment of the present invention, the angle between the straight lines where the matching point pairs are located is calculated, and the erroneous matching point pairs whose angles exceed a predetermined threshold are eliminated, including: the predetermined threshold is 10 degrees, when the angle between the straight lines where the matching point pairs are located is less than the predetermined threshold, it is determined to be a valid match, otherwise the matching point pairs are eliminated.

[0041] Specifically, a pair of corresponding points p ri With p pi For example, calculate the angle between the two points:

[0042]

[0043] All nearest neighbor point pairs are calculated in sequence. If the angle between the two straight lines containing the point pair is less than a predetermined angle γ (set to 10 degrees in another embodiment), the point pair is considered to be matched successfully. Otherwise, the failed matching point pairs are eliminated and no longer participate in the subsequent residual construction.

[0044] Based on the content of the above method embodiment, as an optional embodiment, the camera and lidar target-free calibration method based on straight line feature matching provided in the embodiment of the present invention, the pairs of successfully matched points are classified and counted according to the slope of the straight line, and the point pairs in different directions are weightedly compensated according to the slope distribution, including: calculating the angle θ between the straight line and the X-axis of the camera pixel coordinate system, if θ is less than 30°, it is divided into a horizontal straight line, if θ is greater than 60°, it is divided into a vertical straight line, if 30°≤θ≤60°, it is divided into an inclined straight line, and the number of point pairs of horizontal and vertical straight lines is counted for weighted compensation.

[0045] Specifically, the slope statistics of the matching point pairs are performed

[0046] Let w1 = 0, w2 = 0. Among them, w1 counts the number of horizontal directions, and w2 counts the number of vertical directions. p Point p in the point set pi , calculate p pi The angle between the straight line and the X-axis of the camera pixel coordinate system:

[0047]

[0048] Next, we classify the straight line types: when θ i <α (set to 30 degrees in this paper), the straight line is classified as a horizontal line, w1+1; when θ i When >β (set to 60 degrees in this paper), the straight line is classified as a vertical line, w2+1.

[0049] Based on the content of the above method embodiment, as an optional embodiment, the camera and lidar target-free calibration method based on straight line feature matching provided in the embodiment of the present invention, the residual function is constructed based on the weighted matching point pairs, including: point cloud continuous edge straight line residual, depth projection map straight line residual and intensity projection map straight line residual. During weighted compensation, the horizontal or vertical point pairs are given weights inversely proportional to the number, and the weights of the inclined direction point pairs are set to 1.

[0050] Based on the content of the above method embodiment, as an optional embodiment, the camera and lidar non-target calibration method based on straight line feature matching provided in the embodiment of the present invention, wherein the optimal extrinsic parameter conversion matrix of the camera and lidar is solved by nonlinear optimization, including: using the Ceres optimization library to solve the least squares problem, the objective function is the sum of three types of residuals, and the optimal extrinsic parameter conversion matrix is ​​obtained through iterative optimization.

[0051] Specifically, take the residual construction between the points on the continuous edge line extracted from the laser point cloud and the points on the feature line extracted from the camera image as an example:

[0052]

[0053] Among them, w pi is the weight correction parameter,

[0054] π(p) represents the pinhole projection model, and f(p) is the camera distortion model.

[0055] Similarly, the residual constructed by matching the laser depth projection image and the laser reflection intensity projection image with the camera image is:

[0056]

[0057] Calculate the objective function (optimal external parameter )

[0058]

[0059] Use nonlinear optimization libraries to solve least squares problems, including but not limited to ceres and other optimization libraries to obtain the optimal external parameters

[0060] The object-free camera and lidar calibration method based on line feature matching, provided by the present invention, extracts laser edge feature lines from multiple sources and matches them with camera image lines, improving data matching richness. Matching point pairs are screened using angle thresholds to eliminate false matches and ensure matching accuracy. Classification and weighting based on line slopes compensate for small numbers of point pairs, enhancing the balance of feature optimization in different directions. Finally, a nonlinear optimization solution is used to significantly improve calibration robustness and accuracy. The method requires no manual calibration objects, is highly adaptable to various environments, and is highly generalizable.

[0061] See also Figure 4 , controlling equipment equipped with cameras and lidar to sample indoor and outdoor scenes with edge features. The collected point clouds and image information are used to extract feature lines. Edge line extraction is performed using the camera's RGB image. There are three ways to obtain edge feature lines using lidar: 1. Extract continuous edge lines directly from the laser point cloud: Divide the lidar point cloud into a set voxel size, fit planes within the voxels, and solve for continuous edges where the planes intersect. The edge feature lines are then projected onto a 2D pixel plane. 2. The depth information of the laser point cloud is projected according to the initial extrinsic parameters to obtain a laser depth map, which is then grayscale processed and feature line extraction is performed. 3. Similarly, the reflection intensity information of the laser point cloud is projected according to the initial extrinsic parameters to obtain a laser reflection intensity map, which is then grayscale processed and feature line extraction is performed. The laser feature lines extracted using the three different methods are then matched with the feature lines extracted from the camera image. A k-dimensional tree is constructed for the points on the lines in the camera image, and the nearest neighbors of the laser line points are found. Then, the angle between the straight lines containing the matching point pairs is calculated to see if it is within a set threshold. The matching results are then tested and incorrectly matched point pairs are eliminated. If the point pairs are successfully matched, three camera-lidar residuals are constructed and weighted. Finally, a nonlinear optimization method is used to solve the optimal extrinsic parameter conversion matrix to achieve camera and lidar calibration.

[0062] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with processor functions. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention can be encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a camera and laser radar non-target calibration device based on straight line feature matching, which is used to execute the camera and laser radar non-target calibration method based on straight line feature matching in the above method embodiment. Figure 2The device includes: a first main module for controlling equipment equipped with a camera and a laser radar to collect scene data containing edge line features and extract edge lines from the camera's RGB image; a second main module for extracting three types of edge feature lines from the laser radar point cloud: dividing the laser point cloud into voxels and fitting planes, extracting adjacent plane intersections as continuous edge lines, projecting the laser point cloud depth values ​​into a grayscale image, extracting edge lines from the depth projection image, and projecting the laser point cloud intensity values ​​into a grayscale image, extracting edge lines from the intensity projection image; a third main module for constructing a k-dimensional tree for points on the camera image and the three types of laser edge lines, and generating matching point pairs through nearest neighbor search; a fourth main module for calculating the angle between the lines on which the matching point pairs lie, and eliminating mismatched point pairs whose angles exceed a predetermined threshold; a fifth main module for classifying and counting successfully matched point pairs according to the slope of the line on which they lie, and performing weighted compensation on point pairs on lines in different directions based on the slope distribution; and a sixth main module for constructing a residual function based on the weighted matching point pairs, and solving the optimal extrinsic parameter conversion matrix between the camera and the laser radar through nonlinear optimization.

[0063] The camera and laser radar non-target calibration device based on line feature matching provided by the embodiment of the present invention adopts Figure 2 Several modules in the system extract laser edge feature lines from multiple sources and match them with camera image lines to improve data matching richness. Angle thresholds are used to filter matching point pairs, eliminating false matches and ensuring matching accuracy. Classification and weighting based on line slopes compensate for small numbers of point pairs and enhance the balance of feature optimization in different directions. Finally, a nonlinear optimization solution is used to significantly improve calibration robustness and accuracy. It requires no manual calibration objects, is highly adaptable to various environments, and has strong generalization capabilities.

[0064] It should be noted that the device in the device embodiment provided by the present invention can be used to implement the method in the above-mentioned method embodiment as well as the method in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set. The principle is basically the same as the principle of the above-mentioned device embodiment provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned device embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining technical features, and ensure the practicality of the technical solutions, they can improve the device in the above-mentioned device embodiment to obtain the corresponding device class embodiment, thereby obtaining the corresponding device class embodiment for implementing the methods in other method class embodiments. For example:

[0065] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the camera and lidar target-free calibration device based on straight line feature matching provided in the embodiment of the present invention also includes: a first sub-module, used to realize the voxel division of the laser point cloud and fitting of the plane, including: the voxel size of the indoor scene is set to 0.5 meters, the voxel size of the outdoor scene is set to 1 meter, the plane within the voxel is fitted by the RANSAC algorithm, and the plane pairs with intersection angles in the range of 30 degrees to 150 degrees are retained for extracting continuous edge lines.

[0066] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the camera and laser radar non-target calibration device based on straight line feature matching provided in the embodiment of the present invention also includes: a second sub-module, which is used to realize the construction of a k-dimensional tree for the camera image and the points on the three types of laser edge lines, and generate matching point pairs through nearest neighbor search, including: constructing a k-dimensional tree for the pixel points on the edge line of the camera image, searching for the nearest neighbors of the pixel points of the three types of laser edge lines respectively, and generating a point pair matching relationship between the camera and the laser.

[0067] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the camera and lidar target-free calibration device based on straight line feature matching provided in the embodiment of the present invention also includes: a third sub-module, used to realize the calculation of the angle of the straight line where the matching point pair is located, and eliminate the incorrect matching point pairs whose angle exceeds the predetermined threshold, including: the predetermined threshold is 10 degrees, when the angle of the straight line where the matching point pair is located is less than the predetermined threshold, it is determined to be a valid match, otherwise the matching point pair is eliminated.

[0068] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the camera and lidar target-free calibration device based on straight line feature matching provided in the embodiment of the present invention further includes: a fourth sub-module, which is used to realize the classification and statistics of the successfully matched point pairs according to the slope of the straight line, and perform weighted compensation on the point pairs in different directions according to the slope distribution, including: calculating the angle θ between the straight line and the X-axis of the camera pixel coordinate system, if θ is less than 30°, it is divided into a horizontal straight line, if θ is greater than 60°, it is divided into a vertical straight line, if 30°≤θ≤60°, it is divided into an inclined straight line, and counting the number of point pairs of horizontal and vertical straight lines for weighted compensation.

[0069] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the camera and lidar target-free calibration device based on straight line feature matching provided in the embodiment of the present invention also includes: a fifth sub-module, which is used to realize the construction of the residual function based on the weighted matching point pairs, including: point cloud continuous edge straight line residual, depth projection map straight line residual and intensity projection map straight line residual. During weighted compensation, the horizontal or vertical point pairs are given weights inversely proportional to the number, and the weights of the inclined direction point pairs are set to 1.

[0070] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the camera and lidar target-free calibration device based on straight line feature matching provided in the embodiment of the present invention also includes: a sixth sub-module, which is used to realize the solution of the optimal extrinsic parameter conversion matrix of the camera and lidar through nonlinear optimization, including: using the Ceres optimization library to solve the least squares problem, the objective function is the sum of three types of residuals, and the optimal extrinsic parameter conversion matrix is ​​obtained through iterative optimization.

[0071] The method of the embodiment of the present invention is implemented by electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 3 As shown, the electronic device includes: at least one processor, a communications interface, at least one memory, and a communications bus, wherein the at least one processor, the communications interface, and the at least one memory communicate with each other via the communications bus. The at least one processor can call logic instructions in the at least one memory to execute all or part of the steps of the methods provided in the aforementioned method embodiments.

[0072] In addition, the logic instructions in the at least one memory mentioned above can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0073] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0074] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0075] The flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. Based on this understanding, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or sometimes in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0076] It should be noted that the terms "include", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements. Any "predetermined threshold", "preset threshold" or similar expressions that do not indicate a specific value can be determined by a person of ordinary skill in the art through simple experiments or corresponding debugging.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for non-target calibration of camera and lidar based on line feature matching, characterized in that: include: Control the equipment equipped with cameras and lidar to collect scene data containing edge line features, and extract edge lines in the camera's RGB image; extract three types of edge feature lines from the lidar point cloud: divide the laser point cloud into voxels and fit planes, extract the intersection lines of adjacent planes as continuous edge lines, project the laser point cloud depth values ​​into a grayscale image, extract edge lines in the depth projection image, project the laser point cloud intensity values ​​into a grayscale image, and extract edge lines in the intensity projection image; construct a k-dimensional tree for the camera image and the points on the three types of laser edge lines, and generate matching point pairs through nearest neighbor search; calculate the angle between the lines where the matching point pairs are located, and eliminate the mismatched point pairs whose angles exceed a predetermined threshold; classify and count the successfully matched point pairs according to the slope of the line where they are located, and perform weighted compensation on point pairs on lines in different directions according to the slope distribution; construct a residual function based on the weighted matching point pairs, and solve the optimal extrinsic parameter conversion matrix of the camera and lidar through nonlinear optimization.

2. The method for non-target calibration of camera and laser radar based on line feature matching according to claim 1, characterized in that: The method of dividing the laser point cloud into voxels and fitting planes includes: setting the voxel size of indoor scenes to 0.5 meters, setting the voxel size of outdoor scenes to 1 meter, fitting the planes within the voxels through the RANSAC algorithm, and retaining plane pairs with intersection angles within the range of 30 degrees to 150 degrees for extracting continuous edge lines.

3. The method for non-target calibration of camera and laser radar based on line feature matching according to claim 2, characterized in that: The method constructs a k-dimensional tree for the camera image and the points on the three types of laser edge lines, and generates matching point pairs through nearest neighbor search, including: constructing a k-dimensional tree for the pixel points on the camera image edge lines, searching for the nearest neighbors of the three types of laser edge line pixel points respectively, and generating a point pair matching relationship between the camera and the laser.

4. The method for non-target calibration of camera and laser radar based on line feature matching according to claim 3, characterized in that: The calculating of the angle between the straight lines containing the matching point pairs and the elimination of mismatched point pairs whose angles exceed a predetermined threshold includes: the predetermined threshold is 10 degrees, and when the angle between the straight lines containing the matching point pairs is less than the predetermined threshold, it is determined to be a valid match; otherwise, the matching point pairs are eliminated.

5. The method for calibration of camera and laser radar without target based on line feature matching according to claim 4, characterized in that: The successfully matched point pairs are classified and counted according to the slope of the line on which they are located, and weighted compensation is performed on point pairs on lines in different directions according to the slope distribution, including: calculating the angle θ between the line and the X-axis of the camera pixel coordinate system, if θ is less than 30°, it is classified as a horizontal line, if θ is greater than 60°, it is classified as a vertical line, and if 30°≤θ≤60°, it is classified as an inclined line, and the number of point pairs on the horizontal line and the vertical line is counted for weighted compensation.

6. The method for non-target calibration of camera and laser radar based on line feature matching according to claim 5, characterized in that: The residual function constructed based on the weighted matching point pairs includes: point cloud continuous edge straight line residual, depth projection line residual and intensity projection line residual. During weighted compensation, horizontal or vertical point pairs are given weights inversely proportional to the number, and the weights of oblique direction point pairs are set to 1.

7. The method for non-target calibration of camera and laser radar based on line feature matching according to claim 6, characterized in that: The method of solving the optimal extrinsic parameter conversion matrix of the camera and the lidar through nonlinear optimization includes: using the Ceres optimization library to solve the least squares problem, the objective function is the sum of three types of residuals, and obtaining the optimal extrinsic parameter conversion matrix through iterative optimization.

8. A camera and laser radar calibration device without target based on line feature matching, characterized in that: include: The first main module is used to control equipment equipped with cameras and lidar to collect scene data containing edge line features and extract edge lines from the camera's RGB image. The second main module is used to extract three types of edge feature lines from the lidar point cloud: dividing the laser point cloud into voxels and fitting planes, extracting the intersection lines of adjacent planes as continuous edge lines, projecting the laser point cloud depth values ​​into a grayscale image, extracting edge lines from the depth projection image, and projecting the laser point cloud intensity values ​​into a grayscale image, extracting edge lines from the intensity projection image. The third main module is used to construct a k-dimensional tree for the camera image and points on the three types of laser edge lines, and generate matching point pairs through nearest neighbor search. The fourth main module is used to calculate the angle between the lines on which the matching point pairs lie and eliminate mismatched point pairs whose angles exceed a predetermined threshold. The fifth main module is used to classify and count successfully matched point pairs according to the slope of the line on which they lie, and perform weighted compensation for point pairs on lines in different directions based on the slope distribution. The sixth main module is used to construct a residual function based on the weighted matching point pairs and solve the optimal extrinsic parameter conversion matrix between the camera and lidar through nonlinear optimization.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein, The processor, memory and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which cause a computer to execute the method of any one of claims 1 to 7.

Images