Method for calibrating external parameters of laser radar and camera in target-free environment

Abstract

A method for calibrating external parameters of a laser radar and a camera in a target-free environment comprises the following specific steps: firstly, starting the laser radar and the camera at the same time to carry out data acquisition so as to obtain a radar point cloud and a camera image at the same time, and extracting a depth continuous edge, a depth discontinuous edge and an intensity discontinuous edge from the radar point cloud; extracting natural edge features from the camera image; and matching the extracted radar edge features with the camera edge features to generate a multi-feature edge. The performance of the algorithm in various complex environments is verified through experiments, so that the reliability and robustness of the algorithm are verified. The method has the beneficial effects that the algorithm not only can effectively process the noise problem caused by the depth and intensity discontinuous edge, but also can keep higher matching precision and stability under various environmental conditions, has better accuracy and robustness, and can provide powerful support for a subsequent multi-sensor fusion system.

Description

Technical Field

[0001] The present invention relates to the field of robot perception technology, and in particular to a method for calibrating external parameters of a laser radar and a camera in a target-free environment. Background Art

[0002] Currently, lidar sensors and cameras play a vital role in perception tasks in robotic systems, including applications such as simultaneous localization and mapping (SLAM) and semantic segmentation. Lidar provides three-dimensional measurements of the environment, while cameras provide rich texture information. The fusion of the two can significantly improve the performance of the perception system. Therefore, it is becoming increasingly important to develop robust and accurate lidar-camera fusion methods.

[0003] External parameter calibration is the estimation of the six degrees of freedom (DoF) transformation (rotation and translation) between the LiDAR and camera coordinate systems, which is a key step in sensor fusion. Among the existing external parameter calibration methods, there are mainly target-based calibration methods and target-free calibration methods.

[0004] Existing target-based calibration methods, such as using a checkerboard or a specific pattern, have proven their effectiveness and robustness in experimental environments. However, these methods have the following disadvantages in practical applications: they rely on specific calibration targets and need to be arranged in the scene, which increases the complexity and time cost of the operation and limits their practicality in actual dynamic environments; they must arrange the calibration targets in the scene, which limits their application scope in complex and changing environments; in actual operations, it is not convenient to perform frequent calibration, especially in dynamic or unstructured environments.

[0005] Although targetless calibration methods do not require external calibration targets, they usually have specific requirements for the scene and often only use a single feature, which has the following disadvantages: most methods usually only use a single feature, such as a plane or an edge, which limits their robustness in complex and diverse environments; many targetless methods have specific requirements for the scene, such as requiring the inclusion of significant plane or edge features, which is difficult to meet in some environments; the calibration accuracy is limited when dealing with complex environments and multiple scenes, especially when extracting deep discontinuous edges, there is an expansion problem, which affects the accuracy of the calibration. Summary of the invention

[0006] The purpose of the present invention is to provide a method for calibrating the external parameters of lidar and camera in a target-free environment in view of the shortcomings and defects in the prior art. The algorithm can not only effectively deal with the noise problems caused by discontinuous edges of depth and intensity, but also maintain high matching accuracy and stability under various environmental conditions. It has good accuracy and robustness, and can provide strong support for subsequent multi-sensor fusion systems.

[0007] To achieve the above purpose, the present invention adopts the following technical scheme: a method for calibrating the external parameters of a laser radar and a camera in a target-free environment, a method for calibrating the external parameters of a laser radar and a camera in a target-free environment, characterized in that it includes the following specific steps: S1, first, start the laser radar and the camera at the same time to collect data to obtain the radar point cloud and the camera image at the same time, extract the depth continuous edge, the depth discontinuous edge and the intensity discontinuous edge from the radar point cloud, and extract the natural edge feature from the camera image; S2, match the extracted radar edge feature with the camera edge feature to generate a multi-feature edge: In order to improve the accuracy, a noise model considering the radar spot is established, the expansion error of the depth discontinuous edge is calculated, and it is compensated in the optimization process. By minimizing the reprojection error, the external parameters of the laser radar and the camera are iteratively adjusted using a nonlinear optimization algorithm until they converge to the optimal solution. S3, the performance of the algorithm in a variety of complex environments is verified by experiments, thereby verifying the reliability and robustness of the algorithm.

[0008] Furthermore, the specific steps of S1 are: 1) after data collection is completed; 2) first voxelize the lidar point cloud to construct a structured spatial model, and extract accurate and depth-continuous edges from it; 3) next, convert these accumulated three-dimensional point clouds into depth-continuous edges, and extract two types of key discontinuous edges in the two-dimensional image: depth discontinuous edges and intensity discontinuous edges; 4) in order to further analyze and process these features, remap the two types of discontinuous edges extracted from the two-dimensional image back to the original three-dimensional lidar point cloud, and fuse them with the previously extracted depth-continuous edges, and finally obtain three-dimensional point cloud data containing multi-feature edges constructed in the lidar coordinate system; 5) at the same time, apply classic edge detection algorithms from the camera image to extract natural edge features, which often contain rich semantic information in complex scenes; 6) after completing the multi-feature edge extraction, match and iteratively optimize the extracted data.

[0009] Furthermore, S2 is specifically as follows: using the initial external parameters and the pre-calibrated camera internal parameters, the multi-feature three-dimensional edge in the lidar coordinate system is projected onto the image plane in the camera coordinate system, and the corresponding residual is constructed by matching the projected lidar edge with the natural edge features in the camera image, thereby laying the foundation for subsequent parameter optimization. In order to further improve the matching accuracy of the system, not only the inherent noise in the lidar and camera measurement process is considered, but also the light spot problem formed by the divergence angle of the lidar in different scenes is modeled and analyzed. Due to the divergence effect of the laser beam, the light spot at the depth discontinuous edge will be Expansion and drawing errors are introduced at the edge, which may lead to local inconsistency of the point cloud. This error is quantified through an accurate spot model, and the expansion deviation of the lidar at the depth discontinuous edge is calculated. The deviation is compensated in the optimization process to reduce its impact on the external parameter calibration. On the basis of deviation compensation and edge matching, a nonlinear optimization problem for minimizing the reprojection error is further constructed, and an efficient nonlinear optimization algorithm is used to iteratively adjust the external parameter parameters of the lidar and camera. Through multiple iterations, until convergence to the optimal solution, the external parameter relationship between the lidar and the camera is accurately estimated.

[0010] Furthermore, the extraction of depth continuous edges, depth discontinuous edges and intensity discontinuous edges is specifically as follows: for the extraction of depth continuous edges, based on point cloud voxel cutting and plane fitting, the depth continuous edges are efficiently and accurately extracted; in the process of extracting depth discontinuous edges and intensity discontinuous edges, the data is first mapped to a spherical coordinate system to better adapt to the inherent measurement geometry of the lidar, and then, this spherical representation is converted into a Cartesian coordinate system to generate a two-dimensional intensity image, in which each pixel represents a cluster of points in 3D space, and the average intensity value of these points is used to color each pixel while retaining its depth information, and then, the CANNY edge detection algorithm is applied to the intensity map to extract all edges, and a KD-Tree-based filtering mechanism is then used to distinguish depth discontinuous edges and intensity discontinuous edges on the plane according to locality and depth changes, and the depth discontinuous edges and intensity discontinuous edges extracted in this way are then remapped to the original lidar point cloud, in which all extracted edges are merged to obtain multi-feature edges.

[0011] Furthermore, the iterative optimization includes edge matching, measuring noise and calibration formula and optimization.

[0012] Furthermore, the edge matching is specifically as follows: the edge matching and optimization process is performed in the image space of the camera coordinate system, and the edge points in the laser radar coordinate system are projected onto the image plane of the camera coordinate system by the following formula: C p i =π(L P i ) , (2), p i =f( C p i ), (3), where CPi and LPi represent 3D points in the camera and lidar coordinate systems, respectively. Formula (1) represents the rigid body transformation T applied to point LPi, where R and t represent the rotation matrix and translation vector, respectively. The symbol pi∈R2 represents the position of the point in the camera image space. The function π(CPi) is the camera projection model, and f(CPi) is the camera distortion model. These two models together reflect the internal transformation from CPi=[Xi,Yi,Zi]T to pi=[xi,yi]T. In the KD tree generated by the edge pixels of the image, a query is initiated for point pi to find its nearest κ neighbors, and these κ nearest neighbors are denoted as Si={qij;j=1,…,κ}. These relationships are defined as follows: Under this condition, the line represented by Si is determined by the points qi lying on the line, and is accompanied by a normal vector ni, which is the eigenvector associated with the smallest eigenvalue of Si and summarizes the direction of the line in space.

[0013] Furthermore, the measurement noise is specifically: the laser radar edge points extracted from the image and the corresponding edge features will be affected by the measurement noise. For the inherent measurement noise of the laser radar and the camera, a commonly used noise model is used. For the measurement noise of the camera, the noise Iwi is modeled as a normally distributed variable N(0,IΣi), which will disturb the edge point features qi during the image edge extraction process. For the points extracted from the laser radar, the noise associated with these points is recorded as Lwi. The corresponding mathematical expression is as follows: in, Among them, the relationship between the real point position LPigt and its measured value LPi is obtained, ωi∈S2 represents the measured azimuth, and δωi~N(02×1,Σωi) represents the measurement noise on the ωi tangent plane, and it is assumed to be normally distributed, di represents the depth measurement value, and δdi~N(0,Σdi) represents the ranging error associated with it, which is also assumed to obey the normal distribution; on this basis, the measurement deviation caused by the divergence angle of the laser radar beam is further introduced. In the simplified subsequent optimization model, the shape of the laser radar beam is approximated as a circle to facilitate calculation ; In the subsequent optimization process, the measurement deviation caused by the divergence angle of the LiDAR beam was taken into account. Since the laser pulse is not a perfect point but a beam with a divergence angle, when scanning from the foreground to the background object, part of the laser pulse will be reflected by the foreground, and the rest will be reflected by the background, generating two reflected pulses to reach the LiDAR receiver. When the foreground object has a high reflectivity, even if the center line of the beam has exceeded the foreground object, the signal from the first pulse still dominates, which results in false points outside the actual edge of the foreground object. When the distance between the foreground and the background is close, the two pulses The signals will be fused to produce a comprehensive signal, generating a series of points connecting the foreground and background, called "drawing points"; the expansion points cause abnormal intensity values ​​at the depth discontinuity edge, affecting its accuracy and making the edge look expanded. Although drawing points are also abnormal points, they have little effect on the intensity map of the lidar; the uncertainty of the depth discontinuity edge caused by the expansion points caused by the lidar beam follows a biased Gaussian distribution (μ, LΣi). When the centerline of the lidar beam is exactly on the depth discontinuity edge, the measured value is the depth of the foreground object, which is the key position for generating false points. As the centerline of the beam moves upward, if the foreground object is still within the range of the beam, ideally, the measurement value should return the value of the background object reached by the centerline of the beam, but due to the measurement principle of the lidar, the measurement value will still return the value of the foreground object. When the circular beam is exactly tangent to the depth discontinuity edge, the lidar still measures the value of the foreground object, which corresponds to the expansion point farthest from the depth discontinuity edge of the actual point cloud. When the centerline of the lidar beam moves further upward and the beam is completely on the background object, the returned measurement value is the value of the background object.The deviation mean μ varies with the position of the center line of the beam. When the center line of the beam is tangent to the edge of the foreground object, μ = 0. When the entire beam is tangent to the foreground object, μ = ei(di, θ), where ei represents the radius of the circular beam, which is determined by the depth measurement di and the divergence angle θ of the lidar beam. It is worth noting that when extracting edges from the intensity image of the lidar, the extracted depth discontinuity edges are the expansion points farthest from the true edges. These points are the positions where the entire beam is tangent to the foreground object. These points conform to the biased Gaussian distribution (μ, LΣi). For these expansion points on the depth discontinuity edges, if they are regarded as actual points of the depth discontinuity edges, the following relationship should be satisfied: Among them, E i =e i V, V = N × L, where V is the vector obtained by performing a cross product (×) operation between vectors N and L, and is orthogonal to both N and L. Here, N is the normal vector of the plane where the edge lies, and L is the vector perpendicular from the center of the beam to the actual edge of the surface.

[0014] Furthermore, the calibration formula and optimization are as follows: Figure 6 As shown, let LPi be an edge point obtained from the lidar point cloud. The edge corresponding to this point is identified by the normal ni (in S1), and there is a corresponding point qi∈R2 on this edge. When the noise in LPi is taken into account and the point is mapped to the image plane using the verified external parameters, it should be accurately aligned with the edge represented by (ni,qi) in the image. The ideal observation model is as follows: where the noise components associated with the lidar and camera measurements are denoted as Lwi and Cwi, respectively. Formula (7) shows that since the lidar edge point consists of two independent points, it imposes constraints on the external parameters. In addition, formula (7) establishes a nonlinear equation related to the lidar edge point LPi, the normal vector ni, and the measurement value of the corresponding point qi, which is used to solve the external parameter LCT. To solve this nonlinear equation, an iterative method is used, starting from the current estimate LCT^ and using the definition in SE(3) The operation parameterizes LCT in the tangent space of LCT^, expressed as: Substituting equation (8) into (7) and approximating the resulting equation with a first-order term, we can obtain: in, The dilation factor Ei has been described in detail above. In the case of depth-continuous edges and intensity-discontinuous edges located on a plane, Ei is considered to be zero. Formula (10) illustrates the constraints related to edge alignment. The set of all such edge correspondences leads to the formation of a comprehensive constraint that can be used to establish a maximum likelihood problem that simultaneously minimizes the variance associated with the extrinsic parameters. Subsequently, the Gauss-Newton method is used to optimize the problem, and the Ceres Solver is used as a computational tool to iteratively optimize the extrinsic parameters until they converge to the set threshold.

[0015] After adopting the above technical scheme, the beneficial effects of the present invention are: the algorithm can not only effectively deal with the noise problem caused by discontinuous edges of depth and intensity, but also maintain high matching accuracy and stability under various environmental conditions, with good accuracy and robustness, and can provide strong support for subsequent multi-sensor fusion systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 It is a system overview diagram of the present invention.

[0018] Figure 2 It is a physical picture and design drawing of the experimental platform in the present invention.

[0019] Figure 3 It is a schematic diagram of multiple feature edges in the present invention.

[0020] Figure 4 It is a schematic diagram of the drawing points caused by the light spot caused by the divergence angle of the laser radar in the present invention.

[0021] Figure 5 It is a schematic diagram of the expansion point caused by the light spot caused by the divergence angle of the laser radar in the present invention.

[0022] Figure 6 It is a schematic diagram of edge points projected from the laser radar to the image plane in the present invention.

[0023] Figure 7 It is a schematic diagram of the calibration scene in the present invention (AEF is an outdoor scene, BCD is an indoor scene).

[0024] Figure 8 This is a schematic diagram of the calibration results corresponding to different scenarios in the present invention. Figure 1(The punctuation marks in each group of punctuation marks on the figure from left to right are Scene1, Scene2, Scene3, Scene4, Scene5, Scene6 and Reference).

[0025] Fig. 9 This is a schematic diagram of the calibration results corresponding to different scenarios in the present invention. Figure 2 (The punctuation marks in each group of punctuation marks on the figure from left to right are Scene1, Scene2, Scene3, Scene4, Scene5, Scene6 and Reference). DETAILED DESCRIPTION

[0026] See also Figure 1 As shown, the technical solution adopted in this specific implementation is: a method for calibrating external parameters of a laser radar and a camera in a target-free environment, characterized in that it includes the following specific steps:

[0027] S1, first, start the lidar and camera to collect data at the same time to obtain the radar point cloud and camera image at the same time, extract the depth continuous edge, depth discontinuous edge and intensity discontinuous edge from the radar point cloud, and extract the natural edge feature from the camera image;

[0028] S2, matching the extracted radar edge features with the camera edge features to generate multi-feature edges: To improve accuracy, a noise model that takes radar spots into account is established, the expansion error of depth discontinuous edges is calculated, and compensation is performed during the optimization process. By minimizing the reprojection error, the external parameters of the lidar and camera are iteratively adjusted using a nonlinear optimization algorithm until convergence to the optimal solution.

[0029] S3, through experiments to verify the performance of the algorithm in a variety of complex environments, so as to verify the reliability and robustness of the algorithm.

[0030] In the multi-feature edge extraction stage, initialization preparation is first performed. The transformation relationship between the laser radar coordinate system and the visual camera coordinate system is first derived from the CAD design drawing of the experimental platform as the initial external parameter, and the camera is pre-calibrated with internal parameters using the MATLAB toolbox. Then, data is collected from the laser radar and the visual camera. For non-repetitive scanning solid-state laser radars such as Lanwo's AVIA laser radar, it is left to stand for about 30 seconds to accumulate point clouds, and an image is obtained using the visual camera while it is still. You can refer to Figure 2 :

[0031] The specific steps of S1 are:

[0032] 1) After data collection is completed;

[0033] 2) First, voxelize the LiDAR point cloud to build a structured spatial model, from which accurate and deep continuous edges are extracted;

[0034] 3) Next, these accumulated 3D point clouds are converted into depth continuous edges, and two key types of discontinuous edges are extracted in the 2D image: depth discontinuous edges and intensity discontinuous edges;

[0035] Deep continuous edges: These edges represent smooth transitions and connections on the surface of an object, but pixel gradient changes in the image are usually not obvious;

[0036] Deep discontinuous edges: These edges divide different foreground-background transitions, and the pixel change gradient in the image is large, making them easy to extract;

[0037] Intensity discontinuity edges: These edges are highlighted by intensity changes due to material or color changes, revealing the intrinsic characteristics of the object's surface.

[0038] Analyzing the intensity changes on a uniform plane can effectively extract depth continuous edges, depth discontinuous edges, and intensity discontinuous edges. For multi-feature edges, see Figure 3 shown.

[0039] The extraction of the depth continuous edge, the depth discontinuous edge and the intensity discontinuous edge is specifically as follows:

[0040] For the extraction of depth continuous edges, depth continuous edges are extracted efficiently and accurately based on point cloud voxel cutting and plane fitting; in the process of extracting depth discontinuous edges and intensity discontinuous edges, the data is first mapped to a spherical coordinate system to better adapt to the inherent measurement geometry of the lidar. Subsequently, this spherical representation is converted to a Cartesian coordinate system to generate a two-dimensional intensity image. In this image, each pixel represents a cluster of points in 3D space, and the average intensity value of these points is used to color each pixel while retaining its depth information. Then, the CANNY edge detection algorithm is applied to the intensity map to extract all edges. The KD-Tree based filtering mechanism is then used to distinguish depth discontinuous edges and intensity discontinuous edges on the plane according to locality and depth changes. The depth discontinuous edges and intensity discontinuous edges extracted in this way are then remapped to the original lidar point cloud, in which all extracted edges are merged to obtain multi-feature edges.

[0041] This method makes full use of various edge features in the calibration scene and is performed entirely in the lidar coordinate system, avoiding zero-value mapping and multi-value mapping problems, which significantly enhances the robustness and accuracy of the calibration algorithm, making it suitable for a wider range of application scenarios.

[0042] 4) In order to further analyze and process these features, the two types of discontinuous edges extracted from the two-dimensional image are remapped back to the original three-dimensional lidar point cloud and fused with the previously extracted deep continuous edges, and finally the three-dimensional point cloud data containing multi-feature edges is obtained in the lidar coordinate system;

[0043] 5) At the same time, the classic edge detection algorithm is applied to extract natural edge features from the camera image. These edge features often contain rich semantic information in complex scenes, such as object contours and environmental structures.

[0044] 6) After completing the multi-feature edge extraction, the extracted data is matched and iteratively optimized.

[0045] The S2 is specifically as follows: using the initial external parameters and the pre-calibrated camera internal parameters, the multi-feature three-dimensional edge in the laser radar coordinate system is projected onto the image plane in the camera coordinate system, and by matching the projected laser radar edge with the natural edge features in the camera image, the corresponding residual is constructed to lay the foundation for subsequent parameter optimization. In order to further improve the matching accuracy of the system, not only the inherent noise in the laser radar and camera measurement process is considered, but also the spot problem formed by the divergence angle of the laser radar in different scenes is modeled and analyzed. Due to the divergence effect of the laser beam, expansion and drawing errors will be introduced at the depth discontinuous edge, which may cause local inconsistency of the point cloud. This error is quantified by an accurate spot model, and the expansion deviation of the laser radar at the depth discontinuous edge is calculated. The deviation is compensated in the optimization process to reduce its influence on the external parameter calibration. On the basis of deviation compensation and edge matching, a nonlinear optimization problem of minimizing the reprojection error is further constructed, and an efficient nonlinear optimization algorithm is used to iteratively adjust the external parameter parameters of the laser radar and the camera. Through multiple iterations, until convergence to the optimal solution, the external parameter relationship between the laser radar and the camera is accurately estimated.

[0046] The iterative optimization includes edge matching, measuring noise and calibration formula and optimization.

[0047] The edge matching is specifically as follows:

[0048] The edge matching and optimization process is performed in the image space of the camera coordinate system. The edge points in the lidar coordinate system are projected onto the image plane of the camera coordinate system using the following formula:

[0049]

[0050] C p i =π( L P i ), (2)

[0051] p i =f( C p i ), (3)

[0052] Where CPi and LPi represent 3D points in the camera and lidar coordinate systems, respectively. Formula (1) represents the rigid body transformation T applied to point LPi, where R and t represent the rotation matrix and translation vector, respectively. The symbol pi∈R2 represents the position of the point in the camera image space. The function π(CPi) is the camera projection model, and f(CPi) is the camera distortion model. These two models together reflect the internal transformation from CPi=[Xi,Yi,Zi]T to pi=[xi,yi]T.

[0053] In the KD tree generated by the edge pixels of the image, a query is initiated for point pi to find its nearest κ neighbors, and these κ nearest neighbors are recorded as Si = {qij; j = 1,…,κ}. These relationships are defined as follows:

[0054]

[0055] Under this condition, the line represented by Si is determined by the points qi lying on the line, and is accompanied by a normal vector ni, which is the eigenvector associated with the smallest eigenvalue of Si and summarizes the direction of the line in space.

[0056] The measurement noise is specifically:

[0057] The laser radar edge points and corresponding edge features extracted from the image will be affected by the measurement noise. For the inherent measurement noise of the laser radar and the camera, a common noise model is used. For the camera measurement noise, the noise Iwi is modeled as a normally distributed variable N(0,IΣi), which will disturb the edge point features qi during the image edge extraction process. For the points extracted from the laser radar, the noise associated with these points is recorded as Lwi, and the corresponding mathematical expression is as follows:

[0058]

[0059] in,

[0060] Among them, the relationship between the real point position LPigt and its measured value LPi is obtained, ωi∈S2 represents the measured azimuth, and δωi~N(02×1,Σωi) represents the measurement noise on the ωi tangent plane, and it is assumed to be normally distributed, di represents the depth measurement value, and δdi~N(0,Σdi) represents the ranging error associated with it, which is also assumed to obey the normal distribution;

[0061] On this basis, the measurement deviation caused by the divergence angle of the LiDAR beam is further introduced. In the simplified subsequent optimization model, the shape of the LiDAR beam is approximated as a circle for ease of calculation.

[0062] In the subsequent optimization process, the measurement deviation caused by the divergence angle of the LiDAR beam was taken into account, see Figure 4 As shown, since the laser pulse is not a perfect point but a beam with a divergent angle, when scanning from the foreground to the background object, part of the laser pulse will be reflected by the foreground, and the rest will be reflected by the background, generating two reflected pulses that reach the lidar receiver. When the foreground object has a high reflectivity, even if the center line of the beam has exceeded the foreground object, the signal from the first pulse still dominates, resulting in a false point outside the actual edge of the foreground object. When the distance between the foreground and the background is close, the signals of the two pulses will merge to generate a comprehensive signal, generating a series of points connecting the foreground and the background, which are called "drawing points";

[0063] See also Figure 5 As shown, the dilation points cause abnormal intensity values ​​at the edges of depth discontinuities, affecting their accuracy and making the edges look dilated. Although the wire drawing points are also abnormal points, they have little effect on the intensity map of the LiDAR;

[0064] The uncertainty of the depth discontinuity edge caused by the expansion point caused by the LiDAR beam follows a biased Gaussian distribution (μ, LΣi). When the centerline of the LiDAR beam is exactly on the depth discontinuity edge, the measured value is the depth of the foreground object, which is the key position for generating false points. As the centerline of the beam moves upward, if the foreground object is still within the range of the beam, ideally, the measurement value should return the value of the background object reached by the centerline of the beam, but due to the measurement principle of the LiDAR, the measurement value will still return the value of the foreground object. When the circular beam is exactly tangent to the depth discontinuity edge (see Figure 4-Figure 5 As described above, the LiDAR still measures the value of the foreground object, which corresponds to the expansion point farthest from the actual point cloud depth discontinuity edge. When the center line of the LiDAR beam moves further up and the beam is completely on the background object, the returned measurement value is the value of the background object.

[0065] The deviation mean μ varies with the position of the center line of the beam. When the center line of the beam is tangent to the edge of the foreground object, μ = 0. When the entire beam is tangent to the foreground object, μ = ei(di, θ), where ei represents the radius of the circular beam, which is determined by the depth measurement di and the divergence angle θ of the lidar beam. It is worth noting that when extracting edges from the intensity image of the lidar, the extracted depth discontinuity edges are the expansion points farthest from the true edges. These points are where the entire beam is tangent to the foreground object. These points conform to the biased Gaussian distribution (μ, LΣi). For these expansion points on the depth discontinuity edges, if they are regarded as actual points of the depth discontinuity edges, the following relationship should be satisfied:

[0066]

[0067] Where E i =e i V,V=N×L

[0068] Vector V is obtained by performing a cross product (×) operation between vectors N and L, and is orthogonal to both N and L, where N is the normal vector to the plane where the edge lies, and L is the vector from the center of the beam perpendicular to the actual edge of the surface.

[0069] The calibration formula and optimization are specifically as follows:

[0070] See formula 1-formula 3 and Figure 6 As shown, let LPi be an edge point obtained from the lidar point cloud. In the image, the edge corresponding to this point is identified by the normal ni (in S1), and there is a corresponding point qi∈R2 on this edge. When the noise in LPi is taken into account and the point is mapped to the image plane using the verified external parameters, it should be accurately aligned with the edge represented by (ni,qi) in the image. The ideal observation model is as follows:

[0071]

[0072] where the noise components associated with the lidar and camera measurements are denoted as Lwi and Cwi, respectively. Formula (7) shows that since the lidar edge point consists of two independent points, it imposes constraints on the external parameters. In addition, formula (7) establishes a nonlinear equation related to the lidar edge point LPi, the normal vector ni, and the measurement value of the corresponding point qi, which is used to solve the external parameter LCT. To solve this nonlinear equation, an iterative method is used, starting from the current estimate LCT^ and using the definition in SE(3) The operation parameterizes LCT in the tangent space of LCT^, expressed as:

[0073]

[0074] Substituting equation (8) into (7) and approximating the resulting equation with a first-order term, we can obtain:

[0075]

[0076] in The dilation factor Ei was described in detail above. In the case of depth-continuous edges and intensity-discontinuous edges located on a plane, Ei is considered to be zero. Formula (10) illustrates the constraints related to edge alignment. The set of all such edge correspondences leads to the formation of a comprehensive constraint that can be used to establish a maximum likelihood problem that simultaneously minimizes the variance associated with the extrinsic parameters. The problem is then solved using the Gauss-Newton method for optimization, and the Ceres Solver is used as a computational tool to iteratively optimize the extrinsic parameters until they converge to our threshold.

[0077] The following is the algorithm testing process for indoor and outdoor scenes in the present invention:

[0078] See also Figure 7 As shown, pictures of indoor and outdoor scenes are obtained. Figure 8-Figure 9 As can be seen, in various scenarios, the external parameters from different initial values ​​converge and approximate the reference values, which proves the consistency, robustness and accuracy of the algorithm in different environments.

[0079] The above description is only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for calibrating external parameters of a laser radar and a camera in a target-free environment, characterized in that: It includes the following specific steps: S1, first, start the lidar and camera to collect data at the same time to obtain the radar point cloud and camera image at the same time, extract the depth continuous edge, depth discontinuous edge and intensity discontinuous edge from the radar point cloud, and extract the natural edge feature from the camera image; S2, matching the extracted radar edge features with the camera edge features to generate multi-feature edges: To improve accuracy, a noise model that takes radar spots into account is established, the expansion error of depth discontinuous edges is calculated, and compensation is performed during the optimization process. By minimizing the reprojection error, the external parameters of the lidar and camera are iteratively adjusted using a nonlinear optimization algorithm until convergence to the optimal solution; S3, through experiments to verify the performance of the algorithm in a variety of complex environments, so as to verify the reliability and robustness of the algorithm.

2. The method for calibrating external parameters of a laser radar and a camera in a target-free environment according to claim 1, characterized in that: The specific steps of S1 are: 1) After data collection is completed; 2) First, voxelize the LiDAR point cloud to build a structured spatial model, from which accurate and deep continuous edges are extracted; 3) Next, these accumulated 3D point clouds are converted into depth continuous edges, and two key types of discontinuous edges are extracted in the 2D image: depth discontinuous edges and intensity discontinuous edges; 4) In order to further analyze and process these features, the two types of discontinuous edges extracted from the two-dimensional image are remapped back to the original three-dimensional lidar point cloud and fused with the previously extracted deep continuous edges, and finally the three-dimensional point cloud data containing multi-feature edges is obtained in the lidar coordinate system; 5) At the same time, the classic edge detection algorithm is applied to extract natural edge features from the camera image. These edge features often contain rich semantic information in complex scenes; 6) After completing the multi-feature edge extraction, the extracted data is matched and iteratively optimized.

3. The method for calibrating external parameters of a laser radar and a camera in a target-free environment according to claim 1, characterized in that: The S2 is specifically: Using the initial extrinsic parameters and the pre-calibrated camera intrinsic parameters, the multi-feature three-dimensional edges in the lidar coordinate system are projected onto the image plane in the camera coordinate system. By matching the projected lidar edges with the natural edge features in the camera image, the corresponding residual is constructed, laying the foundation for subsequent parameter optimization. In order to further improve the matching accuracy of the system, not only the inherent noise in the lidar and camera measurement process is considered, but also the spot problem formed by the divergence angle of the lidar in different scenes is modeled and analyzed. Due to the divergence effect of the laser beam, expansion and drawing errors will be introduced at the depth discontinuous edge, which may lead to local inconsistency of the point cloud. This error is quantified through an accurate spot model, and the expansion deviation of the lidar at the depth discontinuous edge is calculated. The deviation is compensated in the optimization process to reduce its impact on the external parameter calibration. On the basis of deviation compensation and edge matching, a nonlinear optimization problem that minimizes the reprojection error is further constructed, and an efficient nonlinear optimization algorithm is used to iteratively adjust the extrinsic parameters of the lidar and camera. Through multiple iterations, until convergence to the optimal solution, the extrinsic relationship between the lidar and the camera is accurately estimated.

4. The method for calibrating external parameters of a laser radar and a camera in a target-free environment according to claim 1, characterized in that: The extraction of the depth continuous edge, the depth discontinuous edge and the intensity discontinuous edge is specifically as follows: For the extraction of depth continuous edges, based on point cloud voxel cutting and plane fitting, the depth continuous edges are extracted efficiently and accurately; In the process of extracting depth discontinuous edges and intensity discontinuous edges, the data is first mapped to a spherical coordinate system to better adapt to the inherent measurement geometry of the lidar. Subsequently, this spherical representation is converted to a Cartesian coordinate system to generate a two-dimensional intensity image. In this image, each pixel represents a cluster of points in 3D space, and the average intensity value of these points is used to color each pixel while retaining its depth information. The CANNY edge detection algorithm is then applied to the intensity map to extract all edges. The KD-Tree based filtering mechanism is then used to distinguish between depth discontinuous edges and intensity discontinuous edges on the plane based on locality and depth changes. The depth discontinuous edges and intensity discontinuous edges extracted in this way are then remapped to the original lidar point cloud, where all extracted edges are merged to obtain multi-feature edges.

5. The method for calibrating external parameters of a laser radar and a camera in a target-free environment according to claim 1, characterized in that: The iterative optimization includes edge matching, measuring noise and calibration formula and optimization.

6. The method for calibrating external parameters of a laser radar and a camera in a target-free environment according to claim 5, characterized in that: The edge matching is specifically as follows: The edge matching and optimization process is performed in the image space of the camera coordinate system. The edge points in the lidar coordinate system are projected onto the image plane of the camera coordinate system using the following formula: C p i =π( L P i ), (2) p i =f( C p i ), (3) Where CPi and LPi represent 3D points in the camera and lidar coordinate systems, respectively. Formula (1) represents the rigid body transformation T applied to point LPi, where R and t represent the rotation matrix and translation vector, respectively. The symbol pi∈R2 represents the position of the point in the camera image space. The function π(CPi) is the camera projection model, and f(CPi) is the camera distortion model. These two models together reflect the internal transformation from CPi=[Xi,Yi,Zi]T to pi=[xi,yi]T. In the KD tree generated by the edge pixels of the image, a query is initiated for point pi to find its nearest κ neighbors. These κ nearest neighbors are denoted as Si = {qij; j = 1,…,κ}. These relationships are defined as follows: Under this condition, the line represented by Si is determined by the points qi lying on the line, and is accompanied by a normal vector ni, which is the eigenvector associated with the smallest eigenvalue of Si and summarizes the direction of the line in space.

7. The method for calibrating external parameters of a laser radar and a camera in a target-free environment according to claim 5, characterized in that: The measurement noise is specifically: The laser radar edge points and corresponding edge features extracted from the image will be affected by the measurement noise. For the inherent measurement noise of the laser radar and the camera, a common noise model is used. For the camera measurement noise, the noise Iwi is modeled as a normally distributed variable N(0,IΣi), which will disturb the edge point features qi during the image edge extraction process. For the points extracted from the laser radar, the noise associated with these points is recorded as Lwi, and the corresponding mathematical expression is as follows: L P i gt ≈ C P i - L w i , (5) in, Among them, the relationship between the real point position LPigt and its measured value LPi is obtained, ωi∈S2 represents the measured azimuth, and δωi~N(02×1,Σωi) represents the measurement noise on the ωi tangent plane, and it is assumed to be normally distributed, di represents the depth measurement value, and δdi~N(0,Σdi) represents the ranging error associated with it, which is also assumed to obey the normal distribution; On this basis, the measurement deviation caused by the divergence angle of the LiDAR beam is further introduced. In the simplified subsequent optimization model, the shape of the LiDAR beam is approximated as a circle for ease of calculation. In the subsequent optimization process, the measurement deviation caused by the divergence angle of the LiDAR beam was taken into account. Since the laser pulse is not a perfect point, but a beam with a divergence angle, when scanning from the foreground to the background object, part of the laser pulse will be reflected by the foreground, and the rest will be reflected by the background, generating two reflected pulses to reach the LiDAR receiver. When the foreground object has a high reflectivity, even if the center line of the beam has exceeded the foreground object, the signal from the first pulse still dominates, which leads to the appearance of false points outside the actual edge of the foreground object. When the distance between the foreground and the background is close, the signals of the two pulses will merge to generate a composite signal, generating a series of points connecting the foreground and the background, which are called "drawing points"; Dilation points cause abnormal intensity values ​​at the edges of depth discontinuities, affecting their accuracy and making the edges look dilated. Although brushed points are also abnormal points, they have little effect on the intensity map of the lidar; The uncertainty of the depth discontinuity edge caused by the expansion point caused by the LiDAR beam follows a biased Gaussian distribution (μ, LΣi). When the centerline of the LiDAR beam is exactly on the depth discontinuity edge, the measured value is the depth of the foreground object, which is the key position for generating false points. As the centerline of the beam moves upward, if the foreground object is still within the range of the beam, ideally, the measured value should return the value of the background object reached by the centerline of the beam, but due to the measurement principle of the LiDAR, the measured value will still return the value of the foreground object. When the circular beam is exactly tangent to the depth discontinuity edge, the LiDAR still measures the value of the foreground object, which corresponds to the expansion point farthest from the depth discontinuity edge of the actual point cloud. When the centerline of the LiDAR beam moves further upward and the beam is completely on the background object, the returned measurement value is the value of the background object. The deviation mean μ varies with the position of the center line of the beam. When the center line of the beam is tangent to the edge of the foreground object, μ = 0. When the entire beam is tangent to the foreground object, μ = ei(di, θ), where ei represents the radius of the circular beam, which is determined by the depth measurement di and the divergence angle θ of the lidar beam. It is worth noting that when extracting edges from the intensity image of the lidar, the extracted depth discontinuity edges are the expansion points farthest from the true edges. These points are the positions where the entire beam is tangent to the foreground object. These points conform to the biased Gaussian distribution (μ, LΣi). For these expansion points on the depth discontinuity edges, if they are regarded as actual points of the depth discontinuity edges, the following relationship should be satisfied: Among them, E i =e i V, V = N × L, where V is the vector obtained by performing a cross product (×) operation between vectors N and L, and is orthogonal to both N and L. Here, N is the normal vector of the plane where the edge lies, and L is the vector perpendicular from the center of the beam to the actual edge of the surface.

8. The method for calibrating external parameters of a laser radar and a camera in a target-free environment according to claim 1, characterized in that: The calibration formula and optimization are specifically as follows: Referring to Equations 1-3 and Figure 6, let LPi be an edge point obtained from the lidar point cloud. The edge corresponding to this point is identified by the normal ni (in S1), and there is a corresponding point qi∈R2 on this edge. When the noise in LPi is taken into account and the point is mapped to the image plane using the verified external parameters, it should be accurately aligned with the edge represented by (ni,qi) in the image. The ideal observation model is as follows: where the noise components associated with the lidar and camera measurements are denoted as Lwi and Cwi, respectively. Formula (7) shows that since the lidar edge point consists of two independent points, it imposes constraints on the external parameters. In addition, formula (7) establishes a nonlinear equation related to the lidar edge point LPi, the normal vector ni, and the measurement value of the corresponding point qi, which is used to solve the external parameter LCT. To solve this nonlinear equation, an iterative method is used, starting from the current estimate LCT^ and using the definition in SE(3) The operation parameterizes LCT in the tangent space of LCT^, expressed as: Substituting equation (8) into (7) and approximating the resulting equation with a first-order term, we can obtain: in The dilation factor Ei has been described in detail above. In the case of depth-continuous edges and intensity-discontinuous edges located on a plane, Ei is considered to be zero. Formula (10) illustrates the constraints related to edge alignment. The set of all such edge correspondences leads to the formation of a comprehensive constraint that can be used to establish a maximum likelihood problem that simultaneously minimizes the variance associated with the extrinsic parameters. Subsequently, the Gauss-Newton method is used to optimize the problem, and the Ceres Solver is used as a computational tool to iteratively optimize the extrinsic parameters until they converge to the set threshold.

Images