A robust line-based camera-radar calibration method without target

Abstract

This invention relates to intelligent driving technology and multi-sensor calibration, and discloses a robust targetless camera-radar calibration method based on line segments. The initial attitude is obtained using orthogonal dominant directions and straight lines in a single scene. A multi-constraint-driven mismatch elimination method is introduced to remove outliers, utilizing collinearity constraints to improve outlier removal capability. To enhance the smoothness of the flow space, smoothing flow constraints are applied to the assumed correspondences in an arc-based regenerating kernel Hilbert space. Local constraints are introduced to preserve the triangular topology between adjacent vector arcs generated from interior points. The unknown variables of the MCMM are solved using an expectation-maximization algorithm. Subsequently, given the linear correspondences of the MCMM, calibration parameters are estimated through iterative refinement. Simulations and practical experiments demonstrate that this method has excellent performance.

Description

Technical Field

[0001] This invention relates to intelligent driving technology and the field of multi-sensor calibration, and in particular to a robust targetless camera-radar calibration method based on line segments. Background Technology

[0002] With the rapid development of autonomous driving and mobile robotics technologies, perception systems typically utilize multiple sensors to acquire various environmental information, including distance measurement and object extraction. LiDAR and cameras are commonly used sensors. While cameras provide visual information that aligns with human sensory experience, they are susceptible to changes in lighting and cannot directly capture 3D spatial data, limiting their practicality. In contrast, LiDAR provides 3D environmental data unaffected by lighting changes, but lacks color and texture information. Therefore, integrating these two sensors is crucial for vehicle perception or robot localization.

[0003] External calibration between LiDAR and cameras is an indispensable prerequisite for multi-sensor fusion. Currently, existing research mainly falls into two categories: target-based calibration and targetless calibration. Although target-based calibration methods have made significant progress in accuracy and robustness, they still have some shortcomings in terms of user experience. On the one hand, these methods heavily rely on manual targets. On the other hand, sensor drift caused by vibration often requires offline calibration, which is not only time-consuming but also very inconvenient. Considering these factors, targetless calibration methods are attracting increasing attention. Feature-based methods are commonly used for targetless camera-LiDAR calibration, with boundary features being particularly popular due to their stable extraction from point clouds and camera images. The main challenge in using edges as features for external estimation lies in establishing effective feature correspondences. Manual methods typically involve manually specifying boundary features in camera images and point clouds, which can be very time-consuming; while most edge-based automatic calibration methods either rely on edge alignment based on spatial geometry or use similarity for calibration. However, these methods heavily rely on initial guesses for given feature matching. Furthermore, due to modal differences and weakly correlated matching methods, the initial matching features often contain many incorrect matches. Current boundary costs or similarity metrics struggle to completely eliminate these erroneous correspondences, negatively impacting external calibration. Recent work has proposed solutions to these challenges, but some limitations remain. For example, parameter initialization is achieved using pairs of parallel lines and a single plane. However, in real-world scenarios, complex planar structures in challenging environments may be encountered. If the normal vectors of the obtained planes do not accurately represent the orientation of the parallel line clusters, rotation estimation may fail or become inaccurate. Furthermore, while the proposed local triangle-preserving matching method improves the inlier rate, the stringent inlier confirmation conditions result in low recall. Due to the limited number of correct line correspondences, calibration results may exhibit local underfitting. Summary of the Invention

[0004] To address the aforementioned problems, the purpose of this invention is to propose a robust targetless camera-radar calibration method based on line segments, which can accurately calibrate the rigid transformation between the camera and the radar.

[0005] The technical solution of this invention is as follows: A robust targetless camera-radar calibration method based on line segments, the specific steps of which are as follows:

[0006] Step 1, Feature Extraction: Using the structural environment as a reference target, statically acquire camera images and radar data; extract 3D boundary line segments from radar data using voxel cutting and plane fitting methods based on point clouds; extract 2D line segments from camera images using a line segment detector (LSD);

[0007] Step 2, Extrinsic parameter initialization: Obtain the initial extrinsic parameters between the camera and radar based on 2D line segments and 3D boundary line segments; identify the main parallel line directions through 3D boundary line segments and determine the initial rotation matrix by maximizing the interior points; obtain the initial translation matrix through grid search and outlier robust multi-scale function; and then construct the initial extrinsic parameters based on the initial rotation matrix and the initial translation matrix.

[0008] Step 3: Based on the initial extrinsic and intrinsic parameters, obtain the projection matrix P; project the 3D line segments onto the camera image according to the projection matrix to obtain 2D projection lines; use the nearest neighbor method to associate the 2D projection lines and the 2D line segments in the camera image to obtain 2D-3D matching line pairs; the 2D-3D matching line pairs contain erroneous matching pairs, and use the multi-constraint driven mismatch elimination method MCMM to remove erroneous matching pairs to obtain high-quality 2D-3D matching line pairs;

[0009] Step 4, Visualization: Based on the high-quality 2D-3D matching line pairs selected by MCMM, the calibration parameters are obtained using the Levenberg-Marquardt algorithm with mismatch removal; the 3D radar data from Step 1 is projected onto the camera image according to the calibration parameters, and the quality of the calibration parameters is judged by the visualized projection contour and image contour.

[0010] The initial rotation matrix is ​​obtained by rotation estimation based on the dominant direction of 2D-3D line segments;

[0011] 3D boundary segments are projected onto the normalized plane of the LiDAR and their normal vectors are determined. Based on the two endpoints and normal vectors of the projected segments, three mutually perpendicular direction vectors are derived using a hybrid vanishing point method. By recognizing that the directions of the 3D boundary segments associated with the 3D vanishing points are parallel to the directions of the 3D vanishing points, the set of 3D boundary segments corresponding to each 3D vanishing point is identified. The two 3D directions with the most 3D boundary segments are considered as the dominant horizontal directions of the 3D boundary segments. The dominant direction perpendicular to 3D line segments

[0012] Similar to calculating the dominant direction of 3D boundary line segments, the dominant 2D horizontal direction is determined based on the parallelism between the vanishing point of the camera image and the direction of the 2D line segment. and two-dimensional vertical dominant direction

[0013] Considering the dominant direction d of the 3D boundary line segment L The dominant direction d of the 2D line segment C The relationship between the two is defined as R. CL d L =d C This equation represents the dominant direction d of the 3D boundary line segment. L via RCL After transformation, the dominant direction d of the 2D line segment C Parallelism; when a rotation matrix can effectively align a three-dimensional direction with its corresponding two-dimensional direction, it is called a potential rotation matrix; for each potential rotation matrix, two sets of direction matching pairs are obtained, namely perpendicular matching pairs. and horizontal matching pairs d C Corresponding two-dimensional line clusters With d C Corresponding three-dimensional line clusters By linking them together, we obtain horizontal and vertical matching pairs. Given a potential rotation matrix and random translations, transform a 3D line segment Projected onto the image plane; based on the matching pairs The interior points are determined by thresholds for the orientation angle and distance between them; the potential rotation matrix with the largest number of interior points is selected as the initial rotation matrix R. CL .

[0014] The initial translation matrix is ​​obtained by translation estimation based on a multi-scale function;

[0015] The specific steps for obtaining the initial translation matrix t are as follows: Given an initial rotation matrix R CL ∈SO(3), the grid-based optimization method estimates the corresponding initial translation matrix t∈R 3 ; Due to R CL The pre-defined, grid-based optimization method reduces the search space to three degrees of freedom for translation, establishes an equidistant grid, and uses point-based reprojection error as the evaluation metric; each 3D point is uniformly sampled along the 3D boundary line segment, using an initial rotation matrix R. CL Translation within an equidistant grid projects 3D sampling points onto the camera image plane; a nearest neighbor method is used to establish a 2D-3D correspondence, with the nearest pixel of each projection point being determined as its matching point, thus generating a reprojection error e.

[0016] To ensure the estimation of translation parameters, the reprojection error is incorporated into the Gaussian field estimation framework. The following error function is applied to points on the radar 3D boundary line segment:

[0017]

[0018] Where U1 represents the number of matching points; U2 represents the scaling factor; The variance of the scaling factor is represented; this cost function with multi-scale variance is called the multi-scale function.

[0019] The initial translation matrix t is obtained by minimizing the proposed multi-scale function.

[0020] The multi-constraint driven mismatch removal method is specifically as follows:

[0021] Establish local constraints, collinearity constraints, and smooth flow constraints. Based on these three constraints, build a probabilistic mixture model and remove the 2D-3D matching line pair set. The outliers in the data are the incorrect matching pairs.

[0022] The probabilistic mixture model is as follows:

[0023]

[0024] in, and P = (p1,…,p N ) T Let E be a point mapped on the Gaussian sphere; E = (∈1,…,∈ N ) T This refers to the geometric matching error based on collinearity constraints. It is a 2D-3D matching line pair set The i-th corresponding point; It is the set of mapping points corresponding to local constraints, used to calculate the weighting factor. γ is the mixing coefficient, 0 ≤ γ ≤ 1; θ is the set of unknown parameters θ ∈ {θ1, θ2, γ}, and θ1 and θ2 represent Gaussian distributions respectively. and The unknown parameter; p represents the probability function; ∈ i Let a1 and a2 represent uniform distributions with areas a1 and a2, respectively; inlier refers to inliers and outliers refer to outliers.

[0025] To reduce overfitting, the regularized prior is represented as follows: λ、 f and f represent the regularization parameter, the Hilbert space of the regeneration kernel, and the learning function, respectively;

[0026] Using Bayes' rule, the posterior estimate is expressed as follows:

[0027]

[0028] Maximize the corresponding objective function formula (3) to obtain the optimal parameter set;

[0029] The objective function optimization process comprehensively considers collinearity constraints, smooth flow constraints, and local constraints to ensure that the final solution is both theoretically robust and accurate. Formula (3) is highly non-convex and involves a complex latent variable structure, transforming the maximum likelihood estimate of the parameter set θ into the minimization of the corresponding negative log-likelihood function. Based on this principle, the negative logarithm of the likelihood expression is taken to obtain the following functional form:

[0030]

[0031] in as well as These correspond to collinearity constraints, smooth flow constraints, and local constraints, respectively; φ(f) is the learning function f in the reproducing kernel Hilbert space. The square of the norm in.

[0032] The collinearity constraint specifically refers to: given a set of 2D-3D matching line pairs S and a projection matrix P, we obtain... in Indicates the projection line; if the line pairs If the interior points correspond, then under the true projection matrix, the projected lines... and camera image 2D line segments Lines that satisfy the collinearity constraint are called collinear pairs; outliers are identified by calculating the algebraic distance from a point to the line in a collinear pair.

[0033] The correspondence between two-dimensional and three-dimensional lines is transformed into a homogeneous linear equation system W. i h = 0; where W i Let h represent the known 2×18 coefficient matrix calculated from the i-th matching line, where h is an 18-dimensional column vector; consider two cases of line pairs: interior point correspondence and outlier point correspondence;

[0034] For the line pairs corresponding to the interior points In the presence of noise, the error of linear matching is ∈ i =W i h; Assume two-dimensional error ∈ i Follows a Gaussian distribution:

[0035]

[0036] Where σ² represents the standard deviation;

[0037] If the lines are... For outliers, the system of equations deviates from a linear structure; in this case, the error output is confined to a rectangular region of area a, where the error ∈ j =W j h follows a uniform distribution.

[0038] The smoothing constraint specifically refers to the i-th camera image line. and projection lines The back projection normal vectors are denoted as n. i and Intersecting at the center of the sphere; point p on the surface of the sphere i ∈S 2 and Representing the back projection normal vector n i and The other end; linear correspondence Points mapped onto the sphere In representing a vector arc When following the right-hand rule, the vector is arced. The direction is defined as the mapping point p from the camera image line. i Mapping point to the projection line Direction; arc Located in n i and On the large circle formed;

[0039] Following the mapping relationship from image to sphere, the two-dimensional line corresponds to... Points mapped onto the sphere Forming a motion field; the three-dimensional vector arcs of interior points exhibit continuous and smooth characteristics, while outliers do not; given a set of observation points P and a set of projection points... Construct a vector field f: Based on the characteristic that the vector field of outliers is not smooth while the vector field of interior points is smooth, outliers or interior points are identified, thereby effectively distinguishing between interior and outlier data.

[0040] Assume that the error associated with the interior points follows a Gaussian distribution, specifically: The errors of outliers follow a uniform distribution 1 / a1, representing the area of ​​their random distribution region; the Gaussian components of the probabilistic mixture model are described as follows:

[0041]

[0042] Where θ1={f,σ1} is the set of unknown parameters; D1 is the dimension of the point set P.

[0043] The local constraint specifically refers to: providing a 2D-3D matching line correspondence. Use two different projection matrices to project 3D line segments Projected onto a virtual plane to obtain line correspondence. and Mapping these 2D line correspondences to point correspondences on a sphere, on the first Gaussian sphere. The above is represented as Second Gaussian sphere The above is represented as The starting point of the arc vector is the mapping point p of the camera image line. i The endpoint of the arc vector is the mapping point of the projection line. or The direction of the arc vector is from p i point to or From the second Gaussian sphere Points on Move to the first Gaussian sphere Upwards, without changing its coordinates, forming an arc; this is formed by the arc vector. and The resulting arc-shaped structure represents a spherical triangle. The edge-angle-edge structure is connected by a line segment to form a spherical triangle; for N corresponding lines, a set of triangular pairs is constructed.

[0044] Based on the principle of invariance of spherical triangles, the correct weight formula is as follows: For each triplet Extract their H nearest neighbors, denoted as and And establish consistent neighborhood correspondences based on neighborhood. Intersection The i-th line correspondence and its neighboring triplet T i Mapping triples Spherical triangles are constructed on the surface of a sphere. To evaluate the similarity of triangles in a neighborhood, the edge-to-edge ratio of the spherical triangle is used as a metric, as defined below:

[0045]

[0046] Where |·| is the length of the arc;

[0047] If the mapping point and its adjacent elements can maintain the invariance of the triangle within the local region, then the above three ratios are close to 1; therefore, the similarity of adjacent triangles is measured by the following formula:

[0048] s △ (i,t)=min(a′(i,t)) / max(a′(i,t)) (8)

[0049] Where a′(i,t)={a1′(i,t),a2′(i,t),a3′(i,t)}, s △ (i,t)∈[0,1]; as s △ As the value of (i,t) approaches 1, the similarity between adjacent triangles increases, making it more likely that the line correspondence is an interior point; based on this, a triangular cost function is introduced for the i-th correspondence. This is used to evaluate the structural consistency of local triangular topologies; to increase the number of reliable correspondences, K iterations are needed to obtain multiple initial costs. And take the average of these initial cost functions; therefore, the correct weights are expressed as:

[0050]

[0051] The probabilistic mixture model solves for the optimal parameter set using the EM algorithm.

[0052] The EM algorithm alternates between the expectation step E and the maximization step M, gradually increasing the likelihood function value to approximate the optimal parameter set θ.

[0053] Binary hidden variable z i Let z represent the i-th corresponding state. i =1 means "inlier", while z i =0 represents "outlier"; based on this modeling method, the probabilistic mixture model in equation (2) is further decomposed into:

[0054]

[0055] Where p(z) i =1)=γ,p(z i =0) =1-γ; then the log-likelihood of the complete data is expressed as:

[0056]

[0057] (1) E-step: According to Bayes' theorem, the posterior probability that the term corresponding to the i-th line is an interior point is:

[0058]

[0059] in

[0060] (2) M-step: In this step, the unknown parameter set θ′ is updated by minimizing equation (11); the parameters are found. To find the optimal value, calculate the partial derivative and set it to 0, we get:

[0061]

[0062] In the same way, obtain the parameters. And γ, as shown below:

[0063]

[0064] Secondly, consider the relationship with h. And transform it into a weight matrix as Least squares method:

[0065]

[0066] The first term, the learning function f, is a non-rigid transformation that maps the moving point to the target point in the reproducing kernel Hilbert space; the second term, as a global regularization term, enhances the smoothness and consistency of the learning function f; according to motion coherence theory, the learning function f has the form of a Gaussian radial basis function, where... The Gaussian radial basis functions are used to construct the kernel in the regenerating kernel Hilbert space; the vector field structure is an arc, and the smoothness of the kernel space Γ is encoded by the arc length; by applying the vector-valued representation theorem, the optimal mapping for any given point is calculated using the following equation:

[0067]

[0068] Where d(·,·) represents the arc length, and c i These are the coefficients that need to be solved; rewritten as:

[0069]

[0070] Where Γ is an N×N matrix. Its (i,j)th element is Taking the derivative of C with respect to the coefficient vector, we obtain the following system of linear equations:

[0071]

[0072] The Levenberg-Marquardt algorithm (LMO) with mismatch removal first performs a global pose optimization on all initial matching line pairs to obtain a more reasonable initial estimate. In each iteration, all matching pairs are sorted according to the residual size, and a small number of abnormal matches with the largest residuals are gradually removed. Through the iterative process of alternating "optimization and removal", the purity of the matching set is gradually improved, and finally the set of high-confidence line segment matching pairs with the smallest error is retained.

[0073] The beneficial effects of this invention are as follows: Addressing three limitations of traditional targetless calibration methods: a) the need for known initial guesses, b) the impact of outliers on matching features affecting calibration accuracy, and c) the low recall rate of recently proposed local triangle-preserving matching methods, resulting in only a limited number of correct linear correspondences during the refinement stage. This limitation increases the risk of parameter estimation degradation. To solve these problems, a robust targetless camera-radar calibration method based on line segments is proposed. First, to enhance the versatility of the initialization method in complex planar structural environments, we utilize only line segments to estimate initial parameters. On one hand, a cluster of 3D line segments is used to identify the main parallel line directions, and the initial rotation is determined by maximizing the interior points. On the other hand, the initial translation is obtained through grid search, and a multi-scale initialization function is developed to mitigate the influence of outliers. Second, regarding mismatch elimination, our goal is to eliminate as many incorrect correspondences as possible without sacrificing the most accurate line correspondences. To achieve this goal, we introduce a multi-constraint-driven mismatch elimination method. This method involves converting outlier-contaminated line correspondences into arc vectors on a Gaussian sphere. Notably, vectors formed by inliers exhibit a consistent flow pattern, while vectors formed by outliers show irregular variations. Leveraging this observation, a maximum likelihood Bayesian framework is employed to associate each correspondence with a latent variable. The expectation-maximization (EM) algorithm is then used to estimate the inlier and outlier sets. To effectively identify outliers, three constraints are introduced. As corresponding spatial transformation constraints, a commonality constraint with linear transformation and a smooth flow constraint with nonlinear transformation are introduced. It is assumed that line correspondences satisfying these constraints follow a Gaussian distribution; otherwise, they follow a uniform distribution. To fully utilize the behavior patterns of arc vectors, local constraints are used as prior weights to guide inlier selection. These local constraints preserve the local structure of adjacent inlier vector arcs. Finally, the effectiveness of this method is verified through testing on Liu_data, Structural_dataset, and a self-recorded dataset. Attached Figure Description

[0074] Figure 1 This is a flowchart illustrating a targetless camera-radar calibration method based on line segments.

[0075] Figure 2 This is a schematic diagram of the main process of a specific embodiment of the method of the present invention.

[0076] Detailed Implementation Figure 1 This is the main flowchart of the technical solution of the present invention.

[0077] For artificially structured environments, a robust targetless camera-radar calibration method based on line segments is invented. This method has two significant advantages: (1) no initial parameters are required, (2) true line matching can be obtained from data containing outliers, and (3) the quality of interior point data can be improved while retaining most correct matches. To achieve this goal, we first obtain the initial parameters of the camera-radar from the structural regularities of the structured scene. The specific operation is as follows: candidate rotation matrices are obtained using the horizontal and vertical dominant directions of 2D-3D lines, and the initial rotation matrix is ​​obtained using the maximum interior point method. Given the initial rotation matrix and parallel lines, the initial translation is obtained by optimizing the three degrees of freedom of the translation mesh and then minimizing the multi-scale function.

[0078] To eliminate false matches, a coarse-to-fine approach is used to find correct correspondences. In the coarse matching stage, line correspondences are mapped to vector arcs on a Gaussian sphere. We observe that the relative directions of adjacent vector arcs mapped from outliers may change significantly, but the neighborhood topology between inliers is well preserved, and the inliers exhibit consistent fluidity. To measure this regularity, three constraints are introduced and embedded in a mixture model. Binary classification of the mixture model is used to identify unknown inlier correspondences. In the fine matching stage, the goal is to remove correspondences with large residuals. A false match removal method based on Levenberg-Marquardt (LM) optimization is also introduced. This method iteratively filters correspondences with large noise or errors by considering the projection constraints from 3D lines to 2D lines in the image plane. Finally, if a line correspondence passes the above checks, it is considered a correct match.

[0079] like Figure 1 As shown, the robust targetless camera-radar calibration method based on line segments proposed in this invention includes the following steps:

[0080] Step 1: Feature Extraction. Ensuring the accuracy and robustness of extrinsic parameter calibration relies on the quality of features extracted from both sensors. Therefore, an environment with rich edge line structures was selected as the calibration reference scene to provide stable and repeatable feature support. In 3D space, a method combining voxel partitioning and plane fitting based on point clouds was used for line feature extraction. Specifically, the point cloud was first divided into regular voxel units, and local plane fitting was performed within each voxel to extract 3D boundary line segment features from adjacent plane boundaries. Compared to traditional methods, this strategy maintains structural continuity while possessing good geometric representation capabilities. In the image domain, 2D line segment extraction was based on the LSD (LineSegment Detector) algorithm, which has advantages such as high computational efficiency and stable accuracy, and can effectively detect real edge line segments in complex image backgrounds. Through the above feature extraction process, we ensured that the line segment features obtained from the two sensors have a good spatial structural correspondence, thus laying a solid foundation for subsequent matching and calibration processes.

[0081] Step 2, Extrinsic Parameter Initialization: The 2D and 3D lines are registered by selecting the maximum number of neighboring points between the projected line and nearby 2D lines. However, this method relies on initial guesses. To improve the versatility and robustness of the initialization method in environments with complex planar structures, we designed an initial parameter estimation strategy that relies solely on line segment features. For example... Figure 1 As shown, the structured environment of the Manhattan world is a scene with dominant horizontal and vertical directions, corresponding to the two dominant vanishing directions in the 2D image. In the rotation initialization stage, line segment clusters in 3D space are first extracted, and their dominant parallel line directions are identified by analyzing their main distribution directions. Subsequently, initial rotation parameters are estimated based on an optimization criterion that maximizes the number of points within a line segment. In the translation initialization stage, a grid search strategy is employed to find the optimal translation solution in the candidate space. Simultaneously, to further suppress interference caused by noise or anomalous structures, a multi-scale initialization function is introduced to regulate the parameter estimation process at multiple scale levels, effectively reducing the impact of outliers on the final result.

[0082] Step 2-a. Rotation estimation based on 2D-3D dominant orientation pairs

[0083] To obtain the dominant 3D orientation, a hybrid vanishing point method was employed (Li, Haoang et al. "Quasi-Globally Optimal and Near / True Real-Time Vanishing Point Estimation in Manhattan World." IEEE Transactions on Pattern Analysis and Machine Intelligence 44(2020):1503-1518). This method, initially designed to estimate vanishing points in images, relies solely on the endpoints and normal vectors of line segments. While primarily used for vanishing point estimation, it can efficiently and almost in real-time estimate 3D orientation angles in Manhattan structures. The process involves projecting 3D line segments onto a LiDAR-normalized plane and determining the normal vectors of the projected line segments. By utilizing the two endpoints and normal vectors of the projected line segments as input, three mutually perpendicular orientation vectors are derived. Recognizing that the 3D line directions associated with the orientation angles are almost parallel to the 3D orientation angles, the set of line segments corresponding to each orientation angle can be identified. Subsequently, the two orientation vectors with the highest number of 3D boundary line segments are considered as the dominant 3D horizontal orientations. and vertical dominant direction For two-dimensional images, a hybrid vanishing point method is used to estimate the vanishing point. Similar to the process of identifying the dominant three-dimensional direction, the parallelism between the vanishing point and the direction of the two-dimensional line segment is used to determine the dominant two-dimensional horizontal direction. and vertical dominant direction

[0084] Considering the dominant two-dimensional orientation of the image With the three-dimensional dominant direction The relationship between them remains unknown, and potential solutions can only be determined through combination methods. It has been determined that the dominant 2D direction corresponding to the vanishing point is a ray, while the dominant 3D direction linked to the 3D line segment has two opposite directions. Therefore, there are a total of 8 candidate rotation matrices; in the initial rotation estimation process, candidate matrices with lower correlation are first screened using direction constraints. Specifically, considering the d of the 3D parallel lines on the image... L With the d of the two-dimensional line segment C They are related. The relationship between them can be defined as R CL d L =d C This equation indicates that the three-dimensional line direction d L via R CL After transformation, with respect to the two-dimensional line direction d CParallel. Without loss of generality, a rotation matrix is ​​called a latent rotation matrix if it can effectively align a three-dimensional direction with its corresponding two-dimensional direction. For each latent rotation matrix, two sets of direction matching pairs are obtained. These are perpendicular matching pairs. and horizontal matching pairs At the same time, d C Corresponding two-dimensional line clusters With d C Corresponding three-dimensional line clusters Connect them. Obtain matching pairs of horizontal and vertical lines. Subsequently, under the conditions of potential rotation and random translation, the three-dimensional line segment is... Projected onto the image plane. Criteria for determining interior points include evaluating line pairs. The orientation angle and distance between them. Then, the rotation matrix with the most interior points is selected as the initial rotation estimate R. CL .

[0085] Step 2-b. Translation estimation based on multi-scale function

[0086] First, an initialization step is performed to obtain the initial rotation matrix; second, each 3D line segment is uniformly sampled, and the 2D-3D correspondence is obtained using the nearest neighbor method; finally, the estimated initial translation matrix t is obtained by minimizing the proposed multi-scale function.

[0087] Given an initial rotation matrix R CL For ∈SO(3), the corresponding initial translation matrix t∈R can be efficiently estimated using a grid-based optimization method. 3 Because of R CL The pre-determined grid-based optimization method reduces the search space to three degrees of freedom in translation, thus significantly improving computational efficiency. Effectively evaluating the quality of each parameter set is crucial during the translation parameter search process. This requires rigorously evaluating calibration quality using a cost function. Given that the projected radar 3D boundary segments are closely aligned with the corresponding 2D image contours when using calibration parameters, they essentially share similar spatial locations. Therefore, point-based reprojection error is used as the evaluation metric. Specifically, each 3D point is uniformly sampled along a 3D collinear segment. An initial rotation matrix R is used... CL The point is projected onto a two-dimensional image plane by an optimized translation t. To establish a two-dimensional-to-three-dimensional correspondence, the nearest pixel of each projected point is determined as its matching point, thus generating a reprojection error e.

[0088] To ensure accurate estimation of the translation parameters, the projection error is incorporated into the Gaussian field estimation framework. Without loss of generality, the following error function is applied to points on the radar 3D boundary segment:

[0089]

[0090] Here, U1 represents the number of matching points; U2 represents the scaling factor. This cost function with multi-scale variance is called the multi-scale initialization function. A larger variance covers all matching features, thus dominating the global orientation; while a smaller variance increases the weight of correctly matched features, effectively mitigating the influence of orientation determined by outliers. In short, a lower multi-scale initialization function results in LiDAR boundaries that are closer to the image contour, thus producing a better translation vector.

[0091] Step 3: When the -D line is projected onto the image plane with initial parameters, the 2D-3D line registration problem is solved by searching for the nearest line with the most pixels. Unlike points and lines in the image, there is no dedicated descriptor between the projected line and the observed line, which may lead to mismatches in assumed correspondences. False correspondences are considered outliers in pose optimization and reduce the accuracy of calibration. It is necessary to remove these erroneous correspondences; based on the initial transformation obtained in Step 2, the line segments in 3D space are projected onto the camera image plane to obtain the corresponding 2D line segments. Subsequently, point sets are uniformly sampled at certain intervals on all 2D and 3D line segments, and a nearest neighbor search strategy is adopted to achieve coarse matching between 3D and 2D line segments. To remove erroneous matches, a multi-constraint driven erroneous match removal method is proposed.

[0092] Step 3-a. Problem Description and Generation of Three Constraints

[0093] Given a set of line correspondences These correspondences are corrupted due to mismatches. Our goal is to remove outliers in S. Driven by various effective constraints that can distinguish between outliers and normal points in S, we propose a robust method based on multi-constraint driving information to eliminate mismatches. Collinearity constraint: Given the linear correspondence S and the projection matrix P, we obtain... in Indicates the projection line. If the line is a pair If the interior points correspond, then under the true projection matrix, the projected lines... and observation line Collinearity constraints are satisfied; these line pairs are called collinear matches. Collinear matches mean that outliers can be identified by calculating the algebraic distance from a point to a line. However, the above outlier removal methods are not practical for external calibration. To address this issue and reduce reliance on prior information about unknown parameters, the classic Perspective-n-Line (PnL) method is employed. This method involves transforming the two-dimensional-to-three-dimensional line correspondences into a homogeneous linear system of equations W. i h = 0. Where W iLet represent the known 2×18 coefficient matrix calculated from the i-th matching line. h is an 18-dimensional column vector. The advantage of this linear equation is that it does not depend on external parameters. Then, consider two cases for line pairs: interior point correspondence and exterior point correspondence. For interior point pairs... In the presence of noise, the error of linear matching is ∈ i =W i h. Assume two-dimensional error ∈ i Follows a Gaussian distribution:

[0094]

[0095] Where σ represents the standard deviation. On the other hand, if the line pair For anomalies, the system of equations deviates from a linear structure. In this case, the error output is confined to a rectangular region of area a, where the error ∈ j =W j h follows a uniform distribution.

[0096] Smoothness Constraints: To better understand the smoothness constraints corresponding to a straight line, it is necessary to first introduce the mapping relationship between a planar straight line and a point on a sphere. If the i-th image line l i and projection lines The back projection normal vectors are denoted as n. i and They intersect at the center of the sphere. Point p on the surface of the sphere. i ∈S 2 and Representing the unit normal vector n i and The other end. Therefore, the linear correspondence. Points mapped onto the sphere In representing a vector arc When following the right-hand rule, the vector is arced. The direction is defined as the direction from the mapping point p of the image line. i Mapping point to the projection line The direction. Clearly, the arc. Located in n i and On the large circle formed.

[0097] Following the mapping relationship from image to sphere, the two-dimensional line corresponds to... Points mapped onto the sphere, corresponding to A motion field is formed. The three-dimensional vector arcs of interior points exhibit coherent and smooth characteristics, while outliers do not. This phenomenon is called smooth spatial flow, and therefore it is necessary to introduce it into the three-dimensional vector field on a Gaussian sphere. Given a set of observation points P and a set of projection points... The goal is to construct a vector field f: This vector field is used to interpolate inliers, effectively distinguishing inliers from data containing outliers. It is assumed that the errors associated with inliers follow a Gaussian distribution, specifically... The errors of outliers follow a uniform distribution 1 / a1, representing the area of ​​their randomly distributed region. The Gaussian components of the mixture model can be described as follows:

[0098]

[0099] Where θ1 = {f, σ1} is the set of unknown parameters. D1 is the dimension of the point set P.

[0100] Local constraints: Specifically, given a line correspondence... Use two different projection matrices to project 3D lines Projected onto a virtual plane to obtain line correspondence. and Then these 2D line correspondences are mapped to point correspondences on the sphere, on the first Gaussian sphere. The above is represented as Second Gaussian sphere The above is represented as The starting point of the arc vector is the mapping point p of the camera image line. i The endpoint of the arc vector is the mapping point of the projection line. (or The direction of the arc is from p i point to (or To create a triangular topology, from the second Gaussian sphere... Points on Move to the first Gaussian sphere Upwards, without changing its coordinates, forming an arc. This is formed by the arc vector. and The resulting arc-shaped structure represents a spherical triangle. The edge-angle-edge structure is formed by corresponding line segments to create a spherical triangle. For N corresponding lines, a set of triangular pairs can be constructed.

[0101] Based on the principle of invariance of spherical triangles, the correct weight formula is as follows: For each triplet First, extract their H nearest neighbors, denoted as HH. and And establish consistent neighborhood correspondences based on neighborhood. Intersection The i-th line correspondence and its neighboring triplet T i Mapping triples Spherical triangles can be formed on a sphere. To evaluate the similarity of triangles in a neighborhood, the edge-to-edge ratio of spherical triangles is used as a metric, specifically defined as follows:

[0102]

[0103] Where |·| is the length of the arc.

[0104] If the mapped point and its adjacent elements can maintain the invariance of the triangle within the local region, then the above three ratios should be close to 1. Therefore, the similarity of adjacent triangles can be measured by the following formula:

[0105] s △ (i,t)=min(a′(i,t)) / max(a′(i,t)) (5)

[0106] Where a′(i,t)={a1′(i,t),a2′(i,t),a3′(i,t)}, s △ (i,t)∈[0,1]. As s △ As the value of (i,t) approaches 1, the similarity between adjacent triangles increases, making it more likely that the line correspondence is an interior point. Based on this, a triangular cost function is introduced for the i-th correspondence. This is used to evaluate the structural consistency of local triangular topologies. To increase the number of reliable correspondences, K iterations are needed to obtain multiple initial costs. And take the average of these initial cost functions. Therefore, the correct weights are expressed as:

[0107]

[0108] Finally, multiple constraint information is integrated to infer model probabilities, improving the effectiveness of mismatch removal. Therefore, the mismatch removal problem can be formulated as a probabilistic mixture model, with the following form:

[0109]

[0110] Where 0≤γ≤1, and P = (p1,…,p N ) T Let E be a point mapped onto the Gaussian sphere. E = (∈1,…,∈ N ) T This refers to the geometric matching error based on collinearity constraints. It is a line correspondence set The i-th corresponding point. It is the set of mapping points corresponding to local constraints, used to calculate the weighting factor. γ is the mixing coefficient. θ is the set of unknown parameters θ∈{θ1,θ2,γ}.

[0111] Furthermore, to reduce overfitting, the regularized prior can be expressed as: Using Bayes' rule, the posterior estimate can be expressed as:

[0112]

[0113] Maximize the corresponding objective function Eq.(7) to obtain the optimal parameter set. This optimization process comprehensively considers collinearity constraints, smooth flow constraints, and local constraints to ensure that the final solution is both theoretically robust and accurate. Generally, the maximum likelihood estimate of the parameter set θ can be transformed into minimizing the corresponding negative log-likelihood function. Based on this principle, taking the negative logarithm of the likelihood expression in Eq.(7) yields the following functional form:

[0114]

[0115] in as well as These correspond to collinearity constraint, smooth flow constraint, and local constraint, respectively.

[0116] Step 3-b. EM Algorithm

[0117] In probabilistic mixture models, directly using Equation (8) to solve for parameters often presents optimization challenges. Specifically, the likelihood function is usually highly nonconvex and contains a complex latent variable structure, making the analytical derivation of the global optimum extremely difficult. To effectively address these challenges, the Expectation-Maximization (EM) algorithm is introduced, which solves the optimization problem iteratively. The EM algorithm alternates between expectation steps (E-steps) and maximization steps (M-steps), gradually increasing the likelihood function value to approximate the optimal parameter set θ.

[0118] The EM algorithm typically introduces latent variables to represent matching relationships. In our problem, the defined function needs to determine whether each correspondence is an interior or exterior point. Therefore, a binary latent variable z is introduced. i Let z represent the i-th corresponding state. i =1 means "inlier", while z i =0 represents "outlier". Based on this modeling method, the probabilistic mixture model in equation (7) is further decomposed into:

[0119]

[0120] Where p(z) i =1)=γ,p(z i =0) =1-γ. Then the log-likelihood of the complete data can be expressed as:

[0121]

[0122] (1) E-step: According to Bayes' theorem, the posterior probability that the term corresponding to the i-th line is an interior point is:

[0123] in

[0124] (2) M-step: In this step, the unknown parameter set θ is updated by minimizing Eq.(11). To find the parameters... To find the optimal value, calculate the partial derivative and set it to 0, we get:

[0125]

[0126] In the same way, the parameters can be obtained. And γ, as shown below:

[0127]

[0128] Secondly, consider the relationship with h. And transform it into a weight matrix as Least squares method:

[0129]

[0130] The first term, the learning function f, is a non-rigid transformation that maps the moving point to the target point in the reproducing kernel Hilbert space (RKHS). The second term, as a global regularization term, enhances the smoothness and consistency of the learning function f. According to motion coherence theory, the learning transformation f has the form of Gaussian radial basis functions (GRBF), where... The GRBF is used to construct the kernel in RKHS. The vector field structure is an arc, and the arc length can be used to encode the smoothness of the kernel space Γ. By applying the vector-valued representation theorem, the optimal mapping for any given point is computed using the following equation:

[0131]

[0132] Where d(·,·) represents the arc length. i These are the coefficients that need to be solved for. Therefore, it can be rewritten as:

[0133]

[0134] Where Γ is an N×N matrix. The (i,j)th element is... Taking the derivative of C with respect to the coefficient vector, we obtain the following system of linear equations:

[0135]

[0136] Step 4: Refine parameters and visualize:

[0137] Although EM filtering was used to optimize the line segment matching results in the preprocessing stage, a certain number of outliers may still remain in the line segment correspondences due to environmental complexity and sensor noise. These outlier matches will adversely affect the accuracy and stability of extrinsic parameter estimation, so it is necessary to introduce a more robust extrinsic parameter calibration strategy. To this end, a robust pose estimation method combining LM (Levenberg-Marquardt) optimization and iterative outlier removal is proposed, abbreviated as LMO (LM-based Optimization with outlier removal). This method first performs a global pose optimization on all initially matched line segment pairs to obtain a more reasonable initial estimate. Subsequently, in each iteration, all matching pairs are sorted according to the residual size, and a small number of outlier matches with the largest residuals are gradually removed. Through this iterative process of alternating "optimization-removal", the purity of the matching set is gradually improved, and finally, the set of high-confidence line segment matching pairs with the smallest error is retained. This strategy not only improves the robustness of extrinsic parameter estimation, but also significantly improves the calibration accuracy in complex scenarios. Based on the correct 2D-3D line pairs, calibration parameters are obtained using the Levenberg-Marquardt algorithm with mismatch removal; the 3D point cloud is projected onto the camera image, and the quality of the calibration parameters is judged by the visualized projection contour and image contour.

[0138] like Figure 2As shown, this process encompasses key steps such as sensor data acquisition, preprocessing, feature association, and outlier removal, systematically improving calibration accuracy and robustness. Specifically, the system uses statically captured camera images and LiDAR data as input. It receives raw multimodal data as the starting point of the calibration process. Before feature association, a series of preprocessing steps must be performed on the raw data. The preprocessing module includes distortion correction and image point cloud feature extraction. When the initial extrinsic parameters are available (denoted as "Y"), the system enters the line matching stage. This stage requires multi-scene data acquisition to ensure sufficient line features are obtained. If the initial parameters are unavailable ("N"), the system enters the external initialization module. To further enhance the robustness of extrinsic parameter estimation, MCMM is introduced to remove outliers. MCMM jointly considers three types of constraints and improves the quality of line data through a soft classification strategy under the EM framework. Then, a nonlinear optimization method for outlier removal is used to optimize the extrinsic parameters, improving calibration accuracy. Finally, to ensure the accuracy and reliability of the current calibration results, the system performs a projection-based verification step. To enhance the distinguishability of the projection effect, we assign different colors to the projection points based on their depth. When there are errors in the calibration parameters, the 3D projection points of an object may not align with the corresponding pixels, causing objects at the same depth to appear in different colors. This difference is particularly noticeable in regions with discontinuous depth.

[0139] Comparative experiments using non-repeating scanning lidar qualitatively and quantitatively demonstrate that the method of this invention is superior to other targetless methods, with an average error of 0.8 and a variance of less than 0.7.

Claims

1. A robust targetless camera-radar calibration method based on line segments, characterized in that, The specific steps are as follows: Step 1, Feature Extraction: Using the structural environment as a reference target, statically acquire camera images and radar data; extract 3D boundary line segments from the radar data based on point cloud voxel cutting and plane fitting methods; 2D line segments are extracted from the camera image using a line segment detector (LSD). Step 2, Extrinsic parameter initialization: Obtain the initial extrinsic parameters between the camera and radar based on 2D line segments and 3D boundary line segments; identify the main parallel line directions through 3D boundary line segments and determine the initial rotation matrix by maximizing the interior points; obtain the initial translation matrix through grid search and outlier robust multi-scale function; and then construct the initial extrinsic parameters based on the initial rotation matrix and the initial translation matrix. Step 3: Based on the initial extrinsic and intrinsic parameters, obtain the projection matrix P; project the 3D boundary line segments onto the camera image according to the projection matrix to obtain 2D projection lines; use the nearest neighbor method to associate the 2D projection lines and the 2D line segments in the camera image to obtain 2D-3D matching line pairs; the 2D-3D matching line pairs contain erroneous matching pairs, and use the Multi-Constraint Driven Mismatch Elimination Method (MCMM) to remove erroneous matching pairs to obtain high-quality 2D-3D matching line pairs; Step 4, Visualization: Based on the high-quality 2D-3D matching line pairs selected by MCMM, the calibration parameters are obtained using the Levenberg-Marquardt algorithm with mismatch removal; the 3D radar data from Step 1 is projected onto the camera image according to the calibration parameters, and the quality of the calibration parameters is judged by the visualized projection contour and image contour. The initial rotation matrix is ​​obtained by rotation estimation based on the dominant direction of 2D-3D line segments; 3D boundary segments are projected onto the normalized plane of the LiDAR and their normal vectors are determined. Based on the two endpoints and normal vectors of the projected segments, three mutually perpendicular direction vectors are derived using a hybrid vanishing point method. By recognizing that the directions of the 3D boundary segments associated with the 3D vanishing points are parallel to the directions of the 3D vanishing points, the set of 3D boundary segments corresponding to each 3D vanishing point is identified. The two 3D directions with the most 3D boundary segments are considered as the dominant horizontal directions of the 3D boundary segments. The dominant direction perpendicular to 3D line segments ; Similar to calculating the dominant direction of 3D boundary line segments, the dominant 2D horizontal direction is determined based on the parallelism between the vanishing point of the camera image and the direction of the 2D line segment. and two-dimensional vertical dominant direction ; Considering the dominant direction of 3D boundary line segments The dominant direction of 2D line segments Related, the relationship between the two is defined as = This equation represents the dominant direction of the 3D boundary line segment. through After transformation, the dominant direction of the 2D line segment Parallel; when a rotation matrix can effectively align a three-dimensional direction with its corresponding two-dimensional direction, it is called a potential rotation matrix; for each potential rotation matrix, two sets of direction matching pairs are obtained, namely, perpendicular matching pairs < , >and horizontal matching pairs< , >; will Corresponding two-dimensional line clusters and Corresponding three-dimensional line clusters By linking them together, we obtain horizontal and vertical matching pairs. , Given a potential rotation matrix and random translations, transform a 3D line segment... Projected onto the image plane; based on the matching pair < , The interior points are determined by thresholds for the orientation angle and distance between them; the potential rotation matrix with the largest number of interior points is selected as the initial rotation matrix. .

2. The robust targetless camera-radar calibration method based on line segments according to claim 1, characterized in that, The initial translation matrix is ​​obtained by translation estimation based on a multi-scale function; The specific steps for obtaining the initial translation matrix t are as follows: Given an initial rotation matrix... ∈SO(3), a grid-based optimization method estimates the corresponding initial translation matrix t∈R. 3 ;because The pre-defined, grid-based optimization method reduces the search space to three degrees of freedom for translation, establishes an equidistant grid, and uses point-based reprojection error as the evaluation metric; each 3D point is uniformly sampled along the 3D boundary line segment, using an initial rotation matrix. Translation within an equidistant grid projects 3D sampling points onto the camera image plane. A nearest neighbor method is used to establish a 2D-3D correspondence, with the nearest pixel of each projected point being its matching point. This process introduces reprojection errors. ; To ensure the estimation of translation parameters, the reprojection error is incorporated into the Gaussian field estimation framework. The following error function is applied to points on the radar 3D boundary line segment: （1） in, Indicates the number of matching points; Indicates the scaling factor; The variance of the scaling factor is represented; this cost function with multi-scale variance is called the multi-scale function. The initial translation matrix t is obtained by minimizing the proposed multi-scale function.

3. The robust targetless camera-radar calibration method based on line segments according to claim 2, characterized in that, The multi-constraint driven mismatch removal method is specifically as follows: Establish local constraints, collinearity constraints, and smooth flow constraints. Based on these three constraints, build a probabilistic mixture model and remove the 2D-3D matching line pair set. The outliers in the data are the incorrect matching pairs. The probabilistic mixture model is as follows: （2） in, and For the point mapped onto the Gaussian sphere; This refers to the geometric matching error based on collinearity constraints. It is a 2D-3D matching line pair set The i-th corresponding point; It is the set of mapping points corresponding to local constraints, used to calculate the weighting factor. ; It is the mixing coefficient. ; It is a set of unknown parameters ,and and Representing Gaussian distributions respectively and Unknown parameters; Represents a probability function; For error, and They represent areas of and uniform distribution Point of view, Points of abnormality; To reduce overfitting, the regularized prior is represented as follows: , , and Let represent the regularization parameter, the regeneration kernel Hilbert space, and the learning function, respectively; Using Bayes' rule, the posterior estimate is expressed as follows: （3） Maximize the corresponding objective function formula (3) to obtain the optimal parameter set; The objective function optimization process comprehensively considers collinearity constraints, smooth flow constraints, and local constraints to ensure that the final solution is both theoretically robust and accurate; Formula (3) is highly non-convex and involves a complex hidden variable structure, which will affect the parameter set. The maximum likelihood estimation is transformed into minimizing the corresponding negative log-likelihood function; based on this principle, taking the negative logarithm of the likelihood expression yields the following functional form: （4） in , ,as well as These are collinearity constraints, smooth flow constraints, and local constraints, respectively. To learn the function f in the reproducing kernel Hilbert space The square of the norm in.

4. The robust targetless camera-radar calibration method based on line segments according to claim 3, characterized in that, The collinearity constraint specifically refers to: given a 2D-3D matching line pair set S and a projection matrix P, obtaining... ,in Indicates the projection line; if the line pairs If the interior points correspond, then under the true projection matrix, the projected lines... and camera image 2D line segments Lines that satisfy the collinearity constraint are called collinear pairs; outliers are identified by calculating the algebraic distance from a point to the line in a collinear pair. The correspondence between two-dimensional and three-dimensional lines is transformed into a system of homogeneous linear equations. ;in, This represents the known information calculated from the i-th matching line. coefficient matrix, It is an 18-dimensional column vector; consider two cases of line pairs: correspondence between interior points and correspondence between outliers; For the line pairs corresponding to the interior points In the presence of noise, the error of linear matching is Assuming two-dimensional error Follows a Gaussian distribution: （5） in Indicates standard deviation; If the lines are... If the outlier corresponds to an outlier, then the system of equations deviates from a linear structure; in this case, the error output is constrained to an area of... Within the rectangular area, the error It follows a uniform distribution.

5. The robust targetless camera-radar calibration method based on line segments according to claim 3, characterized in that, The smooth flow constraint specifically refers to the i-th camera image line. and projection lines The back projection normal vectors are denoted as follows: and Points on the surface of the sphere intersect at the center of the sphere. and They represent the back projection normal vectors respectively. and The other end; linear correspondence Points mapped onto the sphere ; in representing vector arcs When following the right-hand rule, the vector is arced. The direction is defined as the mapping point from the camera image line. Mapping point to the projection line Direction; arc Located in and On the large circle formed; Following the mapping relationship from image to sphere, the two-dimensional line corresponds to... Points mapped onto the sphere This forms a motion field; the three-dimensional vector arcs of interior points exhibit continuous and smooth characteristics, while outliers do not; given a set of observation points... and a set of projection points Construct a vector field : Based on the characteristic that the vector field of outliers is not smooth while the vector field of interior points is smooth, outliers or interior points are identified, thereby effectively distinguishing between interior point and outlier data. Assume that the error associated with the interior points follows a Gaussian distribution, specifically: The errors of outliers follow a uniform distribution. , representing the area of ​​its randomly distributed region; the Gaussian components of the probabilistic mixture model are described as follows: （6） in It is a set of unknown parameters; It is a point set The dimension of.

6. The robust targetless camera-radar calibration method based on line segments according to claim 3, characterized in that, The local constraint specifically refers to: providing a 2D-3D matching line correspondence. Using two different projection matrices to project 3D line segments Projected onto a virtual plane to obtain line correspondence. and Mapping these 2D line correspondences to point correspondences on a sphere, on the first Gaussian sphere. The above is represented as On the second Gaussian sphere The above is represented as The starting point of the arc vector is the mapping point of the camera image line. The endpoint of the arc vector is the mapping point of the projection line. or The direction of the arc vector is from point to or From the second Gaussian sphere Points on Move to the first Gaussian sphere Upwards, without changing its coordinates, forming an arc; This is derived from the arc vector. and The resulting arc-shaped structure represents a spherical triangle. The edge-angle-edge structure is connected by a line segment to form a spherical triangle; for N corresponding lines, a set of triangular pairs is constructed. ; Based on the principle of invariance of spherical triangles, the correct weight formula is as follows: For each triplet Extract their H nearest neighbors, denoted as , and And establish consistent neighborhood correspondences based on neighborhood. Intersection The correspondence between the i-th line and its neighboring triples Mapping triples Spherical triangles are constructed on the surface of a sphere. To evaluate the similarity of triangles in a neighborhood, the edge-to-edge ratio of the spherical triangle is used as a metric, as defined below: （7） in It is the length of the arc; If the mapping point and its adjacent elements can maintain the invariance of the triangle within the local region, then the above three ratios are close to 1; therefore, the similarity of adjacent triangles is measured by the following formula: （8） in , ;along with As the value approaches 1, the similarity between adjacent triangles increases, making it more likely that the corresponding line is an interior point. Based on this, a triangular cost function is introduced for the i-th correspondence. This is used to evaluate the structural consistency of local triangular topologies; to increase the number of reliable correspondences, K iterations are needed to obtain multiple initial costs. And take the average of these initial cost functions; therefore, the correct weights are expressed as: （9）。 7. The robust targetless camera-radar calibration method based on line segments according to claim 3, characterized in that, The probabilistic mixture model solves for the optimal parameter set using the EM algorithm; The EM algorithm alternates between the expectation step (E steps) and the maximization step (M steps), gradually increasing the likelihood function value to approximate the optimal parameter set. ; Binary hidden variable Let i represent the corresponding state: It means "inlier", and The term "outlier" is used to represent the probabilistic mixture model in equation (2). Based on this modeling method, the probabilistic mixture model in equation (2) is further decomposed into: （10） in The log-likelihood of the complete data is then expressed as: （11） (1) E-step: According to Bayes' theorem, the posterior probability that the term corresponding to the i-th line is an interior point is: （12） in ; (2) M-step: In this step, the unknown parameter set is updated by minimizing equation (11). Find the parameters To find the optimal value, calculate the partial derivative and set it to 0, we get: （13） In the same way, obtain the parameters. and As shown below: （14） （15） Secondly, consider with Related And transform it into a weight matrix as Least squares method: （16） The learning function of the first term It is a non-rigid transformation that maps the moving point to the target point in the regenerating kernel Hilbert space; The second term, as a global regularization term, enhances the learning function. Smoothness and consistency; based on the theory of motion coherence, the learning function... It has the form of Gaussian radial basis functions, where Gaussian radial basis functions are used to construct the kernel in the regenerated kernel Hilbert space; the vector field structure is an arc, and the kernel space is encoded by the arc length. The smoothness; by applying the vector-valued representation theorem, the optimal mapping for any given point is calculated using the following equation: （17） in Indicates arc length, These are the coefficients that need to be solved; rewritten as: （18） in It is The matrix has an (i, j)th element as... ; ; Take the coefficient vector The derivative of is obtained by solving the following system of linear equations: （19）。 8. The robust targetless camera-radar calibration method based on line segments according to claim 1, characterized in that, The Levenberg-Marquardt algorithm (LMO) with mismatch removal first performs a global pose optimization on all initial matching line pairs to obtain a more reasonable initial estimate. In each iteration, all matching pairs are sorted according to the residual size, and a small number of abnormal matches with the largest residuals are gradually removed. Through the iterative process of alternating "optimization-removal", the purity of the matching set is gradually improved, and finally the set of high-confidence line segment matching pairs with the smallest error is retained.

Images