Multi-target optimization based camera and lidar extrinsic parameter joint calibration method

Abstract

The application discloses a camera and radar joint calibration method based on multi-target external parameter optimization. The method first divides the laser radar point cloud into voxels and fits a local plane, selects a pair of planes with a normal vector angle of 30°-150°, calculates the space intersection line, and retains points less than 3 cm from the intersection line as a three-dimensional depth continuous edge to suppress the false edge caused by point cloud inflation and multi-return aliasing. At the same time, the camera image is screened in two dimensions, the local PCA main direction and direction credibility weight of each edge are calculated, and the weak texture edge is weighted or removed. Then, a double-target optimization function is constructed, which includes the projection geometric distance error and the direction consistency error with the direction credibility weight. Finally, a multi-target optimization algorithm is adopted, the Pareto front is maintained through external archive and crowding degree mechanism, the six-degree-of-freedom external parameter is directly globally optimized, and a high-precision calibration result is obtained. The application does not require any calibration board and artificial initial value, and can realize a translation error less than 0.025 m and a rotation error less than 0.15° in a complex outdoor scene and a low-density solid-state laser radar point cloud, and is suitable for camera and laser radar online calibration.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot and multi-sensor fusion technology, specifically, a method for online calibration of external parameters of outdoor lidar and cameras without calibration plates. Background Technology

[0002] With the rapid development of computer technology, robotics, and artificial intelligence, the application demand for autonomous vehicle technology is growing daily. In the multi-sensor fusion system of autonomous vehicles, LiDAR and cameras are the two most core environmental perception devices. LiDAR can provide high-precision 3D point cloud information, reflecting the geometric structure of the environment, while cameras can capture rich texture and color information for semantic recognition and target classification. The complementary fusion of the two can significantly improve the robustness and accuracy of environmental perception.

[0003] However, the two sensors differ significantly in imaging mechanism, coordinate system definition, installation location, and sampling frequency. To achieve accurate spatial and temporal alignment between the image and point cloud, extrinsic parameter calibration is necessary. Precise geometric calibration is a prerequisite for multi-sensor data alignment and fusion. The goal of calibration is to align and fuse point cloud data and image data in spatial information, determining the extrinsic parameter matrix between the two sensors—that is, the rotation and translation relationship between the radar and camera coordinate systems—thereby achieving accurate projection of point cloud data onto the image plane.

[0004] Traditional calibration methods are mainly divided into two categories: those with targets and those without. Target-based methods (such as checkerboard or perforated plates) offer high accuracy but rely on specific props, making them unsuitable for online calibration in dynamic scenes and susceptible to lighting and occlusion in complex outdoor environments. Target-free methods utilize natural environmental features (such as edges and planes) to overcome prop dependence, but existing technologies still suffer from the following significant drawbacks: First, pseudo-edges often appear in LiDAR point clouds at object boundaries, caused by laser beam divergence angle, multi-echo aliasing, and point cloud expansion, leading to inaccurate feature extraction and severe mismatches. Second, most methods use only projected geometric distance as the sole optimization objective, making them insensitive to rotational errors and prone to producing low-precision results that are accurate in translation but off-center in rotation. Third, traditional gradient optimization (such as Ceres or Levenberg-Marquardt) is highly dependent on initial values, easily getting trapped in local optima when initial value perturbations are large (±8° / ±0.15m), failing to converge to a global solution. These issues result in the accuracy of existing targetless calibration being less than 0.05m / 0.5° in low-density solid-state lidar (such as Livox HAP) and complex outdoor scenarios, which cannot meet the needs of autonomous vehicles for "on-demand calibration and online correction". Summary of the Invention

[0005] The purpose of this invention is to provide a joint calibration method for camera and lidar extrinsic parameters based on multi-target optimization, in order to solve the defects of existing targetless calibration technology, such as severe pseudo-edge interference of lidar, insufficient rotation error constraints, and strong dependence of optimization algorithms on initial values, and to achieve high-precision and robust online calibration.

[0006] The technical solution to achieve the objective of this invention is: a joint calibration method for camera and lidar extrinsic parameters based on multi-target optimization. It includes the following steps:

[0007] The first step is to divide the lidar point cloud into voxel meshes. Within each voxel, a three-dimensional depth continuous edge extraction method based on local plane fitting and plane intersection line solution is used to extract the depth continuous edge to suppress false edges.

[0008] The second step is to perform two-dimensional edge extraction on the camera image, and to perform PCA analysis on the local region of each image edge to obtain the principal direction vector and the ratio of eigenvalues ​​to calculate the confidence weight, in order to prepare for the third step of constructing a bi-objective optimization function.

[0009] The third step is to construct a dual-objective optimization function, where the first objective is the sum of the pixel Euclidean distances between the three-dimensional depth continuous edge points after projection through the current extrinsic parameters and the nearest two-dimensional edge points, and the second objective is the cosine loss of the angle between the three-dimensional edge direction vector and the corresponding two-dimensional edge principal direction vector, and the second objective is weighted by the direction confidence weights obtained in the second step.

[0010] The fourth step involves using a multi-objective optimization algorithm to directly optimize the six-degree-of-freedom extrinsic parameters of the lidar and camera globally, establishing and maintaining the Pareto optimal set of non-dominated solutions, and selecting the solution with the smallest weighted synthesis error from the final Pareto front as the final extrinsic parameter calibration result of the lidar and camera after the iteration.

[0011] Furthermore, the specific implementation steps of the first step are as follows:

[0012] 1) First, the planar structure needs to be estimated within the local area of ​​the point cloud: First, the point cloud is spatially discretized by dividing it into voxel meshes, so that the point cloud within each voxel exhibits local smoothness; then, the point set contained in each voxel... A local planar model is fitted, and the angle between adjacent planes is calculated, laying the foundation for subsequent depth-continuous edge extraction. The calculation method is as follows:

[0013]

[0014] in It represents the angle between two planes. and These are the normal vectors of the two planes.

[0015] 2) When When the angle is too small or too large, it usually indicates that the two plane pairs are almost parallel or form a sharp angle. Such structures often do not constitute reliable edges for detection in real-world scenarios. Therefore, we next apply an angle constraint to the included angle to filter out plane pairs that do not conform to the true boundary characteristics. The constraint method is as follows:

[0016]

[0017] in For the minimum angle, The maximum angle avoids false intersections within the plane while ensuring that common structural boundaries in architectural scenes can be correctly identified.

[0018] 3) For the plane pairs obtained through angle constraint filtering Further solve for its spatial intersection line The direction vector of the intersection line is the cross product of the normal vectors of the two planes, and any point on the intersection line can be obtained by solving a simultaneous equation for the planes. To generate practical edge points usable for projection and matching, points close to the intersection line from the original point cloud need to be selected as depth-continuous edge points. The specific calculation method is as follows:

[0019]

[0020]

[0021]

[0022]

[0023] in Point coordinates , and These are the normal vectors of the two planes. and This is the offset. As the base point of the intersection, The direction vector of the intersection line is the cross product of the normal vectors of the two planes. This is a scaling factor for the extension direction of the intersection line. The coordinates of a point in a point cloud. For point cloud points to the intersection line The shortest distance, This is a set of depth-continuous edge points. Downsampling based on dynamic voxel networks is performed to improve the uniformity of point cloud edge distribution and reduce the overhead of subsequent projection calculations. The resulting depth edge set exhibits clear geometric boundary features.

[0024] Furthermore, in the second step, the specific steps for performing two-dimensional edge filtering on the camera image are as follows:

[0025] 1) To obtain reliable edge direction information, the input image is first subjected to Gaussian filtering to effectively suppress noise interference and preserve edge details. Then, image gradient magnitude calculation combined with non-maximum suppression is used to extract a binary edge map with clear structure and good continuity. Based on this, the gradient components in the x and y directions are calculated separately. By combining the gradient information in both directions, the initial direction representation of the image edges is accurately calculated and determined. The specific calculation method is as follows:

[0026]

[0027]

[0028] in For image representation, for gradient of direction, for Gradient of direction. This represents the initial direction of the image edges.

[0029] 2) After obtaining the initial edge map in the previous step, a local statistical method is further used to enhance the robustness of the direction estimation: For each edge point, a fixed-size neighborhood window is taken, and the image coordinates of all edge points within the window are collected. First, the mean vector is calculated as the neighborhood center, and then the covariance matrix is ​​constructed. Subsequently, the matrix is ​​decomposed into eigenvalues ​​to obtain the eigenvector corresponding to the largest eigenvalue, which is the more structurally meaningful principal direction vector at that edge point. The specific calculation method is as follows:

[0030]

[0031]

[0032]

[0033] in For edge points The mean coordinates of the local neighborhood point set. The number of neighborhood point set points. The coordinates of the neighborhood point set point. Let covariance matrix be the variance matrix. Indicates the first 1 eigenvector Indicates the first Each feature value.

[0034] 3) Calculate the directional confidence weight using the two eigenvalues ​​obtained in step 2). A larger confidence weight indicates a more pronounced linear structure in the marginal distribution, while a smaller weight indicates a more scattered marginal distribution. The specific calculation method is as follows:

[0035]

[0036] in As a credibility weight, As a regulating factor, and These are the eigenvalues ​​obtained from the covariance matrices in steps 2) and 3 respectively. Through the above process, a direction vector is obtained for each valid edge point. With directional credibility These two elements work together to construct the consistency constraints with the edge direction of the point cloud, providing high-quality directional feature information for extrinsic parameter calibration.

[0037] Furthermore, in the third step, the specific implementation steps for constructing the bi-objective optimization function model are as follows:

[0038] 1) After extracting the continuous edges of the point cloud depth and the direction of the image edges, a extrinsic parameter optimization model that can simultaneously reflect spatial location consistency and structural orientation consistency is further constructed, i.e., a dual-objective modeling based on geometric distance error and orientation consistency error. The specific construction steps are as follows:

[0039] a) For any point cloud edge point, obtain its coordinates in the camera coordinate system using the rotation matrix and translation vector. Then, based on the camera pinhole imaging model, project the point cloud coordinates onto the pixel plane coordinates. The specific calculation method is as follows:

[0040]

[0041]

[0042] in, These are the coordinates of the point cloud in the camera coordinate system. It is the rotation transformation matrix between the camera coordinate system and the radar coordinate system. These are the coordinates of the point cloud in the radar coordinate system. It is the translation transformation vector between the camera coordinate system and the radar coordinate system. These are the coordinates of the projection point. This represents the focal length components in the horizontal and vertical directions. This represents the coordinates of the principal point in the image.

[0043] b) Calculate the pixel distance between the projected point and its nearest neighbor in the image edge set to reflect the degree of spatial misalignment caused by extrinsic parameters. The total geometric distance error is defined as the total degree of spatial misalignment, i.e., the sum of all pixel distances. The specific calculation method is as follows:

[0044]

[0045]

[0046] in, Indicates the first A distance of one pixel, Indicates the projection point. Indicates the nearest neighbor point. This represents the geometric distance error.

[0047] c) Based on the previously extracted edge direction features from the 3D point cloud, and then obtaining the local principal direction from the image edge neighborhood using PCA, the angle between the two is calculated to reflect the accuracy of the rotation. The specific calculation method is as follows:

[0048]

[0049] in, For the first The angle between the edge direction of a point and the local principal direction of its neighborhood. For the first The edge direction of each point, For the first The local principal direction of the edge neighborhood of each point.

[0050] d) Calculate the orientation confidence weights to eliminate the interference of regions with weak textures and blurred edges on the orientation error, and finally calculate the final orientation error. The specific calculation method is as follows:

[0051]

[0052]

[0053] in, It is the first The directional credibility weight of each point and It is the first PCA eigenvalues ​​at each point As a regulating factor, It is the final direction error. For the first The angle between the edge direction of a point and the local principal direction of the edge neighborhood.

[0054] 2) The two types of errors are jointly constructed into a multi-objective extrinsic parameter optimization problem, but instead of directly simplifying it to a single objective function, the independence of the two objectives is maintained. The specific calculation method is as follows:

[0055]

[0056] in, For geometric distance error, For directional error, These are external parameters.

[0057] 3) Using Pareto optimality theory, we search for a set of non-dominated solutions to explore all solutions in the extrinsic parameter space that can achieve a reasonable balance between the two objectives.

[0058] Furthermore, in the fourth step, the specific implementation steps for using a multi-objective optimization algorithm to directly perform global optimization on the six degrees of freedom extrinsic parameters of the lidar and camera, and output the final extrinsic parameter calibration results of the lidar and camera, are as follows:

[0059] 1) Initialize the gray wolf population, for For each gray wolf in the sequence 1,…,N, a random position vector is generated, and the fitness value of each gray wolf is calculated using the biobjective function constructed in step three. The specific calculation method is as follows:

[0060]

[0061]

[0062] in, Let be the rotation angle about the three coordinate axes. To translate from radar to camera, For the first The dual-target value of Sekiro: Shadows Die Twice. For the first Sekiro's position vector For geometric distance error, This represents the direction error. In each iteration, the gray wolf's position will represent a set of extrinsic parameters. The extrinsic parameter search interval is set to a rotation of ±10° and a translation of ±0.2m to avoid convergence difficulties caused by an excessively large search space.

[0063] 2) Perform non-dominated sorting on the population to obtain the rank of each individual and the Pareto front solutions of level 1, and store them in the external archive. Then, calculate the crowding distance for each solution in the archive to maintain solution set diversity. The specific calculation method is as follows:

[0064]

[0065] in: Indicates the first The crowding distance of each solution. Indicates adjacent solutions after sorting. , In the The value on the target. Let represent the maximum and minimum values ​​of the m-th target on the current frontier.

[0066] 3) Update the convergence factor based on the current iteration number to achieve dynamic leader selection. The specific calculation method is as follows:

[0067]

[0068] in, For the first The convergence factor of the generation, Indicates the current iteration number. This indicates the maximum number of iterations.

[0069] 4) Generate a random vector for each gray wolf and calculate the convergence factor. Wolf, Wolf, The candidate positions for wolves are calculated as follows:

[0070]

[0071]

[0072]

[0073] in, For the first Leaders ( , , The exploration coefficient of the location, For the first The convergence factor of the generation, , It is a random number in the interval [0,1]. It is the first Random weighted coefficients for leaders. For the first The weighted distance from Sekiro to the k-th leader. For the first The generation The position of leader For the first The generation Sekiro's position For the first Leader-guided candidate positions.

[0074] 5) Update the positions of all other gray wolves based on the positions of the three leaders. For any dimension that exceeds the search space, prune that dimension according to its boundaries. The specific calculation method is as follows:

[0075]

[0076] in, It updates the location of each gray wolf. They are three leaders ( , , The position of ).

[0077] 6) After recalculating the bi-objective fitness of all individuals, perform non-dominated sorting on the population, update the Pareto rank of each individual and extract the new first-level front. Then, recalculate the crowding distance in the objective space of the new front, select the current three alpha wolves (α, β, δ) by combining the rank and distribution density, and replace the external archived Pareto set P with this front. Dominated solutions are removed and truncated to a preset size according to the crowding, so as to maintain the balance between diversity and convergence.

[0078] 7) Repeat steps 3) to 6) iteratively until the maximum number of iterations is reached. Calculate the weighted synthesis error from the final Pareto set P, and select the optimal solution as the calibration extrinsic parameter. The specific calculation method is as follows:

[0079]

[0080]

[0081] in, The weighted composite error of a solution. The geometric distance error of a solution. For the direction error of a solution, Let P be the target weight, and P be the Pareto optimal solution set. This is the final solution, i.e., the final calibration extrinsic parameter.

[0082] Compared with existing technologies, the significant advantages of this invention are as follows: It employs a planar intersection line solution for 3D continuous depth edge extraction, effectively suppressing false edge interference; it uses PCA directional weights to filter 2D edges, eliminating weak texture noise; it constructs a dual-objective function to simultaneously constrain translation and rotation, achieving sub-degree accuracy; and it uses multi-objective optimization to overcome initial value dependence. This solves problems such as inaccurate features, rotational deviations, and local optima in existing methods, meeting the requirements for online calibration of autonomous driving systems. Attached Figure Description

[0083] Figure 1 This is an image showing the effect of mapping point cloud extracted from continuous 3D depth edges onto a 2D image.

[0084] Figure 2 The result of extracting the principal direction vector from the edges of a 2D image.

[0085] Figure 3 A visual diagram illustrating the principle of global optimization in multi-objective optimization algorithms.

[0086] Figure 4 This is a flowchart of a method for joint calibration of camera and lidar extrinsic parameters based on multi-target optimization. Detailed Implementation

[0087] This invention proposes a joint calibration method for camera and lidar extrinsic parameters based on multi-objective optimization. By using three-dimensional depth continuous edge filtering, two-dimensional image continuous edge filtering, constructing a bi-objective optimization function to enhance rotation constraints, and multi-objective extrinsic parameter optimization to get rid of initial value dependence, it achieves centimeter-level translation and sub-degree-level rotation accuracy.

[0088] This invention relates to a joint calibration method for camera and lidar extrinsic parameters based on multi-target optimization. Its specific implementation includes the following steps:

[0089] The first step is to divide the lidar point cloud into voxel meshes. Within each voxel, a three-dimensional depth continuous edge extraction method based on local plane fitting and plane intersection line solving is used to extract the edge to suppress false edges.

[0090] 1) First, the planar structure needs to be estimated within a local area of ​​the point cloud. The point cloud is spatially discretized using a voxel network, ensuring that the point cloud within each voxel exhibits local smoothness. This involves defining the point set contained in each voxel. A local planar model is fitted, and the angles between the planes are calculated. The calculation method is as follows:

[0091]

[0092] in It represents the angle between two planes. and These are the normal vectors of the two planes.

[0093] 2) When When the angle is too small or too large, it usually indicates that the two plane pairs are almost parallel or form a sharp angle. Such structures often do not constitute reliable edges for detection in real-world scenarios. Therefore, we next apply an angle constraint to the included angle to filter out plane pairs that do not conform to the true boundary characteristics. The constraint method is as follows:

[0094]

[0095] in For the minimum angle, The maximum angle avoids false intersections within the plane while ensuring that common structural boundaries in architectural scenes can be correctly identified.

[0096] 3) For the plane pairs obtained through angle constraint filtering Further solve for its spatial intersection line The direction vector of the intersection line is the cross product of the normal vectors of the two planes, and any point on the intersection line can be obtained by solving a simultaneous equation for the planes. To generate practical edge points usable for projection and matching, points close to the intersection line from the original point cloud need to be selected as depth-continuous edge points. The specific calculation method is as follows:

[0097]

[0098]

[0099]

[0100]

[0101] in Point coordinates , and These are the normal vectors of the two planes. and This is the offset. As the base point of the intersection line, The direction vector of the intersection line is the cross product of the normal vectors of the two planes. This is a scaling factor for the extension direction of the intersection line. The coordinates of a point in a point cloud. For point cloud points to the intersection line The shortest distance, This is a set of depth-continuous edge points. Downsampling based on dynamic voxel networks is performed to improve the uniformity of point cloud edge distribution and reduce the overhead of subsequent projection calculations. The resulting depth edge set exhibits clear geometric boundary features.

[0102] The second step is to perform two-dimensional edge detection on the camera image, and to perform PCA analysis on the local region of each edge to obtain the principal direction vector and the ratio of eigenvalues ​​to calculate the confidence weight. Weak texture edges with weights below a set threshold are downweighted or removed.

[0103] 1) To obtain reliable edge direction information, Gaussian filtering is first applied to the image edges to eliminate noise. Then, image gradient magnitude and non-maximum suppression are used to obtain a structurally stable binary edge map. The image representation is then used to calculate... direction and The gradients in the two directions are used to calculate the initial orientation representation of the image edges. The specific calculation method is as follows:

[0104]

[0105]

[0106] in For image representation, for gradient of direction, for Gradient of direction. This represents the initial direction of the image edges.

[0107] 2) Building upon the previous step, principal component analysis (PCA) is performed on the neighborhood of edge points using local statistical methods to extract principal direction vectors with stronger structural significance. For a given edge point in the image, the mean coordinates of all edge points within its local neighborhood window are calculated. Then, a covariance matrix is ​​constructed, and eigenvalues ​​are obtained through eigenvalue decomposition. The specific calculation method is as follows:

[0108]

[0109]

[0110]

[0111] in For edge points The mean coordinates of the local neighborhood point set. The number of neighborhood point set points. The coordinates of the neighborhood point set point. Let covariance matrix be the variance matrix. Indicates the first 1 eigenvector Indicates the first Each feature value.

[0112] 3) Calculate the directional confidence weight using the two eigenvalues ​​obtained in step 2). A larger confidence weight indicates a more pronounced linear structure in the marginal distribution, while a smaller weight indicates a more scattered marginal distribution. The specific calculation method is as follows:

[0113]

[0114] in As a credibility weight, As a regulating factor, and These are the eigenvalues ​​obtained from the covariance matrices in steps 2) and 3 respectively. Through the above process, a direction vector is obtained for each valid edge point. With directional credibility These two elements work together to construct the consistency constraints with the edge direction of the point cloud, providing high-quality directional feature information for extrinsic parameter calibration.

[0115] The third step is to construct a dual-objective optimization function, where the first objective is the sum of the pixel Euclidean distances between the three-dimensional depth continuous edge points after projection through the current extrinsic parameters and the nearest reliable two-dimensional edge points, and the second objective is the cosine loss of the angle between the three-dimensional intersection direction vector and the corresponding two-dimensional edge principal direction vector.

[0116] 1) After extracting the continuous edges of the point cloud depth and the direction of the image edges, a extrinsic parameter optimization model that can simultaneously reflect spatial location consistency and structural orientation consistency is constructed, i.e., a dual-objective modeling based on geometric distance error and orientation consistency error. The specific construction steps are as follows:

[0117] a) For any point cloud edge point, its coordinates in the camera coordinate system are obtained through rotation matrix and translation vector. Then, the point cloud coordinates are projected onto the pixel plane coordinates according to the camera pinhole imaging model. The specific calculation method is as follows:

[0118]

[0119]

[0120] in, These are the coordinates of the point cloud in the camera coordinate system. It is the rotation transformation matrix between the camera coordinate system and the radar coordinate system. These are the coordinates of the point cloud in the radar coordinate system. It is the translation transformation vector between the camera coordinate system and the radar coordinate system. These are the coordinates of the projection point. This represents the focal length components in the horizontal and vertical directions. This represents the coordinates of the principal point in the image.

[0121] b) Calculate the pixel distance between the projected point and its nearest neighbor in the image edge set to reflect the degree of spatial misalignment caused by extrinsic parameters. The total geometric distance error is defined as the total degree of spatial misalignment, i.e., the sum of all pixel distances. The specific calculation method is as follows:

[0122]

[0123]

[0124] in, Indicates the first A distance of one pixel, Indicates the projection point. Indicates the nearest neighbor point. This represents the geometric distance error.

[0125] c) Based on the previously extracted edge direction features from the 3D point cloud, and then obtaining the local principal direction from the image edge neighborhood using PCA, the angle between the two is calculated to reflect the accuracy of the rotation. The specific calculation method is as follows:

[0126]

[0127] in, For the first The angle between the edge direction of a point and the local principal direction of its neighborhood. For the first The edge direction of each point, For the first The local principal direction of the edge neighborhood of each point.

[0128] d) Calculate the orientation confidence weights to eliminate the interference of regions with weak textures and blurred edges on the orientation error, and finally calculate the final orientation error. The specific calculation method is as follows:

[0129]

[0130]

[0131] in, It is the first The directional credibility weight of each point and It is the first PCA eigenvalues ​​at each point As a regulating factor, It is the final direction error. For the first The angle between the edge direction of a point and the local principal direction of the edge neighborhood.

[0132] 2) The two types of errors are jointly constructed into a multi-objective extrinsic parameter optimization problem, but instead of directly simplifying it to a single objective function, the independence of the two objectives is maintained. The specific calculation method is as follows:

[0133]

[0134] in, For geometric distance error, For directional error, These are external parameters.

[0135] 3) Using Pareto optimality theory, we search for a set of non-dominated solutions to explore all solutions in the extrinsic parameter space that can achieve a reasonable balance between the two objectives.

[0136] The fourth step involves using a multi-objective optimization algorithm to directly optimize the six-degree-of-freedom extrinsic parameters of the lidar and camera globally, establishing and maintaining the Pareto optimal set of non-dominated solutions, and selecting the solution with the smallest weighted synthesis error from the final Pareto front as the final extrinsic parameter calibration result of the lidar and camera after the iteration.

[0137] 1) Initialize the gray wolf population, for For each gray wolf in the sequence 1,…,N, a random position vector is generated, and the fitness value of each gray wolf is calculated using the biobjective function constructed in step three. The specific calculation method is as follows:

[0138]

[0139]

[0140] in, Let be the rotation angle about the three coordinate axes. To translate from radar to camera, For the first The dual-target value of Sekiro: Shadows Die Twice. For the first Sekiro's position vector For geometric distance error, This represents the direction error. In each iteration, the gray wolf's position will represent a set of extrinsic parameters. The extrinsic parameter search interval is set to a rotation of ±10° and a translation of ±0.2m to avoid convergence difficulties caused by an excessively large search space.

[0141] 2) Perform non-dominated sorting on the population to obtain the rank of each individual and the Pareto front solutions of level 1, and store them in the external archive. Then, calculate the crowding distance for each solution in the archive to maintain solution set diversity. The specific calculation method is as follows:

[0142]

[0143] in: Indicates the first The crowding distance of each solution. Indicates adjacent solutions after sorting. , In the The value on the target. Let represent the maximum and minimum values ​​of the m-th target on the current frontier.

[0144] 3) Update the convergence factor based on the current iteration number to achieve dynamic leader selection. The specific calculation method is as follows:

[0145]

[0146] in, For the first The convergence factor of the generation, Indicates the current iteration number. This indicates the maximum number of iterations.

[0147] 4) Generate a random vector for each gray wolf and calculate the convergence factor. Wolf, Wolf, The candidate positions for wolves are calculated as follows:

[0148]

[0149]

[0150]

[0151] in, For the first Leaders ( , , The exploration coefficient of the location, For the first The convergence factor of the generation, , It is a random number in the interval [0,1]. It is the first Random weighted coefficients for leaders. For the first The weighted distance from Sekiro to the k-th leader. For the first The generation The position of leader For the first The generation Sekiro's position For the first Leader-guided candidate positions.

[0152] 5) Update the positions of all other gray wolves based on the positions of the three leaders. For any dimension that exceeds the search space, prune that dimension according to its boundaries. The specific calculation method is as follows:

[0153]

[0154] in, It updates the location of each gray wolf. They are three leaders ( , , The position of ).

[0155] 6) Recalculate the biobjective values ​​for all individuals, perform non-dominated ordination on the population and update the rank of each individual and the new first-level Pareto front. Recalculate the crowding distance of each individual in the objective space. From the new first-level front, reselect individuals based on rank and crowding. Three alpha wolves update the outer Pareto set P with the new first frontier (removing dominated solutions and truncating the scale by crowding).

[0156] 7) Repeat steps 3) to 6) iteratively until the maximum number of iterations is reached. Calculate the weighted synthesis error from the final Pareto set P, and select the optimal solution as the calibration extrinsic parameter. The specific calculation method is as follows:

[0157]

[0158]

[0159] in, The weighted composite error of a solution. The geometric distance error of a solution. For the direction error of a solution, Let P be the target weight, and P be the Pareto optimal solution set. This is the final solution, i.e., the final calibration extrinsic parameter.

[0160] The invention will be further illustrated below with reference to specific implementation examples.

[0161] This embodiment is based on the following hardware platform: ① Data acquisition platform: The LiDAR is a Livox HAP solid-state LiDAR (horizontal field of view 100°, vertical field of view 70.4°, point frequency 300,000 points / second); the camera is a HIKVISION DS-U02; the IMU is an Xsens MTi-300; all sensors are connected to the NVIDIA Jetson AGX Xavier computing platform, and the equipment is powered by the mobile robot's own lithium battery; ② Experimental platform: CPU is Intel® Core i5-8259U, graphics card is Intel® IRIS655, and memory is 16GB.

[0162] This embodiment is based on the following software environment: Computer operating system: Ubuntu 20.04 LTS, Robot operating system: ROS Noetic, Point cloud processing library: PCL 1.12, Optimization library: Ceres Solver 2.1.0.

[0163] Combination Figure 4 The present invention provides a method for joint calibration of camera and lidar extrinsic parameters based on multi-target optimization, and the specific implementation steps are as follows:

[0164] The first step is to divide the LiDAR point cloud into voxel meshes. Within each voxel, a depth continuous edge extraction method based on local plane fitting and plane intersection line solving is used to extract 3D depth continuous edges to suppress false edges; the effect is as follows. Figure 1 As shown.

[0165] The second step involves performing two-dimensional edge detection on the camera image. PCA analysis is then performed on the local region of each edge to obtain the principal direction vector and eigenvalue ratio, which are used to calculate the confidence weight. Weak texture edges with weights below a set threshold are either downweighted or removed. The effect is as follows: Figure 2 As shown.

[0166] The third step is to construct a dual-objective optimization function, where the first objective is the sum of the pixel Euclidean distances between the three-dimensional depth continuous edge points after projection through the current extrinsic parameters and the nearest reliable two-dimensional edge points, and the second objective is the cosine loss of the angle between the three-dimensional intersection direction vector and the corresponding two-dimensional edge principal direction vector, and the second objective is weighted by the direction confidence weight obtained in the third step.

[0167] Fourth step, combine Figure 3 The hunting process employs a multi-objective optimization algorithm to directly optimize the six-degree-of-freedom extrinsic parameters of the lidar and camera globally, establishes and maintains the Pareto optimal set of non-dominated solutions, and selects the solution with the smallest weighted synthesis error from the final Pareto front as the final extrinsic parameter calibration result of the lidar and camera after the iteration ends.

Claims

1. A method for joint calibration of camera and lidar extrinsic parameters based on multi-objective optimization, characterized in that, Includes the following steps: The first step is to divide the lidar point cloud into voxel meshes. Within each voxel, a three-dimensional depth continuous edge extraction method based on local plane fitting and plane intersection line solution is used to extract the depth continuous edge to suppress false edges. The second step is to perform two-dimensional edge extraction on the camera image, and to perform PCA analysis on the local region of each image edge to obtain the principal direction vector and the ratio of eigenvalues ​​to calculate the confidence weight, in order to prepare for the third step of constructing a bi-objective optimization function. The third step is to construct a dual-objective optimization function, where the first objective is the sum of the pixel Euclidean distances between the three-dimensional depth continuous edge points after projection through the current extrinsic parameters and the nearest two-dimensional edge points, and the second objective is the cosine loss of the angle between the three-dimensional edge direction vector and the corresponding two-dimensional edge principal direction vector, and the second objective is weighted by the direction confidence weights obtained in the second step. The fourth step involves using a multi-objective optimization algorithm to directly optimize the six-degree-of-freedom extrinsic parameters of the lidar and camera globally, establishing and maintaining the Pareto optimal set of non-dominated solutions, and selecting the solution with the smallest weighted synthesis error from the final Pareto front as the final extrinsic parameter calibration result of the lidar and camera after the iteration.

2. The method for joint calibration of camera and lidar extrinsic parameters based on multi-objective optimization according to claim 1, characterized in that, The specific steps implemented in the first step are as follows: 1) First, estimate the planar structure within a local area of ​​the point cloud; then, spatially discretize the point cloud using a voxel network, ensuring that the point cloud within each voxel exhibits local smoothness; finally, define the point set contained in each voxel. A local planar model is fitted, and the angles between the planes are calculated; the calculation method is as follows: in It represents the angle between two planes. and These are the normal vectors of the two planes; 2) Apply angular constraints to the included angle to filter out plane pairs that do not conform to the true boundary characteristics. The constraint method is as follows: in For the minimum angle, The maximum angle; 3) For the plane pairs obtained through angle constraint filtering Further solve for its spatial intersection line ; The intersection direction vector is the cross product of the normal vectors of the two planes, and any point on the intersection line can be obtained by solving a simultaneous equation for the planes. Points close to the intersection line are selected from the original point cloud as depth-continuous edge points to form actual edge points that can be used for projection and matching. The specific calculation method is as follows: in Point coordinates , and These are the normal vectors of the two planes. and This is the offset. As the base point of the intersection, The direction vector of the intersection line is the cross product of the normal vectors of the two planes. This is a scaling factor for the extension direction of the intersection line. The coordinates of a point in a point cloud. For point cloud points to the intersection line The shortest distance, For a set of depth-continuous edge points; Perform downsampling based on dynamic voxel networks.

3. The method for joint calibration of camera and lidar extrinsic parameters based on multi-objective optimization according to claim 2, characterized in that, The specific steps to implement the second step are as follows: 1) First, Gaussian filtering is applied to the image edges to eliminate noise. Then, image gradient magnitude and non-maximum suppression are used to obtain a structurally stable binary edge map. The image representation is then used to calculate... direction and The gradients in the two directions are used to calculate the initial directional representation of the image edges; the specific calculation method is as follows: in For image representation, for gradient of direction, for Gradient of direction; This represents the initial direction of the image edges; 2) Principal component analysis (PCA) is performed on the neighborhood of edge points using local statistical methods to extract principal direction vectors with stronger structural significance. For a given edge point in the image, the mean coordinates of all edge points within its local neighborhood window are calculated, and then a covariance matrix is ​​constructed. Eigenvalues ​​are then obtained by eigenvalue decomposition of the covariance matrix. The specific calculation method is as follows: in For edge points The mean coordinates of the local neighborhood point set. The number of neighborhood point set points. The coordinates of the neighborhood point set point. Let covariance matrix be the variance matrix. Indicates the first 1 eigenvector Indicates the first One eigenvalue; 3) Calculate the directional confidence weight using the two feature values ​​obtained in step 2). The specific calculation method is as follows: in As the directional credibility weight, As a regulating factor, and These are the eigenvalues ​​obtained from the covariance matrix in two steps, 2).

4. The method for joint calibration of camera and lidar extrinsic parameters based on multi-objective optimization according to claim 3, characterized in that, The specific steps to achieve the third step are as follows: 1) After extracting the continuous edge of the point cloud depth and the direction of the image edge, construct an external parameter optimization model that can simultaneously reflect the consistency of spatial position and the consistency of structural direction, namely a bi-objective optimization function based on geometric distance error and direction consistency error; The specific construction steps are as follows: a) For any point cloud edge point, its coordinates in the camera coordinate system are obtained through rotation matrix and translation vector. Then, the point cloud coordinates are projected onto the pixel plane coordinates according to the camera pinhole imaging model. The specific calculation method is as follows: in, These are the coordinates of the point cloud in the camera coordinate system. It is the rotation transformation matrix between the camera coordinate system and the radar coordinate system. These are the coordinates of the point cloud in the radar coordinate system. It is the translation transformation vector between the camera coordinate system and the radar coordinate system. These are the coordinates of the projection point. This represents the focal length components in the horizontal and vertical directions. Indicates the coordinates of the principal point of the image; b) Calculate the pixel distance between the projection point and the nearest neighbor in the image edge set to reflect the degree of spatial misalignment caused by extrinsic parameters; the total geometric distance error is defined as the total degree of spatial misalignment, that is, the sum of all pixel distances; the specific calculation method is as follows: in, Indicates the first A distance of one pixel, Indicates the projection point. Indicates the nearest neighbor point. Indicates geometric distance error; c) Based on the previously extracted edge direction features from the 3D point cloud, and then obtaining the local principal direction from the image edge neighborhood PCA, calculate the angle between the two to reflect the accuracy of the rotation; the specific calculation method is as follows: in, For the first The angle between the principal direction of the projection edge and the principal direction of the image edge. For the first The principal direction vector of each projected edge For the first The principal direction vectors of the image edges; d) Calculate the direction confidence weight for each projected edge and image edge using the same method as in step two, and finally calculate the final direction error; the specific calculation method is as follows: in, It is the first The direction confidence weight of each image edge It is the first The directional confidence weight of each projected edge It is the final direction error. For the first The angle between the projection edge and the image edge; 2) Combine the two types of errors into a multi-objective extrinsic parameter optimization problem. The specific calculation method is as follows: in, For geometric distance error, For directional error, These are external parameters. 3) Using Pareto optimality theory, we search for a set of non-dominated solutions to explore all solutions in the extrinsic parameter space that can achieve a reasonable balance between the two objectives.

5. The method for joint calibration of camera and lidar extrinsic parameters based on multi-objective optimization according to claim 1, characterized in that, The specific steps to achieve the fourth step are as follows: 1) Initialize the gray wolf population, for For each gray wolf in the sequence 1,…,N, a position vector is randomly generated, and the fitness value is calculated for each gray wolf using the biobjective function constructed in step three. The specific calculation method is as follows: in, Let be the rotation angle about the three coordinate axes. To translate from radar to camera, For the first The dual-target value of Sekiro: Shadows Die Twice. For the first Sekiro's position vector For geometric distance error, The direction error is represented by the position of the gray wolf in each iteration. The search range for the external parameters is set to rotate ±10° and translate ±0.2m to avoid convergence difficulties caused by an excessively large search space. 2) Perform a non-dominated sort on the population to obtain the rank of each individual and the Pareto front solutions of level 1, and store them in an external archive. Then, calculate the crowding distance for each solution in the archive to maintain solution set diversity. The specific calculation method is as follows: in: Indicates the first The crowding distance of each solution. Indicates adjacent solutions after sorting. , In the The value on the target; This represents the maximum and minimum values ​​of the m-th target on the current frontier; 3) Update the convergence factor based on the current iteration number to achieve dynamic leader selection. The specific calculation method is as follows: in, For the first The convergence factor of the generation, Indicates the current iteration number. Indicates the maximum number of iterations; 4) Generate a random vector for each gray wolf and calculate the convergence factor. Wolf, Wolf, The candidate positions for wolves are calculated as follows: in, For the first Leaders ( , , The exploration coefficient of the location, For the first The convergence factor of the generation, , It is a random number in the interval [0,1]. It is the first Random weighted coefficients for leaders; For the first The weighted distance from Sekiro to the k-th leader. For the first The generation The position of leader For the first The generation Sekiro's position For the first Leader-led candidate positions; 5) Update the position of each other gray wolf based on the positions of the three leaders. For any dimension that exceeds the search space, prune that dimension according to its boundaries. The specific calculation method is as follows: in, It updates the location of each gray wolf. They are three leaders ( , , The position of ); 6) Recalculate the biobjective values ​​for all individuals, perform a non-dominated ordination of the population and update the rank of each individual and the new first-level Pareto front. Recalculate the crowding distance of each individual in the objective space. From the new first-level front, reselect individuals based on rank and crowding. Three-headed wolves, updating the outer Pareto set P with a new first frontier; 7) Repeat steps 3) to 6) iteratively until the maximum number of iterations is reached. Calculate the weighted synthesis error from the final Pareto set P, and select the optimal solution as the calibration extrinsic parameter. The specific calculation method is as follows: in, The weighted composite error of a solution. The geometric distance error of a solution. For the direction error of a solution, Let P be the target weight, and P be the Pareto optimal solution set. This is the final solution, i.e., the final calibration extrinsic parameter.

Images