Joint calibration method, device and system based on rotary laser radar and camera, and storage medium

Abstract

The invention discloses a joint calibration method based on a rotary laser radar and a camera, and the method comprises the following steps: S1, carrying out the kinematic modeling of the camera, the laser radar and a motor rotating shaft based on a DH parameter method; s2, collecting synchronous laser radar point cloud data and camera image data of a motor rotating shaft at different angles; s3, processing the laser radar point cloud data; s4, processing the camera image data; s5, projecting the extracted laser point cloud depth continuous edge features to an image plane based on a current external parameter estimation value, and matching and associating the extracted laser point cloud depth continuous edge features with the image edge features extracted in the step S4; s6, constructing a projection residual equation about the external parameters according to the successfully matched feature pairs; and S7, the residual equation is linearized. According to the joint calibration method based on the rotary laser radar and the camera, high-precision and high-robustness automatic calibration is completed in an environment without an auxiliary target through kinematics modeling of a system and an advanced feature extraction matching technology. The invention further discloses a joint calibration device and system based on the rotary laser radar and the camera, and a storage medium.

Description

Technical Field

[0001] This invention relates to the field of calibration, and more particularly to a joint calibration method, apparatus, storage medium, and system based on a rotating lidar and a camera. Background Technology

[0002] Existing technologies for the joint calibration of color cameras, LiDAR, and rotating motors have several limitations: 1. From a modeling perspective, traditional methods lack a unified geometric framework. Estimates of extrinsic parameters between the rotating LiDAR and the camera rely heavily on fragmented coordinate transformation formulas, failing to employ the DH method for systematic kinematic modeling. This leads to ambiguous rigid body transformation relationships between coordinate systems (LiDAR, rotation axis, camera), chaotic parameter definitions, and a tendency for error accumulation during extrinsic parameter transformation, making it difficult to guarantee high-precision calibration. 2. Regarding calibration methods, existing technologies generally rely on target tools such as calibration boards, resulting in cumbersome procedures: the calibration board must be manually and precisely placed, and its flatness and installation position directly affect calibration accuracy. Furthermore, machining errors, installation deviations, and mechanical wear after long-term use can cause the relative position of the calibration board and sensor to deviate from the ideal state, introducing additional errors. Traditional methods cannot effectively compensate for these uncertainties. 3. Regarding feature extraction, fixed-resolution voxelization is the mainstream method for extracting laser point cloud edges, but it has significant contradictions: high resolution, while preserving details, significantly increases computation time; low resolution may include multiple planar features in the same voxel, leading to confusion between depth-continuous and discontinuous edges, causing subsequent feature matching errors. 4. In image edge extraction, traditional algorithms (such as Canny) lack specificity for LiDAR features, are prone to missed detections in weak-texture scenes, and have weak correlation between extracted image edges and laser point cloud depth features, resulting in numerous mismatches during registration, further reducing calibration accuracy. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, one of the objectives of this invention is to provide a joint calibration method based on rotating lidar and camera, so as to achieve high-precision and robust automatic calibration in an environment without auxiliary targets through systematic kinematic modeling and advanced feature extraction and matching technology.

[0004] The second objective of this invention is to provide a joint calibration device based on a rotating lidar and a camera, so as to achieve high-precision and robust automatic calibration in an environment without auxiliary targets through systematic kinematic modeling and advanced feature extraction and matching technology.

[0005] The third objective of this invention is to provide a storage medium that enables high-precision and robust automatic calibration in an environment without auxiliary targets by using systematic kinematic modeling and advanced feature extraction and matching techniques.

[0006] The fourth objective of this invention is to provide a joint calibration device system based on a rotating lidar and a camera, so as to achieve high-precision and robust automatic calibration in an environment without auxiliary targets through system kinematic modeling and advanced feature extraction and matching technology.

[0007] One of the objectives of this invention is achieved through the following technical solution: The joint calibration method based on rotating lidar and camera includes the following steps: S1. Based on the DH parameter method, kinematic modeling is performed on the camera, lidar and motor rotation axis to define the first rigid body transformation from the lidar to the motor rotation axis coordinate system, the second rigid body transformation from the motor rotation axis coordinate system to the camera coordinate system, and the coordinate transformation relationship caused by the rotation angle of the motor rotation axis. S2. Collect synchronous lidar point cloud data and camera image data of the motor rotation shaft at different angles; S3. Process the lidar point cloud data and use a planar adaptive voxelization strategy to extract the depth continuous edge features of the lidar point cloud. S4. Process the camera image data and use the EdgeNAT edge detection algorithm based on Transformer to extract image edge features; S5. Extract the continuous edge features of the laser point cloud depth, project them onto the image plane based on the current extrinsic parameter estimates, and match and associate them with the image edge features extracted in step S4. S6. Based on the successfully matched feature pairs, construct the projection residual equations about the extrinsic parameters; S7. Linearize the residual equation and use a nonlinear optimization algorithm to iteratively optimize and solve it. Finally, obtain the first rigid body transformation parameters of the laser radar to the motor rotation axis coordinate system and the second rigid body transformation parameters of the motor rotation axis coordinate system to the camera coordinate system, and complete the joint calibration.

[0008] Furthermore, the planar adaptive voxelization strategy described in step S3 specifically includes: S31. Divide the original point cloud into initial voxels; S32. For each voxel, calculate the eigenvalues ​​of the covariance matrix. Let the eigenvalues ​​of the covariance matrix be sorted by size as λ1≥λ2≥λ3. If the eigenvalues ​​satisfy... ,in If the plane threshold is met, the voxel is determined to be a planar voxel; otherwise, the voxel is divided into multiple sub-voxels and step S32 is recursively executed until all voxels satisfy the planarity condition or reach the minimum voxel size. S33. For adjacent planar voxels, calculate the angle between their planar normal vectors. If the angle exceeds the edge determination threshold, determine that there is a depth-continuous edge at the intersection of the two planar voxels, and extract the point cloud of that region as a valid edge feature.

[0009] Furthermore, the EdgeNAT edge detection algorithm described in step S4 uses image edge ground truth labels generated by back projection of continuous edge features of lidar point cloud depth for supervised learning during the training phase, so as to enhance the ability to extract structured edge features of the calibrated scene. During the inference phase, after extracting edge features, a geometric consistency filtering step is further included: the extracted image edge points are back-projected to a virtual 3D space based on camera intrinsic parameters and roughly aligned with the spatial distribution of the LiDAR point cloud to remove outlier edges that deviate from the spatial structure.

[0010] Furthermore, the matching association described in step S5 specifically includes: For the depth-continuous edge features of the laser point cloud projected onto the image plane, search for its K nearest neighbors in the image edge features; Calculate the mean and covariance matrix of the K nearest neighbors, and take the eigenvector corresponding to the smallest eigenvalue of the covariance matrix as the normal vector of the image edge at that point; Verify the orthogonality between the direction of the continuous edge features of the laser point cloud depth and the normal vector to filter out incorrect matching pairs.

[0011] Furthermore, the projection residual equation constructed in step S6 is: in It is the normal vector of the image edge; This represents the coordinates of the continuous edge features of the laser point cloud depth projected onto the image plane. Represents the pixel coordinates of edge features in the camera image; Gaussian white noise representing camera image data; Represents the laser point cloud in the lidar coordinate system; This represents the transformation matrix of the radar point cloud from time i-1 to time i, according to the sampling frequency ƒ. Represents the coordinates of the laser point in the lidar coordinate system at time i-1; Gaussian white noise represents the point cloud data from a lidar system.

[0012] Furthermore, in step S7, the second rigid body transformation matrix is ​​solved iteratively using the Gauss-Newton method based on the linearized residual equation.

[0013] The second objective of this invention is achieved by the following technical solution: The joint calibration device based on rotating lidar and camera includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is a calibration program, and when the processor executes the calibration program, it implements the calibration method steps as one of the objectives of this invention.

[0014] The third objective of this invention is achieved by the following technical solution: A storage medium having a computer program stored thereon, the computer program being a calibration program, which, when executed by a processor, implements calibration method steps as one of the objectives of this invention.

[0015] The fourth objective of this invention is achieved by the following technical solution: A joint calibration system based on a rotating lidar and a camera is used to perform calibration method steps as described in one of the objectives of this invention, the system comprising: The kinematic modeling module is used to perform kinematic modeling of the camera, lidar, and motor rotation axis based on the DH parameter method. It clearly defines the first rigid body transformation from the lidar to the motor rotation axis coordinate system, the second rigid body transformation from the motor rotation axis coordinate system to the camera coordinate system, and the coordinate transformation relationship caused by the rotation angle θ of the motor rotation axis. The data acquisition module is used to acquire synchronous lidar point cloud data and camera image data of the motor rotation shaft at different angles; The point cloud feature extraction module is used to process lidar point cloud data and uses a planar adaptive voxelization strategy to extract the depth continuous edge features of the lidar point cloud. The image feature extraction module is used to process camera image data and uses the Transformer-based EdgeNAT edge detection algorithm to extract image edge features; The feature matching module is used to project the extracted laser point cloud depth continuous edge features onto the image plane based on the current extrinsic parameter estimates, and match and associate them with the image edge features. The residual construction module is used to construct projection residual equations about the extrinsic parameters based on successfully matched feature pairs. The optimization and solution module is used to linearize the residual equation and iteratively optimize the solution using a nonlinear optimization algorithm to finally obtain the rigid body transformation parameters from the motor rotation axis coordinate system to the camera coordinate system, thus completing the joint calibration.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: 1. Target-free calibration: It breaks through the limitation of target calibration in existing technologies, eliminating the need for auxiliary tools such as calibration boards, significantly reducing calibration costs and operational complexity, and is especially suitable for natural scenes with obvious structured features; 2. Accurate modeling: By introducing the DH method for system kinematic modeling, it provides a clear and unambiguous geometric expression framework for extrinsic parameter estimation, fundamentally reducing coordinate transformation confusion and error accumulation; 3. Efficient and accurate feature extraction: The planar adaptive voxelization strategy dynamically adjusts the resolution, effectively solving the contradiction between efficiency and accuracy in fixed resolution, and improving the accuracy of deep continuous edge extraction. The targeted EdgeNAT algorithm enhances the edge detection capability in weak texture scenes and strengthens the correlation with point cloud features through geometric filtering; 4. Strong robustness: Combining multi-period point cloud feature extraction, multi-level data association strategies (such as normal vector orthogonal verification), and an iterative optimization mechanism based on maximum likelihood estimation, it can effectively compensate for parameter drift caused by equipment installation deviations, mechanical wear, and environmental factors, significantly improving the stability and robustness of calibration results. 5. High degree of automation: A complete joint calibration process for non-feature scenarios has been formed, providing a reliable foundation for high-precision online self-calibration of multi-sensor fusion equipment in dynamic working environments. Attached Figure Description

[0017] Figure 1 A flowchart of the joint calibration method based on rotating lidar and camera provided by the present invention; Figure 2 A schematic diagram of the kinematic modeling of the camera, lidar, and motor rotation axis provided by the present invention using the DH method. Figure 3 The flowchart of the planar adaptive voxelization strategy provided by the present invention is shown. Detailed Implementation

[0018] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0019] Example 1 This invention provides a preferred embodiment of a joint calibration method based on a rotating lidar and a camera, such as... Figure 1 As shown, it includes the following steps: S1. Based on the DH parameter method, kinematic modeling is performed on the camera, lidar, and motor rotation axis to define the first rigid body transformation from the lidar to the motor rotation axis coordinate system, the second rigid body transformation from the motor rotation axis coordinate system to the camera coordinate system, and the coordinate transformation relationship caused by the rotation angle of the motor rotation axis.

[0020] The kinematic modeling of the camera, lidar, and motor rotation axes using the DH method is as follows: The lidar coordinate system represents the initial position of the motor; Indicates the coordinate system of the rotation axis; Indicates the camera coordinate system; This represents the rigid body transformation of a laser radar to a rotation axis coordinate system; This represents a rigid body transformation from the rotation axis coordinate system to the camera coordinate system; such as... Figure 2 As shown, the calibration objective is to calibrate the linear transformation matrix from the radar to the camera in a non-feature scene. DH parameters ,in = , This represents the angle of rotation of the motor.

[0021] DH Parameter Table For laser point clouds in lidar coordinate system Its representation in the camera coordinate system It can be given by the following transformation relationship ( , (Representing the cosine and sine functions respectively): S2. Collect synchronous lidar point cloud data and camera image data of the motor rotation shaft at different angles.

[0022] S3. Process the lidar point cloud data and use a planar adaptive voxelization strategy to extract the depth continuous edge features of the lidar point cloud.

[0023] To achieve high-precision extrinsic parameter calibration and ensure the accuracy of data association, the calibration results must be determined by valid data. Due to the significant coordinate system difference between the 3D point cloud and the 2D camera projection plane, the edge feature point cloud acquired by the LiDAR contains both depth-continuous and depth-discontinuous types. However, on the camera's 2D image plane, the system struggles to distinguish between these two edge features. Therefore, to improve the accuracy of data association, it is necessary to filter out depth-discontinuous edge point clouds and use only depth-continuous edge LiDAR point cloud data to support the calibration process.

[0024] S4. Process the camera image data and use the EdgeNAT edge detection algorithm based on Transformer to extract image edge features; S5. Extract the continuous edge features of the laser point cloud depth, project them onto the image plane based on the current extrinsic parameter estimates, and match and associate them with the image edge features extracted in step S4. S6. Based on the successfully matched feature pairs, construct the projection residual equations about the extrinsic parameters; S7. Linearize the residual equation and use a nonlinear optimization algorithm to iteratively optimize and solve it. Finally, obtain the first rigid body transformation parameters of the laser radar to the motor rotation axis coordinate system and the second rigid body transformation parameters of the motor rotation axis coordinate system to the camera coordinate system, and complete the joint calibration.

[0025] By completing the above steps, the non-feature scene extrinsic parameter calibration of the lidar, motor rotation axis, and camera is completed, enabling the mapping equipment that integrates the rotating lidar and camera to achieve self-calibration of extrinsic parameters in any scene with obvious structured features.

[0026] Understandably, in the joint calibration of non-feature scenes using LiDAR and cameras, the accuracy of point cloud edge features directly affects the accuracy of extrinsic parameter estimation. Edge features in 3D point clouds can be divided into two categories: depth-continuous and depth-discontinuous. Depth-discontinuous edges (such as boundaries of dynamic objects or sensor noise points) are prone to incorrect matching with image features, therefore, it is necessary to prioritize the extraction of depth-continuous edge point clouds. Existing fixed-resolution voxelization methods have significant shortcomings: high resolution greatly increases computation time, while low resolution may result in multiple planar features within the same voxel, leading to matching errors. To address this issue, this method employs a planar adaptive voxelization strategy to achieve accurate and efficient extraction of depth-continuous edges.

[0027] The core process of planar adaptive voxelization includes two stages: hierarchical detection and dynamic segmentation. First, the original point cloud is divided into initial small voxels, and then planarity verification is performed on each voxel. By calculating the eigenvalues ​​of the point cloud covariance matrix, it is determined whether the voxel is planar. If the voxel does not meet the requirements, it is segmented into multiple sub-voxels, and the planarity verification is repeated until all sub-voxels meet the planarity condition or reach the minimum voxel size. This dynamic adjustment mechanism overcomes the efficiency-accuracy contradiction of fixed-resolution methods while ensuring the uniqueness of point cloud features within each voxel.

[0028] Furthermore, the planar adaptive voxelization strategy described in step S3 specifically includes: S31. Divide the original point cloud into initial voxels; S32. For each voxel, calculate the eigenvalues ​​of the covariance matrix. Let the eigenvalues ​​of the covariance matrix be sorted by size as λ1≥λ2≥λ3. If the eigenvalues ​​satisfy... ,in If the plane threshold is met, the voxel is determined to be a planar voxel; otherwise, the voxel is divided into 8 sub-voxels and step S32 is recursively executed until all voxels meet the planarity condition or reach the minimum voxel size. S33. For adjacent planar voxels, calculate the angle between their planar normal vectors. If the angle exceeds the edge determination threshold, determine that there is a depth-continuous edge at the intersection of the two planar voxels, and extract the point cloud of that region as a valid edge feature.

[0029] Specifically, for any two adjacent plane voxels, calculate their plane normal vectors. and The included angle If the edge detection threshold is exceeded, a depth-continuous edge is considered to exist at the intersection of the two planar voxels, and the point cloud of the intersection region is extracted as a valid edge feature. This process uses planar geometric relationships to filter out false edges caused by noise or unstructured features, retaining only the real physical edges that exist in the scene, such as wall intersections, the boundary between the ground and columns, etc.

[0030] The final extracted depth-continuous edge point cloud retains structural information in three-dimensional space and is highly correlated with observable edge features (such as lines and contours) in the camera image. Using this method, the signal-to-noise ratio of the LiDAR point cloud's edge features is significantly improved, laying a reliable foundation for subsequent matching and association with camera image edges, and effectively reducing extrinsic parameter calibration deviations caused by feature mismatches.

[0031] Edge feature extraction from camera images is achieved using the EdgeNAT algorithm, which incorporates LiDAR data and is a one-stage edge detector based on Transformer. The EdgeNAT algorithm enhances the global contextual awareness of edge features through the self-attention mechanism of Transformer. Its core computation can be represented as: in This represents the attention mechanism; These represent the query, key, and value matrices, respectively. This means converting attention scores into a probability distribution; Representing vectors Dimensions.

[0032] This mechanism effectively solves the problem of missed detections by edge detectors in weak texture scenes in existing technologies. Its core design includes a multi-scale feature fusion module: first, a CNN backbone network is used to extract multi-level feature maps of the image, then the maps are input into a Transformer encoder to capture long-distance spatial dependencies, and finally a lightweight decoder is used to generate pixel-level edge probability maps.

[0033] To improve the relevance of the calibration scenario, reduce the pressure of filtering calculations, and achieve more accurate registration results, a LiDAR-assisted sample augmentation strategy is adopted during the training phase to ensure that the training data is highly consistent with the structured edges that need to be extracted in the calibration task, such as wall boundaries and door and window outlines.

[0034] Furthermore, the EdgeNAT edge detection algorithm described in step S4 uses image edge ground truth labels generated by back projection of continuous edge features of lidar point cloud depth for supervised learning during the training phase, so as to enhance the ability to extract structured edge features of the calibrated scene. During the inference phase, after extracting edge features, a geometric consistency filtering step is further included: the extracted image edge points are back-projected to a virtual 3D space based on camera intrinsic parameters and roughly aligned with the spatial distribution of the LiDAR point cloud to remove outlier edges that deviate from the spatial structure.

[0035] Specifically, after outputting an edge heatmap from the input image, the algorithm employs adaptive thresholding and non-maximum suppression techniques to extract a sub-pixel precision set of edge points. To address calibration requirements, geometric consistency filtering is further introduced: based on the camera intrinsic parameter model, the extracted edge points are back-projected into a virtual 3D space and roughly aligned with the spatial distribution of the radar point cloud, eliminating outliers that significantly deviate from the spatial structure, such as shadows from dynamic objects and reflective interference. This step significantly improves the initial matching compatibility between image edges and radar depth continuous edges.

[0036] The final output image edge features contain two key types of information: one is the sub-pixel coordinate position. Secondly, the local edge direction vector. The orientation information will be used for orthogonality verification in the subsequent matching stage to ensure the effectiveness of the geometric constraints between the projection points and the image edges, thus forming the basis for the high-precision residual equation.

[0037] The extracted depth-continuous laser point cloud data is projected onto the image plane using the following formula.

[0038] in f(.) represents the image plane coordinates; f(.) represents the camera distortion model; This represents a pinhole camera model; This represents Gaussian white noise. This represents the rigid body transformation from laser radar to the motor's rotation axis coordinate system; This represents a linear transformation from the motor rotation axis coordinate system to the camera coordinate system (using the current extrinsic parameter estimation results). This represents the laser point cloud in the lidar coordinate system.

[0039] Furthermore, the matching association described in step S5 specifically includes: For the depth-continuous edge features of the laser point cloud projected onto the image plane, search for its K nearest neighbors in the image edge features; Calculate the mean and covariance matrix of the K nearest neighbors, and take the eigenvector corresponding to the smallest eigenvalue of the covariance matrix as the normal vector of the image edge at that point; Verify the orthogonality between the direction of the continuous edge features of the laser point cloud depth and the normal vector to filter out incorrect matching pairs.

[0040] Specifically, based on the rotational position of the motor shaft, LiDAR point cloud data and camera image data are sampled at a fixed frequency ƒ at the same time. The search is then performed within the pixel data. of The nearest neighbor image edge points , The label representing the pixel. Calculation mean Covariance .Pick The eigenvector corresponding to the smallest eigenvalue is used as the normal vector of the edge feature. And verify the direction of laser point cloud edge features and The orthogonality of the pairs is used to filter out incorrect matches. The specific formula is: For a successfully matched pair of laser point cloud features and image features, a residual equation is constructed, and the projection point should fall on the matching edge.

[0041] Furthermore, the projection residual equation constructed in step S6 is: in It is the normal vector of the image edge; This represents the coordinates of the continuous edge features of the laser point cloud depth projected onto the image plane. Represents the pixel coordinates of edge features in the camera image; Gaussian white noise representing camera image data; Represents the laser point cloud in the lidar coordinate system; This represents the transformation matrix of the radar point cloud from time i-1 to time i, according to the sampling frequency ƒ. Represents the coordinates of the laser point in the lidar coordinate system at time i-1; Gaussian white noise represents the point cloud data from a lidar system.

[0042] The lidar point cloud data at different sampling times in the residual equation reuses the point cloud data used to match image edge features, which improves the correlation between camera, motor rotation axis and lidar calibration data, reduces the acquisition environment error introduced by using different data for calibration between different sensors, and significantly improves the stability of calibration results.

[0043] Furthermore, in step S7, the second rigid body transformation matrix is ​​solved iteratively using the Gauss-Newton method based on the linearized residual equation.

[0044] Specifically, the linearized residual equation is: in Represents the original residual vector; Represents the residual transformation from the motor's rotation axis to the camera. Jacobian matrix; The Jacobian matrix representing the residual with respect to image feature noise; This represents the residual transformation from laser radar to motor rotation axis. Jacobian matrix; This represents the Jacobian matrix of the residual relative to the noise in the lidar point cloud.

[0045] Based on the obtained linearized residual equation, the solution is obtained iteratively using the Gauss-Newton method. The rigid body transformation matrix from the motor rotation axis coordinate system to the camera coordinate system, i.e., the second rigid body transformation matrix.

[0046] Example 2 This invention also provides a joint calibration device based on a rotating lidar and a camera, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is a calibration program, and when the processor executes the calibration program, it performs the following steps: S1. Based on the DH parameter method, kinematic modeling is performed on the camera, lidar and motor rotation axis to define the first rigid body transformation from the lidar to the motor rotation axis coordinate system, the second rigid body transformation from the motor rotation axis coordinate system to the camera coordinate system, and the coordinate transformation relationship caused by the rotation angle of the motor rotation axis. S2. Collect synchronous lidar point cloud data and camera image data of the motor rotation shaft at different angles; S3. Process the lidar point cloud data and use a planar adaptive voxelization strategy to extract the depth continuous edge features of the lidar point cloud. S4. Process the camera image data and use the EdgeNAT edge detection algorithm based on Transformer to extract image edge features; S5. Extract the continuous edge features of the laser point cloud depth, project them onto the image plane based on the current extrinsic parameter estimates, and match and associate them with the image edge features extracted in step S4. S6. Based on the successfully matched feature pairs, construct the projection residual equations about the extrinsic parameters; S7. Linearize the residual equation and use a nonlinear optimization algorithm to iteratively optimize and solve it. Finally, obtain the first rigid body transformation parameters of the laser radar to the motor rotation axis coordinate system and the second rigid body transformation parameters of the motor rotation axis coordinate system to the camera coordinate system, and complete the joint calibration.

[0047] like Figure 3 As shown, the planar adaptive voxelization strategy described in step S3 further includes: S31. Divide the original point cloud into initial voxels; S32. For each voxel, calculate the eigenvalues ​​of the covariance matrix. Let the eigenvalues ​​of the covariance matrix be sorted by size as λ1≥λ2≥λ3. If the eigenvalues ​​satisfy... ,in If the plane threshold is met, the voxel is determined to be a planar voxel; otherwise, the voxel is divided into multiple sub-voxels and step S32 is recursively executed until all voxels satisfy the planarity condition or reach the minimum voxel size. S33. For adjacent planar voxels, calculate the angle between their planar normal vectors. If the angle exceeds the edge determination threshold, determine that there is a depth-continuous edge at the intersection of the two planar voxels, and extract the point cloud of that region as a valid edge feature.

[0048] Furthermore, the EdgeNAT edge detection algorithm described in step S4 uses image edge ground truth labels generated by back projection of continuous edge features of lidar point cloud depth for supervised learning during the training phase, so as to enhance the ability to extract structured edge features of the calibrated scene. During the inference phase, after extracting edge features, a geometric consistency filtering step is further included: the extracted image edge points are back-projected to a virtual 3D space based on camera intrinsic parameters and roughly aligned with the spatial distribution of the LiDAR point cloud to remove outlier edges that deviate from the spatial structure.

[0049] Furthermore, the matching association described in step S5 specifically includes: For the depth-continuous edge features of the laser point cloud projected onto the image plane, search for its K nearest neighbors in the image edge features; Calculate the mean and covariance matrix of the K nearest neighbors, and take the eigenvector corresponding to the smallest eigenvalue of the covariance matrix as the normal vector of the image edge at that point; Verify the orthogonality between the direction of the continuous edge features of the laser point cloud depth and the normal vector to filter out incorrect matching pairs.

[0050] Furthermore, the projection residual equation constructed in step S6 is: in It is the normal vector of the image edge; This represents the coordinates of the continuous edge features of the laser point cloud depth projected onto the image plane. Represents the pixel coordinates of edge features in the camera image; Gaussian white noise representing camera image data; Represents the laser point cloud in the lidar coordinate system; This represents the transformation matrix of the radar point cloud from time i-1 to time i, according to the sampling frequency ƒ. Represents the coordinates of the laser point in the lidar coordinate system at time i-1; Gaussian white noise represents the point cloud data from a lidar system.

[0051] Furthermore, in step S7, the second rigid body transformation matrix is ​​solved iteratively using the Gauss-Newton method based on the linearized residual equation.

[0052] Example 3 The present invention also provides a storage medium storing a computer program thereon, the computer program being a calibration program, which, when executed by a processor, performs the following steps: S1. Based on the DH parameter method, kinematic modeling is performed on the camera, lidar and motor rotation axis to define the first rigid body transformation from the lidar to the motor rotation axis coordinate system, the second rigid body transformation from the motor rotation axis coordinate system to the camera coordinate system, and the coordinate transformation relationship caused by the rotation angle of the motor rotation axis. S2. Collect synchronous lidar point cloud data and camera image data of the motor rotation shaft at different angles; S3. Process the lidar point cloud data and use a planar adaptive voxelization strategy to extract the depth continuous edge features of the lidar point cloud. S4. Process the camera image data and use the EdgeNAT edge detection algorithm based on Transformer to extract image edge features; S5. Extract the continuous edge features of the laser point cloud depth, project them onto the image plane based on the current extrinsic parameter estimates, and match and associate them with the image edge features extracted in step S4. S6. Based on the successfully matched feature pairs, construct the projection residual equations about the extrinsic parameters; S7. Linearize the residual equation and use a nonlinear optimization algorithm to iteratively optimize and solve it. Finally, obtain the first rigid body transformation parameters of the laser radar to the motor rotation axis coordinate system and the second rigid body transformation parameters of the motor rotation axis coordinate system to the camera coordinate system, and complete the joint calibration.

[0053] Furthermore, the planar adaptive voxelization strategy described in step S3 specifically includes: S31. Divide the original point cloud into initial voxels; S32. For each voxel, calculate the eigenvalues ​​of the covariance matrix. Let the eigenvalues ​​of the covariance matrix be sorted by size as λ1≥λ2≥λ3. If the eigenvalues ​​satisfy... ,in If the plane threshold is met, the voxel is determined to be a planar voxel; otherwise, the voxel is divided into multiple sub-voxels and step S32 is recursively executed until all voxels satisfy the planarity condition or reach the minimum voxel size. S33. For adjacent planar voxels, calculate the angle between their planar normal vectors. If the angle exceeds the edge determination threshold, determine that there is a depth-continuous edge at the intersection of the two planar voxels, and extract the point cloud of that region as a valid edge feature.

[0054] Furthermore, the EdgeNAT edge detection algorithm described in step S4 uses image edge ground truth labels generated by back projection of continuous edge features of lidar point cloud depth for supervised learning during the training phase, so as to enhance the ability to extract structured edge features of the calibrated scene. During the inference phase, after extracting edge features, a geometric consistency filtering step is further included: the extracted image edge points are back-projected to a virtual 3D space based on camera intrinsic parameters and roughly aligned with the spatial distribution of the LiDAR point cloud to remove outlier edges that deviate from the spatial structure.

[0055] Furthermore, the matching association described in step S5 specifically includes: For the depth-continuous edge features of the laser point cloud projected onto the image plane, search for its K nearest neighbors in the image edge features; Calculate the mean and covariance matrix of the K nearest neighbors, and take the eigenvector corresponding to the smallest eigenvalue of the covariance matrix as the normal vector of the image edge at that point; Verify the orthogonality between the direction of the continuous edge features of the laser point cloud depth and the normal vector to filter out incorrect matching pairs.

[0056] Furthermore, the projection residual equation constructed in step S6 is: in It is the normal vector of the image edge; This represents the coordinates of the continuous edge features of the laser point cloud depth projected onto the image plane. Represents the pixel coordinates of edge features in the camera image; Gaussian white noise representing camera image data; Represents the laser point cloud in the lidar coordinate system; This represents the transformation matrix of the radar point cloud from time i-1 to time i, according to the sampling frequency ƒ. Represents the coordinates of the laser point in the lidar coordinate system at time i-1; Gaussian white noise represents the point cloud data from a lidar system.

[0057] Furthermore, in step S7, the second rigid body transformation matrix is ​​solved iteratively using the Gauss-Newton method based on the linearized residual equation.

[0058] Example 4 This invention also provides a joint calibration system based on a rotating lidar and a camera, used to perform the calibration method described in Embodiment 1, the system comprising: The kinematic modeling module is used to perform kinematic modeling of the camera, lidar, and motor rotation axis based on the DH parameter method, in order to define the first rigid body transformation from the lidar to the motor rotation axis coordinate system, the second rigid body transformation from the motor rotation axis coordinate system to the camera coordinate system, and the coordinate transformation relationship caused by the rotation angle of the motor rotation axis. The data acquisition module is used to acquire synchronous lidar point cloud data and camera image data of the motor rotation shaft at different angles; The point cloud feature extraction module is used to process lidar point cloud data and uses a planar adaptive voxelization strategy to extract the depth continuous edge features of the lidar point cloud. The image feature extraction module is used to process camera image data and uses the Transformer-based EdgeNAT edge detection algorithm to extract image edge features; The feature matching module is used to project the extracted laser point cloud depth continuous edge features onto the image plane based on the current extrinsic parameter estimates, and match and associate them with the image edge features. The residual construction module is used to construct projection residual equations about the extrinsic parameters based on successfully matched feature pairs. The optimization and solution module is used to linearize the residual equation and iteratively optimize the solution using a nonlinear optimization algorithm to finally obtain the rigid body transformation parameters from the motor rotation axis coordinate system to the camera coordinate system, thus completing the joint calibration.

[0059] The calibration method used in the joint calibration system based on rotating lidar and camera includes the following steps: S1. Based on the DH parameter method, kinematic modeling is performed on the camera, lidar and motor rotation axis. The first rigid body transformation from the lidar to the motor rotation axis coordinate system, the second rigid body transformation from the motor rotation axis coordinate system to the camera coordinate system, and the coordinate transformation relationship caused by the rotation angle θ of the motor rotation axis are clearly defined. S2. Collect synchronous lidar point cloud data and camera image data of the motor rotation shaft at different angles; S3. Process the lidar point cloud data and use a planar adaptive voxelization strategy to extract the depth continuous edge features of the lidar point cloud. S4. Process the camera image data and use the EdgeNAT edge detection algorithm based on Transformer to extract image edge features; S5. Extract the continuous edge features of the laser point cloud depth, project them onto the image plane based on the current extrinsic parameter estimates, and match and associate them with the image edge features extracted in step S4. S6. Based on the successfully matched feature pairs, construct the projection residual equations about the extrinsic parameters; S7. Linearize the residual equation and use a nonlinear optimization algorithm to iteratively optimize and solve it. Finally, obtain the first rigid body transformation parameters of the laser radar to the motor rotation axis coordinate system and the second rigid body transformation parameters of the motor rotation axis coordinate system to the camera coordinate system, and complete the joint calibration.

[0060] Furthermore, the planar adaptive voxelization strategy described in step S3 specifically includes: S31. Divide the original point cloud into initial voxels; S32. For each voxel, calculate the eigenvalues ​​of the covariance matrix. Let the eigenvalues ​​of the covariance matrix be sorted by size as λ1≥λ2≥λ3. If the eigenvalues ​​satisfy... ,in If the plane threshold is met, the voxel is determined to be a planar voxel; otherwise, the voxel is divided into multiple sub-voxels and step S32 is recursively executed until all voxels satisfy the planarity condition or reach the minimum voxel size. S33. For adjacent planar voxels, calculate the angle between their planar normal vectors. If the angle exceeds the edge determination threshold, determine that there is a depth-continuous edge at the intersection of the two planar voxels, and extract the point cloud of that region as a valid edge feature.

[0061] Furthermore, the EdgeNAT edge detection algorithm described in step S4 uses image edge ground truth labels generated by back projection of continuous edge features of lidar point cloud depth for supervised learning during the training phase, so as to enhance the ability to extract structured edge features of the calibrated scene. During the inference phase, after extracting edge features, a geometric consistency filtering step is further included: the extracted image edge points are back-projected to a virtual 3D space based on camera intrinsic parameters and roughly aligned with the spatial distribution of the LiDAR point cloud to remove outlier edges that deviate from the spatial structure.

[0062] Furthermore, the matching association described in step S5 specifically includes: For the depth-continuous edge features of the laser point cloud projected onto the image plane, search for its K nearest neighbors in the image edge features; Calculate the mean and covariance matrix of the K nearest neighbors, and take the eigenvector corresponding to the smallest eigenvalue of the covariance matrix as the normal vector of the image edge at that point; Verify the orthogonality between the direction of the continuous edge features of the laser point cloud depth and the normal vector to filter out incorrect matching pairs.

[0063] Furthermore, the projection residual equation constructed in step S6 is: in It is the normal vector of the image edge; This represents the coordinates of the continuous edge features of the laser point cloud depth projected onto the image plane. Represents the pixel coordinates of edge features in the camera image; Gaussian white noise representing camera image data; Represents the laser point cloud in the lidar coordinate system; This represents the transformation matrix of the radar point cloud from time i-1 to time i, according to the sampling frequency ƒ. Represents the coordinates of the laser point in the lidar coordinate system at time i-1; Gaussian white noise represents the point cloud data from a lidar system.

[0064] Furthermore, in step S7, the second rigid body transformation matrix is ​​solved iteratively using the Gauss-Newton method based on the linearized residual equation.

[0065] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A joint calibration method based on rotating lidar and camera, characterized in that, Includes the following steps: S1. Based on the DH parameter method, kinematic modeling is performed on the camera, lidar and motor rotation axis to define the first rigid body transformation from the lidar to the motor rotation axis coordinate system, the second rigid body transformation from the motor rotation axis coordinate system to the camera coordinate system, and the coordinate transformation relationship caused by the rotation angle of the motor rotation axis. S2. Collect synchronous lidar point cloud data and camera image data of the motor rotation shaft at different angles; S3. Process the lidar point cloud data and use a planar adaptive voxelization strategy to extract the depth continuous edge features of the lidar point cloud. S4. Process the camera image data and use the EdgeNAT edge detection algorithm based on Transformer to extract image edge features; S5. Extract the continuous edge features of the laser point cloud depth, project them onto the image plane based on the current extrinsic parameter estimates, and match and associate them with the image edge features extracted in step S4. S6. Based on the successfully matched feature pairs, construct the projection residual equations about the extrinsic parameters; S7. Linearize the residual equation and use a nonlinear optimization algorithm to iteratively optimize and solve it. Finally, obtain the first rigid body transformation parameters of the laser radar to the motor rotation axis coordinate system and the second rigid body transformation parameters of the motor rotation axis coordinate system to the camera coordinate system, and complete the joint calibration.

2. The joint calibration method based on rotating lidar and camera according to claim 1, characterized in that, The planar adaptive voxelization strategy described in step S3 specifically includes: S31. Divide the original point cloud into initial voxels; S32. For each voxel, calculate the eigenvalues ​​of the covariance matrix. Let the eigenvalues ​​of the covariance matrix be sorted by size as λ1≥λ2≥λ3. If the eigenvalues ​​satisfy... ,in If the plane threshold is met, the voxel is determined to be a planar voxel; otherwise, the voxel is divided into multiple sub-voxels and step S32 is recursively executed until all voxels satisfy the planarity condition or reach the minimum voxel size. S33. For adjacent planar voxels, calculate the angle between their planar normal vectors. If the angle exceeds the edge determination threshold, determine that there is a depth-continuous edge at the intersection of the two planar voxels, and extract the point cloud of that region as a valid edge feature.

3. The joint calibration method based on rotating lidar and camera according to claim 1, characterized in that, The EdgeNAT edge detection algorithm described in step S4 uses image edge ground value labels generated by back projection of continuous edge features of lidar point cloud depth for supervised learning during the training phase, so as to enhance the ability to extract structured edge features of the calibrated scene. During the inference phase, after extracting edge features, a geometric consistency filtering step is further included: the extracted image edge points are back-projected to a virtual 3D space based on camera intrinsic parameters and roughly aligned with the spatial distribution of the LiDAR point cloud to remove outlier edges that deviate from the spatial structure.

4. The joint calibration method based on rotating lidar and camera according to claim 1, characterized in that, The matching association mentioned in step S5 specifically includes: For the depth-continuous edge features of the laser point cloud projected onto the image plane, search for its K nearest neighbors in the image edge features; Calculate the mean and covariance matrix of the K nearest neighbors, and take the eigenvector corresponding to the smallest eigenvalue of the covariance matrix as the normal vector of the image edge at that point; Verify the orthogonality between the direction of the continuous edge features of the laser point cloud depth and the normal vector to filter out incorrect matching pairs.

5. The joint calibration method based on rotating lidar and camera according to claim 4, characterized in that, The projection residual equation constructed in step S6 is as follows: in It is the normal vector of the image edge; This represents the coordinates of the continuous edge features of the laser point cloud depth projected onto the image plane. Represents the pixel coordinates of edge features in the camera image; Gaussian white noise representing camera image data; Represents the laser point cloud in the lidar coordinate system; This represents the transformation matrix of the radar point cloud from time i-1 to time i, according to the sampling frequency ƒ. Represents the coordinates of the laser point in the lidar coordinate system at time i-1; Gaussian white noise represents the point cloud data from a lidar system.

6. The joint calibration method based on rotating lidar and camera according to claim 5, characterized in that, In step S7, the second rigid body transformation matrix is ​​solved iteratively using the Gauss-Newton method based on the linearized residual equation.

7. A joint calibration device based on a rotating lidar and a camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is a calibration program, characterized in that... When the processor executes the calibration program, it implements the steps of the method as described in any one of claims 1-6.

8. A storage medium storing a computer program thereon, said computer program being a calibration program, characterized in that, When the calibration program is executed by the processor, it implements the steps of the method as described in any one of claims 1-6.

9. A joint calibration system based on a rotating lidar and a camera, used to perform the method as described in any one of claims 1-6, characterized in that, The system includes: The kinematic modeling module is used to perform kinematic modeling of the camera, lidar, and motor rotation axis based on the DH parameter method. It clearly defines the first rigid body transformation from the lidar to the motor rotation axis coordinate system, the second rigid body transformation from the motor rotation axis coordinate system to the camera coordinate system, and the coordinate transformation relationship caused by the rotation angle θ of the motor rotation axis. The data acquisition module is used to acquire synchronous lidar point cloud data and camera image data of the motor rotation shaft at different angles; The point cloud feature extraction module is used to process lidar point cloud data and uses a planar adaptive voxelization strategy to extract the depth continuous edge features of the lidar point cloud. The image feature extraction module is used to process camera image data and uses the Transformer-based EdgeNAT edge detection algorithm to extract image edge features; The feature matching module is used to project the extracted laser point cloud depth continuous edge features onto the image plane based on the current extrinsic parameter estimates, and match and associate them with the image edge features. The residual construction module is used to construct projection residual equations about the extrinsic parameters based on successfully matched feature pairs. The optimization and solution module is used to linearize the residual equation and iteratively optimize the solution using a nonlinear optimization algorithm to finally obtain the rigid body transformation parameters from the motor rotation axis coordinate system to the camera coordinate system, thus completing the joint calibration.

Images