A point cloud calibration method and system

Abstract

The application discloses a kind of point cloud calibration method and system, applied to point cloud calibration technical field, method includes: obtaining the point cloud image and visual image of calibration object, determine initial external parameter;Point cloud image is converted into point cloud depth map based on initial external parameter, and visual image is converted into image depth map;The geometric data of point cloud depth map is extracted, to obtain geometric information, obtain image information;The geometric information and image information contained in each pixel point in depth map are processed, and space credibility distribution is constructed;Initial external parameter is processed based on space credibility distribution, to obtain first external parameter;First external parameter and dense point cloud depth map are input into pre-constructed external parameter correction model and are corrected, to obtain target external parameter;Point cloud image is calibrated based on target external parameter.The point cloud calibration method and system provided in the application effectively improve the accuracy of external parameter setting, and then improve the stability and reliability of point cloud calibration.

Description

Technical Field

[0001] This invention relates to the field of point cloud calibration technology, and in particular to a point cloud calibration method and system. Background Technology

[0002] Point cloud data possesses high-precision 3D geometric information but lacks semantic details, while visual data such as RGB, grayscale, or thermal imaging images provide rich texture and semantic content but lack reliable depth information. Point cloud calibration accurately transforms point cloud data from its original coordinate system to the camera coordinate system by precisely solving the extrinsic parameters between the LiDAR and the camera, providing a reliable pose basis for multimodal perception tasks.

[0003] Existing point cloud calibration methods directly utilize initial extrinsic parameters derived from camera intrinsic parameters for point cloud calibration. For example, they may directly use only the sensor's designed installation position and orientation data on the vehicle or robot as extrinsic parameters for point cloud coordinate system transformation. This leads to uncorrected systematic biases in the calibration results, such as translational offsets, rotational errors, or scale inaccuracies. Consequently, the spatial position of the point cloud data in the target coordinate system, such as the camera coordinate system, becomes inaccurate, manifesting as structural distortion, distance distortion, or attitude deviation. This severely affects the geometric consistency and positioning accuracy of the point cloud itself, reducing its reliability in tasks such as high-precision mapping, obstacle ranging, and pose estimation. Summary of the Invention

[0004] This invention provides a point cloud calibration method and system to solve the technical problem of inaccurate extrinsic parameters in existing point cloud calibration methods, thereby improving the accuracy of extrinsic parameters and thus improving the precision and stability of point cloud calibration.

[0005] To address the aforementioned technical problems, this invention provides a point cloud calibration method and system, the method comprising: Acquire point cloud images and visual images of the calibration object, and determine initial extrinsic parameters based on the geometric parameters of the calibration object; Based on the initial extrinsic parameters, the point cloud image is converted into a point cloud depth map, and the visual image is converted into an image depth map; Extract the geometric data from the point cloud depth map to obtain geometric information, and perform visual image information extraction processing on the image depth map to obtain image information; The geometric information and image information contained in each pixel in the depth map obtained by aligning the point cloud depth map with the image depth map are processed, and a spatial confidence distribution is constructed based on the processing result and the position of the pixel in the depth map. The initial extrinsic parameters are processed based on the spatial credibility distribution to obtain the first extrinsic parameter; The first extrinsic parameter and the dense point cloud depth map obtained by interpolating and completing the point cloud depth map are input into the pre-constructed extrinsic parameter correction model for correction to obtain the target extrinsic parameter. The point cloud image is calibrated based on the target extrinsic parameters.

[0006] Preferably, determining the initial extrinsic parameters based on the geometric parameters of the calibration object includes: Based on the geometric parameters, the calibration object is modeled to obtain the calibration object model; Feature extraction is performed on the visual image of the calibrated object to determine the visual features; Extract the three-dimensional coordinates of the calibration object from the point cloud image to obtain three-dimensional data; Based on the visual features and the three-dimensional data, the calibration model is subjected to spatial transformation to obtain the initial extrinsic parameters.

[0007] Preferably, the step of extracting geometric data from the point cloud depth map to obtain geometric information, and performing visual image information extraction processing on the image depth map to obtain image information, includes: Extract the three-dimensional coordinate data from the point cloud depth map to obtain a set of spatial coordinates; The spatial coordinate set is processed using neighborhood search technology to obtain a set of geometric feature points; The geometric information is obtained based on the set of geometric feature points; Visual image information is extracted from the depth map of the image to obtain a preliminary visual feature map; The preliminary visual feature map is processed using edge detection technology to obtain an edge map; The edge map is segmented, and image information of all segmented regions is extracted to obtain the image information of the image depth map.

[0008] Preferably, the step of processing the geometric information and image information contained in each pixel of the depth map obtained by aligning the point cloud depth map and the image depth map, and constructing a spatial confidence distribution based on the processing result and the position of the pixel in the depth map, includes: The distance between each pixel and the optical center is obtained based on the depth map. The distance is then used to obtain a distance decay weight through a Gaussian function. The distance decay weight is then used to construct a prior space weight matrix. Based on the geometric information of each pixel, a geometric integrity index is obtained; Based on the image information of each pixel and its position in the depth map, a visual sharpness index is obtained; The geometric integrity index and the visual clarity index are processed to generate a credibility score; The credibility score is processed based on the prior space weight matrix to obtain the comprehensive credibility. The spatial confidence distribution is constructed based on the overall confidence level and the position of each pixel in the depth map.

[0009] Preferably, the step of processing the initial extrinsic parameters based on the spatial confidence distribution to obtain the first extrinsic parameters includes: Based on the initial extrinsic parameters, the point cloud image is projected onto the visual image to obtain the point cloud visual image under initial registration. Based on the spatial credibility distribution, all matching point pairs in the point cloud visual image are weighted to obtain a weighted point pair set. Based on the set of weighted point pairs, the weighted projection error is obtained; The first extrinsic parameter is obtained by minimizing the weighted projection error.

[0010] Another aspect of the present invention provides a point cloud calibration system, comprising: The acquisition module is used to acquire point cloud images and visual images of the calibration object, and determine initial extrinsic parameters based on the geometric parameters of the calibration object; The depth map module is used to convert the point cloud image into a point cloud depth map based on the initial extrinsic parameters, and to convert the visual image into an image depth map; The image module is used to extract the geometric data of the point cloud depth map to obtain geometric information, and to perform visual image information extraction processing on the image depth map to obtain image information; The construction module is used to process the geometric information and image information contained in each pixel in the depth map obtained by aligning the point cloud depth map and the image depth map, and to construct a spatial confidence distribution based on the processing result and the position of the pixel in the depth map. The processing module is used to process the initial extrinsic parameters based on the spatial confidence distribution to obtain the first extrinsic parameters; The correction module is used to input the first extrinsic parameter and the dense point cloud depth map obtained by interpolating and completing the point cloud depth map into a pre-constructed extrinsic parameter correction model for correction, so as to obtain the target extrinsic parameter; The calibration module is used to calibrate the point cloud image based on the target extrinsic parameters.

[0011] Preferably, determining the initial extrinsic parameters based on the geometric parameters of the calibration object includes: A modeling unit is used to model the calibration object based on the geometric parameters to obtain a calibration object model; The extraction unit is used to extract features from the visual image of the calibration object and determine the visual features; A three-dimensional unit is used to extract the three-dimensional coordinates of the calibration object in the point cloud image to obtain three-dimensional data. The processing unit is used to perform spatial transformation processing on the calibration object model based on the visual features and the three-dimensional data to obtain the initial extrinsic parameters.

[0012] Preferably, the image module includes: The extraction unit is used to extract the three-dimensional coordinate data from the point cloud depth map to obtain a set of spatial coordinates; Geometric feature units are used to process the set of spatial coordinates using neighborhood search technology to obtain a set of geometric feature points; A geometric information unit is used to obtain the geometric information based on the set of geometric feature points; A visual feature unit is used to extract visual image information from the image depth map to obtain a preliminary visual feature map. The edge map unit is used to process the preliminary visual feature map using edge detection technology to obtain an edge map; The image information unit is used to segment the edge map, extract image information of all regions after segmentation, and obtain the image information of the image depth map.

[0013] Preferably, the building module includes: The weight matrix unit is used to obtain the distance between each pixel and the optical center based on the depth map, obtain the distance decay weight through a Gaussian function using the distance, and construct the prior space weight matrix using the distance decay weight. A geometric unit is used to obtain a geometric integrity index based on the geometric information of each pixel. A visual unit is used to obtain a visual sharpness index based on the image information of each pixel and its position in the depth map. A credibility unit is used to process the geometric integrity index and the visual clarity index to generate a credibility score. The synthesis unit is used to process the credibility score based on the prior space weight matrix to obtain the comprehensive credibility. A credibility distribution unit is used to construct the spatial credibility distribution based on the overall credibility and the position of each pixel in the depth map.

[0014] Preferably, the processing module includes: A registration unit is used to project the point cloud image onto the visual image based on the initial extrinsic parameters to obtain a point cloud visual image under the initial registration. A weighting unit is used to perform weighting processing on all matching point pairs in the point cloud visual image based on the spatial confidence distribution to obtain a weighted point pair set. An error unit is used to obtain the weighted projection error based on the set of weighted point pairs; The first unit is used to minimize the weighted projection error to obtain the first extrinsic parameter.

[0015] Compared with the prior art, the beneficial effects of the present invention are at least one of the following: This invention determines initial extrinsic parameters by introducing the geometric parameters of a calibration object. Based on this, point cloud images and visual images are converted into point cloud depth maps and image depth maps, respectively, and their respective geometric and image information is extracted. By fusing the geometric and image features of each pixel in the aligned depth maps and constructing a spatial reliability distribution based on the spatial location of the pixels, the initial extrinsic parameters are weighted and optimized to obtain the first extrinsic parameter. This is further corrected by combining the interpolated dense point cloud depth map with a pre-constructed extrinsic parameter correction model, ultimately obtaining high-precision target extrinsic parameters. This scheme effectively overcomes the systematic deviations, such as translational offsets, rotational errors, or scale inaccuracies, caused by existing methods relying solely on design and installation parameters or coarse initial extrinsic parameters. It significantly improves the spatial positioning accuracy of point clouds in the camera coordinate system, avoiding problems such as structural distortion, distance distortion, or attitude deviation, thereby enhancing the geometric consistency and reliability of point cloud data in multimodal perception tasks such as high-precision mapping, obstacle ranging, and pose estimation. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a point cloud calibration method in one embodiment of the present invention; Figure 2 This is a schematic diagram of the point cloud calibration system in one embodiment of the present invention; Figure label: The module consists of: 11. Acquisition module; 12. Depth map module; 13. Image module; 14. Construction module; 15. Processing module; 16. Correction module; and 17. Calibration module. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] In the description of this invention, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0019] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0020] In the description of this invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] Point cloud data possesses high-precision geometric information but lacks semantics, while visual images are rich in texture and semantics but lack reliable depth. Point cloud calibration aims to accurately solve for the extrinsic parameters between LiDAR and the camera, achieving alignment of multimodal data in a unified coordinate system. However, existing methods often directly use initial extrinsic parameters derived from installation design or camera intrinsic parameters without correcting for inherent biases. This leads to translational shifts, rotational errors, or scale inaccuracies, causing structural distortion, distance distortion, or attitude deviation in the point cloud within the camera coordinate system. This severely impacts its accuracy and reliability in tasks such as high-precision mapping, obstacle ranging, and pose estimation.

[0022] One embodiment of the present invention provides a point cloud calibration method. For details, please refer to [link to relevant documentation]. Figure 1 , Figure 1 The diagram shown is a schematic flowchart of a point cloud calibration method according to one embodiment of the present invention, including: S1. Acquire the point cloud image and visual image of the calibration object, and determine the initial extrinsic parameters based on the geometric parameters of the calibration object; S2. Based on the initial extrinsic parameters, convert the point cloud image into a point cloud depth map, and convert the visual image into an image depth map; S3. Extract the geometric data of the point cloud depth map to obtain geometric information, and perform visual image information extraction processing on the image depth map to obtain image information; S4. Process the geometric and image information contained in each pixel in the depth map obtained by aligning the point cloud depth map and the image depth map, and construct a spatial confidence distribution based on the processing results and the position of the pixel in the depth map. S5. Process the initial extrinsic parameters based on the spatial credibility distribution to obtain the first extrinsic parameter; S6. Input the first extrinsic parameter and the dense point cloud depth map obtained by interpolating and completing the point cloud depth map into the pre-built extrinsic parameter correction model for correction to obtain the target extrinsic parameter; S7. Calibrate the point cloud image based on the target extrinsic parameters.

[0023] First, point cloud images and visual images of the calibration object are acquired, and initial extrinsic parameters are determined based on the geometric parameters of the calibration object. Based on the geometric parameters, the calibration object is modeled to obtain the calibration object model; features are extracted from the visual image of the calibration object to determine the visual features; the three-dimensional coordinates of the calibration object are extracted from the point cloud image to obtain three-dimensional data; based on the visual features and the three-dimensional data, the calibration object model is subjected to spatial transformation processing to obtain the initial extrinsic parameters. The process involves acquiring point cloud images and visual images of a calibration object and determining initial extrinsic parameters based on the geometric parameters of the calibration object. The calibration object refers to a standard reference object used to calibrate the point cloud image and the visual image. Common examples include checkerboard calibration boards, dot calibration boards, and 3D calibration blocks. Its geometric parameters are the inherent dimensional parameters of the calibration object itself, including specific measurable data such as length, width, height, hole spacing, and corner angles. The initial extrinsic parameters refer to the initial parameters used to describe the relative position and orientation relationship between the point cloud image acquisition device (such as LiDAR) and the visual image acquisition device (such as a camera). These mainly include rotation matrices and translation vectors. The core of determining the initial extrinsic parameters is to establish the spatial association between the two devices through the known geometric features of the calibration object, ensuring accurate alignment of the point cloud and the image in the future.

[0024] Based on the geometric parameters of the aforementioned calibration object, a calibration object model is obtained by modeling it. The modeling method can adopt parametric modeling, which directly constructs a three-dimensional model by combining the geometric parameters of the calibration object. For example, a checkerboard calibration board can be constructed into a three-dimensional model composed of multiple rectangular planes based on its grid size and number of grids. A three-dimensional calibration block can be constructed into a cuboid three-dimensional model based on its length, width, and height parameters. Modeling tools can include Open3D, PCL (Point Cloud Library), and MATLAB, etc. By calling the tool interface through programming and inputting geometric parameters, an accurate calibration object model can be generated. Visual features are extracted from the visual image of the calibration object to determine its visual characteristics. These visual features are key information that makes the calibration object recognizable in the visual image, including corners, edges, textures, and contours. The feature extraction method should be an accurate and efficient algorithm. For corner extraction, the Harris corner detection algorithm and the Shi-Tomasi corner detection algorithm can be used. The Shi-Tomasi algorithm is superior to the Harris algorithm in terms of stability and accuracy in corner detection. For edge extraction, the Canny edge detection algorithm can be used. First, the visual image is denoised by Gaussian blurring. Then, the image gradient is calculated to determine the edge strength and direction. Finally, accurate edge features are obtained by double thresholding. For texture extraction, the Local Binary Mode algorithm can be used. By comparing the gray values ​​of pixels with those of their neighbors to generate binary codes, the texture features of the calibration object's surface can be extracted. The extracted visual features need to be filtered and denoised to remove redundant and erroneous features to improve the accuracy of subsequent processing.

[0025] The 3D data is obtained by extracting the 3D coordinates of the calibration object from the point cloud image. The point cloud image is a discrete data set composed of a large number of 3D points. Each point contains spatial coordinates in three directions: X, Y, and Z. The 3D data is the set of 3D coordinates of all points in the region corresponding to the calibration object. The extraction method can first preprocess the original point cloud image by removing noise points and outliers in the point cloud through pass-through filtering and statistical filtering. Then, a segmentation algorithm is used to separate the point cloud region corresponding to the calibration object. Commonly used segmentation algorithms include RANSAC (Random Sample Consensus) and region growing algorithms. The RANSAC algorithm can fit the geometric plane or surface of the calibration object by random sampling, and then segment the calibration object point cloud. The region growing algorithm can grow from the seed point to obtain the complete calibration object point cloud region based on the normal vector and curvature similarity of the point cloud. After the segmentation is completed, the 3D coordinates of all points in the region are extracted, which is the required 3D data.

[0026] Initial extrinsic parameters are obtained by performing spatial transformation processing on the calibration model based on the extracted visual features and 3D data. Spatial transformation processing refers to matching the pose and position of the calibration model with the 3D data of the calibration object in the point cloud image and the visual features of the calibration object in the visual image through transformation operations such as rotation and translation. Specifically, ICP (Iterative Closed Components) can be used for this purpose. The Point (Iterative Closest Point) algorithm, combined with the Perspective-n-Point (PnP) algorithm, first uses the PnP algorithm to solve for the rotation matrix and translation vector using the 2D coordinates corresponding to visual features and the 3D coordinates in the 3D data, obtaining initial transformation parameters. Then, these initial transformation parameters are applied to the calibration model, and the transformation parameters are iteratively optimized using the ICP algorithm to minimize the distance error between the points on the calibration model and the 3D data points in the point cloud image. At the same time, it ensures that the projection of the calibration model in the visual image is accurately aligned with the extracted visual features. The iteration termination condition is set to the distance error being less than a preset threshold or the number of iterations reaching a preset upper limit. The final optimized transformation parameters are the initial extrinsic parameters. This approach can fully utilize the known geometric features of the calibration object, combined with the 3D information of the point cloud and the 2D features of vision, to quickly and accurately obtain the initial extrinsic parameters, laying a reliable foundation for subsequent extrinsic parameter optimization and point cloud calibration.

[0027] Next, based on the initial extrinsic parameters, the point cloud image is converted into a point cloud depth map, and the visual image is converted into an image depth map. The point cloud depth map is an image obtained by mapping the three-dimensional spatial information of the point cloud image onto a two-dimensional image plane. The grayscale value or pixel value of each pixel corresponds to the depth information of the point cloud at that two-dimensional location, i.e., the Z-axis coordinate value. The image depth map is an image obtained by assigning depth attributes to the two-dimensional pixel information of the visual image; its pixels also contain depth information at their corresponding locations. Both provide a unified format of depth data support for subsequent depth map alignment and information fusion.

[0028] The specific method for converting a point cloud image into a point cloud depth map requires consideration of initial extrinsic parameters. First, using the rotation matrix and translation vector in the initial extrinsic parameters, the 3D coordinates of all points in the point cloud image are transformed from the LiDAR coordinate system to the camera coordinate system, achieving coordinate system unification between the point cloud and the visual image. This transformation is accomplished through matrix multiplication, where the 3D coordinates of each point are multiplied by the rotation matrix and then the translation vector is added to obtain the 3D coordinates in the camera coordinate system. Subsequently, a projection transformation is used to project the 3D points in the camera coordinate system onto the image plane. A pinhole camera model can be used for projection, calculating the coordinates of each point based on the focal length and principal point coordinates in the camera's intrinsic parameters. The depth map calculates the two-dimensional pixel coordinates of a point on the image plane, while retaining the Z-axis depth value of that point. Finally, according to the position of the two-dimensional pixel coordinates, the corresponding depth value is assigned to the corresponding pixel in the point cloud depth map. For multiple three-dimensional points projected onto the same pixel coordinate, the depth value of the pixel can be determined by taking the average, maximum, or minimum value. Commonly used tools include PCL and Open3D. The transformation is completed by calling their coordinate transformation and projection interfaces through programming. After the transformation, the generated point cloud depth map needs to be denoised to remove pixels corresponding to invalid depth values ​​and ensure the validity of the point cloud depth map.

[0029] The core of converting a visual image into a depth map is to assign accurate depth information to each pixel of the visual image. Specific methods can be divided into two types. One is a mapping method based on initial extrinsic parameters and a point cloud depth map. This method utilizes the correspondence between the point cloud and the visual image determined by the initial extrinsic parameters to map the existing depth values ​​in the point cloud depth map to the corresponding pixels in the visual image through pixel matching. Pixel matching can employ algorithms such as the sum of squared differences and the sum of absolute differences. By calculating the sum of grayscale differences between the corresponding pixel regions in the point cloud depth map and the visual image, the optimal matching pixel is found, and its corresponding depth value is assigned to the visual image pixel. For pixels without a matching depth value, bilinear interpolation and nearest neighbor interpolation methods are used to supplement the depth value. The other method is based on a monocular depth estimation algorithm. A pre-trained monocular depth estimation model can be used as the depth estimation model of the backbone network. The visual image is input into the model, and the model learns the texture and contour features of the visual image, directly outputting the depth values ​​of the corresponding pixels to generate the image depth map. This method does not rely on point cloud data and can quickly generate a complete image depth map. In practical applications, an appropriate conversion method can be selected based on the completeness of the point cloud data.

[0030] The purpose of this is to convert point cloud images and visual images of different formats into a unified depth map format, which facilitates the subsequent extraction of geometric and image information. The benefits are that a unified data format reduces the difficulty of subsequent processing, while giving the visual image a depth attribute, providing a foundation for the subsequent fusion processing of geometric and image information of pixels and the construction of spatial reliability distribution, and improving the accuracy of subsequent extrinsic parameter correction and point cloud calibration.

[0031] Then, geometric data is extracted from the point cloud depth map to obtain geometric information, and visual image information extraction processing is performed on the image depth map to obtain image information. Three-dimensional coordinate data is extracted from the point cloud depth map to obtain a spatial coordinate set; the spatial coordinate set is processed using neighborhood search technology to obtain a geometric feature point set; geometric information is obtained based on the geometric feature point set; visual image information is extracted from the image depth map to obtain a preliminary visual feature map; edge detection technology is used to process the preliminary visual feature map to obtain an edge map; the edge map is segmented, and image information of all segmented regions is extracted to obtain the image information of the image depth map.

[0032] Geometric information is obtained by extracting geometric data from the point cloud depth map, and visual image information is extracted from the image depth map to obtain image information. Geometric information is the core data that reflects the spatial geometric features of the point cloud depth map, including the set of spatial coordinates, the set of geometric feature points, and the distance and angular relationships between feature points. Image information is the key data that reflects the visual features of the image depth map, including edge contours, regional textures, and pixel grayscale distribution. Extracting both is for subsequent information fusion processing of pixels after the depth map is aligned. The benefit is that it provides accurate data support for the construction of spatial reliability distribution and improves the accuracy of subsequent extrinsic parameter correction. Extracting 3D coordinate data from a point cloud depth map yields a spatial coordinate set. This 3D coordinate data represents the 3D spatial coordinates corresponding to each valid pixel in the point cloud depth map. The extraction method first reads the pixel values ​​from the point cloud depth map. Each pixel value corresponds to the Z-axis depth value in the 3D coordinate system. Then, combining this with the pixel's 2D coordinates on the image plane, as well as the camera's intrinsic and initial extrinsic parameters, a reverse projection transformation is used to calculate the complete 3D coordinates corresponding to each pixel. The reverse projection transformation is implemented through the inverse operation of the pinhole camera model. That is, based on the 2D pixel coordinates and Z-axis depth value, combined with the camera's focal length and principal point coordinates, the X-axis and Y-axis coordinates in the camera coordinate system are derived, thus forming a complete 3D coordinate system. After extraction, all valid 3D coordinates are organized to form a spatial coordinate set. This can be achieved using the coordinate reading and reverse projection interfaces of PCL and Open3D tools. Invalid 3D coordinates are removed to ensure the validity of the spatial coordinate set. Neighborhood search techniques are used to process a set of spatial coordinates to obtain a set of geometric feature points. Neighborhood search involves taking each point in the set as the center and searching for neighboring points within a certain radius. Representative geometric feature points are selected by analyzing the spatial relationships between neighboring points and the center point. Common neighborhood search methods include the KNN (K-Nearest Neighbor) algorithm and the radius-based neighborhood search algorithm. The KNN algorithm allows setting the value of K (the number of neighboring points), typically between 5 and 15, and searches for the K nearest neighbors of each center point. The radius-based neighborhood search algorithm allows setting a fixed radius, typically between 0.01 meters and 0.05 meters, and searches for all neighboring points within the radius of the center point. After the search is complete, geometric feature points are selected by calculating the normal vector and curvature value of each center point. The normal vector can be calculated using Principal Component Analysis (PCA). Principal Component Analysis (PCA) algorithm decomposes the coordinate matrix of points in the neighborhood into eigenvalues, obtaining three eigenvalues ​​and corresponding eigenvectors. The eigenvector corresponding to the smallest eigenvalue is the normal vector of that point. The curvature value can be calculated from the three eigenvalues. Points with larger curvature values ​​are geometric feature points. After filtering, all geometric feature points are summarized to form a geometric feature point set. Then, by calculating the distances, angles, and clustering relationships between feature points, complete geometric information is obtained.

[0033] Visual image information extraction is performed on the image depth map to obtain a preliminary visual feature map. Visual image information extraction mainly involves extracting basic visual features such as pixel grayscale values, texture features, and grayscale gradients from the image depth map. Extraction methods can employ grayscale histogram statistics and texture feature extraction algorithms. Grayscale histogram statistics can obtain the grayscale distribution features of the image depth map. Texture feature extraction can use the HOG (Histogram of Oriented Gradients) algorithm. The HOG algorithm extracts the contour and shape features of the image by calculating the gradient orientation histogram of local regions of the image. After integrating these basic visual features, a preliminary visual feature map is generated, and the grayscale histogram interface and feature extraction interface are called to complete the operation. Edge detection technology is used to process the preliminary visual feature map to obtain an edge map. Edge detection technology extracts the edge contours of the image by identifying regions with abrupt changes in grayscale values ​​in the preliminary visual feature map. The edge contours are a core component of image information. The Canny edge detection algorithm is preferred. This algorithm first performs Gaussian blur denoising on the preliminary visual feature map. The kernel size of the Gaussian blur can be set to 3x3 or 5x5. Then, the gradient intensity and direction of the image are calculated. Subsequently, redundant edge pixels are removed by non-maximum suppression. Finally, accurate edges are obtained by double thresholding. The double thresholds can be set to a low threshold between 50 and 100 and a high threshold between 150 and 200. Pixels higher than the high threshold and those between the high and low thresholds that are connected to the edge of the high threshold are selected as edge pixels to form the edge map.

[0034] Image segmentation involves dividing the edge map into multiple independent regions with similar characteristics. Each region corresponds to a specific target or area in the image. Common segmentation algorithms include thresholding and region growing. Thresholding determines the optimal threshold based on the grayscale histogram of the edge map, dividing the image into foreground and background regions. Region growing uses edge pixels in the edge map as seed points, gradually growing and merging adjacent similar pixels to form complete regions. After segmentation, texture features, grayscale distribution, area, and edge shape of each region are extracted. Integrating the information from all regions yields complete image information. This approach accurately extracts the spatial geometric features of the point cloud depth map and the visual features of the image depth map, providing a high-quality data foundation for subsequent pixel information processing and spatial reliability distribution construction. It also simplifies subsequent processing and improves overall processing efficiency.

[0035] Secondly, the geometric and image information contained in each pixel of the depth map obtained by aligning the point cloud depth map with the image depth map are processed, and a spatial credibility distribution is constructed based on the processing results and the position of the pixel in the depth map. The distance between each pixel and the optical center is obtained from the depth map, and a distance decay weight is obtained using a Gaussian function. A prior spatial weight matrix is ​​constructed using this distance decay weight. A geometric integrity index is obtained based on the geometric information of each pixel. A visual sharpness index is obtained based on the image information and position of each pixel in the depth map. The geometric integrity and visual sharpness indices are processed to generate a credibility score. The credibility score is processed based on the prior spatial weight matrix to obtain a comprehensive credibility score. A spatial credibility distribution is constructed based on the comprehensive credibility score and the position of each pixel in the depth map.

[0036] The geometric and image information contained in each pixel of the depth map obtained by aligning the point cloud depth map with the image depth map is processed. Based on the processing results and the position of the pixel in the depth map, a spatial reliability distribution is constructed. The spatial reliability distribution is a spatial distribution map reflecting the accuracy and reliability of the information contained in each pixel in the depth map. Each pixel corresponds to a comprehensive reliability value. The higher the reliability value, the more accurate the geometric and image information of the pixel. This can provide targeted data support for subsequent initial extrinsic parameter correction. The purpose of this is to filter out high-quality pixel information and eliminate the interference of unreliable information on subsequent processing. The benefit is to improve the accuracy and stability of extrinsic parameter correction.

[0037] First, the distance between each pixel and the optical center is obtained based on the depth map. The optical center refers to the optical center of the camera lens, which is the reference point for camera imaging. Its coordinates can be determined according to the camera's intrinsic parameters. The method for calculating the distance between a pixel and the optical center is to combine the focal length and principal point coordinates in the camera's intrinsic parameters with the pixel's two-dimensional pixel coordinates in the depth map. First, the offset of the pixel relative to the principal point on the image plane is calculated. Then, combined with the pixel's depth value, i.e., the Z-axis coordinate, the spatial straight-line distance between the two is calculated using the Pythagorean theorem. After the calculation, the distance value corresponding to each pixel is obtained. Subsequently, the distance attenuation weight is obtained by applying a Gaussian function to this distance. The core function of the Gaussian function is to achieve an attenuation effect where the weight decreases as the distance increases, because the imaging error of pixels farther from the optical center is usually larger and the information reliability is lower. The distance attenuation weight of each pixel is calculated by the Gaussian function, and the weight value is controlled between 0 and 1. Then, the distance attenuation weights of all pixels are arranged according to their pixel positions in the depth map to construct a prior space weight matrix. The dimension of the prior space weight matrix is ​​consistent with the resolution of the depth map, and each element in the matrix corresponds to the distance attenuation weight of the corresponding pixel in the depth map.

[0038] Next, a geometric integrity index is obtained based on the geometric information of each pixel. The geometric integrity index is a quantitative indicator that measures the completeness and accuracy of the geometric information corresponding to a pixel. Geometric information includes spatial coordinates, geometric feature points, and feature relationships. The calculation method is to first extract the number of geometric feature points and spatial coordinate deviation corresponding to the pixel. If the number of geometric feature points is more and the spatial coordinate deviation is smaller, the geometric integrity index is higher. Specifically, it can be determined by calculating the fitting error and coordinate consistency of the geometric feature points. The fitting error is calculated using the least squares method, fitting the spatial coordinates of the pixel with the surrounding geometric feature points to obtain the fitting deviation value. The coordinate consistency is obtained by calculating the average value of the difference between the spatial coordinates of the pixel and its neighboring pixels. Then, the fitting deviation value and the coordinate consistency are normalized. The normalization method is linear normalization, mapping the values ​​to between 0 and 1. Finally, the geometric integrity index is obtained by weighted summation. The weight of the fitting deviation value is set to 0.4, and the weight of the coordinate consistency is set to 0.6. The higher the index value, the more complete and reliable the geometric information. Then, based on the image information and position of each pixel in the depth map, a visual sharpness index is obtained. The visual sharpness index is a quantitative indicator that measures the clarity and recognizability of the image information corresponding to a pixel. Image information includes edge contours, texture features, and gray-level distribution. The calculation method can combine edge gradient, texture complexity, and gray-level contrast in the image information. The edge gradient is calculated using the Sobel operator to obtain the gray-level gradient values ​​around the pixel. The larger the gradient value, the clearer the edge. The texture complexity is determined by the texture coding variance calculated by the LBP algorithm. The larger the variance, the richer the texture and the higher the recognizability. The gray-level contrast is obtained by calculating the maximum value of the gray-level difference between the pixel and its neighboring pixels. The larger the contrast, the clearer the image information. Subsequently, the edge gradient, texture complexity, and gray-level contrast are linearly normalized to a range of 0 to 1, and then a weighted summation is used to obtain the visual sharpness index. The weights of the three are set to 0.4, 0.3, and 0.3, respectively. The higher the index value, the clearer and more reliable the image information.

[0039] Next, the geometric integrity index and visual sharpness index are processed to generate a credibility score. The processing method adopts a weighted fusion approach, which assigns weights based on the importance of geometric and image information. Typically, the weight of the geometric integrity index is set to 0.5, and the weight of the visual sharpness index is also set to 0.5. If higher accuracy of geometric information is required, the weight of the geometric integrity index can be adjusted to 0.6, and the weight of the visual sharpness index to 0.4. The credibility score is calculated using the formula S=α×G+β×V, where S is the credibility score, α is the weight of the geometric integrity index, G is the geometric integrity index, β is the weight of the visual sharpness index, and V is the visual sharpness index. The score range is controlled between 0 and 1, with a higher score indicating more reliable pixel information.

[0040] The credibility score is then processed based on the prior spatial weight matrix to obtain the comprehensive credibility. The distance attenuation weight of the corresponding pixel in the prior spatial weight matrix is ​​multiplied by the credibility score of that pixel, i.e., comprehensive credibility = distance attenuation weight × credibility score. This allows for further weight suppression of pixels that are far from the optical center and have low information reliability, thereby improving the rationality of the comprehensive credibility and ensuring that the comprehensive credibility reflects both the information quality of the pixel itself and the error caused by the imaging distance. Finally, a spatial confidence distribution is constructed based on the overall confidence level and the position of each pixel in the depth map. The construction method is to map the overall confidence level of each pixel to its two-dimensional pixel coordinates in the depth map, and then combine it with the depth value of the pixel, i.e., the Z-axis coordinate, to form confidence data in three-dimensional space. All three-dimensional confidence data are arranged according to spatial position, and an interpolation algorithm is used to supplement the confidence difference between adjacent pixels. The interpolation algorithm can be a bilinear interpolation algorithm to ensure the continuity of the spatial confidence distribution. Finally, a spatial confidence distribution covering the entire spatial range of the depth map is formed, which provides an accurate and reliable decision basis for subsequent initial extrinsic parameter correction based on this distribution, and further improves the efficiency and accuracy of subsequent extrinsic parameter correction.

[0041] Preferably, the initial extrinsic parameters are processed based on the spatial confidence distribution to obtain the first extrinsic parameters. Based on the initial extrinsic parameters, the point cloud image is projected onto the visual image to obtain the point cloud visual image under initial registration; all matching point pairs in the point cloud visual image are weighted based on the spatial confidence distribution to obtain a weighted point pair set; based on the weighted point pair set, the weighted projection error is obtained; the weighted projection error is minimized to obtain the first extrinsic parameters.

[0042] The first extrinsic parameter is obtained by processing the initial extrinsic parameter based on the spatial reliability distribution. The first extrinsic parameter is obtained after preliminary optimization of the initial extrinsic parameter. Compared with the initial extrinsic parameter, it has higher accuracy and can better reflect the relative position and attitude relationship between the lidar and the camera. This lays the foundation for subsequent final target extrinsic parameter correction. The purpose of this is to use the spatial reliability distribution to filter reliable information and correct the deviation of the initial extrinsic parameter. The benefit is to improve the accuracy of the extrinsic parameter and reduce the interference of unreliable information on subsequent processing.

[0043] First, based on the initial extrinsic parameters, the point cloud image is projected onto the visual image to obtain the point cloud visual image under initial registration. The point cloud visual image is the image obtained by mapping the 3D point cloud image onto the 2D visual image plane through projection transformation. It can intuitively show the correspondence between the point cloud and the visual image. The projection method needs to be implemented by combining the initial extrinsic parameters and the camera intrinsic parameters. First, the rotation matrix and translation vector in the initial extrinsic parameters are used to transform the 3D coordinates of all points in the point cloud image from the LiDAR coordinate system to the camera coordinate system. Then, the 3D points in the camera coordinate system are projected onto the 2D visual image plane through the pinhole camera model. According to the focal length and principal point coordinates of the camera intrinsic parameters, the 2D pixel coordinates of each 3D point on the visual image are calculated. These projected points are superimposed on the original visual image according to the corresponding pixel coordinates to obtain the point cloud visual image under initial registration. The tools for implementation can be PCL and OpenCV. The operation is completed by calling the coordinate transformation interface and the projection interface. After projection, invalid points with excessive projection deviation need to be removed to ensure the validity of the point cloud visual image. Next, based on the spatial confidence distribution, all matching point pairs in the point cloud visual image are weighted to obtain a weighted point pair set. A matching point pair refers to a point pair formed by a point cloud projection point and a corresponding feature point in the visual image. The method for obtaining the matching point pair is to extract the feature points and feature descriptors of the visual image through the scale-invariant feature transform algorithm, and at the same time extract the geometric feature descriptors corresponding to the point cloud projection points. Then, the fast approximate nearest neighbor search algorithm is used to perform feature matching to find the correspondence between the point cloud projection points and the feature points of the visual image, thus forming matching point pairs. The comprehensive confidence value of each pixel in the spatial confidence distribution is the weight of the corresponding matching point pair. The weighting method is to multiply the feature distance of each matching point pair by the comprehensive confidence value of the corresponding pixel to obtain the weighted feature distance. Then, matching point pairs with a weighted feature distance less than a preset threshold are selected and aggregated to form a weighted point pair set. The threshold is usually set between 0.5 and 1.0, which can give higher confidence matching point pairs a larger weight and reduce the influence of unreliable matching point pairs.

[0044] Then, the weighted projection error is obtained based on the weighted point pair set. The weighted projection error is a quantitative indicator that measures the projection deviation of all matching point pairs in the weighted point pair set. It can reflect the degree of deviation of the initial extrinsic parameters. The calculation method is to calculate the Euclidean distance between the two-dimensional pixel coordinates of the point cloud projection point and the two-dimensional pixel coordinates of the corresponding feature point in the visual image for each matching point pair in the weighted point pair set, so as to obtain the projection error of a single matching point pair. Then, each projection error is multiplied by the weight of the corresponding matching point pair to obtain the weighted projection error. Finally, the weighted projection errors of all matching point pairs are summed and averaged to obtain the weighted projection error. The smaller the weighted projection error, the smaller the deviation of the initial extrinsic parameters and the higher the accuracy of the matching point pair. Finally, the weighted projection error is minimized to obtain the first extrinsic parameter. The core of the minimization process is to adjust the rotation matrix and translation vector of the initial extrinsic parameter through iterative optimization to reduce the weighted projection error to a minimum. During the optimization process, the rotation matrix and translation vector of the initial extrinsic parameter are used as optimization variables, and the weighted projection error is used as the objective function. An iteration termination condition is set, that is, when the difference between the weighted projection error of two adjacent iterations is less than a preset threshold or the number of iterations reaches a preset upper limit, the iteration stops. The preset threshold is usually set between 1e-6 and 1e-8, and the number of iterations is set between 50 and 100. The optimized rotation matrix and translation vector obtained after the iteration is the first extrinsic parameter. This process incorporates reliable information of spatial credibility distribution into the extrinsic parameter optimization, which can accurately correct the deviation of the initial extrinsic parameter and provide high-quality basic extrinsic parameters for further correction of the target extrinsic parameter, while improving the efficiency and stability of extrinsic parameter optimization.

[0045] Furthermore, the first extrinsic parameter and the dense point cloud depth map obtained by interpolating and completing the point cloud depth map are input into the pre-constructed extrinsic parameter correction model for correction to obtain the target extrinsic parameter. The first extrinsic parameter and the dense point cloud depth map obtained by interpolating and completing the point cloud depth map are input into a pre-constructed extrinsic parameter correction model for correction to obtain the target extrinsic parameter. The dense point cloud depth map is a complete depth map obtained by interpolating and completing the missing and invalid depth pixels in the point cloud depth map. Compared with the original point cloud depth map, its pixels are denser and the information is more complete, which can make up for the information gaps caused by invalid or missing pixels in the original point cloud depth map. The extrinsic parameter correction model is a deep learning model used to further optimize the extrinsic parameter and improve the accuracy. The target extrinsic parameter is the final extrinsic parameter obtained after two optimizations. Its accuracy meets the actual needs of point cloud calibration and can achieve accurate registration between point cloud image and visual image. The purpose of this is to use the complete information of the dense point cloud depth map and the fitting ability of the extrinsic parameter correction model to further correct the subtle deviations of the first extrinsic parameter. The benefit is to improve the final accuracy of the extrinsic parameter and ensure the accuracy of subsequent point cloud calibration. First, interpolation is performed on the point cloud depth map to obtain a dense point cloud depth map. The core of the interpolation completion process is to fill in invalid depth values ​​and missing pixels in the point cloud depth map. The bilinear interpolation algorithm calculates the weighted average of the depth values ​​of the four adjacent valid pixels around the missing pixel to obtain the completed value. The weights are determined based on the distance between the adjacent pixels and the missing point; the closer the distance, the greater the weight. The cubic spline interpolation algorithm constructs a cubic polynomial curve between adjacent valid pixels and calculates the depth value of the missing point based on the curve equation. Its completion effect is smoother and more accurate. An appropriate method is selected based on the resolution of the point cloud depth map. After completion, the dense point cloud depth map needs to be smoothed and denoised. A Gaussian filter algorithm is used to remove noise introduced by interpolation. The Gaussian filter kernel size is set to 3x3 to ensure the stability and accuracy of the dense point cloud depth map. The operation is completed by calling the interpolation interface and the filtering interface. Then, a pre-built extrinsic parameter correction model is constructed. During model training, a large number of labeled first extrinsic parameter samples and corresponding dense point cloud depth map samples, as well as the corresponding real and accurate extrinsic parameters as labels, are prepared. The mean squared error is used as the loss function to measure the deviation between the corrected extrinsic parameters output by the model and the real extrinsic parameters. The Adam optimizer is selected, and the learning rate is set between 1e-4 and 1e-5. Iterative training is performed for 50 to 100 epochs until the loss function value converges to below the preset threshold, which is usually set to 1e-6. After training, the pre-built extrinsic parameter correction model can be obtained.Finally, the first extrinsic parameter and the dense point cloud depth map are input into the pre-constructed extrinsic parameter correction model for correction. Before input, the first extrinsic parameter needs to be normalized, mapping the values ​​of the rotation matrix and translation vector to between 0 and 1. At the same time, the dense point cloud depth map is adjusted to the input resolution required by the model, usually set to 640x480 or 1280x720. After input into the model, the encoder extracts the features of the two and fuses them. The decoder processes the fused features and outputs the corrected rotation matrix and translation vector. Then, the output result is denormalized to restore it to the original parameter range, and the target extrinsic parameter is obtained. The whole process combines the preliminary optimization result of the first extrinsic parameter with the complete information of the dense point cloud depth map. Through the depth fitting capability of the extrinsic parameter correction model, the subtle deviations of the first extrinsic parameter are corrected, so that the target extrinsic parameter reaches the accuracy required for point cloud calibration, while improving the stability and efficiency of extrinsic parameter correction.

[0046] Finally, the point cloud image is calibrated based on target extrinsic parameters. Point cloud calibration involves establishing a precise spatial correspondence between the point cloud image and the visual image using target extrinsic parameters. This transforms the point cloud image from the LiDAR coordinate system to the camera coordinate system, ensuring accurate alignment between the point cloud information and the visual image information. This guarantees the smooth execution of subsequent point cloud-image fusion processing. The purpose of this is to eliminate spatial discrepancies between the point cloud image and the visual image using precise target extrinsic parameters. The benefit is improved accuracy in point cloud-image fusion, providing reliable data support for subsequent applications such as target detection and 3D reconstruction. The specific implementation method needs to be carried out step by step in conjunction with the target extrinsic parameters. First, the rotation matrix and translation vector contained in the target extrinsic parameters are read. These two parameters are the core of the coordinate system transformation. Then, the coordinates of all 3D points in the point cloud image are transformed from the LiDAR coordinate system to the camera coordinate system using the target extrinsic parameters. The transformation process is completed through matrix operations, that is, the 3D coordinate matrix of each point is multiplied by the rotation matrix of the target extrinsic parameters, and the translation vector of the target extrinsic parameters is added to calculate the 3D coordinates of the point in the camera coordinate system. By calling the coordinate transformation interface in the tool and inputting the point cloud data and target extrinsic parameters, the coordinate system transformation can be completed quickly. After the transformation, the validity of the transformed point cloud data needs to be verified to remove invalid coordinates and outliers generated during the transformation process. Outliers can be removed by using a statistical filtering algorithm, setting a reasonable standard deviation threshold, usually 1 to 2 times the standard deviation, to filter out the valid point cloud data that meets the requirements. Next, the point cloud data transformed into the camera coordinate system is projected onto the visual image plane. The projection method uses a pinhole camera model. Combining the focal length and principal point coordinates of the camera's intrinsic parameters, the two-dimensional pixel coordinates of each 3D point in the camera coordinate system on the visual image plane are calculated. After the calculation is completed, the point cloud projection points are superimposed on the original visual image according to the corresponding pixel coordinates to form a fused image. Then, the alignment accuracy of the fused image is verified. The verification method is to select multiple feature point pairs, calculate the Euclidean distance between the point cloud projection points and the corresponding feature points in the visual image, and count the average distance error of all feature point pairs. If the average error is less than the preset threshold of 0.5 to 1 pixel, the calibration is considered qualified. If the error exceeds the threshold, the accuracy of the target extrinsic parameters or the point cloud coordinate transformation process needs to be rechecked. If necessary, the target extrinsic parameters need to be corrected again until the calibration accuracy meets the requirements. After successful calibration, the calibrated point cloud data is saved, including the 3D coordinates of the point cloud in the camera coordinate system and the corresponding pixel coordinates of the visual image. At the same time, the target extrinsic parameters and camera intrinsic parameters are saved for subsequent repeated calibration or related processing. The entire calibration process fully incorporates the accuracy of the target extrinsic parameters. Through coordinate system transformation and projection verification, the point cloud image and the visual image are accurately aligned, ensuring that the point cloud information can accurately match the visual image information, and providing high-quality calibrated data for various subsequent fusion applications.

[0047] This paper proposes a high-precision automatic extrinsic parameter calibration method for joint LiDAR and camera sensing systems. The overall process is divided into two tightly coupled stages: a coarse calibration stage and a fine optimization stage. This method makes full use of depth maps as intermediate representations, combines deep learning-driven feature matching and attention mechanisms, and introduces point cloud depth completion technology to effectively overcome the matching difficulties caused by the sparsity of the original point cloud, significantly improving the robustness and accuracy of extrinsic parameter estimation.

[0048] In the first stage (coarse calibration stage), the system first synchronously acquires RGB images and 3D point cloud data of the same scene. To initiate the optimization process, the initial extrinsic parameters (i.e., the rigid body transformation matrix from the lidar to the camera) are randomly generated within a reasonable physical range: the translation component is within [ Uniform sampling is performed within the interval [1.5m, 1.5m], and the rotation angle (expressed as Euler angles or rotation vector) is within [ The depth map is randomly set within the range of [20°, 20°]. Then, a monocular depth estimation network is used to generate a dense image depth map from the RGB image; simultaneously, based on the current initial extrinsic parameters and the known camera intrinsic matrix K, the 3D point cloud is projected onto the image plane to generate a sparse but geometrically accurate point cloud depth map. These two depth maps form the basis input for subsequent feature alignment.

[0049] Next, the image depth map and point cloud depth map are fed into two ResNet-18 backbone networks with shared weights for feature extraction. To balance local details and global semantics, a lightweight feature pyramid structure is introduced in the backend of the network, but multi-resolution feature fusion is only retained in the last three layers to avoid weakening semantic information or excessive computational overhead due to high-resolution processing throughout. The extracted multi-scale feature maps are then stitched together and divided into fixed-size image patches, which are input into a cross-modal matching module based on Vision Transformer. The key innovation lies in the prior attention mechanism embedded in this module: when calculating self-attention weights, regions closer to the optical center (i.e., the principal point) are assigned higher initial attention weights. This design is based on actual observations—regions closer to the optical center are less affected by projection distortion, have denser point cloud coverage, and more reliable depth estimation, so they should be given priority for extrinsic parameter constraints. Through this mechanism, the model can focus on the core region with stronger geometric consistency and output a preliminary but reasonable first extrinsic parameter.

[0050] In the second stage (refinement stage), the system performs resolution completion on the sparse point cloud depth map generated in the first stage. Specifically, a specially trained depth completion network is used, guided by the original RGB image, to interpolate and thin invalid or missing pixels in the point cloud depth map, generating a dense, continuous second point cloud depth map that retains the geometric accuracy of the original point cloud. This process not only improves the spatial integrity of the depth map but also implicitly integrates image texture context information, providing high-quality input for subsequent fine alignment.

[0051] Finally, the first extrinsic parameters obtained in the first stage are used as strong prior constraints and re-inputted into the same extrinsic parameter estimation model. However, the model now operates in "optimization mode": its loss function not only includes the depth map alignment error but also incorporates intermediate supervision signals generated during the depth completion process (such as completion confidence maps, edge-preserving weights, or gradient consistency terms). In this mode, the image depth map and the completed second point cloud depth map undergo feature extraction and attention matching again. The model performs local search and fine-tuning near the prior extrinsic parameters, ultimately outputting high-precision, highly consistent target extrinsic parameters. The entire process forms a closed-loop optimization, utilizing the end-to-end expressive power of deep learning while ensuring the geometric rationality and engineering practicality of the results through physical priors and multi-stage strategies.

[0052] Another embodiment of the present invention provides a point cloud calibration system; for details, please refer to [link to relevant documentation]. Figure 2 , Figure 2 The diagram shown illustrates the structure of a point cloud calibration system according to one embodiment of the present invention, comprising: The acquisition module 11 is used to acquire point cloud images and visual images of the calibration object, and determine initial extrinsic parameters based on the geometric parameters of the calibration object; Depth map module 12 is used to convert point cloud images into point cloud depth maps based on initial extrinsic parameters, and to convert visual images into image depth maps; Image module 13 is used to extract geometric data from the point cloud depth map to obtain geometric information, and to perform visual image information extraction processing on the image depth map to obtain image information; Module 14 is used to process the geometric and image information contained in each pixel in the depth map obtained by aligning the point cloud depth map and the image depth map, and to construct a spatial confidence distribution based on the processing results and the position of the pixel in the depth map. Processing module 15 is used to process the initial extrinsic parameters based on the spatial confidence distribution to obtain the first extrinsic parameter; The correction module 16 is used to input the first extrinsic parameter and the dense point cloud depth map obtained by interpolation and completion of the point cloud depth map into the pre-constructed extrinsic parameter correction model for correction, so as to obtain the target extrinsic parameter; The calibration module 17 is used to calibrate the point cloud image based on the target extrinsic parameters.

[0053] Preferably, determining the initial extrinsic parameters based on the geometric parameters of the calibration object includes: The modeling unit is used to model the calibration object based on geometric parameters to obtain the calibration object model; The extraction unit is used to extract features from the visual image of the calibration object and determine the visual features. Three-dimensional units are used to extract the three-dimensional coordinates of the calibration objects in the point cloud image to obtain three-dimensional data. The processing unit is used to perform spatial transformation processing on the calibration object model based on visual features and 3D data to obtain initial extrinsic parameters.

[0054] Preferably, the image module 13 includes: The extraction unit is used to extract the three-dimensional coordinate data from the point cloud depth map to obtain a set of spatial coordinates; Geometric feature units are used to process a set of spatial coordinates using neighborhood search techniques to obtain a set of geometric feature points. Geometric information unit, used to obtain geometric information based on a set of geometric feature points; The visual feature unit is used to extract visual image information from the image depth map to obtain a preliminary visual feature map. The edge map unit is used to process the preliminary visual feature map using edge detection technology to obtain the edge map; The image information unit is used to segment the edge map, extract the image information of all regions after segmentation, and obtain the image information of the image depth map.

[0055] Preferably, the building module 14 includes: The weight matrix unit is used to obtain the distance between each pixel and the optical center based on the depth map. The distance is used to obtain the distance decay weight through a Gaussian function, and the distance decay weight is used to construct the prior space weight matrix. Geometric units are used to obtain geometric integrity metrics based on the geometric information of each pixel. The visual unit is used to obtain a visual sharpness index based on the image information of each pixel and its position in the depth map. The credibility unit is used to process the geometric integrity index and visual clarity index to generate a credibility score; The comprehensive unit is used to process the credibility score based on the prior space weight matrix to obtain the comprehensive credibility. The confidence distribution unit is used to construct a spatial confidence distribution based on the overall confidence level and the position of each pixel in the depth map.

[0056] Preferably, the processing module 15 includes: The registration unit is used to project the point cloud image onto the visual image based on the initial extrinsic parameters to obtain the point cloud visual image under the initial registration. The weighting unit is used to weight all matching point pairs in the point cloud visual image based on the spatial confidence distribution to obtain a weighted point pair set. Error unit, used to obtain weighted projection error based on weighted point pair set; The first unit is used to minimize the weighted projection error and obtain the first extrinsic parameter.

[0057] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0058] Accordingly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform steps in the point cloud calibration method of the above embodiments, for example... Figure 1 Steps S1 to S7 as described above.

[0059] This invention addresses the systematic bias in calibration results caused by the direct use of unverified initial extrinsic parameters (such as theoretical values ​​based on sensor installation locations) in existing point cloud calibration methods. It proposes a multi-stage extrinsic parameter optimization mechanism that integrates geometric and visual information. First, physically reasonable initial extrinsic parameters are generated based on the known geometric parameters of the calibration object. Then, the point cloud and visual data are uniformly converted into depth maps to achieve modal alignment. Next, the geometric and image features of each pixel are jointly analyzed from the aligned depth map, and a spatial reliability distribution reflecting local matching reliability is constructed based on its spatial position in the image. The initial extrinsic parameters are then weighted and optimized based on this distribution to obtain the first extrinsic parameter. Further fine-tuning is performed using a denser point cloud depth map and an extrinsic parameter correction model, ultimately outputting high-precision target extrinsic parameters. This scheme not only effectively suppresses structural distortion, distance distortion, and attitude deviation caused by initial errors but also significantly improves the registration accuracy between the point cloud and the camera coordinate system, thus providing a more reliable and consistent multimodal perception foundation for downstream tasks such as high-precision mapping, obstacle ranging, and pose estimation.

[0060] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A point cloud calibration method, characterized in that, include: Acquire point cloud images and visual images of the calibration object, and determine initial extrinsic parameters based on the geometric parameters of the calibration object; Based on the initial extrinsic parameters, the point cloud image is converted into a point cloud depth map, and the visual image is converted into an image depth map; Extract the geometric data from the point cloud depth map to obtain geometric information, and perform visual image information extraction processing on the image depth map to obtain image information; The geometric information and image information contained in each pixel in the depth map obtained by aligning the point cloud depth map with the image depth map are processed, and a spatial confidence distribution is constructed based on the processing result and the position of the pixel in the depth map. The initial extrinsic parameters are processed based on the spatial credibility distribution to obtain the first extrinsic parameter; The first extrinsic parameter and the dense point cloud depth map obtained by interpolating and completing the point cloud depth map are input into the pre-constructed extrinsic parameter correction model for correction to obtain the target extrinsic parameter. The point cloud image is calibrated based on the target extrinsic parameters.

2. The point cloud calibration method as described in claim 1, characterized in that, The determination of initial extrinsic parameters based on the geometric parameters of the calibration object includes: Based on the geometric parameters, the calibration object is modeled to obtain the calibration object model; Feature extraction is performed on the visual image of the calibrated object to determine the visual features; Extract the three-dimensional coordinates of the calibration object from the point cloud image to obtain three-dimensional data; Based on the visual features and the three-dimensional data, the calibration model is subjected to spatial transformation to obtain the initial extrinsic parameters.

3. The point cloud calibration method as described in claim 1, characterized in that, The geometric data of the point cloud depth map is extracted to obtain geometric information, and visual image information extraction processing is performed on the image depth map to obtain image information, including: Extract the three-dimensional coordinate data from the point cloud depth map to obtain a set of spatial coordinates; The spatial coordinate set is processed using neighborhood search technology to obtain a set of geometric feature points; The geometric information is obtained based on the set of geometric feature points; Visual image information is extracted from the depth map of the image to obtain a preliminary visual feature map; The preliminary visual feature map is processed using edge detection technology to obtain an edge map; The edge map is segmented, and image information of all segmented regions is extracted to obtain the image information of the image depth map.

4. The point cloud calibration method as described in claim 1, characterized in that, The process of processing the geometric information and image information contained in each pixel of the depth map obtained by aligning the point cloud depth map and the image depth map, and constructing a spatial confidence distribution based on the processing result and the position of the pixel in the depth map, includes: The distance between each pixel and the optical center is obtained based on the depth map. The distance is then used to obtain a distance decay weight through a Gaussian function. The distance decay weight is then used to construct a prior space weight matrix. Based on the geometric information of each pixel, a geometric integrity index is obtained; Based on the image information of each pixel and its position in the depth map, a visual sharpness index is obtained; The geometric integrity index and the visual clarity index are processed to generate a credibility score; The credibility score is processed based on the prior space weight matrix to obtain the comprehensive credibility. The spatial confidence distribution is constructed based on the overall confidence level and the position of each pixel in the depth map.

5. The point cloud calibration method as described in claim 1, characterized in that, The process of processing the initial extrinsic parameters based on the spatial credibility distribution to obtain the first extrinsic parameters includes: Based on the initial extrinsic parameters, the point cloud image is projected onto the visual image to obtain the point cloud visual image under initial registration. Based on the spatial credibility distribution, all matching point pairs in the point cloud visual image are weighted to obtain a weighted point pair set. Based on the set of weighted point pairs, the weighted projection error is obtained; The first extrinsic parameter is obtained by minimizing the weighted projection error.

6. A point cloud calibration system, characterized in that, include: The acquisition module is used to acquire point cloud images and visual images of the calibration object, and determine initial extrinsic parameters based on the geometric parameters of the calibration object; The depth map module is used to convert the point cloud image into a point cloud depth map based on the initial extrinsic parameters, and to convert the visual image into an image depth map; The image module is used to extract the geometric data of the point cloud depth map to obtain geometric information, and to perform visual image information extraction processing on the image depth map to obtain image information; The construction module is used to process the geometric information and image information contained in each pixel in the depth map obtained by aligning the point cloud depth map and the image depth map, and to construct a spatial confidence distribution based on the processing result and the position of the pixel in the depth map. The processing module is used to process the initial extrinsic parameters based on the spatial confidence distribution to obtain the first extrinsic parameters; The correction module is used to input the first extrinsic parameter and the dense point cloud depth map obtained by interpolating and completing the point cloud depth map into a pre-constructed extrinsic parameter correction model for correction, so as to obtain the target extrinsic parameter; The calibration module is used to calibrate the point cloud image based on the target extrinsic parameters.

7. The point cloud calibration system as described in claim 6, characterized in that, The determination of initial extrinsic parameters based on the geometric parameters of the calibration object includes: A modeling unit is used to model the calibration object based on the geometric parameters to obtain a calibration object model; The extraction unit is used to extract features from the visual image of the calibration object and determine the visual features; A three-dimensional unit is used to extract the three-dimensional coordinates of the calibration object in the point cloud image to obtain three-dimensional data. The processing unit is used to perform spatial transformation processing on the calibration object model based on the visual features and the three-dimensional data to obtain the initial extrinsic parameters.

8. The point cloud calibration system as described in claim 6, characterized in that, The image module includes: The extraction unit is used to extract the three-dimensional coordinate data from the point cloud depth map to obtain a set of spatial coordinates; Geometric feature units are used to process the set of spatial coordinates using neighborhood search technology to obtain a set of geometric feature points; A geometric information unit is used to obtain the geometric information based on the set of geometric feature points; A visual feature unit is used to extract visual image information from the image depth map to obtain a preliminary visual feature map. The edge map unit is used to process the preliminary visual feature map using edge detection technology to obtain an edge map; The image information unit is used to segment the edge map, extract image information of all regions after segmentation, and obtain the image information of the image depth map.

9. The point cloud calibration system as described in claim 6, characterized in that, The building module includes: The weight matrix unit is used to obtain the distance between each pixel and the optical center based on the depth map, obtain the distance decay weight through a Gaussian function using the distance, and construct the prior space weight matrix using the distance decay weight. A geometric unit is used to obtain a geometric integrity index based on the geometric information of each pixel. A visual unit is used to obtain a visual sharpness index based on the image information of each pixel and its position in the depth map. A credibility unit is used to process the geometric integrity index and the visual clarity index to generate a credibility score. The synthesis unit is used to process the credibility score based on the prior space weight matrix to obtain the comprehensive credibility. A credibility distribution unit is used to construct the spatial credibility distribution based on the overall credibility and the position of each pixel in the depth map.

10. The point cloud calibration system as described in claim 6, characterized in that, The processing module includes: A registration unit is used to project the point cloud image onto the visual image based on the initial extrinsic parameters to obtain a point cloud visual image under the initial registration. A weighting unit is used to perform weighting processing on all matching point pairs in the point cloud visual image based on the spatial confidence distribution to obtain a weighted point pair set. An error unit is used to obtain the weighted projection error based on the set of weighted point pairs; The first unit is used to minimize the weighted projection error to obtain the first extrinsic parameter.

Images