Laser radar point cloud densification method and device fusing image information and medium

Abstract

The invention relates to a laser radar point cloud densification method and device fusing image information, and a medium. The method comprises the following steps: fixing a visual camera and a laser radar on a rigid tool, and carrying out joint calibration and data alignment; a laser radar is used to collect three-dimensional point cloud data, and a visual camera is used to shoot a scene; performing super-pixel segmentation on the image by using an image brightness linear iterative clustering algorithm; mapping the three-dimensional point cloud data into a segmented two-dimensional image superpixel pattern spot region by using a jointly calibrated camera model conversion relationship to realize feature clustering of the original three-dimensional point cloud of the laser radar; performing curved surface fitting on a clustered result by using a random sampling consistency algorithm; and linear interpolation is carried out on the fitted curved surface, dense points which are not covered by the original point cloud of the laser radar are supplemented and generated, and densification of the point cloud of the laser radar is realized. According to the method, the point cloud densification precision and practicability are improved, and technical support is provided for automatic driving, robot navigation and three-dimensional reconstruction.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and 3D scene perception technology, and in particular to a method, device and medium for densifying LiDAR point clouds by fusing image information. Background Technology

[0002] In modern technological fields such as autonomous driving, robot navigation, and 3D reconstruction, the accurate acquisition and processing of 3D spatial information is a core foundation for supporting technology implementation and ensuring system reliability. Its accuracy and efficiency directly determine the comprehensiveness of scene perception, the accuracy of target recognition, and the completeness of environmental modeling. Taking autonomous driving as an example, when vehicles are driving in dynamic scenarios such as urban roads and highways, they need to perceive the surrounding environment in real time with a millisecond-level response speed. As a core perception device, LiDAR actively emits laser beams and receives reflected signals to generate point cloud data containing the 3D coordinates of targets and reflection intensity, providing vehicles with 3D structural information of key elements such as surrounding vehicles, pedestrians, traffic signs, and road boundaries. Based on this data, the autonomous driving system can accurately calculate the safe distance between itself and the vehicle in front, judge the movement trend of obstacles in the lateral lane, and then plan the optimal driving path, execute decisions such as acceleration, deceleration, or steering to ensure driving safety. In the field of robot navigation, whether it is the cargo grasping and positioning of warehouse robots or the indoor obstacle avoidance of service robots, LiDAR point cloud data can help robots build environmental maps and achieve autonomous positioning and path planning. In the field of 3D reconstruction, LiDAR point cloud is the core data source for building high-precision 3D models of buildings, terrains, cultural relics, etc., providing basic support for applications such as digital twins and cultural heritage protection.

[0003] However, due to inherent limitations in hardware performance, the raw point cloud data collected by LiDAR generally suffers from sparsity. This deficiency severely restricts the effectiveness of the technology in many scenarios. In autonomous driving scenarios, mechanical LiDAR, limited by the fixed number of scan lines, typically only contains tens of thousands to hundreds of thousands of point clouds per frame, making it difficult to cover all the details of a complex environment. For vehicles in adjacent lanes, sparse point clouds can only outline the general shape of the vehicle body, failing to accurately capture detailed features such as door handles and window edges, potentially leading to misjudgments of vehicle type or driving status by the target detection algorithm. When facing pedestrians, sparse point clouds cannot even fully represent the posture of human limbs, increasing the difficulty for the system to recognize pedestrian actions and intentions, directly reducing the accuracy of target recognition and creating potential driving safety hazards. More importantly, for small obstacles at a distance, the laser beam emitted by LiDAR cannot densely cover the target surface. Sparse point clouds can only provide coordinate information for a few scattered points, failing to accurately determine the actual position, size, and shape of the obstacle. This prevents the autonomous driving system from triggering avoidance mechanisms in a timely manner, potentially leading to collisions in extreme cases. In robot navigation scenarios, sparse point clouds can cause "holes" in the environment map built by the robot, making it unable to identify low table legs, thin wires, etc., increasing the risk of robot collisions. In 3D reconstruction scenarios, sparse point clouds will make the surface of the reconstructed model rough and lack details, failing to meet the requirements of high-precision modeling.

[0004] To address the point cloud densification problem, existing technologies have proposed a variety of solutions. Current point cloud densification methods can be divided into two categories: point cloud processing based on the point cloud itself and deep learning based on deep learning [1][2].

[0005] The core of the lidar point cloud densification method based on point cloud processing is to improve density through the geometric relationship or spatial features inside the point cloud data. Nearest neighbor interpolation is based on "finding neighboring points and inserting new points". Although it is simple and easy to implement, it is difficult to ensure that the new points accurately reflect the real shape in complex geometric regions with large curvature changes and sharp features. Moving least squares method [3] is to construct a smooth surface and sample new points by fitting local curved surfaces. It can handle noise and surface irregularities well, but it has problems such as high computational cost and sensitivity to parameters. Inappropriate parameters can easily lead to over-smoothing and loss of detailed features. Overall, this type of method does not need to rely on external sensor data and the operation logic focuses on the point cloud itself. However, it generally has limited adaptability to complex scenes, difficulty in balancing computational cost and parameter sensitivity, and insufficient preservation of details in some scenes.

[0006] Deep learning-based methods: using deep neural networks, especially generative adversarial networks or convolutional neural networks, to predict and add new points [4]. Such methods can learn complex data distributions and generate high-quality results, but require a large amount of training data and have high computational costs. In addition, they may also face problems such as model instability and difficulty in convergence during the training process.

[0007] The problem this invention aims to solve is that methods based on point cloud processing often lack an understanding of the overall structure and semantic information of objects, making them difficult to adapt to complex geometric regions, and some methods are computationally expensive. Deep learning-based methods, on the other hand, require large amounts of training data, have high computational costs, and limited generalization ability.

[0008] References [1] Han Bing, Deng Lixiang, Zheng Yi, et al. A review of three-dimensional point cloud upsampling methods [J]. Computer Science, 2024, 51(07): 167-196. [2] Han Xiaoli. Research on point cloud upsampling method of leafy vegetable plants based on deep learning [D]. Northwest A&F University, 2025. [3] Guan Yunlan, Liu Shaotang, Zhou Shijian, et al. Robust point cloud data plane fitting based on global least squares [J]. Geodesy and Geodynamics, 2011, 031(005): 80-83. [4] L. Yu, X. Li, and P. -A. Heng, et al. PU-Net: Point Cloud Upsampling Network [C]. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018: 2790-279. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to propose a method for densifying lidar point clouds by fusing image information, utilizing the rich texture and semantic features of visual images to assist in the densification of lidar point clouds.

[0010] Compared with existing technologies, this invention designs a method, device, and medium for densifying LiDAR point clouds by fusing image information. It focuses on visual image-guided LiDAR point cloud densification to address the sparseness of LiDAR point clouds. Relying on the precise 3D geometric information provided by the LiDAR, it first establishes a spatial mapping relationship between the LiDAR and the visual camera through joint calibration. Then, it uses a local surface model fitted by a random sampling consensus algorithm as a depth benchmark to address the shortcomings of pure visual methods, such as lack of depth information and high susceptibility to environmental influences. Finally, it utilizes a superpixel segmentation algorithm to overcome the difficulty of balancing geometric accuracy and semantic consistency in traditional point cloud densification methods.

[0011] To address the aforementioned technical problems, this invention provides the following technical solution: a method for densifying lidar point clouds by fusing image information, comprising the following steps: A data acquisition device is constructed, and the vision camera and LiDAR are fixed on a rigid fixture. The relative position and attitude of the two and the parameters of the vision camera are calibrated by a calibration tool to obtain calibration parameters. The lidar is used to collect 3D point cloud data, and the vision camera is used to simultaneously collect scene image information to ensure timestamp alignment. The image is segmented into superpixels using the linear iterative clustering algorithm SLIC based on image brightness values, generating superpixel patch regions with consistency in brightness and spatial proximity. Based on the calibration parameters, point cloud and image mapping and clustering are performed, the three-dimensional point cloud data is mapped to the superpixel patch region of the image, and the three-dimensional points mapped to the same patch region are divided into the same cluster of point cloud. For each cluster of point clouds, a random sampling consensus algorithm is used to fit the surface to obtain a surface model that reflects the geometric characteristics of the cluster. The pixels of the image are traversed, and dense points not covered by the original point cloud are generated by linear interpolation based on the surface model. The dense points are then transformed to the world coordinate system to achieve point cloud densification.

[0012] Furthermore, the calibration parameters include the intrinsic parameter matrix K of the visual camera and the extrinsic parameter matrices of the LiDAR and the visual camera. R is the rotation matrix and t is the translation vector. The calibration tool adopts the Zhang Zhengyou checkerboard calibration method. The calibration is completed by simultaneously acquiring calibration board images and point cloud data in at least 8 postures. During the calibration process, it is ensured that the calibration board is fully displayed in the image and that the lidar can scan at least 3 corner points.

[0013] Furthermore, the specific implementation method of the superpixel segmentation is as follows: The linear iterative clustering algorithm based on image brightness values ​​is adopted. First, the RGB image is converted to the Lab color space. Then, superpixel segmentation is performed on the converted image. Let the total number of pixels in the image be N and the number of target superpixels be H. The value of H is set according to the image resolution and the complexity of scene texture. Calculate the initial mesh size The image is divided into S×S grids. The center of each grid is used as the initial candidate cluster center. Within a 3×3 to 5×5 neighborhood, and with larger neighborhoods for complex textures, the pixel with the smallest gradient value and the smallest brightness variance is selected as the final cluster center. Define the feature vector of each pixel as Where L, a, and b are color components, and x and y are pixel coordinates; the similarity between pixels and cluster centers is calculated using a normalized distance metric formula: ; ; ; in, It is the Euclidean distance between the pixel and the cluster center in the Lab color space. It is the target pixel brightness value. It is the red-green deviation value of the target pixel. It is the yellow-blue deviation value of the target pixel. , , These are the pixel brightness value of the cluster center, the pixel red-green deviation value, and the pixel yellow-blue deviation value; It is the Euclidean distance between the pixel and the cluster center in the image space. These are the image space coordinates of pixels. These are the image space coordinates of the cluster centers; It is the initial grid size of the superpixel. These are weight parameters. Represents the normalized distance metric between a pixel and the cluster center; Weight parameters are adjusted based on scene complexity. Make contextual adjustments. The value ranges from 10 to 40, and is used to balance the weights of color similarity and spatial proximity: for scenes with complex textures, color similarity is given priority to ensure segmentation details. Set the value to 10-20; for scenes with simple textures, prioritize spatial proximity to improve computational efficiency and ensure regular superpixel shapes. Take 25-40; Each cluster center searches for pixels only within a 2S×2S rectangular area. After each iteration, the average feature vector of each cluster is recalculated as the new cluster center until the iteration converges. The iteration stops when the positional change of all cluster centers is less than the pixel-level threshold in two adjacent iterations, and outputs a superpixel patch region with brightness and spatial consistency, ensuring that the pixel depth change within the same superpixel is consistent.

[0014] Furthermore, the H value is set according to the image resolution and scene texture complexity: For an image with a resolution of 1280×720, H is set to 500-1000 for simple texture scenes and 1500-2500 for complex texture scenes; For a 1920×1080 image, H is set to 1000-2000 for simple texture scenes and 3000-5000 for complex texture scenes; For a 2560×1440 image, H is set to 2000-3500 for simple texture scenes and 5000-8000 for complex texture scenes; Furthermore, under the same resolution and scene type, the H value is finely adjusted within the above range according to the sparsity of the original point cloud of the LiDAR.

[0015] Furthermore, the specific methods for mapping and clustering the point cloud and image are as follows: Based on the intrinsic parameter matrix K and extrinsic parameter matrix of the visual camera Formula for constructing a pinhole camera imaging model: ; in, , , U and v are the coordinates of a point in the world coordinate system, and u and v are the pixel coordinates of the corresponding point in the camera coordinate system. It is a scaling factor. It is the camera intrinsic parameter matrix. It is an extrinsic parameter matrix. For rotation matrix, It is a translation vector. This is called a projective camera matrix; Three-dimensional points in the world coordinate system collected by lidar , , Through the external parameter matrix The intrinsic parameter matrix K is mapped to the camera coordinate system, and after correction by radial and tangential distortion coefficients, the pixel coordinates of the two-dimensional image are obtained. ; If pixel coordinates If points fall into the same superpixel patch area, the 3D point is assigned to the corresponding cluster of point clouds to ensure that points within the cluster correspond to the same local structure in the real scene.

[0016] Furthermore, the process of surface fitting using the random sampling consensus algorithm includes: For each cluster of point clouds, at least three non-collinear points are randomly selected using the Random Sampling Consensus (RANSAC) algorithm and substituted into the plane equation. Solve the initial model; Calculate the point cloud density of clustered point cloud clusters ,in, V represents the total number of points within the clustered point cloud, and V is the three-dimensional spatial volume occupied by the point cloud cluster in the world coordinate system. ,in , These are the maximum and minimum coordinates of all points within the cluster along the X-axis in the world coordinate system, respectively. , These are the maximum and minimum coordinates of all points within the cluster along the Y-axis, respectively; , These are the maximum and minimum coordinates of all points within the cluster along the Z-axis, respectively. Based on point cloud density Set a distance threshold with scene accuracy coefficient k ,in, For scene accuracy coefficients, Point cloud density; Calculate the Euclidean distance from the remaining points within the cluster to the initial model, and select points with distances less than [a certain value]. Points that are determined to be interior points conforming to the model are greater than or equal to [a certain value]. Points are identified as external points, and the proportion of internal points to the total number of points in the cluster is calculated simultaneously. Repeat the above process, and finally select the model with the most interior points and the highest proportion as the final fitted surface of the clustered point cloud cluster.

[0017] Furthermore, the specific implementation method of the point cloud densification is as follows: Iterate through all pixels within each superpixel The pixels are mapped to the lidar coordinate system through inverse transformation of camera intrinsic parameters and inverse transformation of extrinsic parameters. Substitute the mapped lidar coordinates into the surface model of the corresponding clustered point cloud. For higher-order polynomials, The form is ,in, These are the coefficients of the demand solution. , It is an error term; Linear interpolation is performed on the fitted surface pixel by pixel within the superpixel patch to calculate the depth value of the corresponding superpixel patch region, generating new 3D points not covered by the original point cloud. The dense points are transformed to the world coordinate system using an extrinsic parameter matrix and then merged with the original point cloud of the lidar to achieve lidar point cloud densification.

[0018] Furthermore, the 3D point cloud data and image information are time-stamp aligned through hardware-triggered PPS signals and software calibration, with an alignment error ≤0.5ms.

[0019] The present invention also provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the lidar point cloud densification method for fusing image information.

[0020] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for densifying lidar point clouds by fusing image information. This significantly improves accuracy and provides a precise data foundation for subsequent surface fitting.

[0021] By employing the above technical solution, the present invention achieves point cloud densification for lidar by fusing image information, and has at least the following beneficial effects: 1. Deep integration of superpixel segmentation and point cloud clustering: The SLIC algorithm is used to segment the image into semantically related superpixel patches. Combined with point cloud-image mapping, cross-modal clustering is achieved, avoiding the cross-structure mixing problem that is prone to occur in traditional spatial distance-based clustering. This ensures that the same cluster of point clouds corresponds to the same local structure in the real scene, significantly improving the clustering accuracy and providing a precise data foundation for subsequent surface fitting.

[0022] 2. Image-guided surface fitting and interpolation strategy: Using the local surface fitted by the RANSAC algorithm as the depth reference, and combining it with pixel-by-pixel linear interpolation to generate new 3D points, it not only relies on the accurate geometric information of LiDAR to solve the problem of large depth error in pure vision methods, but also supplements the areas not covered by the original point cloud through image texture, ultimately improving the point cloud density, and the denser point cloud retains the scene details.

[0023] 3. Dynamic parameter optimization mechanism: By adaptively adjusting the number of superpixels, weight parameters and distance thresholds, it can adapt to different application requirements such as autonomous driving (dynamic scene) and 3D reconstruction (static scene), and balance detail preservation and computational efficiency in complex environments, thereby enhancing the robustness of the algorithm.

[0024] In summary, this invention effectively compensates for the shortcomings of single sensors, improves the accuracy and practicality of point cloud densification, and provides technical support for autonomous driving environmental perception, robot navigation, and high-precision 3D reconstruction. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the data acquisition device of the present invention; Figure 2 This is a schematic diagram of the lidar pose of the present invention; Figure 3 This is a schematic diagram illustrating the mapping from the camera pixel coordinate system to the world coordinate system in this invention; Figure 4 This is an overall flowchart of the method of the present invention; Figure 5 This is a schematic diagram of the coordinate transformation relationship of the present invention; Figure 6 This is a schematic diagram of the superpixel segmentation results of the present invention; Figure 7 This is a schematic diagram of the 3D-2D projection result of the present invention; Figure 8This is a schematic diagram of the clustered point cloud clusters of the present invention; Figure 9 This is a comparison image of point clouds before and after the densification process of this invention. Detailed Implementation

[0026] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding and implementation of how the present application uses technical means to solve technical problems and achieve technical effects.

[0027] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0028] Please refer to Figures 1-9 This illustration shows a specific implementation of the present embodiment. This embodiment achieves efficient densification of sparse LiDAR point clouds through visual semantic-guided clustering, robust surface fitting, and pixel-by-pixel precise point filling. It has significant application value in fields such as autonomous driving and 3D reconstruction, improving target detection accuracy and model detail completeness. Its innovation lies not only in the uniqueness of the technical route, but also in the engineering design of dynamic parameter adjustment and adaptive threshold, which balances accuracy and efficiency, forming a complete and feasible technical solution.

[0029] Please refer to Figure 4 This embodiment proposes a method for densifying LiDAR point clouds by fusing image information. The method includes the following steps: A data acquisition device is constructed, the hardware of which includes a visual camera for acquiring sequential images and a lidar for acquiring 3D point cloud data; the visual camera and lidar are fixed to a rigid fixture to ensure relative stability and avoid pose shift caused by vibration during data acquisition; the relative position and pose of the two and the parameters of the visual camera are calibrated using a calibration tool to obtain calibration parameters. As a preferred embodiment, the specific process includes the following steps: like Figure 1As shown, Zhang Zhengyou's checkerboard calibration method was used during the calibration process. The checkerboard size was (12, 9, 30 mm), ensuring the calibration board was flat and free of reflections. The calibration board was placed in front of the sensor (within a range of 1-3 m), and its orientation was adjusted (at least 8 different orientations, including horizontal, tilted, and vertical). Camera images and LiDAR point clouds were simultaneously acquired in each orientation, ensuring the calibration board was fully displayed in the image and that the LiDAR could scan at least 3 corner points of the calibration board. The relative position and orientation of the vision camera and LiDAR, as well as the intrinsic parameters of the vision camera, were calibrated to obtain calibration parameters, thus completing the relative position and orientation calibration.

[0030] Parameters include intrinsic parameter matrix With extrinsic matrix ; in, It is the camera intrinsic parameter matrix. and The focal length of the camera along the x and y axes (unit: pixels) is calculated from the lens focal length and the physical size of the pixel. , For the lens focal length, (where is the physical width of a single pixel); s is the pixel shear coefficient, used to correct distortion caused by the non-rectangular arrangement of the camera's image sensor, and is usually close to 0 (ideally 0). , These are the camera's principal point coordinates (in pixels), which are the intersection of the lens's optical axis and the image plane; ideally, this is the image center. Called a projective camera matrix, The first row of the rotation matrix is ​​formed by the rotation factors. The second row of the rotation matrix is ​​formed by the rotation factor. The third row of the rotation matrix is ​​formed by the rotation factor. The translation amount corresponding to the x-axis direction. The translation amount corresponding to the y-axis direction. The translation amount corresponding to the z-axis direction.

[0031] In this embodiment, the camera and LiDAR are fixed to a rigid fixture to avoid pose shifts caused by vibration. The Zhang Zhengyou checkerboard calibration method is employed, simultaneously acquiring calibration board data in more than eight poses to ensure the calibration accuracy of intrinsic parameters (focal length, principal point) and extrinsic parameters (rotation matrix R, translation vector t), with a reprojection error ≤ 0.5 pixels, laying the foundation for point cloud-image mapping. Secondly, the original LiDAR point cloud is sparse; for example, distant targets may only have dozens of points, making it impossible to capture detailed features such as vehicle door handles or pedestrian poses. This method utilizes high-resolution texture information from images to provide semantic grouping for the point cloud, avoiding the cross-object mixing problem that easily occurs in traditional point cloud clustering. Purely visual depth estimation relies on image matching and is easily affected by lighting and texture deficiencies; for example, depth estimation errors are large in white wall scenes. This method uses the precise 3D coordinates of the LiDAR point cloud as a benchmark and generates a reliable depth model through surface fitting, ensuring the geometric accuracy of newly added point clouds. This breaks through the limitations of traditional single sensors and proposes a dual-drive framework of visual semantic guidance and lidar geometric constraints to achieve accurate densification of sparse point clouds.

[0032] like Figure 2 As shown, the lidar is used to collect 3D point cloud data, and the vision camera is used to simultaneously collect scene image information to ensure timestamp alignment. As a preferred embodiment of the steps, the specific process includes the following steps: As an active 3D measurement device, LiDAR needs to stably acquire 3D point cloud data at a preset frequency. Specific parameters need to be set according to the application scenario requirements: In autonomous driving scenarios, to balance real-time performance and data accuracy, the LiDAR acquisition frequency is typically set to 10Hz. This frequency ensures that one frame of the surrounding environment's 3D structure information is updated every 0.1 seconds, meeting the dynamic environmental perception needs of vehicles traveling at high speeds while avoiding data redundancy and storage pressure caused by excessively high frequencies. If applied to static building 3D reconstruction scenarios, the frequency can be reduced to 1-2Hz to reduce unnecessary computational power consumption. During the acquisition process, the LiDAR records each point in its own coordinate system, typically with the LiDAR center as the origin, the X-axis pointing directly forward, the Y-axis pointing to the left, and the Z-axis vertically upward. The acquired 3D coordinates are... Simultaneously, the reflection intensity value of each point is recorded, ranging from 0 to 255. The reflection intensity information can help distinguish objects of different materials and improve the semantic discrimination capability of point clouds.

[0033] Meanwhile, the visual camera needs to maintain a synchronized working rhythm with the LiDAR, capturing scene visual images at a preset frame rate. Considering the spatiotemporal consistency requirements during data fusion, the camera frame rate needs to match the LiDAR acquisition frequency, generally set to 15Hz, slightly higher than the LiDAR frequency, to ensure that each frame of LiDAR point cloud corresponds to at least one frame of image data. To avoid misalignment between point cloud and image data caused by device movement, precise alignment of their timestamps is essential. This can be achieved through a combination of hardware triggering and software calibration. At the hardware level, the camera shutter is triggered by the PPS (pulse per second) signal output by the LiDAR, ensuring that the camera captures an image at the same moment the LiDAR begins acquiring a frame of point cloud data, guaranteeing an initial timestamp error ≤1ms. At the software level, by reading the timestamp information of the LiDAR point cloud file and the camera image file, linear interpolation is used to correct for minor time deviations, ultimately controlling the timestamp difference between each frame of point cloud and its corresponding image to within 0.5ms. For example, when an autonomous vehicle travels at 60 km / h, it moves only about 8.3 mm within 0.5 ms. This displacement is far less than the measurement accuracy of LiDAR (typically ≤2 cm), effectively avoiding the mismatch between the object's position in the point cloud and the image due to motion, ensuring the accuracy of subsequent point cloud-pixel mapping. Furthermore, the visual images captured by the camera need to be saved in RGB format, with the resolution set according to the scene requirements. 1920×1080 is commonly used in autonomous driving scenarios, while 2048×1536 is commonly used in 3D reconstruction scenarios. Based on the previously calibrated distortion coefficients, distortion correction is performed immediately after shooting to eliminate the influence of radial and tangential lens distortion on subsequent superpixel segmentation and coordinate mapping.

[0034] Simultaneously, a vision camera captures visual images of the scene, ensuring that the timestamps of the images and point cloud data are aligned to avoid data misalignment caused by motion. The 3D point cloud data and image information achieve timestamp alignment through hardware-triggered PPS signals and software calibration, with an alignment error ≤0.5ms.

[0035] In this embodiment, after completing the joint calibration of multiple sensors, the data acquisition stage begins. This requires the collaborative work of the LiDAR and the vision camera. The vision camera captures visual images of the scene, ensuring that the timestamps of the images and point cloud data are aligned to avoid data misalignment caused by motion. This allows for the acquisition of spatiotemporally synchronized 3D point cloud and 2D image data. The 3D point cloud data and image information are time-stamp aligned through hardware-triggered PPS signals and software calibration, with an alignment error ≤0.5ms, providing good data support for subsequent fusion processing.

[0036] The scene image is segmented into superpixel regions using the linear iterative clustering algorithm SLIC based on image brightness values, generating superpixel patch regions with consistency in brightness and spatial proximity. As a preferred embodiment of the steps, the specific process includes the following steps: like Figure 6 As shown, the linear iterative clustering algorithm based on image brightness values ​​is used to convert RGB images into the Lab color space. This is because the Lab space is device-independent, and the three channels L (brightness), a (red-green), and b (yellow-blue) are independent of each other, which can more accurately describe color differences and avoid the segmentation error caused by the coupling of brightness and color in the RGB space. Perform superpixel segmentation on the converted image. Let the total number of pixels in the image be N, and the number of target superpixels be H. Set the value of H according to the image resolution and scene texture complexity: More specifically, for images with a resolution of 1280×720, H is set to 500-1000 for simple texture scenes and 1500-2500 for complex texture scenes; for images with a resolution of 1920×1080, H is set to 1000-2000 for simple texture scenes and 3000-5000 for complex texture scenes; for images with a resolution of 2560×1440, H is set to 2000-3500 for simple texture scenes and 5000-8000 for complex texture scenes; and under the same resolution and scene type, the H value is finely adjusted within the above range according to the sparsity of the original point cloud of the LiDAR.

[0037] Calculate the initial mesh size The image is divided into S×S grids. The center of each grid is used as the initial candidate cluster center. Within a 3×3 to 5×5 neighborhood, and with a larger neighborhood for complex textures, the pixel with the smallest gradient value and the smallest brightness variance is selected as the final cluster center. This avoids the center falling in the edge region (large gradient) or the noise region (abnormal brightness variance), and ensures that it falls in the texture flat region. Define the feature vector of each pixel as Where L, a, and b are color components, and x and y are pixel coordinates; the similarity between pixels and cluster centers is calculated using a normalized distance metric formula: ; ; ; in, It is the Euclidean distance between the pixel and the cluster center in the Lab color space. It is the target pixel brightness value. It is the red-green deviation value of the target pixel. It is the yellow-blue deviation value of the target pixel. , , These are the pixel brightness value of the cluster center, the pixel red-green deviation value, and the pixel yellow-blue deviation value; It is the Euclidean distance between the pixel and the cluster center in the image space. These are the image space coordinates of pixels. These are the image space coordinates of the cluster centers; It is the initial grid size of the superpixel. These are weight parameters. Represents the normalized distance metric between a pixel and the cluster center; Weight parameters are adjusted based on scene complexity. Make contextual adjustments. The value ranges from 10 to 40, and is used to balance the weights of color similarity and spatial proximity: for scenes with complex textures, color similarity is given priority to ensure segmentation details. Set the value to 10-20; for scenes with simple textures, prioritize spatial proximity to improve computational efficiency and ensure regular superpixel shapes. Take 25-40; Each cluster center searches for pixels only within a 2S×2S rectangular area. After each iteration, the average feature vector of each cluster is recalculated as the new cluster center until the iteration converges. The iteration stops when the positional change of all cluster centers is less than the pixel-level threshold in two adjacent iterations, and outputs a superpixel patch region with brightness and spatial consistency, ensuring that the pixel depth change within the same superpixel is consistent.

[0038] In this embodiment, the SLIC algorithm is used to segment the image into superpixel patches. Pixels within the same patch have consistent brightness and color. For example, pixels on a red car body are clustered into the same patch. By calculating the normalized distance metric between the pixel and the cluster center, both color similarity and spatial proximity are taken into account, ensuring that the pixel depth variation within the same superpixel has consistency.

[0039] Based on the calibration parameters, point cloud and image mapping and clustering are performed, the three-dimensional point cloud data is mapped to the superpixel patch region of the image, and the three-dimensional points mapped to the same superpixel patch region are divided into the same cluster of point cloud. As a preferred embodiment, the specific process includes the following steps:

[0040] like Figure 5 As shown, based on the intrinsic parameter matrix K and extrinsic parameter matrix of the visual camera Formula for constructing a pinhole camera imaging model: ; in, , , The coordinates are the three-dimensional coordinates of a point in space within the world coordinate system. This coordinate system typically uses a fixed reference point in the calibration scene as its origin and is used to uniformly describe the environmental space perceived by the LiDAR and camera. u and v are the pixel coordinates of the corresponding point in the camera coordinate system, reflecting the position of that point on the camera's imaging plane. It is a scaling factor, which is related to the distance from the spatial point to the camera, and is used for scale adjustment when projecting three-dimensional coordinates onto a two-dimensional pixel plane; It is the camera intrinsic parameter matrix, which contains the camera's inherent characteristic parameters. These parameters determine the projection relationship of three-dimensional points on the camera's imaging plane. It is an extrinsic parameter matrix, where This is a rotation matrix used to describe the rotation transformation from the world coordinate system to the camera coordinate system; The translation vector describes the translation transformation from the world coordinate system to the camera coordinate system. Together, they determine the camera's position and orientation in the world coordinate system. Known as the projective camera matrix, it integrates the camera's internal and external parameters, fully representing the projection relationship from the world coordinate system to the camera pixel coordinate system, such as... Figure 3 As shown.

[0041] like Figure 7 As shown, based on the transformation relationship of the visual camera imaging model, the three-dimensional points in the world coordinate system acquired by the lidar are transformed... When mapping to the corresponding superpixel patch region of a two-dimensional image, it is necessary to first use the extrinsic parameter matrix obtained by joint calibration in step (2). The coordinate transformation is completed, where the 3×3 rotation matrix R is responsible for adjusting the pose alignment between the world coordinate system and the camera coordinate system, and the 3×1 translation vector t realizes the position offset of the origin of the two coordinate systems. The coordinate transformation converts the 3D points into the camera coordinate system. ; Substituting the inverse transformation formula of the calibrated camera intrinsic parameter matrix K, and combining it with the principle of similar triangles in pinhole camera imaging, the pixel coordinates of the two-dimensional image are calculated. During this process, pixel coordinates need to be corrected using radial and tangential distortion coefficients.

[0042] Finally, based on the mapping matrix generated by superpixel segmentation, the corresponding image patch is determined by the corrected pixel coordinates, thus completing the accurate mapping between the three-dimensional point in the world coordinate system and the two-dimensional image region.

[0043] like Figure 8 As shown, based on the obtained 3D point-image superpixel patch region mapping results, and using the superpixel segmentation results completed by the linear iterative clustering algorithm of image brightness values, a systematic clustering operation is performed on the 3D point cloud of the LiDAR in the world coordinate system. First, the 3D points in each world coordinate system are defined... The superpixel locations of the mapped 2D image are then used to group the 3D point cloud based on these locations. All 3D points mapped to the same superpixel region are grouped into the same cluster of point clouds, i.e., pixel coordinates. If points fall into the same superpixel patch area, the 3D point is assigned to the corresponding cluster of point clouds to ensure that points within the cluster correspond to the same local structure in the real scene.

[0044] In this embodiment, the projection formula establishes a correspondence between the three-dimensional world coordinate system and the two-dimensional image pixel coordinates, providing a key coordinate transformation basis for the mapping of LiDAR point clouds to images and subsequent point cloud processing based on image information. Relying on image semantic grouping and clustering, it ensures that three-dimensional points within the same cluster have clear superpixel patch region correlations in the two-dimensional image space. This correlation further corresponds to the same local structure in the real scene, effectively avoiding the cross-structure point cloud mixing problem common in traditional clustering methods, and laying a structured data foundation for subsequent accurate surface fitting and point cloud densification. Specifically, after projecting the LiDAR point cloud onto the image through calibration parameters, three-dimensional points can be assigned to superpixel patches, achieving point cloud clustering, such as points on the vehicle body clustering into one cluster and ground points clustering into another. The deep integration of superpixel segmentation and point cloud clustering is reflected in: using the SLIC algorithm to segment the image into semantically correlated superpixel patches, and classifying three-dimensional points to corresponding patches through point cloud-image mapping, forming cross-modal clustering of image semantics and point cloud geometry. Traditional clustering methods (such as K-Means) rely solely on spatial distance, which can easily mix point clouds of adjacent objects, such as vehicles and roadside railings. In contrast, this method uses image semantic grouping to ensure that point clouds within the same patch correspond to the same local structure in the real scene, such as vehicle body and windows, significantly improving clustering accuracy and providing precise data support for subsequent surface fitting.

[0045] For each valid cluster of point cloud, a random sampling consensus algorithm is used to fit a surface to obtain a surface model that accurately reflects the geometric characteristics of the cluster. As a preferred embodiment, the specific process includes the following steps: Randomly select at least 3 non-collinear points from the current clustered point cloud, and determine the world coordinate system coordinates of these 3 points. , The initial model parameters are obtained by solving the problem.

[0046] Then, the Euclidean distance from all other 3D points within the cluster to the initial surface is calculated. A distance threshold is set based on the point cloud density and scene accuracy requirements. Points with a distance less than the threshold are considered interior points conforming to the model, while points with a distance greater than or equal to the threshold are considered exterior points. Simultaneously, the proportion of interior points to the total number of points within the cluster is calculated. The specific operation is as follows: Calculate the point cloud density of clustered point cloud clusters ,in, V represents the total number of points within the clustered point cloud, and V is the three-dimensional spatial volume occupied by the point cloud cluster in the world coordinate system. ,in , These are the maximum and minimum coordinates of all points within the cluster along the X-axis in the world coordinate system, respectively. , These are the maximum and minimum coordinates of all points within the cluster along the Y-axis, respectively; , These are the maximum and minimum coordinates of all points within the cluster along the Z-axis, respectively. Based on point cloud density Set a distance threshold with scene accuracy coefficient k ,in, For scene accuracy coefficients, Point cloud density; By calculating the Euclidean distance from the remaining points within the cluster to the initial model, points with distances less than [a certain value] are [selected]. Points that are determined to be interior points conforming to the model are greater than or equal to [a certain value]. Points are identified as external points, and the proportion of internal points to the total number of points in the cluster is calculated simultaneously. Repeat the above process, and finally select the surface model with the most internal points and the highest proportion as the final fitting surface for the clustered point cloud cluster, so as to ensure that the model can fit the real geometric shape of the point cloud within the cluster to the greatest extent.

[0047] like Figure 9 As shown, after completing the surface fitting for each effective cluster of point clouds, an equation that accurately reflects the geometric characteristics of the point clouds within the cluster is obtained. Then, depth information is obtained based on the surface fitting model, and then the encryption operation of the LiDAR point cloud is realized by using the image as a guide: traversing all pixels (u,v) in each superpixel, generating dense points not covered by the original point cloud through linear interpolation based on the surface model, and transforming the dense points to the world coordinate system to realize the point cloud densification.

[0048] Specifically, the process begins by mapping pixels in the visual image to the LiDAR coordinate system using the inverse intrinsic transformation of the visual camera, combined with the inverse extrinsic transformation. In this process, the inverse intrinsic transformation of the visual camera is used to counteract the influence of the camera's internal parameters on the pixel coordinates, while the inverse extrinsic transformation is used to transform the pixel positions in the camera coordinate system to the LiDAR coordinate system, establishing a correspondence between the pixels and the LiDAR coordinate system.

[0049] Once the pixels are successfully mapped to the LiDAR coordinate system, the mapped LiDAR coordinates are substituted into the surface model of the corresponding clustered point cloud. For higher-order polynomials, The form is ,in, These are the coefficients of the demand solution. , This is the error term. Since the equation fits the true geometric shape of the clustered point cloud, substituting it into the equation allows for linear interpolation of each pixel within the superpixel patch towards the fitted surface, thus solving for the true depth value of the superpixel patch region corresponding to that pixel. This depth value reflects the distance information of the pixel relative to the LiDAR in three-dimensional space.

[0050] Once the true depth value is obtained, new 3D points not covered by the original point cloud can be generated. These new 3D points contain spatial locations with accurate depth information converted from image pixels.

[0051] Finally, the newly generated 3D points are transformed to the world coordinate system using extrinsic parameters. Extrinsic parameter transformation unifies the point cloud data from the LiDAR coordinate system to the world coordinate system, ensuring that the newly generated 3D points and the original LiDAR point cloud data are in the same coordinate system, thus achieving point cloud densification. In this way, the originally sparse LiDAR point cloud is effectively supplemented, and the point cloud density is significantly improved.

[0052] In this embodiment, based on an image-guided surface fitting and interpolation strategy, using superpixel patches as units, a local surface model is fitted using the RANSAC algorithm to automatically eliminate noise points. New points are then generated by interpolation along the image pixel direction onto the surface, achieving accurate supplementation of image details and point cloud density. Compared to traditional interpolation (such as nearest neighbor interpolation), which can only supplement points between existing points, this method actively generates new points using image pixel details, improving point cloud density. Furthermore, it integrates image texture and semantic information, ensuring that newly added points strictly follow the geometric features of the object surface, avoiding the loss of geometric details. Compared to purely visual methods, which rely on image matching to estimate depth and are easily affected by lighting and texture deficiencies (e.g., large depth estimation errors in white wall scenes), this application relies on the accurate three-dimensional coordinates of LiDAR depth as a benchmark to overcome depth errors caused by lighting and texture interference. A reliable depth model is generated through surface fitting, ensuring the geometric accuracy of the newly added point cloud.

[0053] The above description is merely a preferred embodiment of the method for denser LiDAR point clouds that integrates image information according to the present invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the core principle of "visual image-guided LiDAR point cloud denserization" of the present invention. In the superpixel segmentation stage, the number of superpixels K and the weight parameter m of the linear iterative clustering algorithm for image brightness values ​​can be dynamically adjusted in combination with scene complexity to balance segmentation details and computational efficiency. In the surface fitting stage, an adaptive distance threshold strategy can be introduced, that is, based on the point cloud density of each cluster of point clouds... Automatically calculate distance threshold ,in, Distance threshold The strategy is calculated by using the ratio of the total number of points n within a cluster to the 3D spatial volume V occupied by the cluster, where k is a preset scene accuracy coefficient. This strategy is automatically executed when performing surface fitting on each clustered point cloud, and is calculated in real-time. And adjust The method dynamically optimizes the interior point determination criteria to improve the accuracy of surface models in complex scenarios. This enhances the densification accuracy in complex scenarios (such as dynamic traffic and sparse texture regions), ensuring the algorithm's generalization ability in multiple scenarios such as autonomous driving and 3D reconstruction. These improvements also rely on the complementary characteristics of LiDAR and visual cameras and do not deviate from the invention's purpose of "compensating for the deficiencies of a single sensor by using a LiDAR point cloud densification method that fuses image information." Therefore, they should also be considered within the scope of protection of this invention.

[0054] This invention achieves point cloud densification for lidar systems by fusing image information, and its innovative advantages are as follows: 1. Deep integration of superpixel segmentation and point cloud clustering: The SLIC algorithm is used to segment the image into semantically related superpixel patches. Combined with point cloud-image mapping, cross-modal clustering is achieved, avoiding the cross-structure mixing problem that is prone to occur in traditional spatial distance-based clustering. This ensures that the same cluster of point clouds corresponds to the same local structure in the real scene, significantly improving the clustering accuracy and providing a precise data foundation for subsequent surface fitting.

[0055] 2. Image-guided surface fitting and interpolation strategy: Using the local surface fitted by the RANSAC algorithm as the depth reference, and combining it with pixel-by-pixel linear interpolation to generate new 3D points, it not only relies on the accurate geometric information of LiDAR to solve the problem of large depth error in pure vision methods, but also supplements the areas not covered by the original point cloud through image texture, ultimately improving the point cloud density, and the denser point cloud retains the scene details.

[0056] 3. Dynamic parameter optimization mechanism: By adaptively adjusting the number of superpixels, weight parameters and distance thresholds, it can adapt to different application requirements such as autonomous driving (dynamic scene) and 3D reconstruction (static scene), and balance detail preservation and computational efficiency in complex environments, thereby enhancing the robustness of the algorithm.

[0057] In summary, this invention effectively compensates for the shortcomings of single sensors, improves the accuracy and practicality of point cloud densification, and provides technical support for autonomous driving environmental perception, robot navigation, and high-precision 3D reconstruction.

[0058] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0059] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0060] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for densifying point clouds of a lidar system by fusing image information, characterized in that, Includes the following steps: A data acquisition device is constructed, and the vision camera and LiDAR are fixed on a rigid fixture. The relative position and attitude of the two and the parameters of the vision camera are calibrated by a calibration tool to obtain calibration parameters. The lidar is used to collect 3D point cloud data, and the vision camera is used to simultaneously collect scene image information to ensure timestamp alignment. The image is segmented into superpixels using the linear iterative clustering algorithm SLIC based on image brightness values, generating superpixel patch regions with consistency in brightness and spatial proximity. Based on the calibration parameters, point cloud and image mapping and clustering are performed, the three-dimensional point cloud data is mapped to the patch area of ​​the image, and the three-dimensional points mapped to the same patch area are divided into the same cluster of point cloud; For each cluster of point clouds, a random sampling consensus algorithm is used to fit the surface to obtain a surface model that reflects the geometric characteristics of the cluster. The pixels of the image are traversed, and dense points not covered by the original point cloud are generated by linear interpolation based on the surface model. The dense points are then transformed to the world coordinate system to achieve point cloud densification.

2. The method for densifying lidar point clouds by fusing image information according to claim 1, characterized in that: The calibration parameters include the intrinsic parameter matrix K of the vision camera and the extrinsic parameter matrices of the lidar and the vision camera. R is the rotation matrix and t is the translation vector. The calibration tool adopts the Zhang Zhengyou checkerboard calibration method. The calibration is completed by simultaneously acquiring calibration board images and point cloud data in at least 8 postures. During the calibration process, it is ensured that the calibration board is fully displayed in the image and that the lidar can scan at least 3 corner points.

3. The method for densifying lidar point clouds by fusing image information according to claim 1, characterized in that: The specific implementation method of the superpixel segmentation is as follows: The linear iterative clustering algorithm based on image brightness values ​​is adopted. First, the RGB image is converted to the Lab color space. Then, superpixel segmentation is performed on the converted image. Let the total number of pixels in the image be N and the number of target superpixels be H. The value of H is set according to the image resolution and the complexity of scene texture. Calculate the initial mesh size The image is divided into S×S grids. The center of each grid is used as the initial candidate cluster center. Within a 3×3 to 5×5 neighborhood, and with larger neighborhoods for complex textures, the pixel with the smallest gradient value and the smallest brightness variance is selected as the final cluster center. Define the feature vector of each pixel as Where L, a, and b are color components, and x and y are pixel coordinates; the similarity between pixels and cluster centers is calculated using a normalized distance metric formula: ； ； ； in, It is the Euclidean distance between the pixel and the cluster center in the Lab color space. It is the target pixel brightness value. It is the red-green deviation value of the target pixel. It is the yellow-blue deviation value of the target pixel. , , These are the pixel brightness value of the cluster center, the pixel red-green deviation value, and the pixel yellow-blue deviation value; It is the Euclidean distance between the pixel and the cluster center in the image space. These are the image space coordinates of pixels. These are the image space coordinates of the cluster centers; It is the initial grid size of the superpixel. These are weight parameters. Represents the normalized distance metric between a pixel and the cluster center; Weight parameters are adjusted based on scenario complexity. Make contextual adjustments. The value ranges from 10 to 40, and is used to balance the weights of color similarity and spatial proximity: for scenes with complex textures, color similarity is given priority to ensure segmentation details. Set the value to 10-20; for scenes with simple textures, prioritize spatial proximity to improve computational efficiency and ensure regular superpixel shapes. Take 25-40; Each cluster center searches for pixels only within a 2S×2S rectangular area. After each iteration, the average feature vector of each cluster is recalculated as the new cluster center until the iteration converges. The iteration stops when the positional change of all cluster centers is less than the pixel-level threshold in two adjacent iterations, and outputs a superpixel patch region with brightness and spatial consistency, ensuring that the pixel depth change within the same superpixel is consistent.

4. The method for densifying lidar point clouds by fusing image information according to claim 3, characterized in that: The H value is set based on the image resolution and scene texture complexity: For an image with a resolution of 1280×720, H is set to 500-1000 for simple texture scenes and 1500-2500 for complex texture scenes; For a 1920×1080 image, H is set to 1000-2000 for simple texture scenes and 3000-5000 for complex texture scenes; For a 2560×1440 image, H is set to 2000-3500 for simple texture scenes and 5000-8000 for complex texture scenes; Furthermore, under the same resolution and scene type, the H value is finely adjusted within the above range according to the sparsity of the original point cloud of the LiDAR.

5. The method for densifying lidar point clouds by fusing image information according to claim 1, characterized in that: The specific methods for mapping and clustering point clouds and images are as follows: Based on the intrinsic parameter matrix K and extrinsic parameter matrix of the visual camera Formula for constructing a pinhole camera imaging model: ； in, , , U and v are the coordinates of a point in the world coordinate system, and u and v are the pixel coordinates of the corresponding point in the camera coordinate system. It is a scaling factor. It is the camera intrinsic parameter matrix. It is an extrinsic parameter matrix. For rotation matrix, It is a translation vector. This is called a projective camera matrix; Three-dimensional points in the world coordinate system collected by lidar , , Through the external parameter matrix The intrinsic parameter matrix K is mapped to the camera coordinate system, and after correction by radial and tangential distortion coefficients, the pixel coordinates of the two-dimensional image are obtained. ; If pixel coordinates If points fall into the same superpixel patch area, the 3D point is assigned to the corresponding cluster of point clouds to ensure that points within the cluster correspond to the same local structure in the real scene.

6. The method for densifying lidar point clouds by fusing image information according to claim 5, characterized in that: The process of surface fitting using the random sampling consensus algorithm includes: For each cluster of point clouds, at least three non-collinear points are randomly selected using the Random Sampling Consensus (RANSAC) algorithm and substituted into the plane equation. Solve the initial model; Calculate the point cloud density of clustered point cloud clusters ,in, V represents the total number of points within the clustered point cloud, and V is the three-dimensional spatial volume occupied by the point cloud cluster in the world coordinate system. ,in , These are the maximum and minimum coordinates of all points within the cluster along the X-axis in the world coordinate system, respectively. , These are the maximum and minimum coordinates of all points within the cluster along the Y-axis, respectively; , These are the maximum and minimum coordinates of all points within the cluster along the Z-axis, respectively. Based on point cloud density Set a distance threshold with scene accuracy coefficient k ,in, For scene accuracy coefficients, Point cloud density; Calculate the Euclidean distance from the remaining points within the cluster to the initial model, and select points with distances less than [a certain value]. Points that are determined to be interior points conforming to the model are greater than or equal to [a certain value]. Points are identified as external points, and the proportion of internal points to the total number of points in the cluster is calculated simultaneously. Repeat the above process, and finally select the model with the most interior points and the highest proportion as the final fitted surface of the clustered point cloud cluster.

7. The method for densifying lidar point clouds by fusing image information according to claim 1, characterized in that: The specific implementation method of the point cloud densification is as follows: Iterate through all pixels within each superpixel The pixels are mapped to the lidar coordinate system through inverse transformation of camera intrinsic parameters and inverse transformation of extrinsic parameters. Substitute the mapped lidar coordinates into the surface model of the corresponding clustered point cloud. For higher-order polynomials, The form is ,in, These are the coefficients of the demand solution. , It is an error term; Linear interpolation is performed on the fitted surface pixel by pixel within the superpixel patch to calculate the depth value of the corresponding superpixel patch region, generating new 3D points not covered by the original point cloud. The dense points are transformed to the world coordinate system using an extrinsic parameter matrix and then merged with the original point cloud of the lidar to achieve lidar point cloud densification.

8. The method for densifying lidar point clouds by fusing image information according to claim 1, characterized in that: The 3D point cloud data and image information are timestamp aligned by hardware-triggered PPS signals and software calibration, with an alignment error of ≤0.5ms.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to cause the electronic device to perform the lidar point cloud densification method for fusing image information as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the lidar point cloud densification method for fusing image information as described in any one of claims 1-8.

Images