Multi-stage filtering and occlusion elimination method, system and device for laser point cloud and panoramic image fusion and medium

Abstract

The application discloses a kind of multi-stage filtering occlusion elimination methods, systems, equipment and medium for laser point cloud and panoramic image fusion.The method comprises: obtaining laser point cloud, panoramic image and POS / IMU attitude position data, based on panoramic image exposure time and laser point scanning time, time window matching is carried out;Three-level coordinate transformation is executed;Three-dimensional point cloud is mapped to panoramic image pixel plane using spherical projection model;For multiple three-dimensional points projected into the same pixel or adjacent pixel, depth-first filtering, vector angle filtering and gray similarity matching are executed in turn, remove invisible points blocked by foreground objects and correct projection offset;Finally, the color of panoramic image is given to the retained visible points, and color point cloud with few occlusion artifacts and high registration accuracy is generated.The method does not depend on labeled training data, and can improve the realism, reliability and computational efficiency of point cloud texture mapping in mobile measurement, digital twin and three-dimensional reconstruction scenarios.

Description

Technical Field

[0001] This invention belongs to the fields of multi-sensor fusion, 3D point cloud processing and computer vision technology, and relates to a multi-level filtering occlusion elimination method, system, device and medium for laser point cloud and panoramic image fusion. Background Technology

[0002] In applications such as mobile surveying, digital twins, realistic 3D modeling, and urban infrastructure inspection, LiDAR can acquire high-precision 3D geometric structures, while panoramic cameras can acquire high-resolution texture information. Fusing LiDAR point clouds with panoramic imagery can generate color point clouds with both geometric accuracy and accurate color information, providing foundational data for scene understanding, mapping, and subsequent intelligent analysis.

[0003] Current point cloud coloring typically relies on extrinsic parameter calibration results between the LiDAR and camera, as well as pose information at the time of acquisition, to project 3D points onto the corresponding image and read pixel colors. However, the observation centers of the LiDAR and panoramic camera are usually not completely aligned, and foreground and background objects in the scene can occlude each other. During the 3D-to-2D projection process, multiple 3D points at different depths may compete for the same pixel coordinates. If the pixel color is directly assigned to all projected points, distant buildings, walls, or ground features that should be occluded will incorrectly acquire the textures of foreground objects such as trees, vehicles, and poles, resulting in obvious occlusion artifacts.

[0004] Furthermore, the mobile acquisition process also presents challenges such as asynchronous panoramic image exposure time and laser scanning time, POS trajectory errors, extrinsic parameter calibration residuals, point cloud noise, and pixel rounding errors. These factors further amplify projection offset and mismatch, making it difficult to reliably obtain high-quality color point clouds using traditional single geometric projection or single depth screening.

[0005] While deep learning methods can be used for cross-modal feature extraction or occlusion relationship inference, they typically require large amounts of labeled data and significant computational resources, and their generalization ability is limited under different acquisition devices, scene densities, and lighting conditions. Therefore, it is necessary to propose an occlusion removal method that does not require labeled training data, has clear physical meaning, is computationally efficient, and is applicable to the fusion of laser point clouds and panoramic images. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to address the projection occlusion artifact problem caused by viewpoint inconsistency, depth competition, pose error, and pixel rounding error in the fusion of existing laser point clouds and panoramic images, and to provide a multi-level filtering occlusion elimination method, system, device, and medium for laser point cloud and panoramic image fusion.

[0007] Technical solution: The present invention provides a multi-level filtering occlusion elimination method for the fusion of laser point clouds and panoramic images, comprising the following steps:

[0008] S100: Acquire laser point cloud data, panoramic image data, and POS / IMU data for characterizing the position and attitude of the acquisition device in the scene to be fused, and acquire the external parameter calibration results between the lidar and the panoramic camera.

[0009] S200: Based on the exposure time of panoramic images and the scanning time of laser points, time window matching is performed to establish a time correspondence for each frame of panoramic images or each batch of laser point clouds.

[0010] S300. Based on POS / IMU data and external parameter calibration results, the laser point cloud is transformed from the world coordinate system to the lidar coordinate system and the panoramic camera coordinate system in sequence to obtain the projection point in the camera coordinate system.

[0011] S400. Establish a spherical projection model of the panoramic image, and map the points to be projected to the pixel coordinates of the panoramic image to form an initial set of projection points.

[0012] S500. Perform depth-first filtering on the initial projection point set. For multiple 3D points projected to the same pixel coordinate, only the 3D point closest to the observation center is retained to eliminate obvious occlusion points caused by pixel depth competition.

[0013] S600: Perform vector angle filtering on the point set after depth-first filtering, calculate the angle between the point to be processed and the neighboring points relative to the observation center within the preset neighborhood radius, and remove occluded points according to the angle threshold; in order to eliminate occluded points that were not processed by depth-first filtering due to angle error or local density.

[0014] S700: Based on the depth map generated by point cloud projection, perform grayscale similarity matching with the grayscale map of the panoramic image to determine the optimal offset between the panoramic image and the point cloud projection and correct the projection relationship; in order to compensate for the systematic registration offset that cannot be corrected by geometric filtering.

[0015] S800 assigns the color information of the corresponding pixels in the corrected panoramic image to the retained visible 3D points, and outputs the occlusion-removed color point cloud.

[0016] Furthermore, in step S200, the exposure time of the panoramic image is... Set a time window around the center and scan the laser points within the window at specific times. Perform a search and select the appropriate option. The smallest data pair is taken as the time matching result, and this time matching result is used as the data association basis for subsequent coordinate transformation and color assignment.

[0017] Furthermore, in step S300, the points in the world coordinate system are first represented using the global pose matrix. Points converted to LiDAR coordinate system Then, using the extrinsic parameter matrix from the LiDAR to the panoramic camera, Points converted to panoramic camera coordinate system The coordinate transformation satisfies: , ,in This is the pose transformation matrix from the world coordinate system to the lidar coordinate system. This is the extrinsic transformation matrix from the lidar coordinate system to the panoramic camera coordinate system.

[0018] Further, in step S400, let the coordinates of any three-dimensional point P in the camera coordinate system be (x, y, z), and the width and height of the panoramic image be W and H, then the pixel coordinates (u, v) corresponding to this three-dimensional point are determined by... Sure.

[0019] Further, in step S500, for each pixel coordinate (u,v), a set of three-dimensional points projected onto that pixel coordinate is constructed, and... As the depth value corresponding to that pixel coordinate, only those that satisfy... 3D points.

[0020] Furthermore, in step S600, a preset filtration radius is formed with the point to be processed as the center. Search for neighboring points within the area, calculate the vector angles formed by the point to be processed and each neighboring point relative to the observation center, and if the minimum vector angle is less than an angle threshold... If the point to be processed is found to be in an occlusion relationship, it will be removed from the projection matching result.

[0021] Further, in step S700, the depth map grayscale image and the panoramic image grayscale image are divided along the horizontal and / or vertical directions into... For each region, the gray level and proportion of each region are calculated. The gray level region sequence of the panoramic image is cyclically shifted, and the position with the smallest mean square error is taken as the optimal offset. The pixel correspondence between the panoramic image and the point cloud projection is updated according to the optimal offset.

[0022] Another object of the present invention is to provide a multi-level filtering occlusion elimination system for the fusion of laser point clouds and panoramic images, comprising:

[0023] The data acquisition module is used to acquire laser point cloud, panoramic image and POS / IMU data;

[0024] The time window matching module is used to establish the temporal correspondence between laser point clouds and panoramic images;

[0025] The coordinate transformation module is used to perform transformations between the world coordinate system, the lidar coordinate system, and the panoramic camera coordinate system;

[0026] The spherical projection module is used to map 3D points to panoramic image pixel coordinates;

[0027] A multi-stage filtering module is used to perform depth-first filtering and vector angle filtering;

[0028] The grayscale matching optimization module is used to correct the projection offset based on the consistency of grayscale distribution;

[0029] The color point cloud output module is used to output the color point cloud after occlusion removal.

[0030] Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described multi-level filtering occlusion elimination method for the fusion of laser point clouds and panoramic images.

[0031] Another object of the present invention is to provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion by running the computer program stored in the memory.

[0032] Beneficial effects: First, the present invention employs a multi-level filtering mechanism from coarse to fine, which can effectively reduce erroneous coloring caused by same-pixel depth occlusion, neighborhood occlusion, and local registration errors. Second, the present invention does not rely on manually labeled training data, and the algorithm flow has clear geometric meaning and physical interpretability. Third, the present invention combines time window matching, three-level coordinate transformation, spherical projection, depth-first filtering, vector angle filtering, and gray-level similarity matching into a complete closed-loop process, which is suitable for panoramic image point cloud coloring in mobile measurement systems. Fourth, the present invention can eliminate a large number of redundant projection points while maintaining an effective point cloud structure, improving the visual realism of the colored point cloud and the reliability of subsequent applications. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the overall process of a multi-level filtering occlusion elimination method according to an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of time window matching according to an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of a spherical projection model according to an embodiment of the present invention;

[0036] Figure 4 This is a logic diagram of a multi-level filtering occlusion elimination method according to an embodiment of the present invention. Detailed Implementation

[0037] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0038] Example 1

[0039] like Figure 1 As shown, this embodiment provides a multi-level filtering occlusion removal method for the fusion of laser point clouds and panoramic images. The method includes steps such as data acquisition, time window matching, three-level coordinate transformation, spherical projection, depth-first filtering, vector angle filtering, grayscale similarity matching, and color point cloud output, as detailed below:

[0040] S100. Acquire data from the scene to be fused. Specifically, acquire laser point cloud data, panoramic image data, and POS / IMU data using a mobile measurement platform or a fixed acquisition platform. Laser point cloud data includes the position coordinates of multiple 3D points and their scanning times; panoramic image data includes the panoramic image and its exposure times; and POS / IMU data includes the position, attitude, velocity, or angular velocity information of the acquisition device. Simultaneously, obtain the extrinsic parameter transformation relationship between the LiDAR and the panoramic camera through calibration.

[0041] S200, Execution time window matching. For example... Figure 2 As shown, the exposure time of the panoramic image Set a time window around the center and search for the laser point scanning time within the time window. Select time difference The smallest data pair is used as the matching result. This step can reduce geometric correspondence errors caused by sensor asynchrony during mobile data acquisition.

[0042] S300, Perform a three-level coordinate transformation. For three-dimensional points in the world coordinate system. First, the pose matrix given by the POS / IMU is transformed to the lidar coordinate system to obtain... Then, based on the extrinsic parameter matrix between the LiDAR and the panoramic camera, it is transformed to the panoramic camera coordinate system to obtain... This process can be represented as: , .in This is the pose transformation matrix from the world coordinate system to the lidar coordinate system. This is the extrinsic transformation matrix from the lidar coordinate system to the panoramic camera coordinate system.

[0043] S400, Perform spherical projection. For example... Figure 3As shown, let the coordinates of any point P in the panoramic camera coordinate system be (x, y, z), and let P(u, v) represent the projection point of point P(x, y, z) onto the image plane after spherical projection, with its position being pixel coordinates (u, v). The width of the panoramic image is W, and the height is H. Then the pixel coordinates can be obtained from... The calculations show that, using this spherical projection model, a 3D point cloud can be mapped onto the pixel plane of a 2D panoramic image.

[0044] S500 performs depth-first filtering, retaining only the 3D point closest to the observation center for multiple 3D points projected to the same pixel coordinates. Specifically, let... For the point cloud set to be processed, For point The projected pixel coordinates are determined. For each pixel coordinate (u,v), all 3D points projected to that pixel coordinate are collected, and the distance from each point to the observation center is calculated. and with This is used as the depth value for that pixel. Only those values ​​that satisfy the criteria are retained. The three-dimensional points are identified, and the remaining points are determined as occluded points and deleted.

[0045] S600: Perform vector angle filtering on the point set after depth-first filtering. Calculate the angle between the point to be processed and its neighboring points relative to the observation center within a preset neighborhood radius, and remove occluded points based on the angle threshold. Specifically, as follows... Figure 4 As shown, even after depth-first filtering, residual neighboring points may still exist due to azimuth angle errors, zenith angle errors, or pixel rounding errors. Therefore, a preset filtering radius is used to filter the point to be processed. Search for neighboring points within the observation center, calculate the angle between the vectors formed by the point to be processed and its neighboring points relative to the observation center, and record the minimum angle. If the minimum angle is less than the angle threshold... If the point to be processed is found to have an occlusion competition relationship with its neighboring points, then the point to be processed will be removed from the matching results.

[0046] S700: Perform grayscale similarity matching. Convert the depth map generated from the point cloud projection into a grayscale image, and also convert the panoramic image into a grayscale image. Divide the two grayscale images horizontally into... For each region, calculate its grayscale value and proportion. Perform cyclic translation on the grayscale region sequence of the panoramic image, and calculate the mean square error between the translated region proportion sequence and the depth map region proportion sequence. Take the translation amount with the smallest mean square error as the optimal horizontal offset. If necessary, perform the same partitioning statistics and cyclic translation matching along the vertical direction to obtain the optimal vertical offset.

[0047] S800: Perform color assignment and output. Based on the optimal offset obtained from grayscale similarity matching, correct the pixel correspondence between the panoramic image and the point cloud projection. Read the corresponding pixel colors for the retained visible 3D points and write the color information into the point cloud attributes to generate a colored point cloud after occlusion removal. Experimental data shows that this workflow can significantly reduce erroneous projection points caused by occlusion and maintain good registration accuracy without relying on labeled training data.

[0048] Example 2

[0049] This embodiment provides a multi-level filtering occlusion elimination system for the fusion of laser point clouds and panoramic images. The system includes a data acquisition module, a time window matching module, a coordinate transformation module, a spherical projection module, a multi-level filtering module, a grayscale matching optimization module, and a color point cloud output module.

[0050] The system comprises the following modules: a data acquisition module for acquiring laser point clouds, panoramic images, POS / IMU data, and extrinsic parameter calibration results; a time window matching module for establishing a data correspondence based on the exposure time of the panoramic image and the scanning time of the laser points; a coordinate transformation module for performing conversions between the world coordinate system, the lidar coordinate system, and the panoramic camera coordinate system; a spherical projection module for mapping 3D points to pixel coordinates of the panoramic image; a multi-level filtering module for performing depth-first filtering and vector angle filtering; a grayscale matching optimization module for determining the registration offset based on the consistency of grayscale distribution between the depth map and the panoramic image; and a color point cloud output module for outputting the occlusion-removed color point cloud.

[0051] Example 3

[0052] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion as described in Embodiment 1. The computer-readable storage medium can be a read-only memory, random access memory, disk, optical disk, flash memory, or other media that can be used to store a computer program.

[0053] Example 4

[0054] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion described in Embodiment 1. The computer device can be an edge computing device in a mobile measurement platform, a server, a workstation, or other electronic equipment with point cloud processing capabilities.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, combinations, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion, characterized in that, Includes the following steps: S100: Acquire laser point cloud data, panoramic image data, and POS / IMU data for characterizing the position and attitude of the acquisition device in the scene to be fused, and acquire the external parameter calibration results between the lidar and the panoramic camera. S200: Based on the exposure time of panoramic images and the scanning time of laser points, time window matching is performed to establish a time correspondence for each frame of panoramic images or each batch of laser point clouds. S300. Based on POS / IMU data and external parameter calibration results, the laser point cloud is transformed from the world coordinate system to the lidar coordinate system and the panoramic camera coordinate system in sequence to obtain the points to be projected in the camera coordinate system. S400. Establish a spherical projection model of the panoramic image, and map the points to be projected to the pixel coordinates of the panoramic image to form an initial set of projection points. S500. Perform depth-first filtering on the initial projection point set. For multiple 3D points projected to the same pixel coordinate, only retain the 3D point closest to the observation center. S600: Perform vector angle filtering on the point set after depth-first filtering, calculate the angle between the point to be processed and the neighboring points relative to the observation center within the preset neighborhood radius, and remove the occluded points according to the angle threshold. S700: Perform grayscale similarity matching between the depth map generated based on point cloud projection and the grayscale map of the panoramic image to determine the optimal offset between the panoramic image and the point cloud projection and correct the projection relationship. S800 assigns the color information of the corresponding pixels in the corrected panoramic image to the retained visible 3D points, and outputs the occlusion-removed color point cloud.

2. The multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion according to claim 1, characterized in that, In step S200, the exposure time of the panoramic image is used. Set a time window around the center and scan the laser points within the window at specific times. Perform a search and select the appropriate option. The smallest data pair is taken as the time matching result, and this time matching result is used as the data association basis for subsequent coordinate transformation and color assignment.

3. The multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion according to claim 1, characterized in that, In step S300, the points in the world coordinate system are first used to... Points converted to LiDAR coordinate system Then, using the extrinsic parameter matrix from the LiDAR to the panoramic camera, Points converted to panoramic camera coordinate system The coordinate transformation satisfies: , ,in This is the pose transformation matrix from the world coordinate system to the lidar coordinate system. This is the extrinsic transformation matrix from the lidar coordinate system to the panoramic camera coordinate system.

4. The multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion according to claim 1, characterized in that, In step S400, let the coordinates of any 3D point P in the camera coordinate system be (x, y, z), and the width and height of the panoramic image be W and H, respectively. Then, the pixel coordinates (u, v) corresponding to this 3D point are determined by... Sure.

5. The multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion according to claim 4, characterized in that, In step S500, for each pixel coordinate (u,v), a set of 3D points projected onto that pixel coordinate is constructed, and... As the depth value corresponding to that pixel coordinate, only those that satisfy... 3D points.

6. The multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion according to claim 1, characterized in that, In step S600, with the point to be processed as the center, within a preset filtration radius... Search for neighboring points within the area, calculate the vector angles formed by the point to be processed and each neighboring point relative to the observation center, and if the minimum vector angle is less than an angle threshold... If the point to be processed is found to be in an occlusion relationship, it will be removed from the projection matching result.

7. The multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion according to claim 1, characterized in that, In step S700, the depth map grayscale image and the panoramic image grayscale image are divided along the horizontal and / or vertical directions into... For each region, the gray level and proportion of each region are calculated. The gray level region sequence of the panoramic image is cyclically shifted, and the position with the smallest mean square error is taken as the optimal offset. The pixel correspondence between the panoramic image and the point cloud projection is updated according to the optimal offset.

8. A multi-level filtering occlusion elimination system for laser point cloud and panoramic image fusion, characterized in that, include: The data acquisition module is used to acquire laser point cloud, panoramic image and POS / IMU data; The time window matching module is used to establish the temporal correspondence between laser point clouds and panoramic images; The coordinate transformation module is used to perform transformations between the world coordinate system, the lidar coordinate system, and the panoramic camera coordinate system; The spherical projection module is used to map 3D points to panoramic image pixel coordinates; A multi-stage filtering module is used to perform depth-first filtering and vector angle filtering; The grayscale matching optimization module is used to correct the projection offset based on the consistency of grayscale distribution; The color point cloud output module is used to output the color point cloud after occlusion removal.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion as described in any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the multi-level filtering occlusion elimination method for laser point cloud and panoramic image fusion as described in any one of claims 1 to 7 by running the computer program stored in the memory.

Images