A method, system, device and storage medium for Gaussian scene floating-point culling

Abstract

Embodiments of the present application provide a Gaussian scene floating point removal method, system, device and storage medium, relating to the technical field of point cloud images. The method comprises: obtaining three-dimensional point cloud data and image data of a target scene; compressing the point cloud data and retaining key geometric features to obtain processed three-dimensional point cloud data; performing semantic segmentation on the image data to obtain semantic segmented image data; aligning the processed point cloud and the semantic image to generate an initial depth map; correcting the depth values of the initial depth map according to the object categories to which the pixels in the semantic segmented image data belong to obtain an optimized depth map; and inputting the optimized depth map into a Gaussian scene model to remove floating points and obtain three-dimensional Gaussian sputtering data. In this way, by fusing the geometric information of the three-dimensional point cloud and the texture information of the image, a depth map with clear depth is generated, floating points are effectively removed, data authenticity and cleanliness are improved, and the removal effect and efficiency in weak texture and high dynamic lighting scenes are enhanced.

Description

Technical Field

[0001] This invention relates to the field of point cloud image technology, and more specifically, to a method, system, device, and storage medium for removing floating points in a Gaussian scene. Background Technology

[0002] In techniques for generating 3D Gaussian scenes from photographs, optimization errors or data noise during the modeling process often result in a large number of false floaters in the scene, damaging its realism and integrity. These floaters appear as isolated Gaussian spheres detached from the real geometric structure, forming visual artifacts resembling flying fluff, which seriously affect the visual quality of high-precision applications such as digital twins and virtual tours. Therefore, an effective removal mechanism is urgently needed.

[0003] Current mainstream methods rely on monocular depth estimation models (such as MonoDepth and DPT) to generate depth maps as constraints, and identify and remove floating points by comparing the values ​​of the predicted depth of the Gaussian sphere with the corresponding positions in the depth map.

[0004] However, monocular depth estimation is essentially inferring three-dimensional information from two-dimensional images, lacking real geometric basis. In scenes with weak texture, large lighting changes, or severe occlusion, the depth error is significant, which can easily lead to the false deletion or omission of floating points. Moreover, monocular depth estimation algorithms usually involve complex neural network inference processes, and processing a high-resolution image can take hundreds of milliseconds, which significantly slows down the overall modeling process, resulting in poor floating point removal effect and low efficiency. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method, system, device and storage medium for removing floating points in Gaussian scenes. By combining the spatial geometric position information of three-dimensional point clouds with the clear texture of two-dimensional images, a depth map with clear position and depth value is generated, and floating points are effectively removed in Gaussian scenes. This helps to maintain stable depth accuracy in weak texture and high dynamic lighting scenes, thereby enhancing the removal effect of floating points and improving efficiency.

[0006] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a method for removing floating points in a Gaussian scene. The method includes: acquiring three-dimensional point cloud data and image data of a target scene; performing data compression processing on the three-dimensional point cloud data and retaining the key geometric features of the target scene to obtain three-dimensional point cloud data with retained key geometric features; performing semantic segmentation on the image data to obtain semantically segmented image data, wherein the semantically segmented image data includes the object category to which each pixel belongs; aligning the three-dimensional point cloud data with retained key geometric features and the semantically segmented image data to obtain an initial depth map, wherein each pixel in the initial depth map has a corresponding depth value; correcting the depth value of each pixel in the initial depth map according to the object category to which each pixel belongs to to obtain an optimized depth map; and inputting the optimized depth map into a preset Gaussian scene model for floating point removal to obtain three-dimensional Gaussian sputtering data.

[0007] In this embodiment, the 3D point cloud data and image data of the target scene are processed separately. The 3D point cloud data is compressed to obtain 3D point cloud data that retains key geometric features. The image data is semantically segmented to obtain the object category to which each pixel in the image belongs. Then, the processed 3D point cloud data and image data are aligned to obtain an initial depth map, so that each 3D point in the 3D point cloud corresponds to a pixel in the initial depth map, and each pixel in the initial depth map has a corresponding depth value. This facilitates depth value correction of the initial depth map. The corrected depth map is then input into a Gaussian scene model for floating point removal, resulting in highly accurate Gaussian sputtering data. In this way, by combining the spatial geometric position information of the 3D point cloud with the clear texture of the 2D image, a depth map with clear position and well-defined depth values ​​is generated. Floating points are effectively removed in the Gaussian scene, resulting in cleaner and more realistic 3D Gaussian sputtering data. This helps maintain stable depth accuracy even in scenes with weak texture and high dynamic lighting, thereby enhancing the floating point removal effect and improving efficiency.

[0008] In some embodiments, the step of performing data compression processing on the three-dimensional point cloud data and retaining the key geometric features of the target scene to obtain three-dimensional point cloud data with retained key geometric features includes: performing denoising processing on the three-dimensional point cloud data to obtain denoised three-dimensional point cloud data; performing downsampling processing on the denoised three-dimensional point cloud data and retaining the key geometric features of the target scene during the downsampling process to obtain three-dimensional point cloud data with retained key geometric features.

[0009] This setup, through denoising and downsampling, compresses the 3D point cloud data, effectively reducing the amount of data while ensuring the integrity of the geometric details required to support subsequent depth optimization and floating point detection, thus providing a high-quality 3D input foundation for the entire method.

[0010] In some embodiments, performing semantic segmentation on the image data to obtain semantically segmented image data includes: performing distortion correction processing on the image data to obtain corrected image data; performing exposure processing on the corrected image data to obtain exposed image data; and performing semantic segmentation on the exposed image data to obtain the object category to which each pixel in the image data belongs.

[0011] This setup, through distortion correction and exposure processing, significantly improves the consistency and interpretability of image quality, enabling the semantic segmentation process to maintain high robustness and accuracy under diverse shooting conditions, and providing a reliable prerequisite for subsequent depth map optimization based on semantic categories.

[0012] In some embodiments, aligning the 3D point cloud data with key geometric features and the semantic segmentation image data to obtain an initial depth map includes: spatiotemporally aligning the 3D point cloud data with key geometric features and the semantic segmentation image data to obtain spatiotemporally aligned 3D point cloud data and spatiotemporally aligned image data; and obtaining an initial depth map based on the spatiotemporally aligned 3D point cloud data and the spatiotemporally aligned image data.

[0013] This setup, by aligning the 3D point cloud data and semantic segmentation image data in time and space, ensures a one-to-one correspondence between 3D spatial points and 2D image pixels, providing a geometric basis for subsequent depth value assignment.

[0014] In some embodiments, obtaining an initial depth map based on the spatiotemporally aligned 3D point cloud data and the spatiotemporally aligned image data includes: projecting the spatiotemporally aligned 3D point cloud data onto the image coordinate system of the spatiotemporally aligned image data to obtain a plurality of projected pixels with depth values, wherein the projected pixels correspond one-to-one with the 3D points in the spatiotemporally aligned 3D point cloud data; and performing depth completion processing on the regions in the image coordinate system of the spatiotemporally aligned image data that are not covered by projection based on the plurality of projected pixels with depth values ​​and the object category to which each pixel in the semantic segmentation image data belongs, to obtain an initial depth map.

[0015] This setup projects each 3D point in the 3D point cloud data to correspond one-to-one with each projected pixel. By semantically segmenting the object category to which each pixel belongs in the image data, the depth values ​​of other pixels between the projected pixels are filled in, effectively improving the spatial coverage and geometric rationality of the initial depth map.

[0016] In some embodiments, the step of correcting the depth value of each pixel in the initial depth map according to the object category to which each pixel belongs to obtain an optimized depth map includes: correcting the depth value of each pixel in the initial depth map according to a preset geometric prior model corresponding to the object category to which each pixel belongs to obtain a geometrically consistent depth map; filtering out a first pixel belonging to the foreground occluded object category and a second pixel belonging to the background occluded object category according to the object category to which each pixel belongs; and removing the first pixel whose depth value is not within a first preset range and the second pixel whose depth value is not within a second preset range from the geometrically consistent depth map to obtain the optimized depth map.

[0017] This setup filters out the first and second pixels by the object category to which each pixel belongs, and then determines whether there is a logical contradiction by combining their depth distributions. This identifies the pixels with logical contradictions, and then obtains the 3D points corresponding to the logically contradictory pixels. By removing the contradictory pixels, an optimized depth map is obtained, thereby improving the accuracy, continuity, and reliability of the optimized depth map.

[0018] In some embodiments, the preset Gaussian scene model includes multiple Gaussian spheres. The step of inputting the optimized depth map into the preset Gaussian scene model to remove floating points and obtain 3D Gaussian sputtering data includes: inputting the optimized depth map into the preset Gaussian scene model to obtain the true depth value of each Gaussian sphere in the optimized depth map; calculating the absolute deviation based on the depth value of each Gaussian sphere in the preset Gaussian scene model and the true depth value in the optimized depth map; and removing floating points based on the absolute deviation to obtain 3D Gaussian sputtering data.

[0019] This setup, by introducing high-precision, semantically enhanced external deep supervision, enables the quantitative evaluation and targeted removal of spurious Gaussian spheres. The final output of the 3D Gaussian sputtering data shows significant improvements in both visual cleanliness and geometric fidelity.

[0020] Secondly, embodiments of the present invention provide a Gaussian scene floating point removal system, the system comprising: an acquisition module for acquiring 3D point cloud data and image data of a target scene; a processing module for performing data compression processing on the 3D point cloud data to obtain 3D point cloud data that retains the key geometric features of the target scene; performing semantic segmentation on the image data to obtain semantic segmented image data, the semantic segmented image data including the object category to which each pixel belongs; an optimization module for aligning the 3D point cloud data that retains the key geometric features of the target scene and the semantic segmented image data to obtain an initial depth map, each pixel in the initial depth map having a corresponding depth value; correcting the depth value of each pixel in the initial depth map according to the object category to which each pixel belongs to to obtain an optimized depth map; and a removal module for inputting the optimized depth map into a preset Gaussian scene model for floating point removal to obtain 3D Gaussian sputtering data.

[0021] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the Gaussian scene floating point removal method as described in the first aspect.

[0022] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the Gaussian scene floating point culling method as described in the first aspect.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of a method for removing floating points in a Gaussian scene provided in an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of sub-steps S201~S202 of step S200 in step 1 of the figure;

[0027] Figure 3 This is a flowchart of sub-steps S301~S303 of step S300 in step 1 of the figure;

[0028] Figure 4 This is a flowchart of sub-steps S410~S420 of step S400 in step 1 of the figure;

[0029] Figure 5 This is a flowchart of sub-steps S421~S422 of step S420 in step 1 of the figure;

[0030] Figure 6 This is a flowchart of sub-steps S501~S503 of step S500 in step 1 of the figure;

[0031] Figure 7 This is a flowchart of sub-steps S601~S603 of step S600 in step 1 of the figure;

[0032] Figure 8 This is a schematic diagram of the functional modules of the Gaussian scene floating point removal system provided in an embodiment of the present invention;

[0033] Figure 9 A block diagram of an electronic device provided in an embodiment of the present invention.

[0034] Icons: 1000 - Gaussian Scene Floating Point Removal System; 1100 - Acquisition Module; 1200 - Processing Module; 1300 - Optimization Module; 1400 - Removal Module; 2000 - Electronic Device; 2100 - Processor; 2200 - Memory; 2300 - Bus; 2400 - Communication Interface. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0036] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0037] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0038] As described in the background section, monocular depth estimation is essentially inferring three-dimensional information from two-dimensional images. It lacks real geometric basis and has significant depth errors in scenes with weak textures, large lighting variations, or severe occlusion. This can easily lead to the false deletion or omission of floating points. Moreover, monocular depth estimation algorithms usually involve complex neural network inference processes, and processing a single high-resolution image can take hundreds of milliseconds, which significantly slows down the overall modeling process and results in poor floating point removal and low efficiency.

[0039] To address this, embodiments of the present invention provide a method for removing floating points in Gaussian scenes. This method combines the spatial geometric location information of a 3D point cloud with the clear texture of a 2D image to generate a depth map with clear location and depth values. This map effectively removes floating points in Gaussian scenes, helping to maintain stable depth accuracy even in scenes with weak texture and high dynamic range lighting, thereby enhancing the removal effect and improving efficiency. (See also...) Figure 1 , Figure 1 This is a flowchart of a method for removing floating points in a Gaussian scene according to an embodiment of the present invention. The method includes steps S100 to S600:

[0040] S100: Acquire 3D point cloud data and image data of the target scene.

[0041] In this embodiment, a synchronous acquisition system consisting of a visual camera and a LiDAR is used to capture data of the target scene. The 3D point cloud data is generated by LiDAR (such as a 16-line or 32-line LiDAR), reflecting the 3D coordinate distribution of the object's surface in real space. The image data is acquired by a high-resolution visual camera within the same time window, containing rich texture, color, and lighting information. It should be noted that the acquisition of the 3D point cloud data and image data must meet spatiotemporal consistency requirements, meaning they correspond to scene observations at the same time and from the same viewpoint. With hardware support, the main control module sends a trigger signal every 100ms to synchronize the acquisition actions of the camera and LiDAR, ensuring frame alignment accuracy. For devices without hardware synchronization capabilities, software-level time alignment is achieved by recording their respective timestamps and establishing a linear mapping relationship, with synchronization errors controlled within 1ms.

[0042] Furthermore, the acquired 3D point cloud data is stored in the form of discrete 3D point sets, with each point having X, Y, and Z coordinate values, representing its position in the lidar coordinate system; the image data exists in the form of a 2D pixel array with a resolution of 1920×1080 or other adapted specifications, and each pixel contains RGB channel values.

[0043] S200. Perform data compression processing on the 3D point cloud data and retain the key geometric features of the target scene to obtain 3D point cloud data with key geometric features retained.

[0044] In this embodiment, raw 3D point cloud data typically contains a large number of redundant points and noise. Directly using it for depth map generation would increase the computational burden and potentially introduce errors; therefore, data compression is necessary. Thus, isolated and outlier points caused by measurement instability are identified and removed. Selective downsampling is then achieved through point cloud structure analysis. This reduces the overall number of points while ensuring the complete preservation of the key geometric structures required for removing floating points. Here, key geometric features are defined as a set of spatial points that characterize the continuity of object boundaries, surface topological changes, or abrupt changes in local curvature. Specifically, these include four typical structures: planar points, edge points, corner points, and columnar points. This data compression strategy is not simply uniform sampling, but rather dynamically adjusts the sampling density based on local geometric complexity, ensuring accurate representation of the real-world scene structure while reducing the amount of data.

[0045] S300. Perform semantic segmentation on the image data to obtain semantically segmented image data, which includes the object category to which each pixel belongs.

[0046] In this embodiment, unprocessed image data only provides visual appearance information and lacks the ability to understand content, thus failing to support semantically guided deep optimization operations. Therefore, a deep learning model is used to perform pixel-level classification on the image, outputting semantic segmentation image data, where each pixel is assigned a category label to indicate the type of object it belongs to.

[0047] For example, the preset object categories mainly include eight common scene elements: ground, buildings, vegetation, pedestrians, vehicles, furniture, streetlights, and road signs, covering the main objects in indoor and outdoor structured environments. Furthermore, semantic segmentation not only provides category attribution but also implicitly includes occlusion relationship information: when the masks of two instances overlap and have different depths, the hierarchical relationship between the foreground occluder and the occluded background can be determined. For instance, if the average depth of a pedestrian instance is less than that of a wall instance, the former is determined to occlude the latter. This occlusion relationship map will be used in the subsequent depth correction stage to identify erroneous projection points that violate physical laws. In addition, the semantic segmentation results are output in a structured format, with each detected instance containing a category ID, bounding box coordinates, a binary mask array, a confidence score, and a unique identifier, facilitating downstream module calls to enhance the cognitive ability of image content. This allows for not only seeing but also understanding scene composition, providing a logical basis for subsequent refined modeling of the depth map.

[0048] S400: Align the 3D point cloud data and semantic segmentation image data that retain key geometric features to obtain an initial depth map. Each pixel in the initial depth map has a corresponding depth value.

[0049] In this embodiment, the 3D point cloud and the image reside in different representation spaces, necessitating the establishment of a precise mapping relationship between them for effective fusion. Therefore, a unified calibration in the spatiotemporal dimension is first performed. Then, projection and completion operations are executed based on the aligned data, ultimately generating an initial depth map with the same resolution as the image, where each pixel contains depth information. Specifically, the extrinsic parameter matrix between the camera and LiDAR, obtained through calibration, transforms the 3D point cloud data, preserving key geometric features, from the LiDAR coordinate system to the camera coordinate system, ensuring it shares the same reference frame as the image. Simultaneously, a timestamp matching mechanism ensures high consistency in acquisition time between the point cloud frame and the image frame, avoiding misalignment caused by motion blur. Based on this, the camera intrinsic parameter matrix is ​​used to project each 3D point (X,Y,Z) onto the image plane (u,v), with the formulas u=round(fx·X / Z+cx) and v=round(fy·Y / Z+cy), and the Z value of that point is used as the depth value of the corresponding pixel. Since the LiDAR point cloud itself is sparse, with most pixels not directly projected, a semantically guided interpolation strategy is also required for depth completion. For example, plane fitting is used to estimate the missing depth in the ground area, and bilinear interpolation is used to fill the holes in the vegetation area, so as to realize the transformation from sparse three-dimensional measurement to dense two-dimensional depth map, providing an initial version with basic integrity for subsequent optimization.

[0050] S500: Correct the depth value of each pixel in the initial depth map according to the object category to which each pixel belongs, and obtain an optimized depth map.

[0051] In this embodiment, although the initial depth map has been completed, unreasonable phenomena such as local fluctuations, discontinuities across objects, or occlusion penetration may still exist, making it difficult to directly use as a reliable constraint for Gaussian scene training. Therefore, a dual correction mechanism based on semantic categories is introduced. First, depth consistency is improved based on the inherent geometric priors of each category, and then occlusion logic is combined to remove outliers that violate physical rules. Specifically, corresponding optimization strategies are set for different object categories: for horizontal plane regions of the ground category, the RANSAC algorithm is used to fit the plane equation Z=aX+bY+c, and depth values ​​deviating from the plane by more than 5cm are replaced with the fitted results to eliminate local perturbations; for vertical structures of the wall category, all depth values ​​within its mask range are extracted to calculate the average depth d. avg , will be with d avg Pixel depth correction for deviations exceeding 3cm is d avg To achieve depth smoothing, for moving objects such as pedestrians or vehicles, the depth value of the edge pixels of their mask is locked to prevent edge blurring caused by filtering. For non-rigid objects such as vegetation, a 5×5 Gaussian filter (σ=1.5) is applied inside the mask for moderate smoothing to preserve natural undulation characteristics.

[0052] Furthermore, by combining the occlusion relationship information provided by semantic segmentation, potential error points are identified: Foreground occlusion objects that are non-transparent and have solid outlines (such as pedestrians, vehicles, and railings) are selected, with a mask area greater than 500 pixels and a semantic confidence score higher than 0.8; the occluded background areas (such as walls and sky) are located, and it is checked whether there are laser points projected there but with a depth less than or close to the foreground depth. If so, they are determined to be erroneous penetration points and are removed. Thus, this correction process comprehensively utilizes both geometric regularity and physical rationality criteria, significantly improving the overall quality of the depth map.

[0053] S600: Input the optimized depth map into the preset Gaussian scene model and remove floating points to obtain three-dimensional Gaussian sputtering data.

[0054] In this embodiment, during the training process of the Gaussian scene model, false Gaussian spheres that are partially detached from the real geometry are generated, appearing to float in the air or far from the actual object surface, affecting the realism of the final rendering effect. Therefore, an optimized depth map is used as an external supervision signal to quantitatively evaluate the spatial rationality of each Gaussian sphere, and individuals that do not meet the criteria are dynamically removed during training iterations. Specifically, the optimized depth map is introduced into the Gaussian scene training process, and the following judgment is performed in each pruning cycle: First, the center coordinates G of each Gaussian sphere are... world (X w ,Y w Z w From the world coordinate system via camera extrinsic R T Transform to the camera coordinate system to obtain G. cam (X c ,Y c Z c )= R T (G world t). Where t = (t x ,t y ,t z ) is the extrinsic translation vector (the coordinates of the camera origin in the world coordinate system); then it is projected onto the image plane (u,v), with the formula u=fx· +cx, v=fy· +cy, where and The normalized coordinates (dimensionless) of the center of the Gaussian sphere on the camera's imaging plane are multiplied by the focal length (in pixels) and then added to the principal point coordinates (the center pixel of the image) to obtain the final pixel indices u and v. u and v may be non-integers (e.g., u=512.3, v=384.7), and decimal places must be retained for subsequent interpolation (to avoid a 1-pixel error caused by direct rounding). If the projected point falls within the image area and Zc>0, the true depth value d at the corresponding location is extracted from the optimized depth map using bilinear interpolation. gauss Furthermore, the depth Z of the Gaussian sphere itself is calculated. c Its corresponding true depth d gauss The absolute deviation between them is Δd=|Zc d gauss |. When Δd exceeds the dynamic threshold T depth When the average depth of the scene is 5%, the Gaussian sphere is determined to be a floating point.

[0055] For example, in the redundancy pruning stage, all Gaussian spheres marked as floating points are physically removed from the model, spatial index structures such as the KD tree are updated, and their occupied memory resources are released. For cases where the original location belongs to a high-frequency detail region, a dynamic densification compensation mechanism is triggered to generate new spheres to fill the gaps; however, for regions without real structural support, no further additions are made. Specifically, the densification process analyzes the fitting residuals of the Gaussian spheres and the scene detail requirements, and adds new Gaussian spheres in a targeted manner to capture high-frequency structures (such as object edges and texture details); the pruning process removes redundant Gaussian spheres that do not significantly contribute to scene representation (such as excessively overlapping spheres or spheres with weights approaching 0) by setting thresholds (such as volume thresholds or contribution thresholds), thereby achieving model lightweighting. Specifically, to accurately remove Gaussian spheres marked as floating points, this embodiment of the invention deeply integrates the aforementioned floating point determination results (including the `is_floating` label and depth deviation Δd) into the determination logic of the pruning stage, forming a "targeted pruning" mechanism. Specifically, this is achieved through the first aspect: label information transmission. During training, the floating point label information (whether it is a floating point `is_floating`, floating point confidence P) of each Gaussian sphere is written as a core attribute into the Gaussian sphere data structure and updated in real time with the training cycle. (Floating point confidence P) In the middle, T depth This is a dynamic threshold, defaulting to the scene's average depth D. avg 5%, that is, T depth =0.05× Davg。 Δd is the depth deviation of the center of the Gaussian sphere, Δd=|Z c -d gaussThe sigmoid function is used to normalize Δd to [0,1], ensuring that the larger the deviation, the higher the sub-score. k is the sensitivity coefficient (value 5~10, default is 7), when Δd=T... depth At that time, S depth ≈0.5; when Δd≥2×T depth At that time, S depth ≥0.95 (close to 1, strong floating point characteristic); when Δd≤0.5×T depth At that time, S depth ≤0.1 (weak feature).

[0056] During the prune process, for each Gaussian sphere to be evaluated, its floating point label is first verified. If the Gaussian sphere has been labeled as a floating point (is_floating=True, floating point confidence P≥0.5), and satisfies the condition that "depth deviation Δd continuously exceeds the dynamic threshold T", then the prune is considered valid. depth If the condition is strengthened, it will be directly included in the set to be pruned; for Gaussian spheres not marked as floating points, the traditional prune logic will still be used to determine whether to prune, so as not to affect the removal of normal redundant spheres.

[0057] For floating Gaussian spheres included in the pruning set, physical culling is performed: their indices are removed from the Gaussian sphere array, the spatial index structure (such as a KD tree) is updated to exclude neighborhood associations of the sphere, and the memory resources it occupies are released; simultaneously, a culling log (including Gaussian sphere coordinates, culling reason, and corresponding training period) is recorded to provide traceability for subsequent model validation. After the floating Gaussian sphere is culled, if its original position belongs to a high-frequency detail region of the real scene (such as a wall edge), the compensation mechanism of the densification process is triggered: based on the geometric and appearance features (such as scale and color) of the surrounding real Gaussian spheres, new effective Gaussian spheres are generated to fill the gaps, avoiding scene structure breakage caused by culling; if the original position is a redundant region without real structure (such as an isolated region in the air), no compensation is performed to maintain scene realism.

[0058] It can be seen that the floating point removal mechanism relies on high-precision depth maps to achieve accurate identification and targeted removal of false Gaussian spheres. The final output of three-dimensional Gaussian sputtering data is superior to traditional methods in terms of visual cleanliness and geometric fidelity.

[0059] In some embodiments, for step S200, one possible implementation of the present invention is as follows: Figure 2 As shown, Figure 2 This is a flowchart of sub-steps S201~S202 of step S200 in Figure 1. Steps S201~S202 include:

[0060] S201. Denoise the 3D point cloud data to obtain the denoised 3D point cloud data.

[0061] In this embodiment, the original 3D point cloud data is affected by environmental interference and sensor limitations, often resulting in isolated noise points caused by low reflectivity, occlusion, or atmospheric disturbances. These points lack true geometric meaning and, if left unprocessed, will mislead subsequent depth modeling. Therefore, a combination of statistical filtering and radius filtering is used to identify and remove such outliers.

[0062] For example, the point cloud is normalized to transform it to the scene's local coordinate system to reduce numerical calculation errors; then for each point P i The system searches for its k nearest neighbors, where the value of k is adaptively set according to the number of LiDAR lines: k=15 for 16-line LiDAR and k=20 for 32-line LiDAR. The neighborhood search is accelerated using a KD-tree. Next, the average Euclidean distance μi from each point to its neighbors is calculated. The global mean μ and standard deviation σ of all points are calculated. A removal threshold T = μ + ασ (α defaults to 2.5) is set. If μi > T for a point, it is identified as an outlier and marked for deletion; if μi ≤ T, it is retained as a valid point. To further address the issue of missed detections in low-density areas, radius filtering is introduced for supplementary verification: a filtering radius r is set (0.3m for 16-line LiDAR and 0.2m for 32-line LiDAR) and the minimum number of neighboring points k are set. min =5, check the number of neighboring points within the range r for the points retained after statistical filtering. If it is less than k... min Points with low density and isolated values ​​are considered low-density isolated points and deleted. Furthermore, to avoid accidentally deleting real edge structures, points with curvature greater than a threshold (default value 0.8) are marked as edge candidates by calculating their curvature. Even if the μi of an edge candidate is slightly higher than T (e.g., T=0.55m, μi=0.58m), it is not deleted immediately. Subsequent semantic segmentation verification (e.g., whether the edge point belongs to an object contour) is used to avoid accidentally deleting key structural edges of the scene (e.g., the intersection of a wall and the ground). Therefore, this denoising process balances denoising intensity and feature preservation, effectively improving the overall quality of the point cloud.

[0063] S202. Perform downsampling on the denoised 3D point cloud data, and retain the key geometric features of the target scene during the downsampling process to obtain 3D point cloud data with key geometric features retained.

[0064] In this embodiment, although denoising has been completed, the point cloud may still contain too many redundant points, especially in flat areas, which is not conducive to real-time processing and efficient modeling. Therefore, further downsampling is required. Thus, a composite strategy of "voxel grid downsampling as the main method and curvature-first sampling as a supplementary method" is adopted to reduce the number of points while ensuring the integrity of key structures.

[0065] For example, the voxel side length is determined based on the number of LiDAR lines and the scene resolution requirements. The formula is ,in The average point spacing in the point cloud is approximately 0.15–0.3m for 16-line data and approximately 0.08–0.15m for 32-line data. To adjust the coefficients, ensure that each voxel contains 3–5 original points. Then, the space is divided into a 3D voxel mesh. For each non-empty voxel, the average coordinates of all points within it are calculated as the centroid, replacing the original point set to achieve data compression. The formula is: centroid coordinates (X... c ,Y c Z c )=( , , However, to prevent oversimplification in high-curvature regions such as edges and corners, a feature-dense region preservation mechanism is introduced: Three eigenvalues ​​λ1≥λ2≥λ3 are obtained through PCA analysis of the neighborhood point covariance matrix, and the curvature C=λ3 / (λ1+λ2+λ3) is calculated. For planar regions with C≤0.2, the original voxel size is maintained; for edge or corner regions with C>0.2, the voxel side length is reduced to half its original size (e.g., from 4cm to 2cm), and finer-grained sub-voxels are re-divided and new centroids are calculated, thereby increasing the sampling density in that region. This downsampling method can be understood as achieving an intelligent compression mode of "coarse sampling in smooth areas and fine sampling in detailed areas," reducing the overall data volume. This ensures that the preservation of key geometric features avoids blurring of key features such as edges and corners after downsampling, ensuring accurate capture of geometric discontinuities during subsequent Gaussian scene depth map optimization.

[0066] For example, planar points are located on a continuous plane, and their neighborhood normal vectors have the same direction. They can be obtained by fitting a planar model using the RANSAC algorithm and then filtering points with a distance of less than 2 cm. In a Gaussian scene, if the depth value of a Gaussian point deviates from the depth value of its corresponding planar point by more than a threshold, it is considered a floating point. Edge points are located at the intersection of two different geometric surfaces, with large local curvature and abrupt changes in normal vectors, and a local curvature greater than 0.2. They can be obtained by calculating curvature or using edge detection operators (a 3D extension of the Sobel operator) and filtering points with curvature C ≥ 0.2. Edge points form the "structural skeleton" of the scene; if a Gaussian point deviates from the contour line formed by the edge points (e.g., a Gaussian point that should be along a corner drifts into the air), it is considered a floating point. Corner points are locations where three or more geometric surfaces intersect, with significantly different eigenvalues. PCA analysis of the eigenvalues ​​of neighboring points is used to filter points where λ1 >> λ2 >> λ3 (e.g., λ1 / λ2 > 5, λ2 / λ3 > 5). Corner points can be understood as key anchor points in the scene, possessing high 3D coordinate accuracy (error ≤ 1cm), and can serve as benchmark calibration points for Gaussian scene depth maps. If the spatial distance between a Gaussian point and a corner point exceeds a threshold (e.g., 0.3m), it is considered a floating point. Columnar points are distributed on the surface of cylindrical objects. The RANSAC algorithm is used to fit the cylindrical model ((x-x0)). 2 +(y-y0) 2 =r 2 (z is free), and points that are less than a threshold (e.g., 3cm) away from the surface of the cylinder are selected. It can be understood that in a Gaussian scene, if the cylinder radius corresponding to the depth value of a Gaussian point deviates from the radius of the real cylinder point by more than 20% (e.g., the radius of the real pipe is 10cm, and the radius of the Gaussian point is 15cm), it is determined to be a floating point.

[0067] In some embodiments, for step S300, one possible implementation of the present invention is as follows: Figure 3 As shown, Figure 3 This is a flowchart of sub-steps S301~S303 of step S300 in Figure 1. Steps S301~S303 include:

[0068] S301. Perform distortion correction processing on the image data to obtain the corrected image data.

[0069] In this embodiment, the camera lens inevitably suffers from radial distortion (barrel or pincushion) and tangential distortion (caused by mounting offset), resulting in pixel shifts at the image edges and affecting the spatial alignment accuracy with the 3D point cloud. Therefore, the original image is geometrically corrected using the pre-calibrated camera intrinsic parameter matrix K and distortion coefficients D.

[0070] For example, calibration parameters matching the current camera model are loaded, the `undistort` function in the OpenCV library is called, the original image, K, and D are input, and the distortion-corrected image data is output. If black borders (invalid pixel areas) appear in the corrected image, the `getOptimalNewCameraMatrix` function is used to calculate a new intrinsic parameter matrix without black borders, and then combined with the `undistortRectifyMap` and `remap` operations to complete the cropping correction, ensuring that the output image does not contain blank areas. In this embodiment, the distortion-corrected image data is defined as an image representation in which each pixel has been restored to the ideal optical imaging position, and its pixel arrangement strictly conforms to the pinhole camera model, with the projection error of the laser point cloud controlled within 1 pixel (approximately 0.1° viewing angle deviation). Therefore, this processing provides the necessary geometric basis for subsequent laser point cloud texture mapping and image back-projection depth completion, ensuring the reliability of pixel-level alignment between multimodal data.

[0071] S302. Perform exposure processing on the corrected image data to obtain the exposed image data.

[0072] In this embodiment, image data is susceptible to changes in lighting conditions, resulting in overexposure, underexposure, or uneven brightness in certain areas, which can interfere with the feature extraction performance of the semantic segmentation model. Therefore, a global and local exposure normalization strategy is implemented to ensure that images acquired at different times and under different conditions maintain consistent brightness and contrast levels.

[0073] For example, the corrected image is converted to a grayscale image using the following formula: Calculate the average brightness L of the entire image. avg With the standard deviation of luminance L std For L avg ∈[100,150] and L std Images ∈ [40, 80] are considered to be within the normal exposure range by default and require no adjustment; for L... avg For underexposed images with a value <100, gamma correction is used to enhance brightness, with γ ranging from 0.6 to 0.8; for L... avg Overexposed images >150 are darkened using γ = 1.2–1.5; for Lstd For low-contrast images (<40), histogram equalization is performed to enhance dynamic range. Furthermore, to improve uneven local lighting issues such as strong light around windows and indoor shadows, the `createCLAHE` function is called to create an adaptive equalization object with a block size of 8×8, a contrast limit of `clipLimit=2.0`, and `bins=256`, performing local contrast optimization on the grayscale image. Finally, the normalized grayscale adjustment ratio is mapped back to the original RGB image, i.e., an adjustment coefficient is calculated for each pixel (R, G, B). ,in, The output is the normalized grayscale value, where x and y are pixel coordinates; the output is the adjusted RGB value (R×). , G× , B× This process avoids color distortion. Therefore, it is evident that this exposure processing workflow significantly improves image quality stability, providing a unified and reliable input condition for subsequent semantic segmentation.

[0074] S303. Perform semantic segmentation on the exposed image data to obtain the object category to which each pixel in the image data belongs.

[0075] In this embodiment, the image after distortion correction and exposure processing already possesses good geometric and lighting consistency, making it suitable as input for a semantic segmentation model. Therefore, the Mask R-CNN architecture is used to perform instance-level semantic segmentation, outputting the category of each pixel and its corresponding instance information. Specifically, the backbone network uses ResNet-50 instead of the original ResNet-101 to achieve lightweighting, reducing computation by 30% while retaining sufficient feature extraction capabilities; the category branch retains only eight target categories: ground, buildings, vegetation, pedestrians, vehicles, furniture, streetlights, and road signs, removing redundant categories to reduce the risk of classification confusion; the mask branch introduces dilated convolutions for large-scale objects such as "buildings" and "ground" to improve boundary accuracy, ensuring that the alignment error between the mask edge and the real geometric edge does not exceed 2 pixels.

[0076] During the offline training phase, synchronized photo sequences were collected in typical scenes (such as offices, streets, and parks), and instance categories, bounding boxes, and pixel-level contours were manually labeled. For each photo, objects were manually labeled with 8 predefined categories, minimum bounding rectangles, and pixel-level contours. For overlapping objects, "occluding object and occluded object" pairs were labeled, such as "pedestrian and wall," indicating that a pedestrian occludes a wall. The sample dataset was expanded through operations such as random cropping (simulating viewport changes), brightness perturbation (simulating exposure changes), horizontal flipping (expanding data diversity), and Gaussian blur (simulating motion blur) to improve the model's generalization ability. Then, a ResNet-50-FPN model pre-trained on the COCO dataset was used as the initial weights, and the loss function was defined as: Total Loss = Classification Loss + Bounding Box Loss + Mask Loss. Training parameters, including batch size (default 8), learning rate (default 0.001), and number of iterations (default 50), were set to execute the training. Finally, the model was tested on an independent validation set. The mAP (mean accuracy) of instance segmentation for key categories (buildings, ground) was ≥0.90 after training; the average distance between the mask edge and the manually labeled area was ≤1.5 pixels.

[0077] In the online inference phase, the image is first scaled to 800×600 and its RGB channels are normalized. Then, feature extraction is performed sequentially. The backbone network (ResNet-50) extracts multi-scale feature maps of the image (e.g., C2-C5, with resolution decreasing sequentially). The Feature Pyramid Network (FPN) fuses the multi-scale features to generate P2-P6 feature maps (balancing detail and semantics; P2 corresponds to high-resolution edge features, and P6 corresponds to low-resolution global semantics). Candidate region generation involves sliding a 3×3 convolutional kernel across the P2-P6 feature maps to generate 2000 candidate regions (RegionProposals). Each candidate region includes a confidence score for whether it is an object and a preliminary bounding box. Non-maximum suppression (NMS) is used to filter out 300 high-confidence candidate regions (confidence ≥ 0.7), which are then used as input for subsequent processing. The classification prediction and mask output process maps each candidate region onto a feature map, extracting features of a fixed size (7×7). It predicts eight object categories and precise bounding boxes for each candidate region. A 28×28 binary mask is output for each candidate region (upsampled to the original region size), and pixels with a mask value ≥ 0.5 are considered to belong to that object instance. Low-quality instances with a confidence score below 0.8 are then filtered out. Finally, the occlusion relationship is identified and output based on instance masks and laser depth. The area of ​​intersection (IoU) of the masks of any two instances is calculated. If IoU > 0.1, an occlusion relationship is determined. Combining the depth information of the laser point cloud (obtaining the average depth Z of each instance mask through spatial projection), instances with smaller Z values ​​(closer to the camera) are considered occluders, while those with larger Z values ​​are considered occluded objects (e.g., a pedestrian instance Z=3m occludes a wall instance Z=5m). An occlusion relationship map (e.g., pedestrians, walls, vegetation) is generated, and the occluder ID and occluded object ID of each instance are labeled, providing a basis for subsequent floating point removal. Floating points appearing in occluded object areas without corresponding occluders may be false points. This semantic segmentation process fully utilizes the preprocessing results, consistently outputting high-quality pixel-level classification results under diverse shooting conditions.

[0078] In some embodiments, for step S400, one possible implementation of the present invention is as follows: Figure 4 As shown, Figure 4 This is a flowchart of sub-steps S410~S420 of step S400 in Figure 1. Steps S410~S420 include:

[0079] S410. Spatiotemporally align the 3D point cloud data and semantic segmentation image data that retain key geometric features to obtain spatiotemporally aligned 3D point cloud data and spatiotemporally aligned image data.

[0080] In this embodiment, although single-modal preprocessing has been completed, the 3D point cloud and image may still be spatially misaligned due to asynchronous acquisition or coordinate system differences. Precise spatiotemporal alignment is necessary before subsequent projection operations can proceed. Therefore, the first step is to confirm whether time synchronization has been achieved: For hardware-synchronized acquisition devices, visual photographs (2D images) and laser point clouds (3D point cloud data) are synchronously acquired by the main control's synchronization hardware trigger signal. The SLAM system's main control module sends a trigger signal every 100ms. After receiving the signal, the camera takes a photograph (exposure time 20ms), and the LiDAR completes a point cloud scan (32-line LiDAR single-frame scan time is approximately 30ms) after receiving the signal. This ensures that both acquire the scene within the same 100ms time window, eliminating the need for additional time synchronization and allowing the acquisition of 2D image frames and 3D point cloud frames at the same moment. For non-hardware-synchronized acquisition devices, visual photographs (2D images) and laser point clouds (3D point sets) are timestamp aligned via software. The system records the camera's photograph capture timestamp (e.g., the T-timestamp of the camera's internal clock). cam ) and the scan timestamps of LiDAR point clouds (such as the T-times of the LiDAR internal clock) lidar ), by using a synchronization signal with a known time interval (such as a calibration signal triggered every 1 second), T is established cam With T lidar Linear mapping relationship: T lidar =k×T cam +b, calculate the parameters k and b. Finally, unify the timestamps of both to the same clock domain (such as system UTC time), and control the synchronization error within 1ms.

[0081] Next is spatial synchronization. Using calibration algorithms (such as Zhang Zhengyou's calibration method), the extrinsic parameter matrices between the camera and LiDAR are solved, including the rotation matrix R (describing the rotation relationship between the two coordinate systems) and the translation vector T (describing the positional offset between the two coordinate systems). Then, using the LiDAR coordinate system as the world coordinate system reference for the SLAM system, the camera coordinate system corresponding to the 2D image captured by the camera is transformed to the LiDAR world coordinate system using the extrinsic parameters R and T. Ultimately, the shooting perspective of an image and the scanning perspective of a point cloud frame completely correspond in 3D space; that is, a pixel in a certain direction in the image corresponds to a 3D point in a certain direction in the point cloud.

[0082] The resulting spatiotemporally aligned 3D point cloud data and spatiotemporally aligned image data have strict spatiotemporal consistency. The direction of any point in 3D space completely matches the direction of the corresponding pixel in the image, providing a solid spatial mapping foundation for subsequent "laser point cloud texture mapping" and "image back projection depth completion".

[0083] S420. Obtain an initial depth map based on the spatiotemporally aligned 3D point cloud data and spatiotemporally aligned image data.

[0084] In this embodiment, after spatiotemporal alignment is completed, the 3D point cloud and the image have a precise correspondence, and the initial depth map can be constructed. Therefore, using the spatiotemporally aligned 3D point cloud data as the source point set, and based on its projection position in the spatiotemporally aligned image data, a set of pixels with depth values ​​is gradually generated. Specifically, each 3D point P is traversed... i The image coordinates (X, Y, Z) are used to perform perspective projection using the camera intrinsic parameters fx, fy, cx, and cy: u = round(fx·X / Z + cx), horizontal pixel coordinates; v = round(fy·Y / Z + cy), vertical pixel coordinates; d = Z, depth value (unit: m, consistent with the accuracy of the LiDAR point cloud). These coordinates are mapped to the image coordinates (u, v), and the depth value d is assigned to that pixel. If multiple points are projected onto the same pixel, only the one with the smallest Z value (the nearest visible point) is retained, and the rest are discarded to avoid depth ambiguity. Invalid points are also filtered out if u / v exceeds the image resolution range, Z ≤ 0, or Z exceeds the effective detection range of LiDAR (e.g., 0.5–100 m). The resulting depth map is still sparse, with many areas remaining blank. To achieve full coverage, a semantically guided depth completion mechanism is introduced: For ground regions, points semantically pre-labeled as ground or with heights within the range of [0.1m, 0.5m] are selected, and a plane Z=aX+bY+c is fitted using RANSAC. The depth value of pixels without depth is then calculated by substituting these values ​​into the formula. For "wall" regions, points semantically labeled as walls are selected, and a vertical plane Xd is fitted. Z=X=d is then calculated as the depth. For "vegetation" regions, bilinear interpolation is used to calculate the missing depth based on a weighted average of surrounding valid pixels. Thus, this process completes the transition from sparse measurement to dense representation, forming a preliminarily usable initial depth map.

[0085] In some embodiments, for step S420, one possible implementation of the present invention is as follows: Figure 5 As shown, Figure 5 This is a flowchart of sub-steps S421-S422 of step S420 in Figure 1. Steps S421-S422 include:

[0086] S421. Project the spatiotemporally aligned 3D point cloud data onto the image coordinate system of the spatiotemporally aligned image data to obtain multiple projected pixels with depth values. The projected pixels correspond one-to-one with the 3D points in the spatiotemporally aligned 3D point cloud data.

[0087] In this embodiment, this step is the core of the initial depth map construction, responsible for establishing a direct relationship between 3D spatial points and 2D image pixels. Therefore, coordinate transformation is performed strictly according to the camera imaging principle to ensure that each 3D point accurately falls into its proper image position. Specifically, each spatiotemporally aligned 3D point Pi(X,Y,Z) is substituted into the projection formula: u=round(fx·X / Z+cx), v=round(fy·Y / Z+cy), where fx and fy are focal lengths, cx and cy are principal point coordinates, and X / Z and Y / Z are normalized coordinates. Multiplying by the focal length and adding the principal point yields the final pixel index. The depth value is directly taken as Z (the Z-axis component in the camera coordinate system) to avoid additional transformation errors. Each successfully projected point forms a projected pixel, whose (u,v) coordinates correspond to the image position, and d=Z is its depth value, maintaining a one-to-one correspondence with the original 3D point. If multiple points are projected to the same (u,v), only the one with the smallest Z is retained (foreground priority principle), and the rest are considered occluded points and discarded. This process can be understood as constructing the "skeleton" of the initial depth map, providing a reliable anchor point for subsequent completion operations based on this skeleton.

[0088] S422. Based on multiple projected pixels with depth values ​​and the object category to which each pixel belongs in the semantic segmentation image data, perform depth completion processing on the region not covered by projection in the image coordinate system of the spatiotemporally aligned image data to obtain an initial depth map.

[0089] In this embodiment, due to the sparsity of the laser point cloud, only a small number of pixels obtain direct depth values, and the remaining areas need to be filled in through reasonable inference. Therefore, an encryption strategy is implemented differently based on the structural prior knowledge provided by the semantic category. Specifically, for ground-type regions, the existing depth values ​​corresponding to all pixels marked as ground in the semantic mask are extracted, and a plane model Z=aX+bY+c is fitted using RANSAC. Outliers with errors exceeding 5cm are removed, and the depth of missing pixels is then calculated using this plane equation. For wall-type vertical planes, their depth values ​​are extracted to calculate the average depth d. avg , within the region and d avg Pixels with a deviation exceeding 3cm are corrected to d. avg To achieve depth consistency, for vegetation-rich texture areas, bilinear interpolation is used to calculate the missing depth based on the distance weighting of four adjacent valid pixels, with the formula d(u,v)=a·d(u1,v1)+b·d(u2,v2)+c·d(u3,v3)+d·d(u4,v4), where the weights are inversely proportional to the distance. Thus, this completion process fully utilizes the semantically guided structural regularity, enabling the initial depth map to achieve full coverage while maintaining physical plausibility.

[0090] In some embodiments, for step S500, one possible implementation of the present invention is as follows: Figure 6 As shown, Figure 6 This is a flowchart of sub-steps S501~S503 of step S500 in Figure 1. Steps S501~S503 include:

[0091] S501. Based on the preset geometric prior model corresponding to the object category to which each pixel belongs, the depth value of each pixel in the initial depth map is corrected to obtain a geometrically consistent depth map.

[0092] In this embodiment, although the initial depth map has been completed, there may be local deviations from the actual structure, requiring further optimization based on geometric regularities. Therefore, corresponding correction models are used for different object categories. Specifically, for the ground category, the depth values ​​of all pixels within its mask are extracted, and a plane Z=aX+bY+c (the ground's preset geometric prior model) is fitted using RANSAC. Pixel depths deviating from this plane by more than 5cm are replaced with the fitted values ​​to ensure that the ground area depth conforms to the plane. For the wall category, its average depth d is calculated. avg Pixels with deviations exceeding 3cm are corrected, and a cylindrical model (X) is fitted to the cylindrical structure. x0) 2 +(Y y0) 2 =r 2 (Based on the pre-defined geometric prior model of the wall surface), points deviating from the surface are corrected; for vegetation categories, a 5×5 Gaussian filter (σ=1.5) is used to smooth the mask (based on the pre-defined geometric prior model of vegetation), preserving natural variations; for instances such as pedestrians and vehicles, the depth values ​​of their mask edge pixels are locked to prevent edge merging caused by optimization. This correction process can be understood as enhancing the geometric plausibility of the depth map, making it closer to the structure of the real scene.

[0093] S502. Based on the object category of each pixel, filter out the first pixel belonging to the foreground occluded object category and the second pixel belonging to the background occluded object category.

[0094] In this embodiment, step S502 provides the logical premise for occlusion removal, requiring a clear distinction between foreground and background pixels, i.e., the first pixel and the second pixel. Therefore, instances meeting the following conditions are selected from the semantic segmentation results: the foreground occluded object must be a non-transparent category with a solid outline, such as pedestrians, vehicles, leaves, railings, and furniture; pixels corresponding to categories without occlusion capabilities, such as glass and air, are excluded as the first pixel; the instance mask area must be greater than 500 pixels to avoid misjudging small objects; and the semantic confidence score must be higher than 0.8 to ensure recognition reliability. The foreground occluded object list F[f1,f2,…,f] is output.n ], each f i Includes instance ID and mask_fore i (pixel-level binary array), average depth Z_fore i (The Z-mean of all 3D points). For each f i Locate the background area B that it occludes. i Extract mask_fore i The minimum bounding rectangle bbox_fore i Expand outwards by 1.5 times to get bbox_back i Select semantically related to the background category (e.g., sky, wall, ground) and masked by `mask_fore`. i The covered pixels are the second pixel, forming the mask_back of the occluded background. i This screening process can be understood as establishing a clear occlusion relationship map, laying the foundation for subsequent error point identification.

[0095] For example, in a real-world scene, the depth (Z) of the foreground object fore The depth (Z) of the background object it occludes must be less than the depth of the background object it occludes. back (i.e., the foreground is closer and the background is farther away), and the occluded background area should be covered by the mask of the foreground object in the image. If the laser point is projected onto the background area covered by the foreground mask, and its depth Z ≥ Z fore (If a laser point exists that should be occluded), it is considered an incorrectly projected laser point and needs to be removed. The detailed steps for occlusion removal are divided into three stages: occlusion region localization, error point identification, and depth map update. For occlusion region localization, foreground occluders are screened. Foreground objects matching the occlusion characteristics are selected from the semantic segmentation results. The screening criteria, from a category perspective, prioritize non-transparent categories with solid outlines (such as pedestrians, vehicles, leaves, railings, and furniture), excluding transparent / non-occluding categories (such as glass and air). From a size perspective, the instance mask area must be ≥500 pixels (to avoid misjudgment of occlusion by small objects such as birds). From a confidence perspective, the semantic segmentation confidence level must be ≥0.8 (to ensure reliable foreground object recognition). Finally, the foreground occluder list F=[f1,f2,...,f n ], each f i Includes: instance ID, mask (mask_fore) i (pixel-level binary array), average depth Z_fore i (The average Z-value of all laser points in this embodiment of the invention).

[0096] Furthermore, locate the occluded background area, and for each foreground occluder f iLocate the background region B that it occludes in the image. i Therefore, the bounding rectangle of the foreground mask is extracted, and mask_fore is calculated. i The minimum bounding rectangle bbox_fore i =[x1,y1,x2,y2] (pixel coordinates) to determine the spatial extent of the occluding object in the image. Then expand the background search range using bbox_fore i Centered on the background, expand the region by 1.5 times towards the background (the direction of increasing depth, i.e., where there is no clear direction in the image, and it needs to be combined with the laser point cloud) to obtain the bbox_back. i， Formula such as: bbox_back i =[x1-Δx,y1-Δy,x2+Δx,y2+Δy]; where Δx=(x2-x1)·0.25, Δy=(y2-y1)·0.25; to ensure that the obscured background is covered.

[0097] Further, filter the background area. In bbox_back i Within the context, pixels with semantic segmentation labels belonging to the background category (such as sky, wall, ground) and not covered by other foreground objects are extracted to form a mask_back for the occluded background region. i mask_back i Medium pixels satisfy: belong to the background category and mask_fore i The pixel value is 1, indicating it is occluded by the foreground. Finally, the occluded background mask `mask_back` is obtained for each foreground occluder. i .

[0098] S503. Remove the first pixel point whose depth value is not within the first preset range and the second pixel point whose depth value is not within the second preset range from the geometrically consistent depth map to obtain the optimized depth map.

[0099] In this embodiment, occlusion logic verification is performed to eliminate depth values ​​that violate physical laws. For each foreground occluder *fi*, the laser points projected onto the *mask_backi* region are checked: if the depth *Zp* ≤ *Z_forei*, it is determined to be a normal point; if *Zp* > *Z_forei* and its projected pixels are completely covered by *mask_forei* or less than 2 pixels from the edge, it is determined to be an erroneous penetration point and added to the erroneous laser point set *Perror*. Subsequently, all points within *Perror* are deleted from the original point cloud, and a new depth map is generated. The first preset range refers to the reasonable depth range of the foreground object itself, and the second preset range refers to the constraint that the background area should not have shallow depths. Therefore, this elimination mechanism effectively removes false depths caused by occlusion, significantly improving the realism of the depth map.

[0100] For example, for each laser point projected onto the mask_backi region, the occlusion logic is verified one by one, and erroneous projection points are identified. First, the mapping of the laser points to the occluded background region is performed. For each point P(X,Y,Z) in the laser point cloud, it is projected onto the image pixel (u,v) according to the camera intrinsic parameters (same as the projection method for generating the initial depth map above), and it is checked whether (u,v) falls on a certain occluded background mask mask_backi. i Inner (i.e., mask_back) i If [u][v]=1), then mark P as the point to be verified and associate it with its corresponding foreground occlusion f. i Next, error point determination is performed based on occlusion logic. For each point p to be verified (depth Z... p ), combined with its associated foreground occlusion f i Z_fore i For the average depth, perform the following judgment: depth level check, if Z p ≤Z_fore i If the depth of point p is less than or equal to the foreground depth, meaning it's located in front of the foreground, it's considered a normal point (it could be a point within the foreground object itself or an unobstructed background point). Penetration check: If Z... p >Z_fore i If point p is located behind the foreground and should be occluded, then a further check is performed: whether the projected pixels (u,v) of this point are completely obscured by the foreground mask mask_foreground. i Coverage (i.e., mask_fore_i[u][v]=1); if there is complete coverage, it is determined to be an incorrect penetration point and added to the set of incorrect laser points P. error If there is partial coverage, calculate the distance between the point and the foreground edge. Points with a distance ≤ 2 pixels are considered erroneous penetration points and added to the erroneous laser point set P. error Final output: Set of incorrect laser points P error .

[0101] Furthermore, P is removed from the original laser point cloud. error From all points, a laser point cloud with occluded errors is obtained; based on this point cloud, a depth map is regenerated (reusing the depth map generation and encryption steps described above). After the above steps are completed, a dense depth map with the same resolution as the camera photo is generated, with each pixel corresponding to a depth value d (unit: m). The depth map format uses a 16-bit single-channel integer (storage range 0~65535, precision 0.01m, adapted to the depth precision of LiDAR).

[0102] In some embodiments, for step S600, one possible implementation of the present invention is as follows: Figure 7 As shown, Figure 7This is a flowchart of sub-steps S601~S603 of step S600 in Figure 1. Steps S601~S603 include:

[0103] S601. Input the optimized depth map into the preset Gaussian scene model to obtain the true depth value of each Gaussian sphere in the optimized depth map.

[0104] In this embodiment, a mapping relationship is established between the Gaussian sphere and the actual depth. Therefore, the center of each Gaussian sphere, G_world(Xw,Yw,Zw), is mapped to the extrinsic parameter R. T Transform to the camera coordinate system to obtain G_cam(Xc,Yc,Zc), and then project it onto the image plane (u,v). If u∈[0,W], v∈[0,H] and Zc>0, then calculate its true depth value d in the optimized depth map using bilinear interpolation. gauss Otherwise, it is marked as NaN. This can be understood as the process of connecting virtual elements with real measurements.

[0105] S602. Calculate the absolute deviation based on the depth value of each Gaussian sphere in the preset Gaussian scene model and the actual depth value in the optimized depth map.

[0106] In this embodiment, this step quantifies the degree of deviation of the Gaussian sphere. Therefore, Δd = |Zc| is calculated. d gauss |, as the basis for judgment. Floating point confidence P = Sigmoid(k(Δd / T)) depth 0.5), k=7, T depth =0.05×D avg This can be understood as reflecting the severity of spatial misalignment.

[0107] S603. Based on the absolute deviation, floating points are removed to obtain three-dimensional Gaussian sputtering data.

[0108] In this embodiment, if Δd>T depth If P > 0.5, the Gaussian sphere is deleted during the prune phase, the index is updated, and memory is released. If the removed sphere belongs to a high-frequency region, densify compensation is triggered. Therefore, this mechanism achieves precise pruning and outputs clean 3D Gaussian sputtering data.

[0109] Based on the above method, embodiments of the present invention also provide a system corresponding to the above method, such as... Figure 8 As shown, Figure 8This is a schematic diagram of the functional modules of the Gaussian scene floating point removal system provided in this embodiment of the invention. It should be noted that the Gaussian scene floating point removal system 1000 provided in this embodiment has the same basic principle and technical effects as the method embodiment described above. For the sake of brevity, parts not mentioned in this embodiment can be referred to the corresponding content in the method embodiment.

[0110] In this embodiment, the Gaussian scene floating point removal system 1000 includes an acquisition module 1100, a processing module 1200, an optimization module 1300, and a removal module 1400. The acquisition module 1100 is used to acquire 3D point cloud data and image data of the target scene. It can be understood that the acquisition module 1100 is used to perform the above-described step S100.

[0111] The processing module 1200 is used to compress the 3D point cloud data to obtain 3D point cloud data that retains the key geometric features of the target scene; and to perform semantic segmentation on the image data to obtain semantically segmented image data, which includes the object category to which each pixel belongs. It can be understood that the processing module 1200 is used to execute the above steps S200~S300.

[0112] The optimization module 1300 aligns the 3D point cloud data and semantic segmentation image data that retain the key geometric features of the target scene to obtain an initial depth map, where each pixel has a corresponding depth value. The depth value of each pixel in the initial depth map is then corrected according to the object category to which each pixel belongs, resulting in an optimized depth map. It can be understood that the optimization module 1300 performs the steps S400~S500 described above.

[0113] The culling module 1400 is used to remove floating points from the optimized depth map input into a preset Gaussian scene model to obtain three-dimensional Gaussian sputtering data. It can be understood that the culling module 1400 is used to perform the above step S600.

[0114] In some embodiments, the processing module 1200 is used to perform denoising processing on the 3D point cloud data to obtain denoised 3D point cloud data; to perform downsampling processing on the denoised 3D point cloud data, and to retain the key geometric features of the target scene during the downsampling process, to obtain 3D point cloud data with retained key geometric features. It can be understood that the processing module 1200 is used to perform the above steps S201~S202.

[0115] In some embodiments, the processing module 1200 is used to perform distortion correction processing on the image data to obtain corrected image data; to perform exposure processing on the corrected image data to obtain exposed image data; and to perform semantic segmentation on the exposed image data to obtain the object category to which each pixel in the image data belongs. It can be understood that the processing module 1200 is used to perform the above steps S301 to S303.

[0116] In some embodiments, the optimization module 1300 is used to perform spatiotemporal alignment of 3D point cloud data retaining key geometric features and semantic segmentation image data to obtain spatiotemporally aligned 3D point cloud data and spatiotemporally aligned image data; and to obtain an initial depth map based on the spatiotemporally aligned 3D point cloud data and spatiotemporally aligned image data. It can be understood that the optimization module 1300 is used to perform the above steps S410~S420.

[0117] In some embodiments, the optimization module 1300 is used to project the spatiotemporally aligned 3D point cloud data onto the image coordinate system of the spatiotemporally aligned image data to obtain multiple projected pixels with depth values. Each projected pixel corresponds one-to-one with a 3D point in the spatiotemporally aligned 3D point cloud data. Based on the multiple projected pixels with depth values ​​and the object category to which each pixel in the semantically segmented image data belongs, depth completion processing is performed on the regions in the image coordinate system of the spatiotemporally aligned image data not covered by projection to obtain an initial depth map. It can be understood that the optimization module 1300 is used to perform the above steps S421~S422.

[0118] In some embodiments, the optimization module 1300 is used to correct the depth value of each pixel in the initial depth map according to a preset geometric prior model corresponding to the object category to which each pixel belongs, to obtain a geometrically consistent depth map; to filter out a first pixel belonging to the foreground occluded object category and a second pixel belonging to the background occluded object category according to the object category to which each pixel belongs; and to remove the first pixel whose depth value is not within a first preset range and the second pixel whose depth value is not within a second preset range in the geometrically consistent depth map, to obtain an optimized depth map. It can be understood that the optimization module 1300 is used to perform the above steps S501~S503.

[0119] In some embodiments, the elimination module 1400 is used to input the optimized depth map into a preset Gaussian scene model to obtain the true depth value of each Gaussian sphere in the optimized depth map; calculate the absolute deviation based on the depth value of each Gaussian sphere in the preset Gaussian scene model and the true depth value in the optimized depth map; and eliminate floating points based on the absolute deviation to obtain three-dimensional Gaussian sputtering data. Alternatively, the elimination module 1400 can immediately perform the above steps S601 to S603.

[0120] Based on the same inventive concept disclosed above, the present invention also provides a block diagram of an electronic device 2000 performing the above method. Please refer to... Figure 9 , Figure 9 This is a block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 2000 includes a processor 2100, a memory 2200, a bus 2300, and a communication interface 2400. The processor 2100 and the memory 2200 are connected via the bus 2300, and the processor 2100 communicates with external devices via the communication interface 2400.

[0121] Processor 2100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 2100 or through software instructions. The processor 2100 may be a general-purpose processor 2100, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0122] The memory 2200 is used to store computer programs. For example, the Gaussian scene floating point removal system 1000 in this embodiment of the invention includes at least one software function module that can be stored in the memory 2200 in the form of software or firmware. After receiving the execution instruction, the processor 2100 executes the program to implement the Gaussian scene floating point removal method in this embodiment of the invention.

[0123] The memory 2200 may include high-speed random access memory (RAM) or non-volatile memory. Optionally, the memory 2200 may be a storage device built into the processor 2100 or a storage device independent of the processor 2100.

[0124] Bus 2300 can be ISA bus 2300, PCI bus 2300 or EISA bus 2300, etc. Figure 9 It is indicated by only one double-headed arrow, but does not mean that there is only one bus 2300 or one type of bus 2300.

[0125] Electronic devices 2000 can be mobile phones, tablets, laptops, desktop computers, and other computer devices.

[0126] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor 2100, implements the Gaussian scene floating point culling method described above. This computer-readable storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0127] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for Gaussian scene floating-point culling, the method comprising: The method comprises: acquiring three-dimensional point cloud data and image data of a target scene; performing data compression processing on the three-dimensional point cloud data and retaining key geometric features of the target scene to obtain three-dimensional point cloud data retaining key geometric features; performing semantic segmentation on the image data to obtain semantic segmentation image data, the semantic segmentation image data comprising an object category to which each pixel point belongs; aligning the three-dimensional point cloud data retaining key geometric features and the semantic segmentation image data to obtain an initial depth map, each pixel point in the initial depth map having a corresponding depth value; correcting the depth value of each pixel point in the initial depth map according to the object category to which the pixel point belongs to obtain an optimized depth map; and inputting the optimized depth map into a preset Gaussian scene model to perform floating point elimination to obtain three-dimensional Gaussian sputtering data.

2. The method of claim 1, wherein, The data compression processing on the three-dimensional point cloud data and the retaining of the key geometric features of the target scene to obtain the three-dimensional point cloud data retaining key geometric features comprise: performing denoising processing on the three-dimensional point cloud data to obtain denoised three-dimensional point cloud data; performing down-sampling processing on the denoised three-dimensional point cloud data and retaining the key geometric features of the target scene in the down-sampling processing to obtain the three-dimensional point cloud data retaining key geometric features.

3. The method of claim 1, wherein, The semantic segmentation on the image data to obtain the semantic segmentation image data comprises: performing distortion correction processing on the image data to obtain corrected image data; performing exposure processing on the corrected image data to obtain image data after exposure processing; performing semantic segmentation on the image data after exposure processing to obtain an object category to which each pixel point in the image data belongs.

4. The method of claim 1, wherein, The aligning of the three-dimensional point cloud data retaining key geometric features and the semantic segmentation image data to obtain the initial depth map comprises: spatiotemporally aligning the three-dimensional point cloud data retaining key geometric features and the semantic segmentation image data to obtain spatiotemporally aligned three-dimensional point cloud data and spatiotemporally aligned image data; obtaining the initial depth map according to the spatiotemporally aligned three-dimensional point cloud data and the spatiotemporally aligned image data.

5. The method of claim 4, wherein, The obtaining of the initial depth map according to the spatiotemporally aligned three-dimensional point cloud data and the spatiotemporally aligned image data comprises: projecting the spatiotemporally aligned three-dimensional point cloud data into an image coordinate system of the spatiotemporally aligned image data to obtain a plurality of projection pixel points having depth values, the projection pixel points one-to-one corresponding to three-dimensional points in the spatiotemporally aligned three-dimensional point cloud data; based on the plurality of projection pixel points having depth values and the object category to which each pixel point in the semantic segmentation image data belongs, performing depth completion processing on a region in the image coordinate system of the spatiotemporally aligned image data that is not covered by projection to obtain the initial depth map.

6. The method of claim 1, wherein, The correction of the depth value of each pixel point in the initial depth map according to the object category to which the pixel point belongs to obtain the optimized depth map comprises: According to a preset geometric prior model corresponding to the object category to which each pixel point belongs, the depth value of each pixel point in the initial depth map is corrected to obtain a geometric consistency depth map; According to the object category to which each pixel point belongs, first pixel points belonging to a foreground occlusion object category and second pixel points belonging to a background occluded object category are screened out; In the geometric consistency depth map, first pixel points whose depth values are not within a first preset range and second pixel points whose depth values are not within a second preset range are removed to obtain an optimized depth map.

7. The method of claim 1, wherein, The preset Gaussian scene model includes a plurality of Gaussian spheres, and the inputting of the optimized depth map into the preset Gaussian scene model for floating point removal to obtain three-dimensional Gaussian sputtering data includes: The inputting of the optimized depth map into the preset Gaussian scene model to obtain the real depth value of each Gaussian sphere in the optimized depth map; According to the depth value of each Gaussian sphere in the preset Gaussian scene model and the real depth value in the optimized depth map, an absolute deviation is calculated. According to the absolute deviation, floating points are removed to obtain three-dimensional Gaussian sputtering data.

8. A Gaussian scene suspension point culling system, comprising: The system comprises: An acquisition module configured to acquire three-dimensional point cloud data and image data of a target scene; A processing module configured to perform data compression processing on the three-dimensional point cloud data to obtain three-dimensional point cloud data retaining key geometric features of the target scene, and perform semantic segmentation on the image data to obtain semantic segmentation image data, the semantic segmentation image data including an object category to which each pixel point belongs; An optimization module configured to perform alignment processing on the three-dimensional point cloud data retaining key geometric features of the target scene and the semantic segmentation image data to obtain an initial depth map, each pixel point in the initial depth map having a corresponding depth value, and correct the depth value of each pixel point in the initial depth map according to the object category to which each pixel point belongs to obtain an optimized depth map; A removal module configured to input the optimized depth map into a preset Gaussian scene model for floating point removal to obtain three-dimensional Gaussian sputtering data.

9. An electronic device, comprising: A processor and a memory, the memory storing a computer program executable by the processor, the processor being capable of executing the computer program to implement the Gaussian scene floating point removal method of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the Gaussian scene floating point removal method of any one of claims 1-7. The computer program is executed by the processor to implement the Gaussian scene floating point removal method of any one of claims 1-7.

Images