A point projection type three-dimensional reconstruction and segmentation method and system based on semi-gaussian pruning

Abstract

The present application relates to the technical field of computer vision, especially to a point projection type three-dimensional reconstruction and segmentation method and system based on semi-Gaussian pruning. The method comprises the following steps: obtaining SFM point cloud and cross-view label consistency label mask based on the obtained multi-view images respectively; initializing the obtained SFM point cloud as identity semi-Gaussian point cloud and rendering optimization by using a differentiable renderer; densifying the initial sparse point cloud by using a localized semi-Gaussian point management method, and resetting and repairing the local error area; using the obtained consistency label mask to supervise Gaussian identity feature learning by using cross-entropy loss, and using unsupervised 3D regularization loss to force the identity consistency between spatially adjacent Gaussians; the method of the present application adopts the method of identity coding semi-Gaussian kernel, eliminates the inherent representation ambiguity of single opacity formula, and has a positive influence on the identity coding accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly to a point projection type three-dimensional reconstruction and segmentation method and system based on semi-Gaussian pruning. BACKGROUND

[0002] In the intersection field of artificial intelligence and computer vision, 3D scene understanding is a cornerstone technology to drive robot autonomous navigation, AR / VR immersive interaction and automatic driving safety decision. Its core goal is not only to accurately reconstruct the geometric structure of the environment, but also to decompose the complex scene into logical components with semantic meaning. This structured analysis capability plays a decisive role in practical applications: robot systems rely on scene segmentation to identify operating objects and dangerous areas, enabling precise grasping and real-time obstacle avoidance; AR / VR applications need to build a virtual-real integrated interactive environment through semantic segmentation, anchor virtual content to the surface of real objects and support natural operation; and automatic driving vehicles require millisecond-level segmentation of roads, traffic participants and roadside facilities to provide a computable environment model for the decision system. As the core engine to achieve the above goals, 3D scene segmentation assigns semantic / instance labels to point clouds, meshes or voxels, converting raw geometric data into programmable semantic units. Its output directly enables key downstream tasks, such as supporting part assembly simulation for industrial robots or implementing automatic editing of building components in digital twin systems. With the development of multi-modal perception and end-to-end deep learning, this field is shifting from pure geometry-driven to intelligent understanding paradigm that integrates physical attributes and functional semantics.

[0003] Traditional 3D segmentation methods have long relied on dense point cloud data, which can explicitly represent spatial geometry but faces severe practical challenges. The primary limitation is the annotation cost: point clouds need to be manually labeled with semantic classes or instance assignments, which can take hundreds of hours in complex occlusion scenarios. The second is the complexity of the processing pipeline: typical segmentation pipelines require multi-stage operations, making real-time applications difficult to achieve. More critically, the robustness problem: sensor noise, object occlusion, and reconstruction artifacts can easily lead to broken segmentation boundaries, class confusion, and semantic inconsistency, significantly reducing system reliability. Emerging machine learning models such as Neural Radiance Fields (NeRF) and 3D Gaussian Splats (3DGS) offer new possibilities by providing high-fidelity rendering and compact, efficient scene representation. However, NeRF has limitations including slow training and inference speed, and its implicit nature makes object-level segmentation within the scene particularly difficult; 3DGS's simple average gradient thresholding adaptive density control mechanism is too crude to capture fine-grained local details. Periodic opacity resetting during training can also introduce artifacts, compromising reconstruction quality and segmentation accuracy. To address these issues, there is an urgent need to develop an identity-encoding semi-Gaussian splatting 3D scene reconstruction and segmentation method and system that uses a set of multi-view images, an improved identity semi-Gaussian kernel (ID-HGS), localized semi-Gaussian point management (LHPM), and global importance score pruning mechanism to achieve high-quality reconstruction and accurate segmentation of three-dimensional scenes. SUMMARY

[0004] To solve the above-mentioned problems, the present application provides a point projection type three-dimensional reconstruction and segmentation method and system based on semi-Gaussian pruning.

[0005] In a first aspect, the present application provides a point projection type three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning, which adopts the following technical solution:

[0006] A point projection type three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning, comprising:

[0007] Obtaining multi-view image data;

[0008] Based on the obtained multi-view image, SFM point cloud and cross-view label consistency label mask are obtained respectively;

[0009] The obtained SFM point cloud is initialized as an identity semi-Gaussian point cloud and rendered and optimized using a differentiable renderer;

[0010] The initial sparse point cloud is densified using a localized semi-Gaussian point management method, and local error areas are identified for resetting and repair;

[0011] The consistency label mask obtained is used to supervise Gaussian identity feature learning using cross-entropy loss, and a 3D regularization loss is used to force the spatially adjacent Gaussians to maintain identity consistency;

[0012] The Gaussians are pruned using a global importance score pruning mechanism;

[0013] The scene is edited at the object level according to the identity information of the Gaussians.

[0014] Further, the SFM point cloud and the consistency label mask of the cross-view label are obtained based on the obtained multi-view images, including adopting motion estimation COLMAP to realize joint reconstruction of sparse point cloud and camera pose, first, using the feature_extractor module of COLMAP to detect and extract local features of each image based on the RootSIFT algorithm, and storing them in the SQLite database; then, cross-view feature matching is performed through feature_matcher, supplemented by multi-model geometric verification and watermark filtering, to establish the corresponding relationship between the same name points between images, solve the matching ambiguity problem caused by the difference in perspective and texture repetition; then, incremental sparse reconstruction is performed: based on the matching result, the camera internal and external parameters are iteratively optimized through bundle adjustment, and the objective function is At the same time, the three-dimensional coordinates of the same name feature points are calculated using a triangulation algorithm to generate an initial sparse point cloud, and finally the cumulative error is reduced through redundant view compression and global bundle adjustment optimization, and the PLY format point cloud containing point coordinates and RGB color, and the accurate pose parameters of each image are output.

[0015] Further, the SFM point cloud and the consistency label mask of the cross-view label are obtained based on the obtained multi-view images, further including using a pre-trained ViT-H architecture to independently process each input image, first, the resolution of the input image is scaled and filled to 1024x1024 pixels; then, 32x32 uniform grid points are generated on the image as prompt input; 3 candidate masks and confidence scores are predicted for each grid point, then two-stage filtering is performed: first, masks with a confidence score >0.88 are retained, and then mask stability is checked; after filtering, redundant is removed through cross-scale non-maximum suppression; finally, post-processing optimization is performed, and the final anything mask map is output.

[0016] Further, the consistency of the SFM point cloud and the cross-view label mask obtained based on the acquired multi-view images is further included introducing a video tracking model DEVA, realizing cross-frame consistency through a bidirectional propagation mechanism, specifically including two stages of intra-fragment consensus optimization and propagation-consensus fusion, wherein, in the intra-fragment consensus stage, the model aligns the segmentation mask of the future n frames to the current frame, selects the mask set with an IoU support degree exceeding 0.5 through integer programming, and filters out isolated noise masks; in the propagation-consensus fusion stage, the model outputs the historical mask and the current consensus perform bipartite graph matching, establish an association mapping based on a mask IoU threshold, pass apparent features through a ConvGRU hidden state, combine motion consistency constraints, maintain object ID stability in an occlusion scene, and finally output a cross-view Figure one mask sequence.

[0017] Further, the SFM point cloud obtained is initialized as an identity half-Gaussian point cloud and rendered and optimized using a differentiable renderer, including introducing a normal vector to divide the Gaussian distribution function into two halves, and assigning independent opacity values and to each half, significantly improving the use of the normal vector component previously unused in 3DGS by only adding an additional opacity parameter, represented as:

[0018] ,

[0019] wherein is the normal vector of the segmentation plane; the complementary half-Gaussian distribution is obtained by negating the normal vector; the paired half-Gaussian representation is further utilized, wherein the parameters are shared by two parts, and the direction of the segmentation plane and the two opacity parameters are simultaneously learned, the volume alpha blending at each pixel, and the identity-coded half-Gaussian are represented as:

[0020] ,

[0021] wherein represents an ordered ID-HGS set, defined as the integral of the two parts of the ID-HGS pair, represented as:

[0022] ,

[0023] wherein, the ID-HGS kernel divides the Gaussian into two regions with different opacities, wherein the ID-HGS learns the surface normal to make the segmentation plane fit the object geometry, and the double opacity values inside and outside effectively minimize color mixing while maintaining sharp boundaries.

[0024] Furthermore, the learning of Gaussian identity features using the obtained consistent label mask with cross-entropy loss includes calculating the error by comparing the 3DGS rendering result with the real image, dividing the image into several non-overlapping blocks, aggregating the error values ​​within each block, and applying a threshold. Determine whether each block contains a problem area, and generate a 2D error distribution map to highlight key optimization areas, represented as follows:

[0025] The layout is designed to highlight key optimization areas, and is represented as follows:

[0026] ,

[0027] in Represents 2D block coordinates. For pixel coordinates, and To render images and real images, An adaptive threshold is used, and then a feature matching algorithm is applied to extract feature points from the current view A and the adjacent reference view B to obtain a normalized feature set. Correspondences between features are established, and finally, local point operations are performed, using volume segmentation and cloning of high-gradient Gaussians to enhance detail representation. A lower densification threshold is applied in the error region to introduce more Gaussians. For areas located in... Any Gaussian point i within:

[0028] ,

[0029] Where c and s represent the number of Gaussians whose gradients are below the compaction gradient threshold after the cloning and segmentation operations, respectively. This represents the value corresponding to the lowest c+s entries when all Gaussian opacity values ​​are sorted in ascending order.

[0030] Furthermore, the use of unsupervised 3D regularization loss to force spatially neighboring Gaussians to maintain identity consistency includes rendering a 2D image using a differentiable 3D Gaussian splash 3DGS renderer. The influence equation for identity encoding is expressed as: The final 2D mask identity features of each pixel Calculated as the identity code assigned to Gaussian The weighted sum, with weights determined by the influence factor of each Gaussian on that pixel. Provided; using rendered 2D features As input, by adding a linear layer f, the feature dimension is restored to K+1, and then... Softmax is applied for identity classification; Gaussian grouping introduces a 3D regularization loss. , forcing neighboring Gaussians to be close in feature space, ensuring sufficient supervision on occluded or invisible Gaussians during training, denoted as:

[0031] ,

[0032] where P contains unit encodings sampled from a three-dimensional unit Gaussian distribution , the set represents its nearest neighbors in three-dimensional space, the complete loss function is a weighted combination of 2D loss , 3D loss and 3D GS loss : .

[0033] Further, the pruning mechanism using global importance score optimization includes rendering 3D Gaussians into images from arbitrary camera viewpoints by a differentiable rasterizer, the initial importance score of each half-Gaussian is quantified according to its contribution on every pixel ray of all training views, the hit count of each Gaussian is calculated by traversing all training pixels; the score is further refined by adjusted 3D Gaussian volume , opacity and , the volume calculation is , where abc is the 3 dimensions of scaling matrix S, the overall global importance score is summarized as:

[0034] ,

[0035] where j represents Gaussian index, M, H and W represent the number of training views and the height and width of images respectively, then pruning operation is performed at appropriate stages, firstly, the global importance score of all Gaussians is calculated, then the Gaussians of a predetermined proportion are pruned according to ascending score order: , where is the original Gaussian point cloud, represents the importance score of Gaussians , is the pruning proportion, is the quantile of score set , is the pruned Gaussian point cloud.

[0036] ​​​Further, the object-level editing of the scene according to the identity information of Gauss includes realizing accurate segmentation of a target object according to a text prompt word input by a user. After the user prompt word is input into an open vocabulary detection model GroundingDINO, an image encoder of the model extracts multi-scale visual features of a multi-view image, and meanwhile, a text encoder analyzes semantics to generate a feature vector. Through a text-to-image cross attention mechanism, the model focuses on a visual area related to the semantic of the prompt word and outputs a pixel-level mask accurately covering a target contour. After the 2D mask is obtained, an identity code corresponding to a mask with the highest coincidence degree with the 2D mask is used to search for Gauss in the scene. If the cosine similarity of the identity codes meets a threshold, the Gauss is determined as a target Gauss. Finally, a segmented Gauss point set carrying semantic labels is output for downstream editing, and thus a complete closed loop from natural language to 3D structure is completed.

[0037] In a second aspect, a point projection type three-dimensional reconstruction and segmentation system based on semi-Gauss pruning includes:

[0038] A data acquisition module is configured to acquire multi-view image data.

[0039] A mask module is configured to obtain an SFM point cloud and a consistency label mask of a cross-view label based on the acquired multi-view image respectively.

[0040] An optimization module is configured to initialize the obtained SFM point cloud as an identity semi-Gauss point cloud and perform rendering optimization using a differentiable renderer.

[0041] A repair module is configured to densify the initial sparse point cloud using a localized semi-Gauss point management method and identify a local error area for resetting and repairing.

[0042] A supervision module is configured to supervise Gauss identity feature learning using a cross-entropy loss using the obtained consistency label mask and use an unsupervised 3D regularization loss to force Gauss with spatial proximity to maintain identity consistency.

[0043] A pruning module is configured to perform Gauss pruning using a global importance score pruning mechanism.

[0044] An editing module is configured to perform object-level editing of a scene according to the identity information of Gauss.

[0045] In a third aspect, the present application provides a computer readable storage medium having a plurality of instructions stored therein, the instructions being adapted to be loaded by a processor of a terminal device and executed to implement the point projection type three-dimensional reconstruction and segmentation method based on semi-Gauss pruning.

[0046] In a fourth aspect, the present application provides a terminal device, comprising a processor and a computer readable storage medium, the processor is used to implement instructions; the computer readable storage medium is used to store a plurality of instructions, the instructions are suitable for being loaded and executed by the processor to implement the point projection type three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning.

[0047] In summary, the present application has the following beneficial technical effects:

[0048] The method of the present application adopts the method of identity coding semi-Gaussian kernel, divides the identity coded Gaussian into two regions with different opacity, learns a normal to make the segmentation plane fit the object geometry, effectively minimizes color mixing with double opacity value, while maintaining sharp boundaries, thereby eliminating the inherent representation ambiguity of single opacity formula, and positively affecting the identity coding accuracy. In addition, in the area with rich texture, the local contrast is enhanced, and the details lost due to single Gaussian oversimplification are prevented. The use of ID-HGS kernel ensures accurate identity coding acquisition while maintaining the fidelity of Gaussian model.

[0049] The LHPM module used in the present application can now be seamlessly integrated into the complete HGS-3DSeg process, reducing artifacts and holes that may hinder identity coding learning by optimizing Gaussian geometric distribution, while enhancing detail reconstruction. This improvement improves scene understanding and segmentation by refining the spatial arrangement of Gaussian clusters and suppressing artifacts.

[0050] The application of this pruning operation in the training process can prevent GPU out-of-memory (OOM) errors, reduce model redundancy, slightly improve training speed, and reduce storage requirements. The timing of pruning is flexible; however, removing a large number of Gaussians will reduce model accuracy and visual quality. These losses can be repaired in subsequent iterative training. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is the main flowchart of the method of the present application.

[0052] Figure 2 is the initial SFM point cloud graph and the final rendered scene graph of the present application, wherein, Figure 2 (a) in (b) of Figure 2 (c) in (b) of Figure 2 is the initial SFM point cloud graph generated by the present application, Figure 2 (d) in (e) of Figure 2 (f) in (e) of Figure 2 is the final rendered scene graph of the present application.

[0053] Figure 3Figure 1 is a comparison chart of anything mask and Gaussian grouping method of the present application, wherein Figure 3 column (a) of Figure 1 is a real scene picture, Figure 3 column (b) of Figure 1 is an anything mask picture generated using a SAM model; Figure 3 column (c) of Figure 1 is an anything mask picture obtained by Gaussian grouping; Figure 3 column (d) of Figure 1 is an anything mask picture generated by the present application.

[0054] Figure 4 Figure 2 is a comparison chart of semantic recognition accuracy of the present application and Gaussian grouping, wherein Figure 4 column (a) of Figure 2 is the semantic recognition result of the reconstructed scene of Gaussian grouping using GINO model under the prompt word "Stuffed bear", Figure 4 column (b) of Figure 2 is the semantic recognition result of the reconstructed scene of the present application using GINO model under the prompt word "Stuffed bear", Figure 4 column (c) of Figure 2 is the semantic mask of the reconstructed scene of Gaussian grouping using GINO model under the prompt word "Stuffed bear", Figure 4 column (d) of Figure 2 is the semantic mask of the reconstructed scene of the present application using GINO model under the prompt word "Stuffed bear".

[0055] Figure 5 Figure 3 is a comparison chart of reconstruction details of the present application and Gaussian grouping, wherein Figure 5 column (a) of Figure 3 is a real scene picture, and the local details in the green frame are enlarged, Figure 5 column (b) of Figure 3 is a picture rendered by Gaussian grouping, and the local details in the green frame are enlarged, Figure 5 column (c) of Figure 3 is a picture rendered by the present application, and the local details in the green frame are enlarged. DETAILED DESCRIPTION

[0056] The present application will be further described in detail below with reference to the accompanying drawings.

[0057] Example 1

[0058] With reference to Figure 1 , the point projection type three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning of the present embodiment comprises: adopting a modular architecture design, containing five core functional modules: an initial point generation module, a consistency mask module, an identity coding semi-Gaussian model, a localized semi-Gaussian point management module, and a global importance score pruning module. The modules interact and cooperate with each other through standardized interfaces, and a complete fault diagnosis closed loop is formed through seven consecutive or parallel processing steps:

[0059] Step S1, the system first performs SFM motion estimation on the multi-view images of the scene to obtain an SFM point cloud;

[0060] Step S2, in parallel with S1, the multi-view images are predicted using a 2D segmentation model to obtain 2D labels, and a video tracking model is used to match the cross-view labels to obtain a consistent label mask;

[0061] Step S3, the SFM point cloud obtained in step S1 is initialized as an identity Gaussian point cloud, and a differentiable renderer is used for rendering optimization;

[0062] Step S4, the initial sparse point cloud is densified using a localized Gaussian point management method, and local error areas are identified for resetting and repairing;

[0063] Step S5, in parallel with step S4, the consistent label mask obtained in step S2 is used to supervise the learning of Gaussian identity features using a cross-entropy loss and an unsupervised 3D regularization loss to force the identity consistency between Gaussians that are spatially adjacent;

[0064] Step S6, a global importance score pruning mechanism is used to optimize memory usage;

[0065] Step S7, downstream editing tasks, based on the identity information of the Gaussians, the scene is edited at the object level.

[0066] Specifically, it includes

[0067] Step S1, SFM point cloud generation:

[0068] For the multi-view image sequence provided by the user, this stage uses the motion estimation software COLMAP to realize the joint reconstruction of sparse point cloud and camera pose. First, feature extraction and matching are performed: the feature_extractor module of COLMAP detects and extracts local features of each image based on RootSIFT or SuperPoint algorithm, and stores them in SQLite database; then cross-view feature matching is performed by feature_matcher, assisted by multi-model geometric verification (RANSAC estimation of fundamental matrix F, homography H and essential matrix E, combined with threshold and distinguish general / planar / panoramic scenes) and watermark filtering (similar transformation detection, threshold ), to establish the corresponding relationship between the same points between images, and solve the matching ambiguity problem caused by perspective difference and texture repetition.

[0069] Then incremental sparse reconstruction is performed: COLMAP's mapper module iteratively optimizes camera intrinsics (focal length, distortion coefficients) and extrinsics (rotation matrix R, translation vector T, objective function ) based on matching results by Bundle Adjustment, while computing 3D coordinates of corresponding feature points using triangulation algorithm (minimum set of two views, enforcing angle constraints and depth positivity ) to generate initial sparse point cloud. This point cloud, though sparse, has fully represented the underlying geometry and spatial topology of the scene through optimal view selection strategy (multi-resolution pyramid scoring , weight balancing point quantity and distribution) and iterative re-triangulation (Post-BA RT).

[0070] Finally, accumulated errors are reduced by redundant view compression (camera grouping common visibility ) and global bundle adjustment optimization, outputting PLY format point cloud containing point coordinates and RGB colors, as well as accurate pose parameters of each image. This process can efficiently handle hundreds of images under CPU / GPU hybrid acceleration (PCG iteration method for large-scale problems), but the reconstruction robustness of low-texture areas still faces challenges and relies on scene graph enhancement (non-panoramic image pair initialization) for relief.

[0071] As shown in Figure 2 , (a) in Figure 2 , (b) in Figure 2 and (c) in Figure 2 are the initial SFM point cloud graphs generated by the embodiment, (d) in Figure 2 , (e) in Figure 2 and (f) in Figure 2 are the final rendered scene graphs of the embodiment.

[0072] Step S2, cross-view consistency mask generation: Figure one

[0073] ​The goal of this stage is to generate semantically consistent object masks for multi-view images. First, single-view mask initialization is performed: the ViT-H architecture of the pre-trained Segment Anything Model (SAM) is used to prompt independent processing of each input image by the irrelevant anything model. SAM first resizes and fills the input image to a resolution of 1024x1024 pixels; then generates 32x32 uniform grid points (a total of 1024 foreground points) on the image as prompt input; then, 3 candidate masks and confidence scores are predicted for each grid point (to solve single-point ambiguity), with about 3072 original outputs; then, two-stage filtering is performed: first, masks with a confidence score greater than 0.88 are retained, and then mask stability is checked (binary mask IoU threshold of 0.45 and 0.55, IoU≥95%); after filtering, redundant masks are removed by cross-scale non-maximum suppression (NMS, IoU threshold of 0.7), with high-confidence masks being prioritized; finally, post-processing optimization is performed: isolated fragments with an area of less than 100 pixels are removed, and internal holes of the same size are filled, and the final anything mask image is output. The output of all views is a set of segmentation masks.

[0074] Cross-view mask association is then performed: to solve the problem of inconsistent multi-view mask IDs, the video tracking model DEVA is introduced. This model achieves cross-frame consistency through a bidirectional propagation mechanism (arranging multi-view images in order as video frames in the model), which includes two core technical stages: intra-fragment consensus optimization and propagation-consensus fusion. In the intra-fragment consensus stage, the model aligns the segmentation masks of the next n frames (default n=3) to the current frame, and filters out isolated noise masks (such as low-support masks that are removed after spatial alignment) by selecting masks with an IoU support of more than 0.5 through integer programming. The current consensus is matched with the past consensus through bipartite graph matching based on a mask IoU threshold (>0.5): matching successful mask pairs are fused into new instances, and unmatched masks are injected into the propagation memory as new objects, and unmatched Figure one masks are managed dynamically through a counter mechanism (deleted after L=5 consecutive frames of non-matching). This process transfers appearance features through ConvGRU hidden state transmission, and maintains object ID stability in occlusion scenarios by combining motion consistency constraints, finally outputting a cross-view mask sequence.

[0075] Step S3, identity half-Gaussian kernel initialization and optimization

[0076] First, a normal vector is introduced to divide the Gaussian distribution function into two halves, and each half is assigned an independent opacity value and This configuration maintains backward compatibility (when = is equivalent to the standard Gaussian blob), while introducing only minimal computational overhead. The modification adds only one extra opacity parameter, exploiting the previously unused normal vector component in 3DGS. This design achieves significant improvements with negligible parameter / cost increment, as described below.

[0077] ,

[0078] where is the normal vector of the split plane; the complementary half-Gaussian distribution is obtained by negating this normal vector.

[0079] The rasterization process of ID-HGS is similar to 3DGS, exploiting paired half-Gaussian representations, where parameters (mean, rotation, scale, color, and identity code) are shared by two parts, while learning the direction of the split plane and two opacity parameters. Volume alpha blending at each pixel and identity-coded half-Gaussian can be expressed as:

[0080] ,

[0081] where denotes the ordered set of ID-HGS, is defined as the integral of two parts of ID-HGS pair.

[0082] ,

[0083] ID-HGS kernels split the Gaussian into two regions with different opacities, where ID-HGS learns the surface normal to make the split plane fit the object geometry, and the dual opacity values (o inside, outside) effectively minimize color blending while maintaining sharp boundaries, thus eliminating the inherent representation ambiguity of single-opacity formulas and positively impacting identity code accuracy. Moreover, in texture-rich regions, the difference of enhances local contrast, preventing detail loss caused by single-Gaussian oversimplification. Employing ID-HGS kernels ensures accurate identity code acquisition while maintaining Gaussian model fidelity.

[0084] Step S4, Localized Half-Gaussian Point Management

[0085] Localized Half-Gaussian Point Management (LHPM) method matches local features between the current view and the reference view, and combines the rendering-derived error map to identify error regions, and pinpoint the 3D regions that need optimization by back-projecting these regions to 3D space. Within these regions, it employs more aggressive strategies to optimize the poorly reconstructed points to fit scene details, and reset the ill-conditioned points that occlude these regions to reduce the impact of occlusions on training, and adaptively prune low-opacity Gaussians to alleviate GPU memory constraints.

[0086] The specific method is as follows: First, we compare the 3D GS rendering result with the real image to calculate the error. Then, we divide the image into several non-overlapping blocks of size and aggregate the error values within each block. Then, based on the threshold , we determine whether each block contains a problem region, and generate a 2D error distribution map to visually highlight the areas that need to be optimized.

[0087] ,

[0088] where represents the 2D block coordinates ( ), represents the pixel coordinates, and represent the rendered image and the real image, respectively, is an adaptive threshold affected by the block size.

[0089] Then, apply a feature matching algorithm (such as LightGlue) to extract feature points from the current view A and the adjacent reference view B, resulting in a normalized feature set (the coordinates are normalized by image size). Then establish a correspondence between the two sets of features, resulting in a set of matches . These matches, combined with the error map, are used to identify error regions between pairs of views. Next, project a conical light from the camera center of the current view and the reference view to the respective error regions, covering all related pixels.

[0090] Finally, local point operations include: based on volume segmentation and cloning high-gradient Gaussians to enhance the representation of details; applying a lower densification threshold in the error region to introduce more Gaussians; resetting the opacity of high-opacity points near the camera (identified as ill-conditioned occluders) to correct the incorrect occlusion effect; and pruning redundant Gaussian points in ascending order of opacity to prevent the model from growing too much. For any Gaussian point i located within :

[0091] ,

[0092] where c and s represent the number of Gaussians whose gradients are below the densification gradient threshold after the cloning and splitting operations, respectively, represents the value corresponding to the lowest c + s entries when all Gaussian opacity values are sorted in ascending order.

[0093] Unlike the Local Point Management (LPM) plugin originally designed for the standard 3DGS kernel, which differs in both the number and dimension of parameters, we made LPM compatible with our method by integrating the identity encoding and adding an extra opacity channel. We also removed the shielding of the nx / ny / nz parameters and redefined the threshold trigger to use the maximum value in each opacity pair. In the Gaussian pruning stage, we adopted a more efficient strategy to replace the traditional global importance score method, avoiding the repeated calculation of scores.

[0094] Step S5, Identity Learning Supervision:

[0095] Identity Encoding is designed as a 16-dimensional learnable compact vector. Similar to the way spherical harmonic (SH) coefficients represent colors, this encoding can be rendered into a 2D image by a differentiable 3D Gaussian Spray (3DGS) renderer. This identity encoding is optimized using the same method as color optimization to represent the scene instance ID. Similar to the color optimization in 3DGS, The influence equation of the identity encoding can be derived as follows:

[0096] ,

[0097] The final 2D mask identity feature of each pixel is computed as the weighted sum of the identity encodings assigned to the Gaussians (each of length 16), with the weights given by the influence factor of each Gaussian on that pixel.

[0098] Specifically, the Gaussian grouping learns a discrete and grouped 3D representation using multi-view images and 2D masks generated by SAM

[16] , enabling the reconstruction and segmentation of any object in a 3D scene. To further improve grouping accuracy, in addition to using the standard cross-entropy loss for identity classification.

[0099] Since the mask recognition labels are two-dimensional, they cannot directly supervise the three-dimensional Gaussian identity encoding . Therefore, the rendered 2D features in equation (1) are used as input. By adding a linear layer f, the feature dimension is restored to K + 1 (K is the total number of masks), and then the Identity classification is performed using softmax.

[0100] Gaussian grouping introduces a 3D regularization loss that enforces neighboring Gaussians to be close in feature space, ensuring sufficient supervision on occluded or invisible Gaussians during training.

[0101] ,

[0102] where P contains unit encodings sampled from a three-dimensional unit Gaussian distribution , the set represents its nearest neighbors in three-dimensional space . The complete loss function is a weighted combination of the 2D loss , the 3D loss and the 3D GS loss :

[0103] .

[0104] Step S6, global importance score pruning:

[0105] Global importance computation: Relying solely on Gaussian opacity as the importance criterion leads to suboptimal Gaussian pruning, so a more effective formula is needed. Inspired by equation (3), these 3D Gaussians can be rendered into images from arbitrary camera viewpoints by a differentiable rasterizer. Then, the initial importance score of each half-Gaussian can be quantified according to its contribution over all the pixel rays ( ) of all training views. Thus, we iterate over all training pixels to count the hit times of each Gaussian.

[0106] The score is further refined by the adjusted 3D Gaussian volume as well as the opacity and . The volume calculation is , where abc are the 3 dimensions of the scaling matrix S in equation (1). Overall, the global importance score can be summarized as:

[0107] ,

[0108] where j denotes the Gaussian index, and M, H and W represent the number of training views and the height and width of the image, respectively. The indicator function B() judges whether a given ray whether intersect with a specific Gaussian. However, directly using the Gaussian volume often overestimates the contribution of background Gaussians, leading to over-pruning of Gaussians representing complex geometries. We replace the volume term in the above equation with the following expression :

[0109] ,

[0110] where the volume is first normalized by the 90th percentile value of all ascendingly sorted Gaussian distributions to limit the range within (0, 1) and introduce a parameter for adjustment.

[0111] In the open scene training containing a large number of Gaussian points, we perform the pruning operation at the appropriate stage. First, calculate the global importance score of all Gaussians, and then prune a predetermined proportion of Gaussians according to the ascending score order (from low to high):

[0112] ,

[0113] where is the original Gaussian point cloud, represents the importance score of the Gaussian, is the pruning proportion, is the quantile of the score set , is the pruned Gaussian point cloud.

[0114] Step S7, downstream editing operation:

[0115] As a key link in the 3D Gaussian scene editing process, this stage realizes the accurate segmentation of target objects according to user input text prompts, laying the foundation for subsequent object removal, attribute modification, etc. Its technical process begins with semantic-driven cross-modal mask generation: after the user prompt word is input into the open vocabulary detection model Grounding DINO, the image encoder of the model extracts multi-scale visual features of the multi-view image, while the text encoder analyzes the semantics to generate a feature vector. Through the text→image cross-attention mechanism, the model focuses on the visual area related to the semantic of the prompt word and outputs a pixel-level mask that accurately covers the target contour.

[0116] After obtaining the 2D mask, the system performs cross-dimensional matching: the identity code corresponding to the mask with the highest overlap degree with the 2D mask is retrieved for the Gaussians in the scene, and if the cosine similarity of its identity code meets the threshold, it is determined as the target Gaussian. The final output of the segmented Gaussian point set will carry semantic labels waiting for downstream editing, thus completing the complete closed loop from natural language to 3D structure.

[0117] To further verify the feasibility and effectiveness of the method of the present application, we carried out experiments in multiple dimensions. To evaluate the segmentation and fine-grained localization performance in open-world scenarios, we used the LERF-Mask dataset introduced by Gaussian grouping

[18] , which provides accurate mask annotations. We further used the three scenes in LERF-Localization, which only provide coarse bounding boxes. To evaluate the reconstruction quality, we benchmarked Gaussian grouping on seven out of the nine complete scenes of Mip-NeRF 360, with flower and tree scenes excluded due to non-public access.

[0118] The present application performs semantic segmentation experiments using mIoU and mBIoU two indicators, as shown in Table 1:

[0119] Table 1 Quantitative comparison on LERF-Localization dataset

[0120]

[0121] The present application also performs reconstruction accuracy tests on the mipnerf360 dataset, as shown in Table 2:

[0122] Table 2 Quantitative comparison on MIP-NERF360 dataset (PSNR↑) - indicates reconstruction failure

[0123] Method stump garden counter Bicycle Kitchen room bonsai Gaussian grouping — 27.0 28.5 — 30.7 28.0 31.3 We 26.8 27.4 29.3 26.4 31.7 29.7 32.1

[0124] Embodiment 2

[0125] The present embodiment provides a point projection type three-dimensional reconstruction and segmentation system based on semi-Gaussian pruning, comprising:

[0126] The data acquisition module is configured to:

[0127] A computer-readable storage medium, wherein a plurality of instructions are stored, the instructions are adapted to be loaded and executed by the processor of the terminal device, and the instructions are adapted to be loaded and executed by the processor of the terminal device.

[0128] A terminal device, comprising a processor and a computer-readable storage medium, the processor is used to implement the instructions; the computer-readable storage medium is used to store a plurality of instructions, the instructions are adapted to be loaded and executed by the processor, and the instructions are adapted to be loaded and executed by the processor.

[0129] The above are preferred embodiments of the present application, which do not limit the protection scope of the present application, therefore: any equivalent changes made on the structure, shape, principle of the present application shall be covered within the protection scope of the present application.

Claims

1. A point projection based three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning, characterized in that, The method comprises the following steps: acquiring multi-view image data; obtaining SFM point cloud and consistency label mask of cross-view label based on the acquired multi-view image respectively; initializing the obtained SFM point cloud as identity Gaussian point cloud and rendering optimization by using a differentiable renderer; densifying the initial sparse point cloud by using a localized Gaussian point management method, and identifying a local error area for resetting and repairing; supervising Gaussian identity feature learning by using cross-entropy loss by using the obtained consistency label mask, and forcing spatially adjacent Gaussians to maintain identity consistency by using unsupervised 3D regularization loss; pruning Gaussians by using a global importance score pruning mechanism; editing the scene at the object level according to the identity information of the Gaussians; The consistency of the acquired multi-view images respectively obtains SFM point cloud and cross-view label, including adopting motion estimation COLMAP to realize joint reconstruction of sparse point cloud and camera pose, first using the feature_extractor module of COLMAP to detect and extract local features of each image based on RootSIFT algorithm, and storing it in SQLite database; then through feature_matcher, cross-view feature matching is carried out, supplemented by multi-model geometric verification and watermark filtering, the corresponding relationship of the same name points between images is established, and the matching ambiguity problem caused by the difference of view angle and texture repetition is solved; then incremental sparse reconstruction is carried out: based on the matching result, the camera internal and external parameters are iteratively optimized by bundle adjustment, and the objective function is At the same time, the three-dimensional coordinates of the same name feature points are calculated by using the triangulation algorithm, the initial sparse point cloud is generated, and finally the cumulative error is reduced through redundant view compression and global bundle adjustment optimization, and the PLY format point cloud containing point coordinates and RGB color and the accurate pose parameters of each image are output. the method of obtaining SFM point cloud and consistency label mask of cross-view label based on the acquired multi-view image respectively further comprises the following steps: independent processing of each input image is performed by using a pre-trained ViT-H architecture, the resolution of the input image is first scaled and filled to 1024x1024 pixels, and then 32x32 uniform grid points are generated on the image as prompt input; 2. The point projection based three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning according to claim 1, characterized in that, The consistency label mask of the SFM point cloud and the cross-view label obtained based on the acquired multi-view image respectively further comprises introducing a video tracking model DEVA, realizing cross-frame consistency through a bidirectional propagation mechanism, specifically comprising two stages of intra-fragment consensus optimization and propagation-consensus fusion, wherein, in the intra-fragment consensus stage, the model aligns the segmentation masks of future n frames to the current frame, selects the mask set with an IoU support degree exceeding 0.5 through integer programming, and filters out isolated noise masks; in the propagation-consensus fusion stage, the model outputs the historical masks and the current consensus perform bipartite graph matching, establish an association mapping based on a mask IoU threshold, transfer the appearance features through a ConvGRU hidden state, maintain the object ID stability in a shielding scene by combining a motion consistency constraint, and finally output a cross-view consistent mask sequence.

3. The point projection based three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning according to claim 2, characterized in that, The resulting SFM point cloud is initialized as an identity half-Gaussian point cloud and rendered using a differentiable renderer with optimization, including introducing a normal vector to split the Gaussian distribution function into two halves and assigning independent opacity values to each half and With only one additional opacity parameter, a significant improvement is achieved using the normal vector component that was not previously used in 3DGS, denoted as: ， where is the normal vector of the segmentation plane; the complementary half-Gaussian is obtained by taking the negative of this normal vector; the paired half-Gaussian representation is exploited again, where the parameters are shared by two parts, while learning the direction of the segmentation plane and the two opacity parameters, the volume alpha blending at each pixel and the identity- encoded half-Gaussian representation are formulated as: ， wherein denotes the ordered set of ID-HGSs, defined as the integral of the two parts of the ID-HGS pair, denoted by ， where the ID-HGS kernel splits the Gaussian into two regions with different opacities, where the ID-HGS learns the surface normals to make the segmentation plane fit the object geometry, the dual opacity values internal and external effectively minimize color bleeding while preserving sharp boundaries.

4. The point projection based three-dimensional reconstruction and segmentation method according to claim 3, wherein, The consistent label mask obtained by the use is supervised Gaussian identity feature learning using cross-entropy loss, including dividing the image into several non-overlapping blocks by comparing the 3D GS rendering result with the real image to calculate the error, and aggregating the error values in each block, and determining whether each block contains a problem area based on a threshold generating a 2D error distribution map to highlight the key optimization area, represented as: ， wherein represents 2D block coordinates, is a pixel coordinate, and are the rendered image and the real image, is an adaptive threshold, then a feature matching algorithm is applied to extract feature points from the current view A and the neighboring reference view B, to obtain a normalized feature set, and to establish a correspondence between the features, and finally a local point operation is performed based on volume segmentation and cloned high gradient Gaussians to enhance the detailed representation; a lower densification threshold is applied in the error area to introduce more Gaussians; for any Gaussian point i located within ​ ， where c and s represent the number of Gaussians whose gradients are below the densification gradient threshold after the cloning and splitting operations, respectively, represents the value corresponding to the lowest c + s entries when all Gaussian opacity values are sorted in ascending order.

5. The point projection based three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning according to claim 4, characterized in that, three candidate masks and confidence scores are predicted for each grid point, then two-stage filtering is performed: first, masks with a confidence score greater than 0.88 are retained, and then mask stability is checked; after filtering, redundancy is removed by cross-scale non-maximum suppression; finally, post-processing optimization is performed, and the final anything mask map is output. ， Final 2D mask identity feature for each pixel is computed as the weighted sum of identity encodings assigned to Gaussians, with weights given by each Gaussian's influence factor at that pixel ; using the rendered 2D features as input, the feature dimension is restored to K+1 by adding a linear layer f, then identity classification is applied using softmax; Gaussian grouping introduces a 3D regularization loss , forcing neighboring Gaussian points to be close in feature space, ensuring sufficient supervision on occluded or invisible Gaussians during training, denoted as: ​​ ， where P comprises unit encodings sampled from a three-dimensional unit Gaussian distribution , the set represents its position in three-dimensional space nearest neighbors, the complete loss function is a weighted combination of the 2D loss , the 3D loss and the 3D GS loss .​ 6. The point projection based three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning according to claim 5, characterized in that, The using global importance score pruning mechanism optimization for Gaussian pruning includes rendering 3D Gauss into images from arbitrary camera viewpoints by a differentiable rasterizer, the initial importance score of each half-Gauss is quantified according to its contribution on every pixel ray of all training views The hit count of each Gauss is calculated by traversing all training pixels; the score is further refined by the adjusted 3D Gauss volume and opacity and The volume calculation is where abc is the 3 dimensions of the scaling matrix S, and the total global importance score is summarized as:​ ， the method of forcing spatially adjacent Gaussians to maintain identity consistency by using unsupervised 3D regularization loss comprises the following steps: ， wherein is the original Gaussian point cloud, denotes a Gaussian of importance score, is the pruning ratio, is the set of scores quantiles, is the pruned Gaussian point cloud.​ 7. The point projection based three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning according to claim 6, characterized in that, a 3D Gaussian splash 3DGS renderer is used to render 2D images, and the influence equation of the identity code is represented as:

8. A point-projection based 3D reconstruction and segmentation system based on semi-Gaussian pruning, performing a point-projection based 3D reconstruction and segmentation method based on semi-Gaussian pruning as claimed in claim 1, characterized in that, where j represents the Gaussian index, M, H and W represent the number of training views and the height and width of the image respectively, then pruning operation is performed at a suitable stage, first, the global importance score of all Gaussians is calculated, then a predetermined proportion of Gaussians is pruned according to the ascending order of the score: the method of editing the scene at the object level according to the identity information of the Gaussians comprises the following steps: precise segmentation of target objects is realized according to user input text prompts, the user prompt is input into a grounding DINO open vocabulary detection model, the image encoder of the model extracts multi-scale visual features of the multi-view image, the text encoder analyzes the semantics to generate a feature vector, the model focuses on the visual area related to the semantic of the prompt by using the text-to-image cross attention mechanism, and outputs a pixel-level mask that accurately covers the target contour, after obtaining the 2D mask, the identity code corresponding to the mask with the highest overlap degree with the 2D mask is searched, if the identity code cosine similarity meets a threshold, it is determined as the target Gaussian, and finally the segmented Gaussian point set carries semantic labels and waits for downstream editing, thus completing the complete closed loop from natural language to 3D structure. The method comprises the following steps: a data acquisition module configured to acquire multi-view image data; a mask module configured to obtain SFM point cloud and consistency label mask of cross-view label based on the acquired multi-view image respectively; An optimization module is configured to initialize the obtained SFM point cloud as an identity Gaussian point cloud and render optimization using a differentiable renderer; A repair module is configured to densify the initial sparse point cloud using a localized Gaussian point management method, and identify local error areas for resetting and repairing; A supervision module is configured to supervise Gaussian identity feature learning using a cross-entropy loss using the obtained consistency label mask, and use an unsupervised 3D regularization loss to force identity consistency between spatially adjacent Gaussians; A pruning module is configured to use a global importance score pruning mechanism to prune Gaussians; An editing module is configured to perform object-level editing on the scene according to the identity information of the Gaussians.

Images