Point projection type three-dimensional reconstruction and segmentation method and system based on semi-Gaussian pruning

Abstract

The invention relates to the technical field of computer vision, in particular to a point projection type three-dimensional reconstruction and segmentation method and system based on semi-Gaussian pruning. The method comprises the following steps: respectively obtaining an SFM point cloud and a consistency label mask of a cross-view label based on an obtained multi-view image; initializing the obtained SFM point cloud into an identity semi-Gaussian point cloud, and performing rendering optimization by using a differentiable renderer; densifying the initial sparse point cloud by using a localized semi-Gaussian point management method, and identifying a local error region for resetting and repairing; using the obtained consistency label mask to supervise Gaussian identity feature learning by using cross entropy loss, and using unsupervised 3D regularization loss to force spatially adjacent gauss to maintain identity consistency; according to the method, the identity coding semi-Gaussian kernel method is adopted, the inherent representation fuzziness of a single opacity formula is eliminated, and the positive influence on the identity coding precision is generated.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a point projection three-dimensional reconstruction and segmentation method and system based on semi-Gaussian pruning. Background Art

[0002] At the intersection of artificial intelligence and computer vision, 3D scene understanding is a cornerstone technology driving autonomous robot navigation, immersive AR / VR interaction, and safe decision-making in autonomous driving. Its core goal is not only to accurately reconstruct the geometric structure of the environment, but also to deconstruct complex scenes into semantically meaningful logical components. This structured parsing capability is crucial for practical applications: Robotic systems rely on scene segmentation to identify operational objects and hazardous areas, enabling precise grasping and real-time obstacle avoidance. AR / VR applications require semantic segmentation to build interactive environments that blend virtual and real objects, anchoring virtual content to real-world surfaces and supporting natural manipulation. Autonomous vehicles require millisecond-level segmentation of roads, traffic participants, and roadside infrastructure to provide a computable environmental model for decision-making systems. As the core engine for achieving these goals, 3D scene segmentation transforms raw geometric data into programmable semantic units by assigning semantic / instance labels to point clouds, meshes, or voxels. Its output directly powers critical downstream tasks, such as supporting robotic arm assembly simulation in industrial scenarios or enabling automated editing of building components in digital twin systems. With the development of multimodal perception and end-to-end deep learning, the field is shifting from a purely geometry-driven paradigm to an intelligent understanding paradigm that integrates physical properties and functional semantics.

[0003] Traditional 3D segmentation methods have long relied on dense point cloud data. While these methods can explicitly represent spatial geometry, they face significant practical challenges. The primary limitation is the cost of annotation: point clouds must be manually labeled with semantic categories or instance affiliations, which can take hundreds of hours in complex scenes with occlusion. A secondary limitation is the complexity of the processing pipeline: a typical segmentation pipeline involves multiple stages, and the computational overhead makes real-time applications difficult. More critical is robustness: sensor noise, object occlusion, and reconstruction artifacts can easily lead to fragmented segmentation boundaries, category confusion, and semantic inconsistency, significantly reducing system reliability. Emerging machine learning models such as Neural Radiance Fields (NeRF) and 3D Gaussian Splattering (3DGS) offer new possibilities by providing high-fidelity rendering and compact and efficient scene representations. However, NeRF has limitations, including slow training and inference speed, and its implicit nature makes object-level segmentation within a scene particularly difficult. 3DGS, due to its coarse adaptive density control mechanism using a simple mean gradient threshold, cannot capture fine-grained local details. Periodic opacity resets during training can also introduce artifacts, compromising reconstruction quality and segmentation accuracy. To solve these problems, it is urgent to propose a 3D scene reconstruction and segmentation method and system based on identity-encoded half-Gaussian sputtering. Using a set of multi-view images, through the improved identity half-Gaussian kernel (ID-HGS), localized half-Gaussian point management (LHPM) and global importance score pruning mechanism, high-quality reconstruction and accurate segmentation of three-dimensional scenes can be achieved. Summary of the Invention

[0004] In order to solve the above-mentioned problems, the present invention provides a point projection 3D reconstruction and segmentation method and system based on semi-Gaussian pruning.

[0005] In a first aspect, the present invention provides a point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning, which adopts the following technical solutions: A point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning, comprising: Acquire multi-view image data; Based on the acquired multi-view images, the SFM point cloud and the consistent label mask of the cross-view label are obtained respectively; The obtained SFM point cloud is initialized as the identity semi-Gaussian point cloud and rendered using a differentiable renderer; Use the localized half-Gaussian point management method to densify the initial sparse point cloud and identify local error areas for reset and repair; The obtained consistent label mask is used to supervise Gaussian identity feature learning using cross entropy loss, and an unsupervised 3D regularization loss is used to enforce identity consistency between spatially adjacent Gaussians. Gaussian pruning using a global importance score pruning mechanism; Perform object-level editing of scenes based on Gaussian identity information.

[0006] Furthermore, the SFM point cloud and the consistent label mask of the cross-view label are obtained based on the acquired multi-view images, including the joint reconstruction of the sparse point cloud and the camera pose using the motion estimation COLMAP. First, the feature_extractor module of COLMAP is used to detect and extract the local features of each image based on the RootSIFT algorithm and store them in the SQLite database; then, cross-view feature matching is performed through feature_matcher, supplemented by multi-model geometry verification and watermark filtering, to establish the correspondence between the same-name points in the images and solve the matching ambiguity problem caused by perspective difference and texture repetition; then, incremental sparse reconstruction is performed: based on the matching results, the camera intrinsic and extrinsic parameters are iteratively optimized through bundle adjustment, and the objective function is At the same time, the triangulation algorithm is used to calculate the three-dimensional coordinates of the feature points with the same name to generate the initial sparse point cloud. Finally, the cumulative error is reduced through redundant view compression and global bundle adjustment optimization, and the PLY format point cloud containing point coordinates and RGB colors, as well as the precise pose parameters of each image, is output.

[0007] Furthermore, the method obtains SFM point clouds and consistent label masks across view labels based on the acquired multi-view images, and also includes independently processing each input image using the pre-trained ViT-H architecture. First, the input image resolution is scaled and padded to 1024×1024 pixels; then a 32×32 uniform grid of points is generated on the image as a prompt input; three candidate masks and confidence scores are predicted for each grid point, and then a two-stage filtering is performed: masks with confidence > 0.88 are first retained, and then the mask stability is checked; after filtering, redundancy is removed through cross-scale non-maximum suppression; finally, post-processing optimization is performed to output the final anything mask map.

[0008] Furthermore, the method obtains SFM point clouds and consistent label masks of cross-view labels based on the acquired multi-view images, and also includes introducing a video tracking model DEVA to achieve cross-frame consistency through a two-way propagation mechanism. Specifically, it includes two stages: intra-segment consensus optimization and propagation-consensus fusion. In the intra-segment consensus stage, the model aligns the segmentation masks of the next n frames to the current frame, and uses integer programming to screen the mask set with an IoU support greater than 0.5 to filter out isolated noise masks; in the propagation-consensus fusion stage, the historical mask output by the model is With the current consensus Perform bipartite graph matching, establish association mapping based on mask IoU threshold, transfer appearance features through ConvGRU hidden state, combine motion consistency constraints, maintain object ID stability in occlusion scenes, and finally output cross-view Figure 1 consistent mask sequence.

[0009] Furthermore, the obtained SFM point cloud is initialized as an identity half-Gaussian point cloud and rendered using a differentiable renderer for optimization, including introducing a normal vector to divide the Gaussian distribution function into two halves and assigning an independent opacity value to each half. and , a significant improvement is achieved by adding only an extra opacity parameter, utilizing the normal vector component that was previously unused in 3DGS, expressed as: , in is the normal vector of the segmentation plane; the complementary half-Gaussian distribution is obtained by inverting this normal vector; then using the paired half-Gaussian representation, where the parameters are shared by the two parts, the direction of the segmentation plane and the two opacity parameters are learned at the same time, the volume alpha blending at each pixel and the half-Gaussian representation of the identity encoding are: , in represents an ordered set of ID-HGS, It is defined as the integral of the two parts of the ID-HGS pair and is expressed as: , The ID-HGS kernel separates the Gaussian into two regions with different opacities, where ID-HGS learns the surface normal to make the segmentation plane fit the object geometry, and the dual opacity value Internal and The exterior effectively minimizes color mixing while maintaining sharp boundaries.

[0010] Furthermore, the consistent label mask obtained is used to supervise Gaussian identity feature learning using cross entropy loss, including calculating the error by comparing the 3DGS rendering result with the real image, dividing the image into several non-overlapping blocks, and aggregating the error values ​​in each block, based on the threshold Determine whether each block contains a problem area and generate a 2D error distribution map to highlight the key optimization areas, expressed as: The layout is designed to highlight key optimization areas, represented as: , in represents 2D block coordinates, is the pixel coordinate, and For rendered and real images, is the adaptive threshold, and then the feature matching algorithm is applied to extract feature points from the current view A and the adjacent reference view B to obtain a normalized feature set and establish the correspondence between features. Finally, local point operations are performed based on volume segmentation and cloning of high-gradient Gaussians to enhance detail representation; a lower densification threshold is applied in the error area to introduce more Gaussians; for the Any Gaussian point i in: , where c and s are the number of Gaussians whose gradients are below the densification gradient threshold after the cloning and splitting operations, respectively. Represents the values ​​corresponding to the lowest c+s entries when all Gaussian opacity values ​​are sorted in ascending order.

[0011] Furthermore, the unsupervised 3D regularization loss is used to enforce identity consistency between spatially adjacent Gaussians, including rendering into a 2D image through a differentiable 3D Gaussian splattering (3DGS) renderer. The influence equation of the identity encoding is expressed as: , the final 2D mask identity feature for each pixel is calculated as the identity code assigned to the Gaussian The weighted sum of the weights is determined by the influence factor of each Gaussian on the pixel Given; using rendered 2D features As input, we add a linear layer f to restore the feature dimension to K+1, and then Apply softmax for identity classification; Gaussian grouping introduces a 3D regularization loss , forcing adjacent Gaussian points to be close in the feature space, ensuring sufficient supervision for occluded or invisible Gaussians during training, expressed as: , where P contains the unit code sampled from a three-dimensional unit Gaussian distribution ,gather Represents its position in three-dimensional space nearest neighbors, the complete loss function By 2D loss , 3D loss and 3DGS loss The weighted combination is: .

[0012] Furthermore, the Gaussian pruning is optimized using a global importance score pruning mechanism, including rendering 3D Gaussians into images from any camera viewpoint through a differentiable rasterizer, and the initial importance score of each half-Gaussian is calculated based on its per-pixel ray in all training views. Contributions To quantify, traverse all training pixels to count the number of hits for each Gaussian; the score is calculated by adjusting the 3D Gaussian volume and opacity and Further refinement, the volume is calculated as , where abc are the 3 dimensions of the scaling matrix S, the overall global importance score is summarized as: , Where j represents the Gaussian index, M, H, and W represent the number of training views and the height and width of the image, respectively. The pruning operation is then performed at the appropriate stage by first computing the global importance scores of all Gaussians and then sequentially pruning a predetermined proportion of Gaussians based on the ascending scores: ,in is the original Gaussian point cloud, Represents Gaussian The importance score of is the trim ratio, is a fractional set of Quantile, is the pruned Gaussian point cloud.

[0013] Furthermore, the scene is edited at the object level based on the Gaussian's identity information, including accurate segmentation of the target object based on the text prompt word input by the user. After the user prompt word is input into the open vocabulary detection model GroundingDINO, the model's image encoder extracts multi-scale visual features of the multi-view image, while the text encoder parses the semantics to generate feature vectors. Through the text-to-image cross-attention mechanism, the model focuses on the visual area related to the semantics of the prompt word and outputs a pixel-level mask that accurately covers the target contour. After obtaining the 2D mask, the identity code corresponding to the mask with the highest overlap with the 2D mask is used to retrieve the Gaussian in the scene. If the cosine similarity of its identity code meets the threshold, it is determined to be the target Gaussian. The final output segmented Gaussian point set will carry semantic labels and wait for downstream editing, thus completing the complete closed loop from natural language to 3D structure.

[0014] In the second aspect, a point projection 3D reconstruction and segmentation system based on semi-Gaussian pruning is provided, comprising: A data acquisition module is configured to acquire multi-view image data; The mask module is configured to obtain the SFM point cloud and the consistent label mask of the cross-view label based on the acquired multi-view images; The optimization module is configured to initialize the obtained SFM point cloud as the identity semi-Gaussian point cloud and render the optimization using a differentiable renderer; The repair module is configured to densify the initial sparse point cloud using a localized half-Gaussian point management method and identify local error areas for reset and repair; The supervision module is configured to supervise the Gaussian identity feature learning using the obtained consistent label mask using cross entropy loss, and use unsupervised 3D regularization loss to enforce identity consistency between spatially adjacent Gaussians; The pruning module is configured to perform Gaussian pruning using a global importance score pruning mechanism; The editing module is configured to perform object-level editing on the scene based on the identity information of the Gaussian.

[0015] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a point projection three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning.

[0016] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to implement the point projection three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning.

[0017] In summary, the present invention has the following beneficial technical effects: The proposed method employs an identity-encoded half-Gaussian kernel to split the identity-encoded Gaussian into two regions of varying opacity. By learning a normal to conform the segmentation plane to the object geometry, the dual opacity value effectively minimizes color mixing while maintaining sharp boundaries, eliminating the inherent ambiguity of the single-opacity formulation and positively impacting identity encoding accuracy. Furthermore, in texture-rich areas, local contrast is enhanced, preventing the loss of detail caused by the oversimplification of a single Gaussian. The ID-HGS kernel ensures accurate identity encoding while maintaining the fidelity of the Gaussian model.

[0018] The LHPM module used in this paper can now be seamlessly integrated into the complete HGS-3DSeg pipeline. By optimizing the Gaussian geometry distribution, it mitigates artifacts and holes that can hinder identity encoding learning while enhancing detail reconstruction. This improvement improves scene understanding and segmentation by refining the spatial arrangement of Gaussian clusters and suppressing artifacts.

[0019] Applying this pruning operation during training can prevent GPU out-of-memory (OOM) errors, reduce model redundancy, slightly increase training speed, and reduce storage requirements. The timing of pruning is flexible; however, removing a large number of Gaussians can reduce model accuracy and visual quality. These losses can be corrected in subsequent training iterations. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the main flow chart of the method of the present invention.

[0021] Figure 2 is the initial SFM point cloud map and the final rendered scene map of the present invention, where Figure 2 (a) in Figure 2 (b) and Figure 2 (c) is the initial SFM point cloud generated by the present invention. Figure 2 (d) Figure 2 (e) and Figure 2 (f) in FIG. 5 is the scene graph finally rendered by the present invention.

[0022] Figure 3 is a comparison diagram of the anything mask of the present invention and the Gaussian grouping method, wherein Figure 3 The (a) column is a real scene picture. Figure 3 Column (b) is the anything mask map generated using the SAM model; Figure 3 The (c) column is the anything mask map obtained by Gaussian grouping; Figure 3 Column (d) is the anything mask map generated by the present invention.

[0023] Figure 4 This is a comparison chart of the accuracy of semantic recognition between the present invention and Gaussian grouping, where Figure 4 (a) is the semantic recognition result obtained by using the GINO model under the prompt word "Stuffed bear" in the scene of Gaussian group reconstruction. Figure 4 (b) is the semantic recognition result obtained by the GINO model under the prompt word "Stuffed bear" in the scene reconstructed by the present invention. Figure 4 (c) is the semantic mask obtained by the Gaussian grouping reconstruction scene using the GINO model under the "Stuffed bear" prompt word. Figure 4 (d) is the semantic mask obtained by the scene reconstructed by the present invention using the GINO model under the prompt word "Stuffed bear".

[0024] Figure 5 This is a comparison chart of the present invention and Gaussian grouping in terms of reconstruction details. Figure 5 (a) The green box is a magnified image of a real scene. Figure 5 Column (b) is the image obtained by Gaussian group rendering. The green box is the magnified local details. Figure 5 Column (c) is the image rendered by the present invention, where the green box is an amplification of local details. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings.

[0026] Example 1 Reference Figure 1 This embodiment of the present invention employs a point-projection 3D reconstruction and segmentation method based on semi-Gaussian pruning. The method employs a modular architecture and comprises five core functional modules: an initial point generation module, a consistency mask module, an identity encoding semi-Gaussian model, a localized semi-Gaussian point management module, and a global importance score pruning module. These modules interact with each other through standardized interfaces, enabling data exchange and process collaboration. Seven sequential or parallel processing steps form a complete fault diagnosis closed loop: In step S1, the system first performs SFM motion estimation on the multi-view images of the scene to obtain SFM point clouds; Step S2, in parallel with S1, uses a 2D segmentation model to predict 2D labels for multi-view images and uses a video tracking model to match cross-view labels to obtain consistent label masks; Step S3, initializing the SFM point cloud obtained in step S1 to an identity semi-Gaussian point cloud and rendering and optimizing it using a differentiable renderer; Step S4, using the localized half-Gaussian point management method to densify the initial sparse point cloud and identify local error areas for reset and repair; Step S5, in parallel with step S4, uses the consistent label mask obtained in step S2 to use the cross entropy loss Supervised Gaussian identity feature learning and unsupervised 3D regularization loss Enforce identity consistency between spatially adjacent Gaussians; Step S6, using a global importance score pruning mechanism to optimize memory usage; Step S7, downstream editing task, performs object-level editing on the scene based on Gaussian identity information.

[0027] Specifically, including Step S1: SFM point cloud generation: For the multi-view image sequences provided by the user, this stage uses the motion estimation software COLMAP to realize the joint reconstruction of sparse point cloud and camera pose. First, feature extraction and matching are performed: COLMAP's feature_extractor module detects and extracts local features of each image based on RootSIFT or SuperPoint algorithm and stores them in SQLite database; then cross-view feature matching is performed through feature_matcher, supplemented by multi-model geometric verification (including RANSAC estimation of basic matrix F, homography matrix H and essential matrix E, combined with threshold and Distinguishing between general / flat / panoramic scenes) and watermark filtering (similar transformation detection, threshold ), establish the correspondence between the same-name points between images, and solve the matching ambiguity problem caused by perspective differences and texture duplication.

[0028] Then perform incremental sparse reconstruction: COLMAP's mapper module iteratively optimizes the camera's intrinsic parameters (focal length, distortion coefficient) and extrinsic parameters (rotation matrix R, translation vector T) based on the matching results through bundle adjustment. The objective function is ), and using triangulation algorithms (minimum set sampling of two views, enforcing triangulation angle constraints and depth positive constraint ) calculates the 3D coordinates of the feature points with the same name and generates an initial sparse point cloud. Although the point cloud has a low density, it is optimized by the optimal viewpoint selection strategy (multi-resolution pyramid scoring , weight Balanced point number and distribution) and iterative retriangulation (Post-BA RT) have fully characterized the basic geometric structure and spatial topological relationship of the scene.

[0029] Finally, redundant view compression (camera grouping common viewing rate ) and global bundle adjustment optimization to reduce cumulative error, outputting a PLY-formatted point cloud containing point coordinates and RGB colors, as well as precise pose parameters for each image. This process can efficiently process hundreds of images using hybrid CPU / GPU acceleration (PCG iterations are used for large-scale problems). However, robust reconstruction of low-texture areas remains challenging, requiring scene graph augmentation (initialization with non-panoramic image pairs) to mitigate this challenge.

[0030] like Figure 2 As shown, Figure 2 (a) Figure 2 (b) and Figure 2 (c) is the initial SFM point cloud generated by this embodiment. Figure 2 (d) Figure 2 (e) and Figure 2 (f) in FIG. 5 is the final rendered scene graph of this embodiment.

[0031] Step S2: cross-view Figure 1 Consistency mask generation: The goal of this stage is to generate semantically consistent object masks for multi-view images. First, perform single-view mask initialization: use the pre-trained Segment Anything The ViT-H architecture of the SAM model (SAM) processes each input image independently in a cue-independent "anything" mode. SAM first scales the input image resolution to 1024×1024 pixels; then generates a 32×32 uniform grid of 1024 foreground points on the image as cue input; then, it predicts three candidate masks and confidence scores for each grid point (to resolve single-point ambiguity), with an original output of approximately 3072 masks; then performs two-stage filtering: first retains masks with confidence > 0.88, and then checks mask stability (interference over union (IoU) ≥ 95% between binary masks with thresholds of 0.45 and 0.55); after filtering, redundancy is removed through cross-scale non-maximum suppression (NMS, IoU threshold 0.7), and high-confidence masks are retained first; finally, post-processing optimization is performed: isolated fragments with an area of ​​<100 pixels are removed, and internal holes of the same size are filled, and the final "anything" mask map is output. The output of all views obtains a set of segmentation masks.

[0032] Then cross-view mask association is performed: In order to solve the problem of inconsistent mask IDs in multiple views, the video tracking model DEVA is introduced. This model achieves cross-frame consistency through a two-way propagation mechanism (arranging multi-view images in sequence as video frames in the model), which specifically includes two core technical stages: intra-fragment consensus optimization and propagation-consensus fusion. In the intra-fragment consensus stage, the model aligns the segmentation masks of the next n frames (n=3 by default) to the current frame, and uses integer programming to filter the mask set with an IoU support greater than 0.5, filtering out isolated noise masks (such as low-support masks that are eliminated after spatial alignment). In the propagation-consensus fusion stage, the historical mask output by the temporal propagation model With the current consensus Perform bipartite graph matching and establish an association map based on the mask IoU threshold (>0.5): the successfully matched mask pairs are fused into new instances, and the unmatched ones are The mask is injected into the propagation memory as a new object, unmatched The mask is dynamically managed in memory through a counter mechanism (deleted after L=5 consecutive frames without matching). This process transmits the appearance features through the ConvGRU hidden state, combines the motion consistency constraint, maintains the stability of the object ID in the occlusion scene, and finally outputs the cross-view Figure 1 consistent mask sequence.

[0033] Step S3: Identity semi-Gaussian kernel initialization and optimization First, a normal vector is introduced to divide the Gaussian distribution function into two halves, and an independent opacity value is assigned to each half. and This configuration maintains backward compatibility (when = , which is equivalent to standard Gaussian grouping when θ is less than θ ( θ = θ ), while incurring minimal computational overhead. This modification simply adds an additional opacity parameter, utilizing a normal vector component previously unused in 3DGS. As described below, this design achieves significant improvements with negligible increase in parameters / cost.

[0034] , in is the normal vector to the splitting plane; the complementary half-Gaussian distribution is obtained by negating this normal vector.

[0035] The rasterization process of ID-HGS is similar to that of 3DGS, using a paired half-Gaussian representation, where the parameters (mean, rotation, scale, color, and identity encoding) are shared by the two parts, while learning the direction of the segmentation plane and two opacity parameters. The volume alpha blending and identity encoding half-Gaussian at each pixel can be expressed as: , in represents an ordered set of ID-HGS, Defined as the integral of the two parts of the ID-HGS pair.

[0036] , The ID-HGS kernel separates the Gaussian into two regions with different opacities, where ID-HGS learns the surface normal to make the segmentation plane fit the object geometry, and the dual opacity value ( internal, The exterior) effectively minimizes color mixing while maintaining sharp boundaries, thus eliminating the representation ambiguity inherent in the single opacity formulation and having a positive impact on identity encoding accuracy. In addition, in texture-rich areas, The difference enhances local contrast and prevents detail loss caused by oversimplification of a single Gaussian. The ID-HGS kernel ensures accurate identity encoding while maintaining the fidelity of the Gaussian model.

[0037] Step S4: Localized half-Gaussian point management The Localized Half-Gaussian Point Management (LHPM) method matches local features between the current view and the reference view and combines the error map derived from the rendering To identify the error areas, it accurately locates the 3D areas that need to be optimized by back-projecting these areas into 3D space. In these areas, it adopts a more aggressive strategy to optimize the points where the reconstruction is insufficient. To fit the scene details, reset the pathological points that occlude these areas To reduce the impact of occlusion on training, and adaptively prune low-opacity Gaussians to alleviate GPU memory constraints.

[0038] The specific method is as follows: First, we compare the 3DGS rendering result with the real image to calculate the error. Then, we divide the image into and aggregate the error values ​​within each block. Then, based on the threshold ,We determine whether each block contains a problem area and generate a 2D error distribution map to ,intuitively highlight the areas that need to be optimized.

[0039] , in represents 2D block coordinates ( ), represents pixel coordinates, and Represent the rendered image and the real image respectively, is an adaptive threshold affected by the block size.

[0040] Then, a feature matching algorithm (such as LightGlue) is applied to extract feature points from the current view A and the adjacent reference view B to obtain a normalized feature set. (The coordinates are normalized by the image size.) Then a correspondence is established between these two sets of features, producing a set of matching These matches are combined with the error map to identify pairwise error regions between views. Next, cone rays are cast from the camera centers of the current and reference views to their respective error regions, covering all relevant pixels.

[0041] Finally, local point operations include: volume-based segmentation and cloning of high-gradient Gaussians to enhance detail representation; applying a lower densification threshold in the error region to introduce more Gaussians; resetting the opacity of high-opacity points close to the camera (identified as pathological occluders) to correct incorrect occlusion effects; and pruning redundant Gaussian points in ascending order of opacity to prevent excessive model growth. Any Gaussian point i in: , where c and s are the number of Gaussians whose gradients are below the densification gradient threshold after the cloning and splitting operations, respectively. Represents the values ​​corresponding to the lowest c+s entries when all Gaussian opacity values ​​are sorted in ascending order.

[0042] Unlike the Local Point Management (LPM) plugin originally designed for the standard 3DGS kernel—which differs in both parameter count and dimensionality—we make LPM compatible with our approach by integrating identity encoding and adding an additional transparency channel. We also remove masking from the nx / ny / nz parameters and redefine the threshold trigger to use the maximum value of each transparency pair. During the Gaussian pruning phase, we adopt a more efficient strategy instead of the traditional global importance scoring method, avoiding repeated calculation of score statistics.

[0043] Step S5, identity learning supervision: The identity encoding is designed as a 16-dimensional learnable compact vector. Similar to how spherical harmonics (SH) coefficients represent colors, this encoding can be rendered into a 2D image using a differentiable 3D Gaussian splatter (3DGS) renderer. The identity encoding is optimized using the same method as color optimization to represent the scene instance ID. Similar to the 3DGS in -The influence equation for hybrid, identity encoding can be derived as follows: , Final 2D mask identity feature for each pixel is calculated as the identity code assigned to the Gaussian The weighted sum of each Gaussian is 16, where the weight is determined by the influence factor of each Gaussian on the pixel. given.

[0044] Specifically, Gaussian grouping utilizes multi-view images and 2D masks generated by SAM

[16] to learn a discrete and grouped 3D representation, which can reconstruct and segment any object in the 3D scene. In order to further improve the grouping accuracy, in addition to using the standard cross entropy loss In addition to identity classification.

[0045] Since the mask identification labels are 2D, they cannot directly supervise the 3D Gaussian identity encoding Therefore, using the 2D features rendered in formula (1) As input. By adding a linear layer f, the feature dimension is restored to K+1 (K is the total number of masks), and then Apply softmax for identity classification.

[0046] Gaussian grouping introduces a 3D regularization loss ,This loss forces adjacent Gaussian points to be close in the feature space, ensuring sufficient supervision for occluded or invisible Gaussians during training.

[0047] , where P contains the unit code sampled from a three-dimensional unit Gaussian distribution ,gather Represents its position in three-dimensional space Nearest neighbors. The complete loss function By 2D loss , 3D loss and 3DGS loss The weighted combination is: .

[0048] Step S6: Global importance score pruning: Global Importance Calculation: Relying solely on Gaussian opacity as the importance criterion leads to suboptimal Gaussian pruning, so a more efficient formulation is needed. Inspired by Equation (3), these 3D Gaussians can be rendered into images from arbitrary camera viewpoints using a differentiable rasterizer. Then, the initial importance score of each half-Gaussian can be calculated based on its importance for each pixel ray across all training views ( ) Therefore, we iterate over all training pixels to count the number of hits for each Gaussian.

[0049] Score through the resized 3D Gaussian volume and opacity and Further refinement. The volume is calculated as , where abc are the three dimensions of the scaling matrix S in formula (1). Overall, the global importance score can be summarized as: , Where j represents the Gaussian index, M, H and W represent the number of training views and the height and width of the image respectively. The indicator function B() determines the given light Whether it intersects with a specific Gaussian. However, directly using the Gaussian volume will usually overestimate the contribution of the background Gaussian, resulting in the Gaussian representing complex geometry being over-clipped. Replace the volume term in the above equation with the following expression : , where the volume is first calculated by taking the 90th percentile value of all ascending sorted Gaussian distributions Normalize the range to (0,1) and introduce the parameter Make adjustments.

[0050] In open scene training with a large number of Gaussian points, we perform pruning at the appropriate stage. First, we calculate the global importance scores of all Gaussians, and then prune a predetermined proportion of Gaussians according to the ascending score order (from low to high): , in is the original Gaussian point cloud, represents the Gaussian importance score, is the trim ratio, is a fractional set of Quantile, is the pruned Gaussian point cloud.

[0051] Step S7: Downstream editing operation: As a key step in the 3D Gaussian scene editing process, this stage achieves precise segmentation of the target object based on the user-entered text prompt, laying the foundation for subsequent operations such as object removal and attribute modification. The technical process begins with semantically driven cross-modal mask generation: After the user prompt is input into the open vocabulary detection model Grounding DINO, the model's image encoder extracts multi-scale visual features from the multi-view image, while the text encoder parses the semantics to generate feature vectors. Using a text-to-image cross-attention mechanism, the model focuses on visual areas semantically relevant to the prompt word and outputs a pixel-level mask that accurately covers the target outline.

[0052] After obtaining the 2D mask, the system performs cross-dimensional matching: the identity code corresponding to the mask with the highest overlap with the 2D mask is used to retrieve the Gaussian in the scene. If the cosine similarity of its identity code meets the threshold, it is determined to be the target Gaussian. The final output segmented Gaussian point set will carry semantic labels waiting for downstream editing, thus completing the complete closed loop from natural language to 3D structure.

[0053] To further verify the feasibility and effectiveness of our method, we conduct experiments in various dimensions. To evaluate the segmentation and fine-grained localization performance in open-world scenes, we use the LERF-Mask dataset introduced by Gaussian Grouping

[18] , which provides accurate mask annotations. We further use three scenes from LERF-Localization, which only provide rough bounding boxes. To evaluate the reconstruction quality, we benchmark Gaussian Grouping on seven of the nine complete scenes of Mip-NeRF 360, excluding the flower and tree scenes because they are not publicly accessible.

[0054] The present invention experiments on the fine-grained segmentation of three-dimensional scenes using the mIoU and mBIoU indicators for semantic segmentation, as shown in Table 1: Table 1 Quantitative comparison on the LERF-Localization dataset

[0055] The present invention also conducts reconstruction accuracy tests on the mipnerf360 dataset, as shown in Table 2: Table 2 Quantitative comparison on the MIP-NERF360 dataset (PSNR↑) - indicates reconstruction failure method stump garden counter Bicycle Kitchen room bonsai Gaussian grouping — 27.0 28.5 — 30.7 28.0 31.3 us 26.8 27.4 29.3 26.4 31.7 29.7 32.1

[0056] Example 2

[0057] This embodiment provides a point projection 3D reconstruction and segmentation system based on semi-Gaussian pruning, including: The data acquisition module is configured as follows: A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a point projection three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning.

[0058] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the point projection three-dimensional reconstruction and segmentation method based on semi-Gaussian pruning.

[0059] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning, characterized in that: include: Acquire multi-view image data; Based on the acquired multi-view images, the SFM point cloud and the consistent label mask of the cross-view label are obtained respectively; The obtained SFM point cloud is initialized as the identity semi-Gaussian point cloud and rendered using a differentiable renderer; Use the localized half-Gaussian point management method to densify the initial sparse point cloud and identify local error areas for reset and repair; The obtained consistent label mask is used to supervise Gaussian identity feature learning using cross entropy loss, and an unsupervised 3D regularization loss is used to enforce identity consistency between spatially adjacent Gaussians. Gaussian pruning using a global importance score pruning mechanism; Perform object-level editing of scenes based on Gaussian identity information.

2. The point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning according to claim 1, characterized in that: The method obtains SFM point clouds and consistent label masks of cross-view labels based on the acquired multi-view images, including the joint reconstruction of sparse point clouds and camera poses using motion estimation COLMAP. First, the feature_extractor module of COLMAP is used to detect and extract local features of each image based on the RootSIFT algorithm and store them in an SQLite database; then, cross-view feature matching is performed through feature_matcher, supplemented by multi-model geometry verification and watermark filtering, to establish the correspondence between the same-name points in the images and solve the matching ambiguity problem caused by perspective difference and texture repetition; then, incremental sparse reconstruction is performed: based on the matching results, the camera intrinsic and extrinsic parameters are iteratively optimized through bundle adjustment, and the objective function is: At the same time, the triangulation algorithm is used to calculate the three-dimensional coordinates of the feature points with the same name to generate the initial sparse point cloud. Finally, the cumulative error is reduced through redundant view compression and global bundle adjustment optimization, and the PLY format point cloud containing point coordinates and RGB colors, as well as the precise pose parameters of each image, is output.

3. The point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning according to claim 2, characterized in that: The method further comprises obtaining the SFM point cloud and the consistent label mask of the cross-view label based on the acquired multi-view images, and independently processing each input image using the pre-trained ViT-H architecture, first scaling the input image resolution to 1024×1024 pixels; and then generating a 32×32 uniform grid of points on the image as a prompt input; For each grid point, three candidate masks and confidence scores are predicted, and then a two-stage filtering is performed: masks with confidence > 0.88 are retained, and then the mask stability is checked; after filtering, redundancy is removed through cross-scale non-maximum suppression; finally, post-processing optimization is performed to output the final anything mask map.

4. The point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning according to claim 3, characterized in that: The method obtains SFM point clouds and consistent label masks across view labels based on the acquired multi-view images, and also introduces the video tracking model DEVA to achieve cross-frame consistency through a two-way propagation mechanism. Specifically, it includes two stages: intra-segment consensus optimization and propagation-consensus fusion. In the intra-segment consensus stage, the model aligns the segmentation masks of the next n frames to the current frame, and uses integer programming to screen the mask set with an IoU support greater than 0.5 to filter out isolated noise masks. In the propagation-consensus fusion stage, the historical mask output by the model is With the current consensus Perform bipartite graph matching, establish an association map based on the mask IoU threshold, transfer appearance features through ConvGRU hidden state, combine motion consistency constraints, maintain object ID stability in occlusion scenarios, and finally output a mask sequence that is consistent across views.

5. The point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning according to claim 4, characterized in that: The obtained SFM point cloud is initialized as an identity half-Gaussian point cloud and rendered using a differentiable renderer, including introducing a normal vector to divide the Gaussian distribution function into two halves and assigning an independent opacity value to each half. and , a significant improvement is achieved by adding only an extra opacity parameter, utilizing the normal vector component that was previously unused in 3DGS, expressed as: ， in is the normal vector of the segmentation plane; the complementary half-Gaussian distribution is obtained by inverting this normal vector; then using the paired half-Gaussian representation, where the parameters are shared by the two parts, the direction of the segmentation plane and the two opacity parameters are learned at the same time, the volume alpha blending at each pixel and the half-Gaussian representation of the identity encoding are: ， in represents an ordered set of ID-HGS, It is defined as the integral of the two parts of the ID-HGS pair and is expressed as: ， The ID-HGS kernel separates the Gaussian into two regions with different opacities, where ID-HGS learns the surface normal to make the segmentation plane fit the object geometry, and the dual opacity value Internal and The exterior effectively minimizes color mixing while maintaining sharp boundaries.

6. The point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning according to claim 5, characterized in that: The consistent label mask obtained is used to supervise Gaussian identity feature learning using cross entropy loss, including calculating the error by comparing the 3DGS rendering result with the real image, dividing the image into several non-overlapping blocks, and aggregating the error values ​​in each block based on the threshold. Determine whether each block contains a problem area and generate a 2D error distribution map to highlight the key optimization areas, expressed as: ， in represents 2D block coordinates, is the pixel coordinate, and For rendered and real images, is the adaptive threshold, and then the feature matching algorithm is applied to extract feature points from the current view A and the adjacent reference view B to obtain a normalized feature set and establish the correspondence between features. Finally, local point operations are performed based on volume segmentation and cloning of high-gradient Gaussians to enhance detail representation; a lower densification threshold is applied in the error area to introduce more Gaussians; for the Any Gaussian point i in: ， where c and s are the number of Gaussians whose gradients are below the densification gradient threshold after the cloning and splitting operations, respectively. Represents the values ​​corresponding to the lowest c+s entries when all Gaussian opacity values ​​are sorted in ascending order.

7. The point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning according to claim 6, characterized in that: The use of unsupervised 3D regularization loss to enforce identity consistency between spatially adjacent Gaussians includes rendering into 2D images through a differentiable 3D Gaussian splattering 3DGS renderer. The influence equation of identity encoding is expressed as: ， Final 2D mask identity feature for each pixel is calculated as the identity code assigned to the Gaussian The weighted sum of the weights is determined by the influence factor of each Gaussian on the pixel Given; using rendered 2D features As input, we add a linear layer f to restore the feature dimension to K+1, and then Apply softmax for identity classification; Gaussian grouping introduces a 3D regularization loss , forcing adjacent Gaussian points to be close in the feature space, ensuring sufficient supervision for occluded or invisible Gaussians during training, expressed as: ， where P contains the unit code sampled from a three-dimensional unit Gaussian distribution ,gather Represents its position in three-dimensional space Nearest neighbors, complete loss function By 2D loss , 3D loss and 3DGS loss The weighted combination is: .

8. The point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning according to claim 7, characterized in that: The proposed method uses a global importance score pruning mechanism to optimize Gaussian pruning, which involves rendering 3D Gaussians into images from arbitrary camera viewpoints via a differentiable rasterizer. The initial importance score of each half-Gaussian is calculated based on its per-pixel ray across all training views. Contributions To quantify, traverse all training pixels to count the number of hits for each Gaussian; the score is calculated by adjusting the 3D Gaussian volume and opacity and Further refinement, the volume is calculated as , where abc are the 3 dimensions of the scaling matrix S, the overall global importance score is summarized as: ， Where j represents the Gaussian index, M, H, and W represent the number of training views and the height and width of the image, respectively. The pruning operation is then performed at the appropriate stage by first computing the global importance scores of all Gaussians and then sequentially pruning a predetermined proportion of Gaussians based on the ascending scores: ， in is the original Gaussian point cloud, Represents Gaussian The importance score of is the trim ratio, is a fractional set of Quantile, is the pruned Gaussian point cloud.

9. The point projection 3D reconstruction and segmentation method based on semi-Gaussian pruning according to claim 8, characterized in that: The object-level editing of the scene based on the Gaussian's identity information includes achieving accurate segmentation of the target object based on the text prompt word input by the user. After the user prompt word is input into the open vocabulary detection model Grounding DINO, the model's image encoder extracts multi-scale visual features of the multi-view image, while the text encoder parses the semantics to generate feature vectors. Through the text-to-image cross-attention mechanism, the model focuses on the visual area related to the semantics of the prompt word and outputs a pixel-level mask that accurately covers the target contour. After obtaining the 2D mask, the identity code corresponding to the mask with the highest overlap with the 2D mask is used to retrieve the Gaussian in the scene. If the cosine similarity of its identity code meets the threshold, it is determined to be the target Gaussian. The final output segmented Gaussian point set will carry semantic labels and await downstream editing, thus completing the complete closed loop from natural language to 3D structure.

10. A point projection 3D reconstruction and segmentation system based on semi-Gaussian pruning, characterized in that: include: A data acquisition module is configured to acquire multi-view image data; The mask module is configured to obtain the SFM point cloud and the consistent label mask of the cross-view label based on the acquired multi-view images; The optimization module is configured to initialize the obtained SFM point cloud as the identity semi-Gaussian point cloud and render the optimization using a differentiable renderer; The repair module is configured to densify the initial sparse point cloud using a localized half-Gaussian point management method and identify local error areas for reset and repair; The supervision module is configured to supervise the Gaussian identity feature learning using the obtained consistent label mask using cross entropy loss, and use unsupervised 3D regularization loss to enforce identity consistency between spatially adjacent Gaussians; The pruning module is configured to perform Gaussian pruning using a global importance score pruning mechanism; The editing module is configured to perform object-level editing on the scene based on Gaussian identity information.

Images