An interactive three-dimensional instance segmentation method and system based on 3DGS

Abstract

The application relates to an interactive three-dimensional instance segmentation method and system based on a 3DGS, and the method comprises the following steps: step 1, Gaussian semantic feature initialization based on a two-dimensional component prior; step 2, scale-gated two-dimensional mask supervision and contrast learning; and step 3, interactive real-time segmentation reasoning fused with visibility weight. The application has the beneficial effects that: 1. The application can be seamlessly applied to any trained 3DGS model, and has strong generalization ability; 2. The application completely solves the distortion and semantic confusion problem of two-dimensional supervision to three-dimensional promotion; 3. The application meets the needs of different application scenarios; 4. The reasoning time consumption is as low as 3-5 ms, which is much better than that of the prior art; the segmentation result relies on the original visibility and occlusion modeling of the 3DGS, and consistent rendering can be realized at any viewing angle, without semantic drift and boundary breakage problems; 5. Only two-dimensional images are needed as input, without the need for dense three-dimensional labeling, so that the training cost is greatly reduced, and the application is suitable for complex occlusion and sparse viewing angle real scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D scene reconstruction, semantic segmentation and interactive graphics processing, specifically to an interactive 3D instance segmentation method and system based on 3DGS. Background Technology

[0002] 3D Gaussian Splatting (3DGS), with its explicit representation of reconstruction followed by rendering, achieves near real-time compositing performance while maintaining high rendering quality, becoming a core technology foundation for immersive interaction, 3D scene editing, and real-time visualization. It uses a set of differentiable Gaussian primitives to jointly model scene geometry and appearance. Compared to implicit neural field representations such as NeRF, it has advantages such as clear explicit structure, high rendering efficiency, and easy integration with the graphics pipeline, showing broad prospects in various 3D applications.

[0003] However, existing 3DGS models only focus on geometry-appearance modeling and rendering. Semantic information in the scene is usually injected in the form of post-processing additional attributes, lacking explicit modeling of cross-view semantic consistency, and failing to form searchable and interactive 3D instance representations. At the same time, existing 3DGS semantic segmentation methods mostly remain at the stage of per-view 2D segmentation propagation, directly projecting 2D segmentation masks into 3D space, without solving the problems of multi-view occlusion and mismatch between mask scale and semantic granularity, which easily leads to semantic drift and unstable boundaries, and the segmentation results are difficult to support real-time interactive operation by users.

[0004] Existing research on 3D segmentation and 3DGS semantic understanding still has the following key shortcomings:

[0005] The lack of effective priors in 3D semantic initialization: directly learning semantic features on 3DGS Gaussian units can easily lead to ambiguity due to geometrically similar but semantically different regions, and does not fully utilize the mature semantic capabilities of the 2D basic visual model, resulting in insufficient feature discriminativeness and weak generalization ability.

[0006] Instability of 2D supervision to 3D: When introducing masks from 2D segmentation models such as SAM as supervision, the correspondence between mask scale and semantic granularity is not considered. Indiscriminate fusion of multi-scale masks can easily lead to semantic confusion. Furthermore, cross-view occlusion and visibility changes can cause 2D supervision to propagate in 3D space with distortion.

[0007] Lack of inference mechanisms for real-time interaction: Existing methods mostly focus on offline automatic segmentation and have not designed an interactive inference process adapted to 3DGS. Users cannot quickly select 3D instances through simple operations (such as clicking), and the segmentation results are difficult to guarantee consistent rendering under any viewpoint.

[0008] Insufficient robustness of clustering and interactive segmentation: Traditional 3D clustering does not consider the visibility weight of Gaussian elements, and Gaussian elements that have been occluded for a long time are prone to introducing clustering noise; feature similarity retrieval during interactive segmentation does not undergo normalization and gating modulation, and is prone to decreased retrieval accuracy due to scale drift.

[0009] Furthermore, traditional 3D segmentation methods (such as point cloud and voxel segmentation) rely on dense 3D annotation, which has high annotation costs, limited generalization ability, and difficulty in deep coupling with the real-time rendering pipeline of 3DGS. On the other hand, existing 3DGS semantic segmentation methods (such as Feature 3DGS and Gaussian Grouping) tend to focus on offline feature distillation or automatic segmentation quality improvement, and lack explicit modeling at the methodological level for the core requirements of interactive stability and cross-view consistency, which cannot meet the needs of practical applications such as immersive interaction and 3D scene editing.

[0010] In summary, there is an urgent need to propose a method for interactive 3D instance segmentation that achieves high discriminative Gaussian semantic feature learning, cross-viewpoint consistent instance representation, and supports queryability, rendering, and high real-time performance without changing the original geometry and rendering process of 3DGS. Summary of the Invention

[0011] The technical problem to be solved by this invention is to provide an interactive 3D instance segmentation method and system based on 3DGS, which solves the following technical problems:

[0012] 1. To address the problems in existing 3DGS semantic segmentation, such as the lack of effective semantic priors in Gaussian meta-semantic feature initialization, insufficient feature discriminativeness, and susceptibility to semantic ambiguity due to geometric similarity;

[0013] 2. To address the issues of mismatch between mask scale and semantic granularity, cross-view occlusion leading to distortion of supervision signals, semantic confusion, and poor cross-view consistency when upscaling 2D segmentation masks to 3DGS 3D space;

[0014] 3. This solves the problem that traditional 3DGS segmentation methods only support offline automatic segmentation, lack an inference mechanism for real-time interaction, and cannot achieve rapid selection of 3D instances driven by simple user operations;

[0015] 4. This addresses the issues of not considering visibility weights in 3D Gaussian clustering, introducing clustering noise from occluded Gaussians, and the decrease in accuracy of feature similarity retrieval due to scale drift in interactive segmentation.

[0016] This invention provides an interactive 3D instance segmentation method and system based on 3DGS. The core idea is to freeze the 3DGS geometric and rendering parameters, learn discriminative semantic features only at the Gaussian meta-level, and stably align the semantic priors and segmentation supervision of the 2D basic visual model to a unified 3D Gaussian feature space. This achieves consistent instance / component-level semantic understanding across viewpoints. Furthermore, a similarity retrieval mechanism that integrates clustering with visibility weights and gated normalization is designed to support automatic instance discovery and click-based interactive segmentation. The segmentation results can be rendered consistently from any viewpoint using the 3DGS rendering pipeline.

[0017] The overall technical solution is divided into three core stages: Gaussian semantic feature initialization, scale-gated mask supervision and contrastive learning, and interactive real-time segmentation inference. The system architecture includes a 3D reconstruction and feature initialization module, a scale-conditional supervised training module, and a clustering and interactive inference module. Each module is decoupled and can be trained end-to-end.

[0018] The technical solution of this invention to solve the above-mentioned technical problems is as follows: First, an interactive 3D instance segmentation method based on 3DGS is provided, including the following steps:

[0019] Step 1: Based on the trained 3DGS model, define an independent semantic feature vector for each Gaussian unit. Using DINOv2 two-dimensional part-level self-supervised features as a prior, fuse Gaussian geometric information, and output 32-dimensional compact Gaussian initial semantic features through lightweight MLP dimensionality reduction and residual connection.

[0020] Step 2: Using the 32-dimensional Gaussian initial semantic features output in Step 1 as input, introduce the two-dimensional segmentation mask generated by SAM as a weak supervision signal, align the mask scale with the semantic granularity through the scale-conditional channel gating mechanism, design the mask correlation comparison learning target to optimize the features, and finally output the optimized Gaussian semantic features.

[0021] Step 3: Using the optimized Gaussian semantic features from Step 2 as input, design two inference modes: unsupervised automatic instance clustering and click-based interactive binary segmentation. Integrate 3DGS visibility weights to improve inference accuracy and finally output the 3D instance segmentation results.

[0022] Based on the above technical solution, the present invention can be further improved as follows:

[0023] Furthermore, step 1 specifically involves:

[0024] Step 1.1, Definition of Gaussian Meta-Semantic Features: Let the scene be defined by... It consists of three-dimensional Gaussian elements, each Gaussian element ,in, Indicates the center of Gauss. The covariance is used to control the spatial expansion shape of the Gaussian element. and Control color and opacity separately; additional semantic feature vectors are introduced. ;

[0025] Step 1.2, Two-dimensional component-level feature aggregation: Sample corresponding semantic features from the dense feature map output by DINOv2, combine them with the visibility weights of 3DGS at this viewpoint for filtering and weighting, and aggregate all visible viewpoint features to obtain the corresponding Gaussian elements. Component-level semantic embedding ;

[0026] Step 1.3, Geometric and Semantic Feature Fusion: Selecting Gaussian Center Coordinates As the most basic and stable geometric description, it is concatenated with the features of two-dimensional components to form a joint input vector: ;

[0027] Step 1.4, Lightweight MLP Dimensionality Reduction and Residual Connection: Designing a Lightweight MLP as a Semantic Fusion Header Its output is defined as: Introducing residual connections Finally, we obtained the initial semantic features of a 32-dimensional Gaussian system.

[0028] Furthermore, step 2 specifically involves:

[0029] Step 2.1, SAM 2D Mask Generation: In the 3DGS training view Above, SAM outputs a set of k masks. It leverages its zero-sample generalization capability to provide high-quality weak supervision without requiring additional annotations;

[0030] Step 2.2, Mask Scale Normalization: Calculate the normalized scale for each mask. This characterizes the relative spatial range and semantic granularity of the mask;

[0031] Step 2.3, Scale-Conditioned Channel Gating: Design a lightweight gating network to map the scalar scale to a weight vector consistent with the number of 32-dimensional semantic feature channels. ,in This represents the element-wise Sigmoid function. For learnable parameters, The channel dimension of the Gaussian semantic channel is used. The weight vector is applied to the pixel-level semantic feature map obtained by 3DGS rendering through Hadamard product to achieve alignment between scale and semantic granularity features.

[0032] Step 2.4, Contrastive learning and feature optimization based on mask correlation:

[0033] ① Construct a binary correlation matrix for pixel pairs using SAM masks. ;

[0034] ② Normalize the gated pixel-level semantic features and calculate the cosine similarity matrix. ;

[0035] ③ Construct a positive sample aggregation term Negative sample separation term Feature regularization term The total loss function The formula is:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] in To regularize the weights, the feature norm is constrained to remain stable near the unit sphere, preventing feature collapse.

[0041] ④ Relying on the 3DGS differentiable splatting rendering pipeline, the loss gradient is backpropagated to the 32-dimensional Gaussian initial semantic features in step 1 to complete iterative optimization and finally obtain the optimized Gaussian semantic features.

[0042] Furthermore, step 3 specifically involves:

[0043] Step 3.1, Unsupervised Automatic Instance Clustering Reasoning:

[0044] ① Definition of multimodal clustering distance: By fusing three modalities—Gaussian geometric center, color, and the optimized Gaussian semantic features from step 2—a normalized clustering distance is defined to eliminate dimensional differences.

[0045]

[0046] in Scale parameters for geometry, color, and feature domain;

[0047] ② Gaussian Global Visibility Weight Calculation: Based on the transmissivity and transparency of 3DGS rendering, calculate the global visibility confidence weight of each Gaussian primitive under multiple viewpoints:

[0048]

[0049] ③ Weighted DBSCAN Clustering: Incorporates visibility weights into DBSCAN neighborhood counting to define weighted neighborhoods. The core point is determined when the number of weighted neighborhood points exceeds a threshold, ultimately resulting in Gaussian-level instance cluster labels. The noise Gaussian is marked as ;

[0050] Step 3.2, Click-based interactive binary segmentation reasoning:

[0051] ① Gating and normalization processing: Reuse the scale-gated network from step 2, combine it with the optimized Gaussian semantic features from step 2, and perform normalization → gating → re-normalization operations:

[0052] , ;

[0053] in For pixel-level contrastive feature tensors, For the first Three-dimensional features of a Gaussian node;

[0054] ② Query vector generation and similarity calculation: This involves generating the pixel data clicked by the user. Corresponding gated normalized features As a query vector, calculate the cosine similarity between all pixels and Gaussians and the query vector;

[0055] ③ Thresholding to obtain segmentation labels: Linearly map pixel-level similarity to [0,1] and set a threshold. A two-dimensional segmentation preview mask is generated; Gaussian similarity is thresholded to obtain two-class segmentation labels of three-dimensional Gaussian units. This refers to interactive instance segmentation results; relying on 3DGS's native visibility and occlusion modeling, segmentation labels are integrated into the rendering pipeline to output segmentation rendering results consistent from any viewpoint.

[0056] This invention also provides an interactive 3D instance segmentation system based on 3DGS, used to apply the above-mentioned interactive 3D instance segmentation method based on 3DGS, comprising:

[0057] The 3D reconstruction and feature initialization module completes the 3DGS geometry reconstruction of the scene, generates compact 3D initial semantic features with spatial awareness for each 3DGS Gaussian unit, provides a foundation for subsequent semantic feature optimization, and does not change the original geometry, color and rendering parameters of 3DGS.

[0058] The scale-conditional supervised training module is based on the Gaussian initial semantic features output in step 1. It uses the two-dimensional mask weak supervision of SAM and completes the end-to-end optimization of Gaussian semantic features through scale gating mechanism and contrastive learning. This allows the features to have cross-viewpoint and cross-scale intra-class aggregation and inter-class separation characteristics, while stably propagating the two-dimensional supervision signal to the three-dimensional Gaussian domain.

[0059] The clustering and interactive reasoning module, based on optimized Gaussian semantic features, enables two reasoning modes: unsupervised automatic instance clustering and click-based interactive binary segmentation. Relying on the visibility and occlusion modeling of 3DGS, it allows the segmentation results to be rendered consistently from any viewpoint, meeting the requirements of real-time interactive 3D instance segmentation.

[0060] Based on the above technical solution, the present invention can be further improved as follows:

[0061] Furthermore, the 3D reconstruction and feature initialization module includes:

[0062] The 3DGS geometry reconstruction unit performs 3D Gaussian sputtering reconstruction of a scene based on an input sequence of 2D images, and outputs a sequence of Gaussian meta-parameters. The 3DGS model outputs geometrically relevant information such as visibility weights, transmittance, and transparency from various viewpoints.

[0063] The DINOv2 feature extraction unit performs DINOv2 inference on the input two-dimensional image sequence and outputs a dense two-dimensional part-level self-supervised feature map, providing two-dimensional semantic priors for Gaussian semantic features.

[0064] The feature aggregation and fusion unit aggregates and splices two-dimensional component features and Gaussian geometric features. It completes feature dimensionality reduction and enhancement through lightweight MLP and residual connections, and outputs 32-dimensional Gaussian initial semantic features.

[0065] Furthermore, the scale-conditional supervised training module includes:

[0066] The SAM mask generation unit generates a two-dimensional segmentation mask and corresponding scale information for the 3DGS training view, providing a high-quality weak supervision signal without additional annotations.

[0067] The scale-gated unit aligns the mask scale with the Gaussian semantic feature channels and dynamically adjusts the channel weights of the semantic features according to the mask granularity to highlight global / local semantic information.

[0068] The contrastive learning optimization unit constructs a mask correlation contrastive learning loss, which propagates the two-dimensional supervision to the three-dimensional Gaussian domain through the 3DGS differentiable splatting rendering pipeline, thus completing the iterative optimization of Gaussian semantic features.

[0069] Furthermore, the clustering and interactive reasoning module includes:

[0070] The multimodal clustering unit integrates Gaussian geometric, color, and semantic features, calculates normalized clustering distance, and combines global visibility weights to achieve weighted DBSCAN clustering, thus completing unsupervised automatic instance discovery.

[0071] The interactive segmentation unit reuses a scale-gated network to complete feature gating normalization, generates query vectors in response to user clicks, and achieves fast selection of 3D instances through similarity calculation and thresholding, outputting pixel-level preview masks and Gaussian-level segmentation labels.

[0072] The 3DGS Consistent Rendering Unit combines Gaussian labels obtained from clustering / interactive segmentation with the 3DGS rendering pipeline. Relying on 3DGS's native visibility and occlusion modeling, it achieves consistent rendering of segmentation results from any viewpoint.

[0073] The beneficial effects of this invention are:

[0074] 1. This invention utilizes the fusion initialization of DINOv2's two-dimensional component-level priors and Gaussian geometric information to solve the ambiguity problem of directly learning semantic features. The features are compact (32-dimensional), have low computational overhead, can be seamlessly applied to any trained 3DGS model, and have strong generalization ability.

[0075] 2. This invention proposes a scale-conditional channel gating mechanism, which explicitly models the correspondence between mask scale and semantic granularity. Combined with contrastive learning based on mask correlation, it enables Gaussian semantic features to maintain good intra-class aggregation and inter-class separation across perspectives and scales, thus completely solving the distortion and semantic confusion problems in the upgrade from two-dimensional supervision to three-dimensional supervision.

[0076] 3. This invention simultaneously supports visibility-weighted unsupervised clustering and gated normalized click-based interactive segmentation. Automatic clustering can quickly obtain candidate instances / components, while interactive segmentation allows users to quickly select 3D instances through single / multi-point clicks, meeting the needs of different application scenarios.

[0077] 4. The semantic feature learning and reasoning of this invention are completely decoupled from 3DGS geometric modeling. The segmentation reasoning only involves feature similarity calculation and threshold determination. The reasoning time is as low as 3-5ms, which is far superior to existing methods. The segmentation results rely on the native visibility and occlusion modeling of 3DGS, and can achieve consistent rendering from any viewpoint without semantic drift or boundary breakage issues.

[0078] 5. This invention utilizes the zero-shot segmentation capability of SAM and the self-supervised features of DINOv2, requiring only two-dimensional images as input and eliminating the need for dense three-dimensional annotations, thus significantly reducing training costs and making it suitable for real-world scenes with complex occlusion and sparse viewpoints. Attached Figure Description

[0079] Figure 1 Initialization process for Gaussian semantic features based on prior knowledge of two-dimensional components;

[0080] Figure 2 The results show the model splitting in portions of the replica and DTU datasets;

[0081] Figure 3 Experimental results for splitting a portion of the LLFF dataset (a is fern, b is fortress, c is hornsr);

[0082] Figure 4 This is a diagram showing the results of an interactive splitting operation on the LEGO dataset. Detailed Implementation

[0083] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0084] An interactive 3D instance segmentation method based on 3DGS includes the following steps:

[0085] Step 1: Initialize Gaussian semantic features based on prior knowledge of two-dimensional components

[0086] Without altering the original geometry, color, and rendering parameters of 3DGS, an independent, learnable semantic feature vector is introduced for each Gaussian primitive. Utilizing DINOv2's 2D part-level self-supervised features as priors, and combined with Gaussian geometric information, a compact and transferable 3D semantic initialization representation is constructed using a lightweight MLP. Specific steps are as follows: Figure 1 As shown. Step 1 specifically involves:

[0087] Step 1.1, Definition of Gaussian Meta-Semantic Features: Let the scene be defined by... It consists of three-dimensional Gaussian elements, each Gaussian element It is described by the following parameters: ,in, Indicates the center of Gauss. The covariance is used to control the spatial expansion shape of the Gaussian element. and Color and opacity are controlled separately. Semantic feature vectors are also introduced. (In the experiment, d=32). This feature only participates in semantic branch learning and does not affect 3DGS geometric reconstruction and color rendering.

[0088] Step 1.2, Two-dimensional component-level feature aggregation: Semantic features are sampled from the dense feature map output by DINOv2, and filtered and weighted using the visibility weights of 3DGS at that viewpoint. After aggregating the features from all visible viewpoints, the corresponding Gaussian elements are obtained. Component-level semantic embedding .

[0089] Step 1.3, Geometric-Semantic Feature Fusion: Selecting Gaussian Center Coordinates As the most basic and stable geometric description, it is concatenated with the features of two-dimensional components to form a joint input vector: This design enables semantic features to have spatial awareness during the encoding stage, which helps the network distinguish between semantically similar Gaussian units that are spatially different.

[0090] Step 1.4, Lightweight MLP Dimensionality Reduction and Residual Connection: To map joint features to a compact and discriminative 3D semantic space, this invention designs a lightweight multilayer perceptron (MLP) as the semantic fusion head. Its output is defined as: Furthermore, residual connections are introduced to enhance the flexibility of feature representation: In the specific implementation, the fusion head is composed of two layers of linear mapping and ReLU activation function, which transforms the semantic features from DINOv2 and Gaussian spatial location fusion features from 384 dimensions to 32 dimensions.

[0091] Step 2: Scale-gated 2D mask supervision and contrastive learning

[0092] A two-dimensional segmentation mask generated by SAM is introduced as a weakly supervised signal. A scale-conditional channel gating mechanism is proposed to explicitly model the correspondence between mask scale and semantic granularity. A contrastive learning objective based on mask correlation is designed to achieve end-to-end optimization of Gaussian semantic features, enabling features to maintain good intra-class aggregation and inter-class separation properties across perspectives and scales. Step 2 specifically involves:

[0093] Step 2.1, SAM 2D Mask Generation: On the 3DGS training view, for each training view... SAM outputs a set of k masks. Without additional annotation, it provides high-quality weak supervision by leveraging SAM's zero-shot generalization capability.

[0094] Step 2.2, Mask Scale Normalization: Calculate the normalized scale for each mask. It describes the relative spatial range and semantic granularity of the mask (large scale corresponds to the whole object, small scale corresponds to local components).

[0095] Step 2.3, Scale-Conditioned Channel Gating: Design a lightweight gating network to map the scalar scale to a weight vector consistent with the number of semantic feature channels. ,in This represents the element-wise Sigmoid function. For learnable parameters, Let be the channel dimension of the Gaussian semantic channel. This weight vector is applied to the pixel-level semantic feature map obtained from 3DGS rendering through a Hadamard product, achieving scale-semantic granularity feature alignment. Large-scale masks preserve global semantic channels, while small-scale masks enhance local detail channels.

[0096] Step 2.4, Contrastive learning and feature optimization based on mask correlation:

[0097] ① Construct a binary correlation matrix between pixel pairs using a SAM mask. The training loss is constructed by consisting of a positive sample aggregation term, a negative sample separation term, and a feature regularization term.

[0098] ② Normalize the gated pixel-level semantic features and calculate the cosine similarity matrix. It characterizes the similarity of pixels in the feature space.

[0099] ③ Construct a positive sample aggregation term Negative sample separation term Feature regularization term The total loss function The formula is:

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] in As a regularization weight, it constrains the feature norm to stabilize near the unit sphere, preventing feature collapse.

[0105] ④ By using the differentiable splatting rendering pipeline of 3DGS, the two-dimensional mask supervision is stably propagated to the three-dimensional Gaussian domain, avoiding supervision distortion caused by direct projection.

[0106] Step 3: Interactive real-time segmentation inference with visibility weights

[0107] In the trained Gaussian semantic feature space, two inference modes are designed: unsupervised automatic instance clustering and click-based interactive binary segmentation. Both modes share the same set of 3D Gaussian semantic features, and the segmentation results rely on 3DGS visibility and occlusion modeling to achieve consistent rendering from any viewpoint. Step 3 specifically involves:

[0108] Step 3.1, Unsupervised Automatic Instance Clustering Reasoning:

[0109] ① Definition of multimodal clustering distance: Integrating the geometric center, color, and semantic features of Gaussian clustering, a normalized clustering distance is defined to eliminate dimensional differences. The formula is:

[0110] ;

[0111] in, The scale parameters for geometry, color, and feature domain are used to normalize the dimensions of different modes, making the three contributions comparable.

[0112] ② Gaussian Global Visibility Weight Calculation: Based on the transmittance and transparency of 3DGS rendering, the global visibility confidence weight of each Gaussian unit in multiple viewpoints is calculated to characterize the contribution of Gaussians to real observation. The formula is as follows:

[0113]

[0114] ③ Weighted DBSCAN Clustering: Incorporates visibility weights into the neighborhood count of DBSCAN, defining a weighted neighborhood. The core point is determined when the number of weighted neighborhood points exceeds a threshold, ultimately resulting in Gaussian-level instance cluster labels. The noise Gaussian is marked as This enables unsupervised automatic instance discovery.

[0115] Step 3.2, gated normalization click-based interactive segmentation:

[0116] ① Gating and Normalization Processing: A scale-gating network shared with Stage 2 is introduced to perform unitization → gating → re-unitization operations on Gaussian semantic features and pixel-level semantic features. The formula is as follows: , ,in For pixel-level contrastive feature tensors, For the first Three-dimensional features of Gaussian nodes.

[0117] ② Query vector generation and similarity calculation: This involves generating the pixel data clicked by the user. Corresponding gated normalized features As a query vector, calculate the cosine similarity between all pixels and Gaussians and the query vector.

[0118] ③ Thresholding to obtain segmentation labels: Linearly map pixel-level similarity to [0,1] and set a threshold. (Default τ=0.4) Generate a 2D segmentation preview mask; threshold the Gaussian similarity to obtain the 3D Gaussian trigonometric unit binary segmentation labels. (1 represents the target instance, 0 represents the background).

[0119] This invention also provides an interactive 3D instance segmentation system based on 3DGS, used to apply the above-mentioned interactive 3D instance segmentation method based on 3DGS, comprising:

[0120] 1. 3D Reconstruction and Feature Initialization Module

[0121] It completes the 3DGS geometry reconstruction of the scene, generating compact 3D initial semantic features with spatial awareness for each 3DGS Gaussian unit, providing a foundation for subsequent semantic feature optimization, without changing the original 3DGS geometry, color, and rendering parameters. Specifically, it includes...

[0122] 1.1 3DGS Geometric Reconstruction Unit

[0123] Based on the input sequence of two-dimensional images, it performs three-dimensional Gaussian sputtering reconstruction of the scene, and the output contains all Gaussian meta-parameters. The 3DGS model outputs geometrically relevant information such as visibility weights, transmittance, and transparency from various viewpoints.

[0124] 1.2 DINOv2 Feature Extraction Unit

[0125] It performs DINOv2 inference on the input two-dimensional image sequence and outputs a dense two-dimensional component-level self-supervised feature map, providing two-dimensional semantic priors for Gaussian semantic features.

[0126] 1.3 Feature Aggregation and Fusion Unit

[0127] It achieves the aggregation and splicing of two-dimensional component features and Gaussian geometric features, and completes feature dimensionality reduction and enhancement through lightweight MLP and residual connection, outputting 32-dimensional Gaussian initial semantic features.

[0128] The 3DGS geometry reconstruction unit performs the standard 3DGS reconstruction process on the input 2D image sequence, obtaining the Gaussian set of the scene and geometric information related to each viewpoint. The DINOv2 feature extraction unit extracts dense feature maps from the same batch of 2D image sequences, samples semantic features according to the correspondence between viewpoints and Gaussian units, and performs weighted feature aggregation based on 3DGS visibility weights to obtain the Gaussian unit's... The feature aggregation and fusion unit will... With Gauss Center spliced ​​as The input is fed into a lightweight MLP consisting of two layers of linear mappings and ReLU activation functions, and the output is... And by introducing residual connections, we obtain That is, the Gaussian initial semantic features, which are associated with and stored in relation to the 3DGS Gaussian meta-parameters.

[0129] 2-Scale Conditioned Supervised Training Module

[0130] Based on the initial Gaussian semantic features output in step 1, it utilizes two-dimensional masked weak supervision of SAM, and completes end-to-end optimization of the Gaussian semantic features through scale gating and contrastive learning. This allows the features to possess cross-viewpoint and cross-scale intra-class aggregation and inter-class separation characteristics, while stably propagating the two-dimensional supervision signal to the three-dimensional Gaussian domain. Specifically, it includes:

[0131] 2.1 SAM Mask Generation Unit

[0132] It generates two-dimensional segmentation masks and corresponding scale information for 3DGS training views, providing high-quality weak supervision signals without additional annotations.

[0133] 2.2 Scale Gating Unit

[0134] It aligns the mask scale with the Gaussian semantic feature channels and dynamically adjusts the channel weights of semantic features according to the mask granularity, highlighting global / local semantic information.

[0135] 2.3 Comparative Learning Optimization Unit

[0136] It constructs a mask correlation contrastive learning loss and propagates the two-dimensional supervision to the three-dimensional Gaussian domain through the 3DGS differentiable splatting rendering pipeline to complete the iterative optimization of Gaussian semantic features.

[0137] SAM mask generation unit for 3DGS training view Inference, output mask set And calculate the normalized scale for each mask. The scale-gated unit will Input a lightweight gating network to generate a weight vector. Scale-semantic granularity alignment is achieved by applying Hadamard product to the pixel-level semantic feature map rendered by 3DGS. The contrastive learning optimization unit utilizes SAM masks to construct a binary correlation matrix for pixel pairs. The gated pixel-level semantic features are normalized to obtain the cosine similarity matrix. , construct containing , , The total loss function is calculated. Leveraging the 3DGS differentiable splatting rendering pipeline, the loss gradient is backpropagated to the initial Gaussian semantic features, completing end-to-end iterative optimization of the features and outputting the optimized Gaussian semantic features.

[0138] 3. Clustering and Interactive Reasoning Module

[0139] Based on optimized Gaussian semantic features, it achieves two inference modes: unsupervised automatic instance clustering and click-based interactive binary segmentation. Leveraging 3DGS's visibility and occlusion modeling, it enables consistent rendering of segmentation results from any viewpoint, meeting the requirements for real-time interactive 3D instance segmentation. Specifically, it includes:

[0140] 3.1 Multimodal Clustering Unit

[0141] It integrates Gaussian geometric, color, and semantic features to calculate normalized clustering distance and combines global visibility weights to achieve weighted DBSCAN clustering, thus completing unsupervised automatic instance discovery.

[0142] 3.2 Interactive Segmentation Unit

[0143] Its reusable scale-gated network completes feature gating normalization, generates query vectors in response to user clicks, and achieves fast selection of 3D instances through similarity calculation and thresholding, outputting pixel-level preview masks and Gaussian-level segmentation labels.

[0144] 3.3 3DGS Consistent Rendering Unit

[0145] It combines Gaussian labels obtained from clustering / interactive segmentation with the 3DGS rendering pipeline, and relies on 3DGS's native visibility and occlusion modeling to achieve consistent rendering of segmentation results from any viewpoint.

[0146] The multimodal clustering unit extracts the geometric center, color, and semantic features of each Gaussian element from the optimized Gaussian semantic features, and calculates the normalized multimodal clustering distance. Based on the translucency and transparency of 3DGS rendering, calculate the global visibility weight of each Gaussian primitive. This data is then incorporated into DBSCAN neighborhood counting, and weighted DBSCAN clustering is performed to obtain Gaussian-level instance cluster labels. The 3DGS Consistent Rendering Unit associates instance cluster tags with 3DGS Gaussian elements, assigning different visual identifiers to different instances based on the tags during rendering. It achieves consistent rendering from any viewpoint by relying on 3DGS visibility and occlusion modeling.

[0147] The interactive segmentation unit responds to user clicks from any viewpoint and acquires pixels. Reusing scale-gated networks for pixel-level semantic features Gaussian semantic features Perform gating normalization to obtain and Clicking the pixel corresponding to... For the query vector, calculate the cosine similarity between all pixels, Gaussian similarities, and the vector. Threshold the pixel-level similarity to generate a 2D preview mask, and threshold the Gaussian similarity to obtain binary segmentation labels. The 3DGS Consistent Rendering Unit integrates Gaussian segmentation tags into the 3DGS rendering pipeline. When switching viewpoints, it relies on 3DGS's native depth sorting and occlusion relationship modeling to allow the segmentation results to naturally adapt to the viewpoint changes, achieving consistent rendering without drift or breaks. At the same time, this unit supports 60FPS real-time rendering, with no delay in response to user clicks and instant updates to the segmentation results.

[0148] The beneficial effects of the present invention will be illustrated below through specific experiments.

[0149] The experiment validated the algorithm based on a public dataset. The hardware environment consisted of an NVIDIA RTX 5000 GPU (16GB VRAM) and an Intel i7-12700K CPU. The software environment included Python 3.8, PyTorch 1.12.1, and CUDA 11.3. The 3DGS basic framework was implemented based on the official open-source code.

[0150] Regarding evaluation metrics, this invention performs quantitative evaluation in both the two-dimensional pixel domain and the three-dimensional Gaussian domain. The main metrics include mean Intersection over Union (mIoU) and mean Accuracy (mAcc), where mIoU is the average of the intersection-union ratios across categories, and mAcc is the average of the recall rates across categories. Confusion matrices are statistically analyzed at the pixel and Gaussian levels, respectively. Furthermore, to assess the fidelity of the segmentation boundaries, a tolerance-based boundary F-score is introduced in some experiments. For interactive segmentation scenarios, this invention uses a click-to-quality curve to evaluate interaction efficiency. Under a fixed click budget, the mIoU after each interaction is recorded, and its area or the number of clicks required to reach a specified threshold is calculated.

[0151] Experiment 1: Qualitative Analysis of Replica and DTU Datasets

[0152] Experimental objective: To verify the segmentation effect, cross-view consistency, and boundary fidelity of the method of the present invention in complex indoor scenes and controlled object scanning scenes.

[0153] Experimental dataset:

[0154] The Replica dataset contains complex indoor scenes with multiple objects in contact and occlusion, with test scenarios including office and living room; the DTU dataset contains controlled multi-view object scanning scenes, with test scenarios including DTU_11 and DTU_40, covering the segmentation requirements of small objects and parts.

[0155] After completing 3DGS geometric reconstruction and semantic feature training of the method of this invention on the above dataset, unsupervised automatic clustering segmentation is performed, and segmentation rendering results under different perspectives are output. Visual qualitative analysis is performed from four dimensions: instance integrity, boundary clarity, cross-view consistency, and occlusion region processing.

[0156] Figure 2 This paper presents the qualitative segmentation results of the proposed method on the Replica and DTU datasets. The proposed method effectively decomposes a target object into multiple semantically consistent components on a 3D Gaussian representation, maintaining clear instance boundaries in most visible areas. The segmentation results show no drift or breakage during cross-view rendering. In areas of object contact and occlusion, only a slight "spiky" structure appears; this phenomenon stems from the unobservability of the true geometry and semantics of the occluded area, not from inherent defects in the method itself. In contrast, the compared methods are prone to semantic confusion in object contact areas, exhibit significant boundary breaks and semantic drift during cross-view rendering, and lack complete component segmentation.

[0157] Experiment 2: Experimental Analysis of NVOS and SPIN-NeRF Datasets

[0158] Experimental objective: To verify the segmentation accuracy of the method of the present invention at the pixel level and Gaussian level, as well as its generalization ability in sparse viewpoint scenes.

[0159] Experimental dataset:

[0160] • NVOS dataset: derived from the LLFF dataset, containing real-world outdoor scenes, and is a multi-view densely labeled dataset; SPIN-NeRF dataset: derived from a subset of NeRF-related datasets, representing sparse-view scenes, simulating limited-view input scenarios in real-world applications.

[0161] The proposed method and various comparative methods were trained on two datasets respectively. The segmentation results were output on the test set. The mIoU and mAcc indices of each method were calculated and statistically analyzed. The segmentation accuracy and sparse view generalization ability of the methods were compared and analyzed.

[0162] Table 1. Quantitative results on the NVOS and SPIN-NeRF datasets.

[0163]

[0164] Table 1 presents the quantitative comparison results of the method of this invention with several representative methods on the NVOS (LLFF) and SPIN-NeRF datasets. The method of this invention achieves an mIoU of 93.3% and mAcc of 99.3% on NVOS, and an mIoU of 93.6% and mAcc of 99.3% on the SPIN-NeRF sparse viewpoint dataset. Compared with the comparison methods, this represents a 0.4%–6.5% improvement in mIoU, demonstrating that the method of this invention significantly outperforms existing methods in both segmentation accuracy and sparse viewpoint generalization ability.

[0165] From a qualitative perspective, such as Figure 3 As shown, the segmentation results of this invention in LLFF scenes demonstrate clear, continuous instance regions and stable cross-view consistency. The final feature map shown in the figure indicates that different objects form clearly separated semantic clusters in the Gaussian feature space, with clear boundaries between the background and foreground, maintaining consistency under multi-view rendering. This phenomenon is highly consistent with the improvement in mIoU in the quantitative metric, indicating that the semantic prior initialization, mask scale conditional modulation, and mask correlation comparison learning introduced in this invention can effectively enhance the discriminativeness of Gaussian features, enabling stable segmentation in 3D space through simple similarity retrieval.

[0166] Furthermore, as can be seen from the comparison, some methods are prone to segmentation and semantic leakage in elongated structures, occluded edges, or areas with similar backgrounds, while the method of this invention can still maintain a relatively complete structural outline in these areas. This further verifies the necessity and effectiveness of unifying two-dimensional prompts to a three-dimensional Gaussian feature space and completing instance reasoning within this space.

[0167] Experiment 3: Analysis of GUI Interaction Experiment Results

[0168] Experimental objective:

[0169] Verify the interactive segmentation real-time performance, cross-view consistency, and interactive efficiency of the method of this invention.

[0170] Test datasets: NeRF synthetic datasets (lego, ship), both of which are typical scenes that have been reconstructed using 3DGS;

[0171] Test operation: The user performs a single-point click / multi-point click prompt from any viewpoint, triggering target instance segmentation, and then switches to different viewpoints to verify the consistency of segmentation.

[0172] Figure 4The paper demonstrates the segmentation rendering results of the same object from multiple viewpoints. It can be observed that despite significant changes in viewpoint, including switching between front and rear views, lateral viewpoint rotation, and changes in partial occlusion relationships, the system can still stably render the independent segmented regions corresponding to the same instance in all viewpoints. This phenomenon indicates that the segmentation results generated by the method of this invention are not based on two-dimensional post-processing from a single viewpoint, but rather on semantic annotation at the three-dimensional Gaussian meta-level.

[0173] Table 2. Inference results on the Nerf synthetic dataset.

[0174]

[0175] As shown in Table 2, the method of the present invention has a single inference time as low as 3-5ms, which is far superior to existing methods. Moreover, the training time is comparable to lightweight methods such as SAGA and ISRF, achieving dual advantages in training efficiency and inference real-time performance.

[0176] The GUI interface of this invention achieves real-time rendering at 60FPS. User clicks trigger segmentation responses without delay, and segmentation results can be updated instantly. When switching perspectives, the segmentation results rely on the native visibility and occlusion modeling of 3DGS to naturally adapt to depth sorting and occlusion relationships. Boundaries remain continuous, instance integrity is not lost, and there are no perspective-dependent breaks or drifts.

[0177] In the interactive segmentation of the Lego and Garden datasets, the method of this invention can accurately select target instances (such as Lego truck beds, tracks, garden plants, and table legs) with a single click, and can achieve joint selection of multiple instances with multiple clicks. The segmentation results remain consistent in scenarios such as switching between front and back views and rotating the side view. In contrast, the comparison method is prone to missing targets and misselecting backgrounds, and the segmentation boundary shows obvious breaks when rendering across views.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An interactive 3D instance segmentation method based on 3DGS, characterized in that, The steps include the following: Step 1: Based on the trained 3DGS model, define an independent semantic feature vector for each Gaussian unit. Using DINOv2 two-dimensional part-level self-supervised features as a prior, fuse Gaussian geometric information, and output 32-dimensional compact Gaussian initial semantic features through lightweight MLP dimensionality reduction and residual connection. Step 2: Using the 32-dimensional Gaussian initial semantic features output in Step 1 as input, introduce the two-dimensional segmentation mask generated by SAM as a weak supervision signal, align the mask scale with the semantic granularity through the scale-conditional channel gating mechanism, design the mask correlation comparison learning target to optimize the features, and finally output the optimized Gaussian semantic features. Step 3: Using the optimized Gaussian semantic features from Step 2 as input, design two inference modes: unsupervised automatic instance clustering and click-based interactive binary segmentation. Integrate 3DGS visibility weights to improve inference accuracy and finally output the 3D instance segmentation results.

2. The interactive 3D instance segmentation method based on 3DGS according to claim 1, characterized in that, Step 1 is as follows: Step 1.1, Definition of Gaussian Meta-Semantic Features: Let the scene be defined by... It consists of three-dimensional Gaussian elements, each Gaussian element ,in, Indicates the center of Gauss. The covariance is used to control the spatial expansion shape of the Gaussian element. and Control color and opacity separately; additional semantic feature vectors are introduced. ; Step 1.2, Two-dimensional component-level feature aggregation: Sample corresponding semantic features from the dense feature map output by DINOv2, combine them with the visibility weights of 3DGS at this viewpoint for filtering and weighting, and aggregate all visible viewpoint features to obtain the corresponding Gaussian elements. Component-level semantic embedding ; Step 1.3, Geometric and Semantic Feature Fusion: Selecting Gaussian Center Coordinates As the most basic and stable geometric description, it is concatenated with the features of two-dimensional components to form a joint input vector: ; Step 1.4, Lightweight MLP Dimensionality Reduction and Residual Connection: Designing a Lightweight MLP as a Semantic Fusion Header Its output is defined as: Introducing residual connections Finally, we obtained the initial semantic features of a 32-dimensional Gaussian system.

3. The interactive 3D instance segmentation method based on 3DGS according to claim 1, characterized in that, Step 2 is as follows: Step 2.1, SAM 2D Mask Generation: In the 3DGS training view Above, SAM outputs a set of k masks. It leverages its zero-sample generalization capability to provide high-quality weak supervision without requiring additional annotations; Step 2.2, Mask Scale Normalization: Calculate the normalized scale for each mask. This characterizes the relative spatial range and semantic granularity of the mask; Step 2.3, Scale-Conditioned Channel Gating: Design a lightweight gating network to map the scalar scale to a weight vector consistent with the number of 32-dimensional semantic feature channels. ,in This represents the element-wise Sigmoid function. For learnable parameters, The channel dimension of the Gaussian semantic channel is used. The weight vector is applied to the pixel-level semantic feature map obtained by 3DGS rendering through Hadamard product to achieve alignment between scale and semantic granularity features. Step 2.4, Contrastive learning and feature optimization based on mask correlation: ① Construct a binary correlation matrix for pixel pairs using SAM masks. ; ② Normalize the gated pixel-level semantic features and calculate the cosine similarity matrix. ; ③ Construct a positive sample aggregation term Negative sample separation term Feature regularization term The total loss function The formula is: ； ； ； ； in To regularize the weights, the feature norm is constrained to remain stable near the unit sphere, preventing feature collapse. ④ Using the 3DGS differentiable splatting rendering pipeline, the loss gradient is backpropagated to the 32-dimensional Gaussian initial semantic features from step 1 to complete iterative optimization and finally obtain the optimized Gaussian semantic features.

4. The interactive 3D instance segmentation method based on 3DGS according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1, Unsupervised Automatic Instance Clustering Reasoning: ① Definition of multimodal clustering distance: By fusing three modalities—Gaussian geometric center, color, and the optimized Gaussian semantic features from step 2—a normalized clustering distance is defined to eliminate dimensional differences. in Scale parameters for geometry, color, and feature domain; ② Gaussian Global Visibility Weight Calculation: Based on the transmissivity and transparency of 3DGS rendering, calculate the global visibility confidence weight of each Gaussian primitive under multiple viewpoints: ③ Weighted DBSCAN Clustering: Incorporates visibility weights into DBSCAN neighborhood counting to define weighted neighborhoods. The core point is determined when the number of weighted neighborhood points exceeds a threshold, ultimately resulting in Gaussian-level instance cluster labels. The noise Gaussian is marked as ; Step 3.2, Click-based interactive binary segmentation reasoning: ① Gating and normalization processing: Reuse the scale-gated network from step 2, combine it with the optimized Gaussian semantic features from step 2, and perform normalization → gating → re-normalization operations: ， ； in For pixel-level contrastive feature tensors, For the first Three-dimensional features of Gaussian nodes; ② Query vector generation and similarity calculation: This involves generating the pixel data clicked by the user. Corresponding gated normalized features As a query vector, calculate the cosine similarity between all pixels and Gaussians and the query vector; ③ Thresholding to obtain segmentation labels: Linearly map pixel-level similarity to [0,1] and set a threshold. A two-dimensional segmentation preview mask is generated; Gaussian similarity is thresholded to obtain two-class segmentation labels of three-dimensional Gaussian units. This refers to interactive instance segmentation results; relying on 3DGS's native visibility and occlusion modeling, segmentation labels are integrated into the rendering pipeline to output segmentation rendering results consistent from any viewpoint.

5. An interactive 3D instance segmentation system based on 3DGS, used to apply the interactive 3D instance segmentation method based on 3DGS as described in any one of claims 1-4, characterized in that, include: The 3D reconstruction and feature initialization module completes the 3DGS geometry reconstruction of the scene, generates compact 3D initial semantic features with spatial awareness for each 3DGS Gaussian unit, provides a foundation for subsequent semantic feature optimization, and does not change the original geometry, color and rendering parameters of 3DGS. The scale-conditional supervised training module is based on the Gaussian initial semantic features output in step 1. It utilizes the two-dimensional mask weak supervision of SAM and completes the end-to-end optimization of Gaussian semantic features through scale gating mechanism and contrastive learning. This allows the features to have cross-viewpoint and cross-scale intra-class aggregation and inter-class separation characteristics, while stably propagating the two-dimensional supervision signal to the three-dimensional Gaussian domain. The clustering and interactive reasoning module, based on optimized Gaussian semantic features, enables two reasoning modes: unsupervised automatic instance clustering and click-based interactive binary segmentation. Relying on the visibility and occlusion modeling of 3DGS, it allows the segmentation results to be rendered consistently from any viewpoint, meeting the requirements of real-time interactive 3D instance segmentation.

6. The interactive 3D instance segmentation system based on 3DGS according to claim 5, characterized in that, The 3D reconstruction and feature initialization module includes: The 3DGS geometry reconstruction unit performs 3D Gaussian sputtering reconstruction of a scene based on an input sequence of 2D images, and outputs a sequence of Gaussian meta-parameters. The 3DGS model outputs geometrically relevant information such as visibility weights, transmittance, and transparency from various viewpoints. The DINOv2 feature extraction unit performs DINOv2 inference on the input two-dimensional image sequence and outputs a dense two-dimensional part-level self-supervised feature map, providing two-dimensional semantic priors for Gaussian semantic features. The feature aggregation and fusion unit aggregates and splices two-dimensional component features and Gaussian geometric features. It completes feature dimensionality reduction and enhancement through lightweight MLP and residual connections, and outputs 32-dimensional Gaussian initial semantic features.

7. The interactive 3D instance segmentation system based on 3DGS according to claim 5, characterized in that, The scale-conditional supervised training module includes: The SAM mask generation unit generates a two-dimensional segmentation mask and corresponding scale information for the 3DGS training view, providing a high-quality weak supervision signal without additional annotations. The scale-gated unit aligns the mask scale with the Gaussian semantic feature channels and dynamically adjusts the channel weights of the semantic features according to the mask granularity to highlight global / local semantic information. The contrastive learning optimization unit constructs a mask correlation contrastive learning loss, which propagates the two-dimensional supervision to the three-dimensional Gaussian domain through the 3DGS differentiable splatting rendering pipeline, thus completing the iterative optimization of Gaussian semantic features.

8. The interactive 3D instance segmentation system based on 3DGS according to claim 5, characterized in that, The clustering and interactive reasoning module includes: The multimodal clustering unit integrates Gaussian geometric, color, and semantic features, calculates normalized clustering distance, and combines global visibility weights to achieve weighted DBSCAN clustering, thus completing unsupervised automatic instance discovery. The interactive segmentation unit reuses a scale-gated network to complete feature gating normalization, generates query vectors in response to user clicks, and achieves fast selection of 3D instances through similarity calculation and thresholding, outputting pixel-level preview masks and Gaussian-level segmentation labels. The 3DGS Consistent Rendering Unit combines Gaussian labels obtained from clustering / interactive segmentation with the 3DGS rendering pipeline. Relying on 3DGS's native visibility and occlusion modeling, it achieves consistent rendering of segmentation results from any viewpoint.

Images