A 3D Gaussian scene semantic segmentation method and system based on uncertainty-aware collaborative optimization

Abstract

This invention provides a 3D Gaussian scene semantic segmentation method and system based on uncertainty-aware collaborative optimization. The method involves: reconstructing a 3D Gaussian scene to obtain a 3D scene represented by Gaussian points; modeling the semantic features of the 3D Gaussian scene by configuring a feature vector for each Gaussian primitive; obtaining 2D segmentation masks for multi-view images using a 2D segmentation model, constructing pixel-level pseudo-confidence labels, and back-projecting them into 3D space to obtain Gaussian-level pseudo-confidence labels; designing an uncertainty estimation module to predict the confidence of each Gaussian semantic label; training the model using a confidence-weighted contrastive loss combined with a mask-constrained semantic diffusion loss; and achieving interactive segmentation during the inference phase based on the similarity between point cue features and Gaussian point features. This invention improves the accuracy, stability, and robustness of 3D Gaussian scene semantic segmentation and is applicable to multiple fields such as digital twins and virtual reality.

Description

Technical Field

[0001] This invention relates to the fields of 3D vision and virtual reality technology, and in particular to a 3D Gaussian scene semantic segmentation method and system based on uncertainty perception collaborative optimization. Background Technology

[0002] Scene representation methods based on 3D Gaussian Splatting (3DGS) have become a core technological foundation for 3D scene modeling and editing due to their advantages of high real-time rendering efficiency and strong geometric representation capabilities. Their applications in intelligent scene understanding, 3D editing, autonomous driving environmental perception, and virtual reality interaction all rely on accurate semantic segmentation of objects in 3D scenes. Existing 3D Gaussian semantic segmentation techniques mainly fall into three categories:

[0003] The first method is the SAGA method proposed by Shanghai Jiao Tong University and Huawei Noah's Ark Lab. It distills the segmentation capability of the two-dimensional SAM model into three-dimensional Gaussian features to achieve interactive semantic segmentation. However, it relies on two-dimensional pseudo-label supervision, is easily affected by occlusion and viewpoint changes, leading to semantic noise propagation, and does not model the credibility of labels.

[0004] The second method is the RT-GS2 method proposed by the Free University of Brussels in Belgium. It improves generalization ability by learning perspective-independent features through self-supervised learning and fusing cross-perspective features. However, it assumes that all semantic supervision information has the same credibility, lacks an uncertainty evaluation mechanism, and has insufficient segmentation stability.

[0005] Thirdly, the OVGaussian method proposed by the Chinese University of Hong Kong and Nanyang Technological University constructs a dedicated dataset for open vocabulary segmentation and trains it using two-dimensional and three-dimensional features. However, it still relies on the supervision signal of the two-dimensional visual model, is susceptible to pseudo-label noise, and does not explicitly model the uncertainty of semantic prediction.

[0006] In summary, existing technologies all suffer from common problems such as large noise in two-dimensional weakly supervised labels, semantic instability in object boundary regions, and semantic information drift in object internal regions. The core reason is that the credibility of semantic prediction is not modeled and evaluated, and reliable and unreliable semantic supervision information cannot be distinguished. This leads to noisy samples interfering with feature learning during training, and boundary regions are difficult to achieve stable semantic convergence due to low label credibility. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a semantic segmentation method and system for 3D Gaussian scenes based on uncertainty-aware collaborative optimization. This invention can model the semantic credibility of each Gaussian unit, jointly optimize the semantic feature learning and inference process, suppress noise supervision and enhance boundary consistency during the training phase, and improve the robustness of point prompts during the inference phase, ultimately achieving high-precision and high-stability semantic segmentation of 3D Gaussian scenes.

[0008] In a first aspect, the present invention provides a three-dimensional Gaussian scene semantic segmentation method based on uncertainty-aware collaborative optimization, comprising the following steps:

[0009] S1) The 3D Gaussian splashing algorithm 3DGS is used to reconstruct the 3D scene using multi-view images, and a 3D Gaussian scene represented by a set of Gaussian points is obtained.

[0010] S2) Use a two-dimensional segmentation model to segment the multi-view Gaussian scene image to obtain a set of two-dimensional segmentation masks. Select the maximum segmentation mask for each object by comparing the overlapping area of ​​the masks.

[0011] S3) Construct Gaussian pseudo-confidence labels using the maximum segmentation mask;

[0012] S4) Predict the semantic label confidence of each Gaussian point using a lightweight neural network, and supervise it with the Gaussian pseudo-confidence label;

[0013] S5) Construct a confidence-weighted contrast loss function. and semantic diffusion loss function based on mask constraints The overall loss function To train a 3D Gaussian semantic segmentation model;

[0014] S6) Receive the point prompt features input by the user, and calculate the cosine similarity score between the point prompt features and all Gaussian point features. Based on the similarity score Implement interactive object segmentation in a 3D Gaussian scene.

[0015] Preferably, in step S1), the attributes of each Gaussian point include position, scale, orientation, opacity, and color features encoded by spherical harmonic coefficients.

[0016] Preferably, in step S3), a Gaussian pseudo-confidence label is constructed using a two-dimensional segmentation mask, specifically including:

[0017] S31) Perform a Euclidean distance transformation on the maximum segmentation mask to obtain the mask boundary distance map;

[0018] S32) Construct pixel-level pseudo-confidence labels based on the mask boundary distance map;

[0019] S33) The pixel-level pseudo-confidence labels are back-projected into three-dimensional space using known camera parameters, and Gaussian-level pseudo-confidence labels are obtained through nearest neighbor search and weighted average aggregation.

[0020] Preferably, in step S4), a lightweight neural network is used to predict the semantic label confidence of each Gaussian point; specifically as follows:

[0021] S41) Take the spatial location, color features, and semantic feature vector of each Gaussian unit as input;

[0022] S42) Obtain each Gaussian point through K-nearest neighbor (KNN) search. Local neighborhood structure in three-dimensional space;

[0023] S43) Calculate the mean of the neighborhood semantic features of Gaussian in the scene. Then, the semantic mean of the Gaussian neighborhood obtained so far is concatenated with its semantic features, spatial location and color information to obtain the concatenated features.

[0024] S44) Input the concatenated features into a lightweight neural network for confidence prediction, and use pseudo-confidence labels for supervision:

[0025] ;

[0026] In this embodiment, the MLP is a multilayer perceptron network. The MLP has two layers, and the activation functions are ReLU and Sigmoid, respectively. Gaussian point The confidence level of semantic labels.

[0027] Secondly, the present invention provides a three-dimensional Gaussian scene semantic segmentation system based on uncertainty-aware collaborative optimization, comprising:

[0028] The Gaussian scene reconstruction module uses the 3D Gaussian splashing algorithm 3DGS to reconstruct a 3D scene from multiple view images, resulting in a 3D Gaussian scene represented by a set of Gaussian points.

[0029] The mask segmentation module uses a two-dimensional segmentation model to segment multi-view Gaussian scene images, obtaining a set of two-dimensional segmentation masks. The maximum segmentation mask for each object is selected by comparing the overlapping areas of the masks.

[0030] The pseudo-confidence label construction module uses a two-dimensional segmentation mask to construct Gaussian-level pseudo-confidence labels.

[0031] The uncertainty estimation module uses a lightweight neural network to predict the semantic label confidence of each Gaussian point and supervises it with the Gaussian pseudo-confidence label.

[0032] The model training module constructs a model including a confidence-weighted contrastive loss function. and semantic diffusion loss function based on mask constraints The overall loss function To train a 3D Gaussian semantic segmentation model;

[0033] The semantic segmentation module receives point hint features input by the user and calculates the cosine similarity score between the point hint features and all Gaussian point features. Based on the similarity score Implement interactive object segmentation in a 3D Gaussian scene;

[0034] The visual editing module is used to render 3D Gaussian scenes in real time, providing scene editing operations such as segmentation, deletion, movement, and rotation.

[0035] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional Gaussian scene semantic segmentation method.

[0036] The beneficial technical effects of this invention are as follows:

[0037] 1. This invention constructs Gaussian pseudo-confidence labels, designs uncertainty estimation for the confidence of each Gaussian predicted semantic label, and adjusts the contribution of the supervision signal according to the confidence in the confidence-weighted contrast loss. This solves the problem of the prior art assuming that all labels have the same confidence, automatically identifies and reduces the impact of low-confidence noise samples on training, and improves the robustness of the model to label noise.

[0038] 2. This invention designs a semantic diffusion loss based on mask constraints, which limits the semantic propagation within the same mask region, allowing the high-confidence central region to transmit reliable semantic information to the low-confidence boundary region, driving the Gaussian semantic features of the boundary region to gradually converge, solving the problem of semantic instability in the boundary region in the prior art, and realizing the stable construction of object-level semantic structure;

[0039] 3. This invention constructs a Gaussian local neighborhood structure through K-nearest neighbors, which strengthens the spatial correlation between Gaussians during the semantic feature learning and diffusion process. Combined with a confidence-weighted supervision mechanism, it makes the semantic features of high-confidence regions within the object the learning leader, effectively avoiding semantic information drift within the internal regions and improving the accuracy of semantic segmentation.

[0040] 4. This invention is based on semantic tag confidence modulation similarity calculation, which reduces the interference of low confidence Gaussian points on the segmentation results. Even if the user point prompt falls on the boundary or noise area, it can still achieve accurate object segmentation based on the features of high confidence Gaussian points, thus improving the reliability of interactive operation. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the method of an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram illustrating the process of constructing Gaussian-level pseudo-confidence labels in an embodiment of the present invention. Detailed Implementation

[0043] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0044] like Figure 1 As shown, this embodiment provides a 3D Gaussian scene semantic segmentation method based on uncertainty-aware collaborative optimization, including the following steps:

[0045] S1) Acquire multi-view images and use the 3D Gaussian Splashing Algorithm (3DGS) to reconstruct a 3D scene from the multi-view images, resulting in a 3D Gaussian scene represented by a set of Gaussian points. ;

[0046] Each Gauss point The attribute is determined by its position. ,scale ,direction Opacity and color features encoded by spherical harmonic coefficients. Defined.

[0047] In this embodiment, to obtain the color of a specific pixel during rendering, 3DGS employs a splatter technique. In each camera pose, a 3D Gaussian vector is projected onto the 2D image space through a world-to-image transformation matrix, projecting to the same coordinates. The Gaussian distribution is mixed in order of depth and according to its opacity. Weighting is performed to generate the color of each pixel. :

[0048]

[0049] In the formula, coordinate position The color characteristics of the pixels; Indicates the number of Gaussian points; Indicates the first Color information of a Gaussian point; Indicates the first The final transparency of a Gaussian point; Indicates the first The final transparency at Gaussian points.

[0050] S2) Use a two-dimensional segmentation model to segment the multi-view Gaussian scene image to obtain a set of two-dimensional segmentation masks. Select the maximum segmentation mask for each object by comparing the overlapping area of ​​the masks.

[0051] In this embodiment, the two-dimensional segmentation model utilizes the SAM or SAM2 model to automatically segment the acquired multi-view Gaussian scene images, obtaining a two-dimensional segmentation mask set:

[0052] ;

[0053] in, and These represent the height and width of the Gaussian scene image, respectively. This represents the number of masks obtained by segmenting the Gaussian scene image from the current viewpoint using a 2D segmentation model; Indicates the first A segmentation mask; To segment the mask set.

[0054] In this embodiment, the maximum segmentation mask for each object is selected by comparing the size of the mask overlap area. ,Right now:

[0055]

[0056] In the formula, Indicates the current segmentation mask; Represents a two-dimensional segmentation mask set All segmentation masks except the current mask; This indicates a series of multiplication operations; Indicates an indicator function; This is the overlap threshold.

[0057] S3) Construct Gaussian pseudo-confidence labels using the maximum segmentation mask; such as... Figure 2 As shown, the details are as follows:

[0058] S31) Perform a Euclidean distance transformation on the maximum segmentation mask to obtain a mask boundary distance map; the mask boundary distance map Represented as:

[0059] ;

[0060] In the formula, Indicates pixel position, Represents the maximum segmentation mask The boundary;

[0061] S32) Construct pixel-level pseudo-confidence labels based on the mask boundary distance map; the pixel-level pseudo-confidence labels Represented as:

[0062] ;

[0063] In the formula, The preset boundary distance threshold; Indicates an indicator function;

[0064] S33) The pixel-level pseudo-confidence labels are back-projected to the three-dimensional space through known camera parameters, and Gaussian-level pseudo-confidence labels are obtained through nearest neighbor search and weighted average aggregation;

[0065] In this embodiment, for image pixels Its corresponding depth is If the camera intrinsic parameter matrix is ​​K, then the three-dimensional coordinates of this pixel in the camera coordinate system are... for:

[0066] ;

[0067] Its position in the world coordinate system is then obtained through extrinsic parameter transformation. :

[0068] ;

[0069] in, The three-dimensional points are obtained by backprojection of pixels. Let be the rotation matrix of the camera's extrinsic parameters. Let be the translation vector of the camera's extrinsic parameters;

[0070] After obtaining the 3D points after pixel backprojection, they are associated with the nearest Gaussian point in space through nearest neighbor search. superior:

[0071] ;

[0072] In the formula, Indicates the first The three-dimensional position of a Gaussian point;

[0073] By back-projecting pixels into 3D space, pixel-level pseudo-confidence labels are assigned to corresponding Gaussians. However, since one Gaussian may correspond to multiple pixels, a weighted average aggregation is used to obtain Gaussian-level pseudo-confidence. :

[0074]

[0075] In the formula, Indicates projection onto Gaussian point A set of pixels.

[0076] Through the above process, the pseudo-confidence information in the two-dimensional image can be propagated to the three-dimensional Gaussian representation, thereby obtaining the Gaussian-level confidence supervision signal: the Gaussian corresponding to the center region of the mask has a higher confidence; the Gaussian corresponding to the boundary region of the mask has a lower confidence.

[0077] S4) Predict the semantic label confidence level of each Gaussian point using a lightweight neural network, and supervise the process using the Gaussian-level pseudo-confidence labels; specifically as follows:

[0078] S41) Take the spatial location, color features, and semantic feature vector of each Gaussian unit as input;

[0079] In this embodiment, each Gaussian element Represented as:

[0080] ;

[0081] In the formula, This represents the three-dimensional spatial position of the nth Gaussian point; This represents the color characteristic of the nth Gaussian point; This represents the semantic feature vector corresponding to the nth Gaussian point;

[0082] In this embodiment, to evaluate the reliability of each Gaussian semantic label in subsequent training, a confidence estimation mechanism is introduced to predict a semantic label confidence value for each Gaussian point. , The larger the value, the more reliable the semantic label of the Gaussian point; The smaller the value, the more likely the Gaussian point is located in a boundary or noise region.

[0083] S42) Obtain each Gaussian point through K-Nearest Neighbor (KNN) search. Local neighborhood structure in three-dimensional space ,Right now:

[0084] ;

[0085] S43) Calculate the mean of the neighborhood semantic features of Gaussian in the scene. Then, the semantic mean of the currently obtained Gaussian neighborhood is concatenated with its semantic features, spatial location, and color features, i.e.:

[0086] ;

[0087] ;

[0088] In the formula, Features of splicing; Indicates a splicing operation; For the first Semantic feature vectors of Gaussian points; The mean of the semantic features of the neighborhood; Indicates the first The three-dimensional spatial position of a Gaussian point; For the first Color information of a Gaussian point;

[0089] S44) Input the concatenated features into a lightweight neural network for confidence prediction, and use pseudo-confidence labels for supervision:

[0090] ;

[0091] In this embodiment, the MLP is a multilayer perceptron network. The MLP has two layers, and the activation functions are ReLU and Sigmoid, respectively. Gaussian point The confidence level of semantic labels.

[0092] S5) Construct a confidence-weighted contrast loss function. and semantic diffusion loss function based on mask constraints The overall loss function To train a 3D Gaussian semantic segmentation model;

[0093] In this embodiment, a confidence-weighted contrastive loss function is constructed. The contribution of the supervision signal is adjusted according to the confidence level of the semantic label to suppress the interference of low-confidence noise samples;

[0094] ;

[0095] In the formula, For pixels The confidence level of the semantic label; To reduce the impact of low-confidence samples on training, it can suppress the interference of noisy samples on feature learning, preserve the strong supervision signal of high-confidence regions, and allow semantic information to be transferred from high-confidence regions to low-confidence regions. Spatial weights are used to enhance the constraints between neighboring samples in space; Indicates cosine similarity; This indicates the mask to which the pixel belongs; For indicator functions; Represents the pixels after Gaussian feature field rendering Semantic features at the location.

[0096] in,

[0097]

[0098] In the formula, These are spatial scale control parameters;

[0099] In the embodiment, a semantic diffusion loss function based on mask constraints is designed. This enables semantic propagation from high-confidence regions to low-confidence boundary regions; combining the two loss functions yields the overall loss function, namely:

[0100]

[0101] In the formula, Indicates the first One mask area; Indicates the first A neighborhood set of Gaussians;

[0102] Overall loss function Set to:

[0103] .

[0104] S6) Receive the point prompt features input by the user, and calculate the cosine similarity score between the point prompt features and all Gaussian point features. Based on the similarity score Implement interactive object segmentation in a 3D Gaussian scene;

[0105] ;

[0106] In the formula, Indicates the features of the point prompt; This represents the cosine similarity.

[0107] Example 2

[0108] This embodiment provides a 3D Gaussian scene semantic segmentation system based on uncertainty-aware collaborative optimization, including:

[0109] The Gaussian scene reconstruction module uses the 3D Gaussian splashing algorithm 3DGS to reconstruct a 3D scene from multiple view images, resulting in a 3D Gaussian scene represented by a set of Gaussian points.

[0110] The mask segmentation module uses a two-dimensional segmentation model to segment multi-view Gaussian scene images, obtaining a set of two-dimensional segmentation masks. The maximum segmentation mask for each object is selected by comparing the overlapping areas of the masks.

[0111] The pseudo-confidence label construction module uses the maximum segmentation mask to construct Gaussian pseudo-confidence labels;

[0112] The uncertainty estimation module uses a lightweight neural network to predict the semantic label confidence of each Gaussian point and supervises it with the Gaussian pseudo-confidence label.

[0113] The model training module constructs a model including a confidence-weighted contrastive loss function. and semantic diffusion loss function based on mask constraints The overall loss function To train a 3D Gaussian semantic segmentation model;

[0114] The semantic segmentation module receives point hint features input by the user and calculates the cosine similarity score between the point hint features and all Gaussian point features. Based on the similarity score Implement interactive object segmentation in a 3D Gaussian scene;

[0115] The visual editing module is used to render 3D Gaussian scenes in real time, providing scene editing operations such as segmentation, deletion, movement, and rotation.

[0116] Example 3

[0117] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the three-dimensional Gaussian scene semantic segmentation method.

[0118] In this embodiment, the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. A processor, coupled to the memory, is used to execute computer programs stored in the memory.

[0119] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form.

[0120] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A semantic segmentation method for 3D Gaussian scenes based on uncertainty-aware collaborative optimization, characterized in that, Includes the following steps: S1) The 3D Gaussian splashing algorithm 3DGS is used to reconstruct the 3D scene using multi-view images, and a 3D Gaussian scene represented by a set of Gaussian points is obtained. S2) Use a two-dimensional segmentation model to segment the multi-view Gaussian scene image to obtain a set of two-dimensional segmentation masks. Select the maximum segmentation mask for each object by comparing the overlapping area of ​​the masks. S3) Construct Gaussian pseudo-confidence labels using the maximum segmentation mask; S4) Predict the semantic label confidence of each Gaussian point using a lightweight neural network, and supervise it with the Gaussian pseudo-confidence label; S5) Construct a confidence-weighted contrast loss function. and semantic diffusion loss function based on mask constraints The overall loss function To train a 3D Gaussian semantic segmentation model; S6) Receive the point prompt features input by the user, and calculate the cosine similarity score between the point prompt features and all Gaussian point features. Based on the similarity score Implement interactive object segmentation in a 3D Gaussian scene.

2. The three-dimensional Gaussian scene semantic segmentation method based on uncertainty-aware collaborative optimization according to claim 1, characterized in that, In step S3), a Gaussian pseudo-confidence label is constructed using a two-dimensional segmentation mask, specifically including: S31) Perform a Euclidean distance transformation on the maximum segmentation mask to obtain the mask boundary distance map; S32) Construct pixel-level pseudo-confidence labels based on the mask boundary distance map; S33) The pixel-level pseudo-confidence labels are back-projected into three-dimensional space using known camera parameters, and Gaussian-level pseudo-confidence labels are obtained through nearest neighbor search and weighted average aggregation.

3. The three-dimensional Gaussian scene semantic segmentation method based on uncertainty-aware collaborative optimization according to claim 2, characterized in that, In step S34), Gaussian pseudo-confidence is obtained by weighted average aggregation. : In the formula, Indicates projection onto Gaussian point The set of pixels; This represents pixel-level pseudo-confidence labels.

4. The three-dimensional Gaussian scene semantic segmentation method based on uncertainty-aware collaborative optimization according to claim 3, characterized in that, In step S4), a lightweight neural network is used to predict the semantic label confidence of each Gaussian point; specifically as follows: S41) Take the spatial location, color features, and semantic feature vector of each Gaussian unit as input; S42) Obtain each Gaussian point through K-nearest neighbor (KNN) search. Local neighborhood structure in three-dimensional space; S43) Calculate the mean of the neighborhood semantic features of Gaussian in the scene. Then, the semantic mean of the Gaussian neighborhood obtained so far is concatenated with its semantic features, spatial location and color features to obtain the concatenated features; S44) Input the concatenated features into a lightweight neural network for confidence prediction, and use pseudo-confidence labels for supervision: ； MLP stands for Multilayer Perceptron Network. Gaussian point The confidence level of semantic labels.

5. A three-dimensional Gaussian scene semantic segmentation method based on uncertainty-aware collaborative optimization according to claim 4, characterized in that, In step S4), the splicing feature for: ； In the formula, Indicates a splicing operation; For the first Semantic feature vectors of Gaussian points; The mean of the semantic features of the neighborhood; Indicates the first The three-dimensional spatial position of a Gaussian point; For the first Color characteristics of a Gaussian point; The mean of the neighborhood semantic features Represented as: ； In the formula, Gaussian point Local neighborhood structure in three-dimensional space.

6. The three-dimensional Gaussian scene semantic segmentation method based on uncertainty-aware collaborative optimization according to claim 5, characterized in that, In step S5), the confidence-weighted contrastive loss function Represented as: ； In the formula, For pixels The confidence level of the semantic label; To reduce the impact of low-confidence samples on training, it can suppress the interference of noisy samples on feature learning, preserve the strong supervision signal of high-confidence regions, and allow semantic information to be transferred from high-confidence regions to low-confidence regions. Spatial weights are used to enhance the constraints between neighboring samples in space; Indicates cosine similarity; This indicates the mask to which the pixel belongs; For indicator functions; Represents the pixels after Gaussian feature field rendering Semantic features at the location.

7. A three-dimensional Gaussian scene semantic segmentation method based on uncertainty-aware collaborative optimization according to claim 6, characterized in that, In step S5), the semantic diffusion loss function based on mask constraints... Represented as: In the formula, Indicates the first One mask area; Indicates the first A neighborhood set of Gaussians.

8. A semantic segmentation method for 3D Gaussian scenes based on uncertainty-aware collaborative optimization according to claim 7, characterized in that, In step S6), the cosine similarity score between the point hint feature and all Gaussian point features is calculated. The expression is: ； In the formula, Indicates the features of the point prompt; Indicates cosine similarity; For the first Semantic feature vectors of Gaussian points.

9. A 3D Gaussian scene semantic segmentation system based on uncertainty-aware collaborative optimization, characterized in that, include: The Gaussian scene reconstruction module uses the 3D Gaussian splashing algorithm 3DGS to reconstruct a 3D scene from multiple view images, resulting in a 3D Gaussian scene represented by a set of Gaussian points. The mask segmentation module uses a two-dimensional segmentation model to segment multi-view Gaussian scene images, obtaining a set of two-dimensional segmentation masks. The maximum segmentation mask for each object is selected by comparing the overlapping areas of the masks. The pseudo-confidence label construction module uses the maximum segmentation mask to construct Gaussian pseudo-confidence labels; The uncertainty estimation module uses a lightweight neural network to predict the semantic label confidence of each Gaussian point and supervises it with the Gaussian pseudo-confidence label. The model training module constructs a model including a confidence-weighted contrastive loss function. and semantic diffusion loss function based on mask constraints The overall loss function To train a 3D Gaussian semantic segmentation model; The semantic segmentation module receives point hint features input by the user and calculates the cosine similarity score between the point hint features and all Gaussian point features. Based on the similarity score Implement interactive object segmentation in a 3D Gaussian scene; The visual editing module is used to render 3D Gaussian scenes in real time, providing scene editing operations such as segmentation, deletion, movement, and rotation.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the three-dimensional Gaussian scene semantic segmentation method as described in any one of claims 1-8.

Images