A method and system for scene segmentation and editing based on three-dimensional Gaussian sputtering

Abstract

This invention provides a scene segmentation and editing method and system based on 3D Gaussian sputtering. The method includes performing 2D instance segmentation on a multi-view 2D image of a 3D scene to obtain a 2D instance mask; initializing 3D reconstruction through 3D Gaussian sputtering to obtain a set of 3D Gaussian volumes; associating each Gaussian volume with cross-view consistent semantic features extracted by a pre-trained visual language model; constructing and iterating a 3D instance prototype library based on the 2D instance mask to obtain a cross-view consistent 2D supervision signal; adding a learnable instance identity code to each Gaussian volume; constructing a multi-dimensional loss function by combining the 2D supervision signal, spatial neighborhood relationship, and cross-view consistent semantic features; optimizing the learnable parameters containing the instance identity code; and grouping Gaussian volume instances based on the optimized instance identity code to complete instance-level scene segmentation and editing. This invention solves the technical problems of instance identity association and high-quality instance-level generation under sparse viewpoints and closely adjacent objects.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for scene segmentation and editing based on three-dimensional Gaussian sputtering. Background Technology

[0002] Scene segmentation is the cornerstone of understanding the 3D world, while scene editing, based on this understanding, allows for flexible and precise modification of the geometry, appearance, style, and other attributes of a 3D scene. Traditional methods suffer from high computational costs and limited detail rendering capabilities, making it difficult to meet the ever-increasing demands for processing speed and detail accuracy in recent years. Currently, 3D Gaussian Splatting (3DGS) has achieved significant breakthroughs in training efficiency and real-time rendering quality, thus becoming the mainstream technology for scene segmentation and editing tasks. However, 3DGS itself does not encode semantic information and cannot distinguish object instances, which limits its development in applications requiring instance-level manipulation, such as robot interaction, autonomous driving simulation, and virtual reality.

[0003] To imbue 3DGS feature representations with instance-level semantics, the core idea of ​​existing technologies is to introduce a learnable instance identity encoding for each 3D Gaussian particle, treating multi-view images as a continuous video sequence. A video object tracker is used to associate the encodings of the same object in different views, serving as a semantic supervision signal to automatically group different object instances in 3D space. However, this method heavily relies on the assumption that "viewpoint changes between input images are small and continuous." When processing real-world scenes with large viewpoint changes or sparse sampling, the tracker is prone to drift or failure, leading to incorrect instance identity associations, biased semantic supervision, and ultimately affecting the accuracy of scene segmentation masks, resulting in misalignment or incompleteness in subsequent scene editing. Furthermore, objects in real-world scenes are often closely adjacent, causing the same Gaussian volume to cover multiple objects, and a single object is easily assigned multiple instance labels, causing boundary adhesion. Under sparse views, tracker drift or failure exacerbates instance label aliasing, and the boundary adhesion problem remains unresolved. Summary of the Invention

[0004] Therefore, the purpose of this invention is to provide a scene segmentation and editing method and system based on three-dimensional Gaussian sputtering, which aims to solve the problem that there is no existing scene segmentation and editing method based on three-dimensional Gaussian sputtering that can correctly associate instance identities and ensure high-quality instance-level generation effects under sparse viewpoints and closely adjacent objects.

[0005] A scene segmentation and editing method based on three-dimensional Gaussian sputtering according to an embodiment of the present invention includes:

[0006] Multi-view of 3D scenes Figure 2 The multi-view image set performs two-dimensional instance segmentation to obtain a two-dimensional instance mask, and then performs three-dimensional Gaussian sputtering on the multi-view image set. Figure 2 The three-dimensional image set is initialized for 3D reconstruction to obtain a set of three-dimensional Gaussian volumes;

[0007] Based on the set of three-dimensional Gaussian volumes, each three-dimensional Gaussian volume is associated with cross-view data extracted from a pre-trained visual language model. Figure 1 Based on the semantic features, a three-dimensional instance prototype library is constructed and iteratively updated to be built based on the two-dimensional instance mask, so as to transform the two-dimensional instance mask into a cross-view object according to the three-dimensional instance prototype library. Figure 1 Two-dimensional monitoring signal;

[0008] A learnable instance identity code is added to each 3D Gaussian volume, and based on the 2D supervision signal, the spatial neighborhood relationship of the 3D Gaussian volume, and the cross-view... Figure 1 A multidimensional loss function is constructed based on semantic features to optimize learnable parameters, wherein the learnable parameters include at least instance identity encoding.

[0009] The similarity between Gaussian volumes is calculated based on the optimized instance identity encoding, and the instances of the 3D Gaussian volumes are grouped. The instance grouping results are then used to complete instance-level scene segmentation, local deletion, and repair.

[0010] In addition, the scene segmentation and editing method based on three-dimensional Gaussian sputtering according to the above embodiments of the present invention may also have the following additional technical features:

[0011] Furthermore, the multi-view image is processed using three-dimensional Gaussian sputtering technology. Figure 2 The steps for initializing a set of 3D images to perform 3D reconstruction and obtain a set of 3D Gaussian volumes include:

[0012] Capture 3D scenes using a camera with a sparse viewpoint to obtain multi-view data. Figure 2 The system generates a 2D image sequence and extracts and matches feature points from the 2D image sequence using a motion reconstruction structure algorithm to recover the camera pose and generate a sparse 3D point cloud.

[0013] Each point in the sparse 3D point cloud is initialized as a 3D Gaussian volume, and the parameters of the 3D Gaussian volume include the center point coordinates, covariance matrix, upper limit of opacity, and spherical harmonic coefficients.

[0014] The three-dimensional Gaussian volume is projected onto each two-dimensional image plane by differentiable rendering. The difference between the reconstructed image and the input real image is calculated as the reconstruction loss. The parameters of the three-dimensional Gaussian volume are iteratively optimized using an optimization algorithm until the loss converges, thus obtaining the set of three-dimensional Gaussian volumes.

[0015] Furthermore, based on the set of three-dimensional Gaussian volumes, each three-dimensional Gaussian volume is associated with cross-view data extracted by a pre-trained visual language model. Figure 1 The steps to generate semantic features include:

[0016] Obtain image patches of the projected region of each 3D Gaussian volume in all visible views;

[0017] Each projection region image patch is input into a pre-trained visual language model to extract multimodal embedding features, while the parameters of the visual language model remain fixed.

[0018] The aggregation operation combines multiple multimodal embedded features to form the cross-view of the three-dimensional Gaussian volume. Figure 1 To achieve semantic features.

[0019] Furthermore, a three-dimensional instance prototype library is constructed and iteratively updated based on the two-dimensional instance mask, so as to transform the two-dimensional instance mask into a cross-view object according to the three-dimensional instance prototype library. Figure 1 The steps for generating a two-dimensional monitoring signal include:

[0020] For each view in the two-dimensional image set, the depth information of the center point of the three-dimensional Gaussian volume with semantic features is statistically analyzed, and Gaussian volumes whose depth falls within a specified interval are selected to form a set of foreground three-dimensional Gaussian volumes.

[0021] Based on the two-dimensional instance mask of the first view, the foreground Gaussian volume set of the view is grouped according to the instance label, and a unique instance ID is assigned to each group to obtain the initial group set;

[0022] Traverse the remaining views, calculate the geometric correlation index between the current view's foreground Gaussian subset and the existing groups in the 3D embodiment prototype library, and determine whether to assign the subset to an existing group or create a new group based on the index value, so as to iteratively update the 3D embodiment prototype library;

[0023] Using the grouping information and instance IDs of the final 3D implementation prototype library, the instance labels of identical objects in the 2D instance masks of all views are uniformly updated to the instance IDs of their respective groups, thus obtaining cross-view... Figure 1 Two-dimensional monitoring signal.

[0024] Furthermore, the instance identity is encoded as a low-dimensional vector, initialized as a random vector, and dynamically adjusted during parameter optimization using a gradient descent algorithm. This vector is used to uniquely identify the instance identity of the three-dimensional Gaussian body, ensuring the consistency of the identity of the same instance from different perspectives.

[0025] Furthermore, based on the two-dimensional supervision signal, the spatial neighborhood relationship of the three-dimensional Gaussian volume, and the cross-view... Figure 1 The steps for constructing a multidimensional loss function based on semantic features include:

[0026] The instance identity is encoded and rendered onto a 2D view plane. The segmentation result is predicted by a linear classification layer, based on cross-view... Figure 1 The cross-entropy loss is calculated using the two-dimensional supervision signal to determine the two-dimensional group supervision loss;

[0027] Calculate the symmetric KL divergence of the identity code distributions of each 3D Gaussian volume and its neighboring Gaussian volumes, and constrain the consistency of identity codes of neighboring Gaussian volumes to determine the 3D neighborhood consistency regularization loss:

[0028] Instance identity encoding is mapped to the semantic space using a learnable mapping head, and the mapped encoding is calculated in relation to cross-view... Figure 1 Cosine similarity of semantic features is used to minimize cosine distance to constrain semantic consistency and determine semantic consistency regularization loss;

[0029] The multidimensional loss function is constructed based on the two-dimensional grouping supervision loss, the three-dimensional neighborhood consistency regularization loss, and the semantic consistency regularization loss.

[0030] Furthermore, the steps for completing instance-level scene segmentation, partial deletion, and repair based on the instance grouping results include:

[0031] Calculate the instance identity encoding similarity between any two 3D Gaussian volumes, construct an adjacency matrix and determine the connectivity between Gaussian volumes, and divide connected Gaussian volumes into the same instance group through connected component analysis;

[0032] Based on the instance-level scene segmentation results, the set of pixels corresponding to the instance to be deleted is located, and then the initial set of three-dimensional Gaussian bodies projected onto the set of pixels is determined. After outlier removal and region integrity processing, the final set of Gaussian bodies to be deleted is removed to complete the deletion.

[0033] The final set of Gaussian volumes to be deleted is projected onto a specified view to generate a deletion mask. Based on the repair loss function composed of reconstruction loss and perception loss, the Gaussian volume parameters corresponding to the repair area are optimized by gradient descent algorithm until the repaired content achieves visual coherence with the surrounding environment.

[0034] Another objective of this invention is to provide a scene segmentation and editing system based on three-dimensional Gaussian sputtering, for implementing the aforementioned scene segmentation and editing method based on three-dimensional Gaussian sputtering, the system comprising:

[0035] The data processing module is used for multi-view processing of 3D scenes. Figure 2 The multi-view image set performs two-dimensional instance segmentation to obtain a two-dimensional instance mask, and then performs three-dimensional Gaussian sputtering on the multi-view image set. Figure 2 The three-dimensional image set is initialized for 3D reconstruction to obtain a set of three-dimensional Gaussian volumes;

[0036] The prototype library building module is used to associate each 3D Gaussian volume with cross-view data extracted from a pre-trained visual language model, based on the set of 3D Gaussian volumes. Figure 1 Based on the semantic features, a three-dimensional instance prototype library is constructed and iteratively updated to be built based on the two-dimensional instance mask, so as to transform the two-dimensional instance mask into a cross-view object according to the three-dimensional instance prototype library. Figure 1 Two-dimensional monitoring signal;

[0037] The parameter optimization module is used to add a learnable instance identity code to each 3D Gaussian volume, and based on the 2D supervision signal, the spatial neighborhood relationship of the 3D Gaussian volume, and the cross-view... Figure 1 A multidimensional loss function is constructed based on semantic features to optimize learnable parameters, wherein the learnable parameters include at least instance identity encoding.

[0038] The scene processing module is used to calculate the similarity between Gaussian bodies based on the optimized instance identity encoding, realize the instance grouping of 3D Gaussian bodies, and complete instance-level scene segmentation, local deletion and repair based on the instance grouping results.

[0039] Another objective of this invention is to provide a storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described scene segmentation and editing method based on three-dimensional Gaussian sputtering.

[0040] Another objective of this invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described scene segmentation and editing method based on three-dimensional Gaussian sputtering.

[0041] This invention constructs a 3D instance prototype library and uses iterative matching based on geometric and semantic similarity to associate a unified set of 3D Gaussian volumes for the same instance from different perspectives, generating cross-viewpoint... Figure 1 The consistent monitoring signal ensures the uniqueness and stability of the same instance identity label, avoiding the label drift problem caused by tracking failures in existing methods. This is achieved by introducing cross-view monitoring of instance identities. Figure 1By aligning consistency constraints with multimodal semantic features, the representation of 3D Gaussian volume integrates geometric, appearance, and high-level semantic information, enabling more effective differentiation of geometrically adjacent but semantically distinct object instances and significantly alleviating boundary adhesion problems. Based on the optimized instance identity encoding, the 3D Gaussian volume set can be grouped into object-level instances, allowing users to directly manipulate the grouped 3D Gaussian volumes to perform local instance-level deletion, repair, and other operations, expanding its application value in virtual reality, film and television post-production, digital twins, and other fields. Therefore, this invention solves the problem in the prior art of lacking a scene segmentation and editing method based on 3D Gaussian sputtering that can correctly associate instance identities and ensure high-quality instance-level generation effects under sparse viewpoints and closely adjacent object conditions. Attached Figure Description

[0042] Figure 1 This is a flowchart of the scene segmentation and editing method based on three-dimensional Gaussian sputtering in the first embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the results of the scene segmentation and editing system based on three-dimensional Gaussian sputtering in the second embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the structure of the electronic device in the third embodiment of the present invention;

[0045] Figure 4 This is a two-dimensional image under a specified view in one embodiment of the present invention;

[0046] Figure 5 for Figure 4 A schematic diagram of the masking effect of a 2D image instance after segmentation by a 2D segmentation model;

[0047] Figure 6 for Figure 4 A schematic diagram illustrating the effect of rendering identity encoding onto a two-dimensional view plane;

[0048] Figure 7 for Figure 4 A schematic diagram illustrating the effect of deletion operation performed using the method of the present invention;

[0049] Figure 8 for Figure 7 A schematic diagram illustrating the effect of repair operation performed using the method of this invention;

[0050] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0051] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0053] Example 1

[0054] Please see Figure 1 The figure shows a scene segmentation and editing method based on three-dimensional Gaussian sputtering in the first embodiment of the present invention, which specifically includes steps S01-S04.

[0055] S01, Multi-view of 3D Scenes Figure 2 The multi-view image set performs two-dimensional instance segmentation to obtain a two-dimensional instance mask, and then performs three-dimensional Gaussian sputtering on the multi-view image set. Figure 2 The three-dimensional image set is initialized for three-dimensional reconstruction to obtain a three-dimensional Gaussian volume set.

[0056] Specifically, this involves capturing 3D scenes using a camera at sparse viewpoints to obtain multi-view data. Figure 2 A 2D image sequence is generated, and feature points are extracted and matched from the 2D image sequence using a motion reconstruction algorithm to recover the camera pose and generate a sparse 3D point cloud. Each point in the sparse 3D point cloud is initialized as a 3D Gaussian volume, and the parameters of the 3D Gaussian volume include the center point coordinates, covariance matrix, upper limit of opacity, and spherical harmonic coefficients. The 3D Gaussian volume is projected onto each 2D image plane through differentiable rendering, and the difference between the reconstructed image and the input real image is calculated as the reconstruction loss. An optimization algorithm is used to iteratively optimize the parameters of the 3D Gaussian volume until the loss converges, resulting in the set of 3D Gaussian volumes.

[0057] More specifically, in practical implementation, firstly, a multi-view image sequence of the 3D scene is acquired. These images can be obtained by taking pictures from different angles with a camera, ensuring coverage of multiple perspectives of the scene to provide sufficient geometric information. Secondly, in one embodiment of the invention, these images are processed using the Structure for Motion Recovery (SfM) algorithm. Specifically, feature points are extracted from the images using feature extraction and matching techniques (such as SIFT or ORB algorithms), and the correspondence between feature points in different images is found. Subsequently, these correspondences are used to calculate the camera pose and sparse point cloud. Next, based on the generated initial sparse point cloud, each 3D point in the point cloud is expanded into a 3D Gaussian volume (the 3D point is denoted as ). Three-dimensional Gaussian solid is denoted as ), to initialize a set of three-dimensional Gaussian volumes, denoted as:

[0058]

[0059] Among them, the superscript " "Indicates the initial stage (0 iterations), subscript Indicates the first There are Gaussian solids, where N represents the total number of three-dimensional Gaussian solids, and the parameter... Let G be the coordinates of the center point of the three-dimensional Gaussian solid in three-dimensional space, with initial values... for The coordinates are then iteratively updated; parameters The learnable covariance matrix represents a 3D Gaussian volume, describing the shape, size, and orientation of the Gaussian volume in 3D space; The learnable upper bound of opacity for each 3D Gaussian volume is determined by learnable parameters. via the Sigmoid function Mapped to obtain; The learnable, spherical harmonic-based color and radiation characteristics are represented and computed through the expansion of the spherical harmonic function (highest order L):

[0060]

[0061] in Let be the spherical harmonic basis functions, l be the order of the spherical harmonic function (L is the maximum order), and m be the degree of the spherical harmonic function. and The viewing angle is the direction angle. For the first Learnable spherical harmonic coefficients of a Gaussian body.

[0062] Furthermore, based on parameters and Definition of the first Distribution of Gaussian bodies in space:

[0063] ,

[0064] in, It is a certain one in space, and T represents the transpose of a vector.

[0065] Based on distribution and parameters Define a Gaussian body at a point Opacity at:

[0066] .

[0067] For the Gaussian volumes in the coverage set, sort them by depth from near to far, and then use forward blending to merge their color values ​​and opacity to obtain the points. Color of the location:

[0068] ,

[0069] in, Represents all points The set of indices of intersecting Gaussian bodies; This represents the cumulative product of the transparency of the preceding Gaussian volume to ensure correct occlusion relationships.

[0070] In subsequent rendering processes, by adjusting these parameters, the contribution of each Gaussian volume to the final image can be optimized, thereby improving the visual effect of the rendering. Specifically, differentiable rendering projects the 3D Gaussian volume onto the 2D image plane of each acquired viewpoint, and the difference between the reconstructed image and the input real image is calculated as the reconstruction loss; the Adam optimization algorithm is used to iteratively optimize the parameters of each 3D Gaussian volume until the loss converges. Furthermore, in the... During rounds of iteration:

[0071] (1) Using the projection function of the camera model about the kth viewpoint Calculate the coordinate parameters of the center point of a three-dimensional Gaussian solid. Projection position in the two-dimensional pixel plane:

[0072] ,

[0073] Among them, the superscript " " indicates iteration" Second-rate, For projection functions;

[0074] (2) Calculate the two-dimensional covariance after projection. First, calculate the projection function. right Jacobian matrix Let the transformation matrix that transforms the parameters of the 3D Gaussian volume in the world coordinate system to the k-th camera view be . First, the covariance parameters of the three-dimensional Gaussian body are... Convert to:

[0075]

[0076] Take its top-left 2×2 submatrix as the two-dimensional covariance after projection:

[0077]

[0078] The subscript "1:2,1:2" indicates that the first and second rows and the first and second columns of the matrix are extracted.

[0079] (3) Calculate the number of seconds. Pixels in a two-dimensional image plane from a single perspective Opacity .based on The definition formula can be derived by analogy:

[0080]

[0081] in, It is iteration Next time point Two-dimensional Gaussian kernel at the location;

[0082] (4) Calculate the number of pixels in the two-dimensional image plane. The color of the location. Similarly, based on The definition formula can be derived by analogy:

[0083]

[0084] (5) Based on each pixel color Reconstruct the scene of a specified view in a two-dimensional image plane and calculate the reconstruction loss. For the... Images from different perspectives Let its corresponding reconstructed image be In one embodiment of the present invention, the loss function is defined as:

[0085] ,

[0086] in, This represents the total number of pixels in all view images;

[0087] (6) Calculate using the backpropagation algorithm about The gradient is calculated, and the four parameters are updated using the Adam optimization algorithm;

[0088] (7) Order Then repeat process (1)-(6) until the loss converges. Let the first step be the first step. If convergence occurs in the nth iteration, the final set of three-dimensional Gaussian volumes is: .

[0089] Furthermore, for each image in the set of 2D images of the 3D scene, an open-world, zero-shot 2D image instance segmentation model (EntitySeg, etc.) is used to segment each image, obtaining a corresponding 2D instance mask. This mask only identifies different individuals and does not distinguish semantic categories. For the k-th 2D view... The corresponding two-dimensional instance mask is:

[0090] ,

[0091] in, This represents the open-world zero-sample two-dimensional image instance segmentation model.

[0092] In one embodiment of the present invention, given a two-dimensional image from a certain viewpoint, the segmentation model performs instance segmentation on it as follows: Figure 4 and Figure 5 As shown.

[0093] Because segmentation is performed independently on each view, the masks are inconsistent between different views. This problem is addressed in subsequent step S02.

[0094] S02, based on the set of three-dimensional Gaussian volumes, associate each three-dimensional Gaussian volume with cross-view data extracted from a pre-trained visual language model. Figure 1 Based on the semantic features, a three-dimensional instance prototype library is constructed and iteratively updated to be built based on the two-dimensional instance mask, so as to transform the two-dimensional instance mask into a cross-view object according to the three-dimensional instance prototype library. Figure 1 Two-dimensional monitoring signal.

[0095] Specifically, image patches of the projected region of each 3D Gaussian volume in all visible views are obtained; each projected region image patch is input into a pre-trained visual language model to extract multimodal embedding features, while the parameters of the visual language model remain fixed; an aggregation operation is performed to combine multiple multimodal embedding features to form the cross-view image of the 3D Gaussian volume. Figure 1 To achieve semantic features.

[0096] In practice, firstly, the projection region position of each 3D Gaussian volume in all its visible 2D views is obtained, and the image patch corresponding to that region position is extracted. For the nth 3D Gaussian volume, let its position in the kth 2D view be... The location of the projection area in the middle is The corresponding two-dimensional image block is:

[0097] ,

[0098] Among them, crop represents the function of intercepting an image patch. In one embodiment of the present invention, can be a quadruple is the upper left corner coordinate of the projection area position, and respectively represent the length and width of the area;

[0099] Secondly, each projected image patch is fed into a pre-trained language-vision model to extract multi-modal embedding features. In one embodiment of the present invention, CLIP is selected as the pre-trained language-vision model, and the corresponding features can be expressed as:

[0100] ,

[0101] Among them, represents the pre-trained language-vision model, and the model parameters are fixed and not updated throughout the process;

[0102] Finally, obtain cross-view Figure 1 consistent semantic representations. Preferably, the multi-modal embedding features of all view projection areas belonging to the same three-dimensional Gaussian volume are averaged to obtain the cross-view Figure 1 consistent semantic representation of the three-dimensional Gaussian volume:

[0103] ,

[0104] Among them, represents the total number of all visible two-dimensional views of the three-dimensional Gaussian volume.

[0105] After associating the cross-view Figure 1 consistent semantic representations, the three-dimensional Gaussian volume set is updated to:

[0106] .

[0107] Furthermore, depth information of the center points of 3D Gaussian volumes with semantic features is statistically analyzed for each view within the 2D image set, and Gaussian volumes whose depth falls within a specified interval are selected to form a foreground 3D Gaussian volume set. Based on the 2D instance mask of the first view, the foreground Gaussian volume set of that view is grouped according to instance labels, and a unique instance ID is assigned to each group to obtain an initial group set. The remaining views are traversed, and the geometric correlation index between the foreground Gaussian volume subset of the current view and the existing groups in the 3D embodiment prototype library is calculated. Based on the index value, the subset is assigned to an existing group or a new group to iteratively update the 3D embodiment prototype library. Using the grouping information and instance IDs of the final 3D embodiment prototype library, the instance labels of the same objects in the 2D instance masks of all views are uniformly updated to the instance IDs of their respective groups to obtain cross-view results. Figure 1 The two-dimensional monitoring signal is obtained.

[0108] In practical implementation, during the preprocessing stage, the depth information of the center point of the 3D Gaussian volume with semantic features in each view within the 2D image set is statistically analyzed. Gaussian volumes whose center point depth falls within a specified interval are then selected to form a foreground 3D Gaussian volume set. For the first... The first view For each instance (marked by a 2D instance mask), the set of depth values ​​projected onto the center point of the 3D Gaussian volume within that instance is denoted as . The corresponding depth range is denoted as In one embodiment of the present invention, the following is defined:

[0109] ,

[0110] in, Represents a set The first quartile, Represents a set The third quartile; and They are sets The minimum and maximum depth values ​​in the range; This is the outlier factor, which defaults to 1.0. This represents the interquartile range, used to remove outliers.

[0111] For any Gaussian body If its center point is located In the The projection of the view belongs to the first view. There are 10 instances, and their depth values ​​fall within the range 10. Then it is believed It is a foreground Gaussian body. After traversing the set... After processing all Gaussian bodies, we obtain the first... The set of foreground Gaussian volumes of all instances of a view, denoted as .in, This represents the instance mask corresponding to the view. The number of instances of the marked object. Indicates that it belongs to the first in this view A subset of the foreground Gaussian body of an instance.

[0112] During the initialization phase, based on the first view Corresponding two-dimensional instance mask According to its markings each The corresponding instance ID is used as a unique identifier. Then, the collection of foreground 3D Gaussian volumes corresponding to this view is traversed. Each three-dimensional Gaussian body ,like object instances (i.e.) ), Assigned to group After the traversal is complete, the initial group set is obtained. That is, the initial 3D instance prototype library, where the superscript " " indicates the initial stage (0 iterations).

[0113] During the iterative update phase, based on grouping rules of geometric overlap and semantic similarity, the assignment of the foreground 3D Gaussian volume outside the group is continuously performed until all remaining views (except the first view) have been traversed. Specifically, in the... In the round of iteration, process the first... A view The process includes:

[0114] (1) Calculation The corresponding foreground Gaussian set Neutron set Compared with each group in the previous update results The geometric correlation index between them is determined by both geometric overlap and semantic similarity, and calculated as a weighted sum. In one embodiment of the invention, geometric overlap is defined as:

[0115] , when and hour,

[0116] in, The intersection operation represents the set operation. Indicates the number of elements in the set;

[0117] Semantic similarity is defined as:

[0118] , when and hour,

[0119] in, The cosine similarity between two vectors is represented by the following expression: The L2 norm of a vector; Represents the foreground Gaussian set The corresponding semantic representation is preferably defined as the cross-view relation of all Gaussian volumes. Figure 1 The average of the semantic representations of the same type, i.e.:

[0120] , when and hour;

[0121] Similarly, This indicates the previous round of grouping. The corresponding semantic representation is preferably defined as the average of the semantic representations of all Gaussian body associations, i.e.:

[0122] ;

[0123] The geometric correlation index is defined as:

[0124] ,

[0125] in, and All are weighted coefficients;

[0126] (2) Update the group set. For the current view, update the group set... Foreground Gaussian subset of an instance Its geometric correlation index set Take the maximum value from the list and record the corresponding group ID, denoted as . .if Then Assigned to the previous round Groups And update the following two items:

[0127] The first round Groups, ,

[0128] Grouping Corresponding semantic representation ,

[0129] Among them, the update coefficient The preferred value is 0.05;

[0130] if ,but:

[0131] Create a new group ,

[0132] Update new groups Corresponding semantic representation ,

[0133] Update the total number of groups (i.e., the total number of instances). ;

[0134] Among them, threshold The preferred value is 0.1. Since the calculation of geometric overlap is based on... Normalization is performed based on a benchmark, so it is not affected by the size of existing groups in the prototype library, thus maintaining the threshold in different scenarios. Consistency and stability in settings.

[0135] To avoid duplicate counting, it is stipulated that the same Gaussian body can only belong to one group, which ensures that the instance identifier is unique and does not drift.

[0136] (3) Let Then repeat process (1)-(2) until all remaining views have been traversed. Let the first view be the first view. The iteration ends at the nth iteration, and the final grouping is: For each group Calculate the corresponding semantic representation And use it as the 3D instance prototype corresponding to that group; finally, combine all of them. This yields the final three-dimensional instance prototype library.

[0137] Using the grouping information and corresponding instance IDs from the final instance prototype library, the instance labels of identical objects in the 2D instance masks of all views are uniformly updated to the instance IDs of their respective groups, resulting in 2D instance masks with consistent identity across views. Specifically, for each group... Calculate the projection of all 3D Gaussian solids onto the first... The position coordinates of the two-dimensional image plane of each view are determined, and the instance label of the pixel at the coordinate is set to the group containing the corresponding Gaussian volume. The instance ID is used to obtain the first Two-dimensional instance mask of a view Similarly, this mask only identifies different individuals and does not distinguish semantic categories. Since each group's instance ID is unique, in different views, as long as the 3D Gaussian volume corresponding to a pixel belongs to the same group... Their instance tags are the same, thus ensuring identity consistency across views. As the first Two-dimensional pixel-level supervision signals of view segmentation results can constrain cross-view Figure 1 To the point of being responsive.

[0138] S03, add a learnable instance identity code to each 3D Gaussian volume, and based on the 2D supervision signal, the spatial neighborhood relationship of the 3D Gaussian volume, and the cross-view... Figure 1 A multidimensional loss function is constructed based on semantic features to optimize learnable parameters, wherein the learnable parameters include at least instance identity encoding.

[0139] Specifically, the instance identity is encoded as a low-dimensional vector, initialized as a random vector, and dynamically adjusted during parameter optimization using a gradient descent algorithm. This vector is used to uniquely identify the instance identity of the three-dimensional Gaussian body, ensuring the consistency of the identity of the same instance from different perspectives.

[0140] Specifically, instance identity encoding It is a low-dimensional vector that is dynamically optimized during subsequent training, used to uniquely identify the instance of the Gaussian body and ensure its consistency across different viewpoints; at the beginning of training, It is initialized as a random vector, denoted as .

[0141] Add learnable instance identity coding Afterwards, a set of three-dimensional Gaussian volumes Updated to:

[0142]

[0143] Among them, the superscript " " indicates the initial stage (0 iterations).

[0144] Furthermore, based on the two-dimensional supervision signal, the spatial neighborhood relationship of the three-dimensional Gaussian volume, and the cross-view... Figure 1 The steps for constructing a multidimensional loss function based on semantic features include: encoding and rendering instance identities onto a two-dimensional view plane, predicting segmentation results through a linear classification layer, and then performing cross-view... Figure 1 The cross-entropy loss is calculated using the consistent 2D supervisory signal to determine the 2D grouping supervisory loss; the symmetric KL (Kullback-Leibler, KL) divergence of the identity encoding distribution of each 3D Gaussian body and its neighboring Gaussian bodies is calculated, and the identity encoding consistency of neighboring Gaussian bodies is constrained to determine the 3D neighborhood consistency regularization loss; the instance identity encoding is mapped to the semantic space through a learnable mapping head, and the mapped encoding and cross-view... Figure 1The cosine similarity of semantic features is used to minimize the cosine distance to constrain semantic consistency and determine the semantic consistency regularization loss; and the multidimensional loss function is constructed based on the two-dimensional grouping supervision loss, the three-dimensional neighborhood consistency regularization loss, and the semantic consistency regularization loss.

[0145] In practical implementation, during the training iteration process, the total loss function is first constructed and calculated. Then, based on backpropagation and the Adam optimization algorithm, the learnable instance identity code and other learnable parameters of each 3D Gaussian volume are updated end-to-end. Training ends when the preset convergence condition is met. During round iteration:

[0146] (1) Calculate the two-dimensional grouped supervision loss First, use forward... Hybrid technology will enable learnable instance identity encoding Render to the A two-dimensional image plane from multiple perspectives is used to obtain the midpoint of the plane. Rendering characteristics at the location:

[0147]

[0148] Among them, the superscript " " indicates s iterations; To cover pixels from near to far according to depth The set of indices of the Gaussian body; Point Learnable opacity, when hour That is, the final product generated in step S01 ,when hour Iterative updates; This represents the cumulative product of the transparency of the preceding Gaussian volume to ensure correct occlusion relationships;

[0149] After all pixels have been rendered, the result is... The The rendering feature map from each perspective produces a rendering effect as follows: Figure 6 As shown.

[0150] Next, the two-dimensional feature map Each pixel is fed into a linear classification layer to predict its value. The probability vector of the segmentation result:

[0151] ,

[0152] in, and This represents the learnable weights and bias parameters of the linear classification layer. This indicates that Softmax normalization is performed on the feature vector along the instance channel. The dimension is equal to the total number of object instances in this view;

[0153] Furthermore, regarding the cross-viewpoint of the kth perspective... Figure 1 To two-dimensional pixel-level supervision signal Convert the instance label corresponding to each pixel into a one-hot label vector, where the vector dimension equals the total number of instances. Let... If the number of instances is C, then medium pixel The label vector is denoted as ,satisfy:

[0154] ,

[0155] in, express The union of all pixels in the set; If and only if point Belongs to the One instance, otherwise .

[0156] Finally, calculate the prediction vector. With label vector The cross-entropy is used to constrain the learnable instance identity encoding. Cross-view identity consistency:

[0157]

[0158] in, This represents the union of all view indices.

[0159] (2) Calculate the three-dimensional neighborhood consistency regularization loss First, from a set of three-dimensional Gaussian volumes with learnable instance identity encodings... Sample a subset from the middle, denoted as Then, for each three-dimensional Gaussian volume in the subset... Searching for it in three-dimensional space There are 10 neighbors, and the set of neighbors is denoted as _ . Then, calculate With each of his neighbors The KL divergence of the identity coding distribution is used to constrain the consistency of the identity coding distribution of adjacent Gaussian volumes in three-dimensional space, preventing identity drift.

[0160]

[0161] in, Represents a set Index of a three-dimensional Gaussian body Represents probability distribution and The KL divergence is used to measure the degree of difference between two probability distributions. This indicates that the instance identity code of the input 3D Gaussian solid is extracted first. Then, the Softmax function is used to map the instance identity encoding into a probability distribution.

[0162] (3) Calculate the semantic consistency regularization loss First, by using a learnable mapping head... Learnable instance identity encoding Mapping to semantic space:

[0163]

[0164] Secondly, normalization The predicted semantic vector is obtained:

[0165]

[0166] in, The L2 norm of a vector;

[0167] At the same time, normalize in the same way. Cross-view corresponding to Gaussian body association Figure 1 semantic representation of :

[0168]

[0169] Finally, calculate the prediction vector. With semantic representation Cosine similarity is used to constrain instance identity encoding to maintain semantic consistency with CLIP semantic features:

[0170]

[0171] in, It is a calculation A subset of samples taken at the time. This represents the cosine similarity between two vectors.

[0172] (4) Calculate the total loss function In one embodiment of the present invention, the total loss function is defined as:

[0173]

[0174] in, and These are weighting parameters used to balance the contributions of each part of the loss.

[0175] (5) Calculate the total loss function using the backpropagation algorithm. Regarding learnable parameters gradient ,in You can then learn instance identity coding. This represents the total number of three-dimensional Gaussian solids; The learnable upper limit parameter for opacity is used for calculation. Needed ,in It is a point The coordinates, the final generated in step S01 Therefore, it will not be updated; and That is, calculation Learnable weights and bias parameters of the linear classification layer at that time; That is, calculation Learnable mapping headers at that time.

[0176] (6) The gradient obtained based on process (5) The Adam optimization algorithm is used to update the learnable parameters. :

[0177]

[0178] in, This means that the Adam optimization algorithm is used to calculate new values ​​for all learnable parameters. The learning rate is preferably 0.001.

[0179] (7) Order Then repeat process (1)-(6) until the iteration stopping condition is met. Let the first step be the first step. After the iteration stops, the set of 3D Gaussian volumes with learnable instance identity codes is updated to:

[0180]

[0181] use When reconstructing a two-dimensional image from all three-dimensional Gaussian volumes, based on the formula defined in step S01, the first... A three-dimensional Gaussian body In the Pixels in each view The opacity at this location is:

[0182]

[0183] in, It is the number of iterations at the end of step S01;

[0184] point The color at this location is:

[0185] .

[0186] S04. Calculate the similarity between Gaussian volumes based on the optimized instance identity encoding to achieve instance grouping of 3D Gaussian volumes, and complete instance-level scene segmentation, local deletion, and repair based on the instance grouping results.

[0187] Specifically, the similarity of instance identity codes between any two 3D Gaussian volumes is calculated, an adjacency matrix is ​​constructed, and the connectivity between Gaussian volumes is determined. Connected Gaussian volumes are grouped into the same instance group through connected component analysis. Based on the instance-level scene segmentation results, the set of pixels corresponding to the instance to be deleted is located, and then the initial set of 3D Gaussian volumes projected onto this set of pixels is determined. After outlier removal and region integrity processing, the final set of Gaussian volumes to be deleted is removed to complete the deletion. The final set of Gaussian volumes to be deleted is projected onto a specified view to generate a deletion mask. Based on the repair loss function composed of reconstruction loss and perception loss, the parameters of the Gaussian volume corresponding to the repair area are optimized through gradient descent algorithm until the repaired content achieves visual coherence with the surrounding environment.

[0188] In practice, the first step is to group the learned instance information:

[0189] (1) For the final generated set of three-dimensional Gaussian volumes with learnable instance identity encoding Any two Gaussian bodies and Define the similarity between the two:

[0190]

[0191] in, It is a Gaussian body Instance identity encoding, It is a Gaussian body Instance identity encoding;

[0192] (2) Construct the adjacency matrix:

[0193]

[0194] Among them, the subscript " " represents a matrix The line, number List, and ; Indicates an indicator function, The threshold is set to 0.7; if the condition is met, the return value is 1, indicating that the two Gaussian bodies are "connected" and belong to the same instance; otherwise, the return value is 0.

[0195] (3) Grouping by connected components:

[0196]

[0197] in, This represents the connected component extraction operator, which, based on the connectivity relations in the adjacency matrix, marks connected Gaussian volumes with the same connected component label ID, indicating that they belong to the same instance, thus obtaining... Grouping of instances .

[0198] Then, based on the grouping, the subsequent instance-level scene segmentation operation is completed:

[0199] (1) Based on grouping Assign labels to all Gaussian bodies, that is, for each group Set the label of all Gaussian bodies in the group to 0. ;

[0200] (2) The projection of the center point of the three-dimensional Gaussian solid onto the 3rd... Position coordinates of the two-dimensional image plane of each view (Defined and finally updated in step S01), the pixel corresponding to this position. Set the instance label to the label of the 3D Gaussian body:

[0201]

[0202] in, It is the first Groups The first in A Gaussian body.

[0203] After traversing all the center points of the three-dimensional Gaussian solid, we obtain the first... Initial scene segmentation mask for a 2D image of a view (only marking different object instances, without distinguishing the semantic category of the instances);

[0204] (3) The parameters finally learned Substitution The calculation formula yields the first... Pixels of a two-dimensional image in a view Predicted probability vector ;

[0205] (4) Obtain the initial instance mask of the view at the pixel point Instance tags, denoted as Take pixels Prediction as an example probability ,like , If it is a threshold, then let ;otherwise, ;

[0206] (5) After traversing the first... Two-dimensional image of a view After analyzing all pixels, the final scene segmentation mask for that view is obtained. (Only different object instances are marked, without distinguishing the semantic category of the instances).

[0207] Based on the Final scene segmentation mask for each view Delete the specified object instance :

[0208] (1) In Find the tag as pixel set ;

[0209] (2) For pixels Find the projection at the 1st Points in a view The location of the center point of a 3D Gaussian solid; a set consisting of all 3D Gaussian solid center points that meet the requirements. ;

[0210] (3) Use the interquartile range technique based on the outlier tolerance coefficient to detect and remove outliers from the set. Outliers in the subset are obtained. ;

[0211] (4) Further process subsets using the three-dimensional convex hull operator. This yields the final set of three-dimensional Gaussian bodies to be deleted. ;

[0212] (5) From the final updated set of three-dimensional Gaussian volumes In the middle, delete a set The target Gaussian body included:

[0213]

[0214] (6) Using the subset after deletion All parameters of the Gaussian body, given the required viewpoint. Calculation points The color of the place Rebuild the instance after deleting the specified instance. Two-dimensional images from different perspectives The effect is as follows Figure 7 As shown.

[0215] Repair deleted areas:

[0216] (1) Following the projection operation described in step S01, the resulting set of three-dimensional Gaussian bodies to be deleted is... The three-dimensional Gaussian volume is projected onto the first... Given a 2D image plane with multiple viewpoints, find the pixels to be deleted in the plane and generate a deletion mask accordingly. .in, Point Deleted.

[0217] (2) Calculation Minimum bounding box of the region with a mean of 1 In one embodiment of the present invention, the function The return value can be a quadruple. It is the coordinate of the top-left corner of the smallest bounding box. and These represent the length and width of the minimum bounding box, respectively.

[0218] (3) Based on the first Images to be repaired from various perspectives Compared with the original image collected in step S01 Calculate repair losses The aim is to ensure that the area outside the deleted region remains consistent with the original image, making the restored content visually consistent with the surrounding environment. In one embodiment of the invention, [the following is implied:] Defined as:

[0219] ,

[0220] in, A collection of background pixels. Original image The collection of all pixels for The set of non-zero pixels, To fall within the minimum bounding box The set of pixels within; It is an L1 norm. For the perceptual loss function; This indicates that pixels in a specified image are extracted. The intensity value at that location;

[0221] Calculated using the backpropagation algorithm Gradient of learnable parameters of a 3D Gaussian volume The gradient descent algorithm is used to optimize the parameters of the three-dimensional Gaussian volume that falls within a specified region in small steps:

[0222]

[0223] in, This represents the learning rate (step size), preferably 0.001;

[0224] (5) Repeat process (3)-(4) until the iteration stops. Based on the three-dimensional Gaussian volume parameters learned at the time of iteration stop, project these three-dimensional Gaussian volumes onto the first Gaussian volume according to the projection operation described in step S01. Two-dimensional image plane from multiple perspectives, reconstructed and restored two-dimensional image The effect is as follows Figure 8 As shown.

[0225] In summary, the scene segmentation and editing method based on 3D Gaussian sputtering in the above embodiments of the present invention, by constructing a 3D instance prototype library and iteratively matching based on geometric and semantic similarity, associates a unified set of 3D Gaussian volumes for the same instance from different viewpoints, generating cross-view... Figure 1 The consistent monitoring signal ensures the uniqueness and stability of the same instance identity label, avoiding the label drift problem caused by tracking failures in existing methods. This is achieved by introducing cross-view monitoring of instance identities. Figure 1 By aligning consistency constraints with multimodal semantic features, the representation of 3D Gaussian volume integrates geometric, appearance, and high-level semantic information, enabling more effective differentiation of geometrically adjacent but semantically distinct object instances and significantly alleviating boundary adhesion problems. Based on the optimized instance identity encoding, the 3D Gaussian volume set can be grouped into object-level instances, allowing users to directly manipulate the grouped 3D Gaussian volumes to perform local instance-level deletion, repair, and other operations, expanding its application value in virtual reality, film and television post-production, digital twins, and other fields. Therefore, this invention solves the problem in the prior art of lacking a scene segmentation and editing method based on 3D Gaussian sputtering that can correctly associate instance identities and ensure high-quality instance-level generation effects under sparse viewpoints and closely adjacent object conditions.

[0226] Example 2

[0227] Please see Figure 2 The diagram shows the structural block diagram of the scene segmentation and editing system based on three-dimensional Gaussian sputtering proposed in the second embodiment of the present invention. The scene segmentation and editing system 200 based on three-dimensional Gaussian sputtering includes: a data processing module 21, a prototype library construction module 22, a parameter optimization module 23, and a scene processing module 24, wherein:

[0228] Data processing module 21 is used for multi-view processing of 3D scenes. Figure 2The multi-view image set performs two-dimensional instance segmentation to obtain a two-dimensional instance mask, and then performs three-dimensional Gaussian sputtering on the multi-view image set. Figure 2 The three-dimensional image set is initialized for 3D reconstruction to obtain a set of three-dimensional Gaussian volumes;

[0229] Prototype library construction module 22 is used to associate each 3D Gaussian volume with cross-view data extracted by a pre-trained visual language model based on the set of 3D Gaussian volumes. Figure 1 Based on the semantic features, a three-dimensional instance prototype library is constructed and iteratively updated to be built based on the two-dimensional instance mask, so as to transform the two-dimensional instance mask into a cross-view object according to the three-dimensional instance prototype library. Figure 1 Two-dimensional monitoring signal;

[0230] The parameter optimization module 23 is used to add a learnable instance identity code to each 3D Gaussian volume, and based on the 2D supervision signal, the spatial neighborhood relationship of the 3D Gaussian volume, and the cross-view... Figure 1 A multidimensional loss function is constructed based on semantic features to optimize learnable parameters, wherein the learnable parameters include at least instance identity encoding.

[0231] The scene processing module 24 is used to calculate the similarity between Gaussian bodies based on the optimized instance identity encoding, realize the instance grouping of 3D Gaussian bodies, and complete instance-level scene segmentation, local deletion and repair based on the instance grouping results.

[0232] Example 3

[0233] In another aspect, the present invention also proposes an electronic device, please refer to [link to relevant documentation]. Figure 3 The diagram shows an electronic device according to the third embodiment of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, it implements the scene segmentation and editing method based on three-dimensional Gaussian sputtering as described above.

[0234] In some embodiments, the processor 10 may be a central processing unit (CPU), controller, microcontroller, microprocessor or other data processing chip, used to run program code stored in memory 20 or process data, such as executing access restriction programs.

[0235] The memory 20 includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 20 can be an internal storage unit of an electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 20 can include both internal and external storage units of the electronic device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or will be output.

[0236] It should be pointed out that, Figure 3 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0237] This invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the scene segmentation and editing method based on three-dimensional Gaussian sputtering as described above.

[0238] Those skilled in the art will understand that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0239] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0240] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0241] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0242] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A scene segmentation and editing method based on three-dimensional Gaussian sputtering, characterized in that, The method includes: Two-dimensional instance segmentation is performed on a multi-view two-dimensional image set of a three-dimensional scene to obtain a two-dimensional instance mask, and three-dimensional reconstruction is performed on the multi-view two-dimensional image set using three-dimensional Gaussian sputtering technology to obtain a three-dimensional Gaussian volume set; Based on the set of 3D Gaussian volumes, each 3D Gaussian volume is associated with cross-view consistent semantic features extracted by a pre-trained visual language model. A 3D instance prototype library is constructed and iteratively updated based on the 2D instance mask, so that the 2D instance mask is transformed into a cross-view consistent 2D supervision signal according to the 3D instance prototype library. This includes statistically analyzing the depth information of the center points of 3D Gaussian volumes with semantic features for each view within the 2D image set, and selecting Gaussian volumes whose depth falls within a specified interval to form a foreground 3D Gaussian volume set. Based on the 2D instance mask of the first view, the foreground Gaussian volume of that view is determined by instance label. The Gaussian body set is grouped, and a unique instance ID is assigned to each group to obtain an initial group set. The remaining views are traversed, and the geometric association index between the foreground Gaussian body subset of the current view and the existing groups of the 3D instance prototype library is calculated. Based on the index value, the subset is assigned to an existing group or a new group to iteratively update the 3D instance prototype library. Using the grouping information and instance ID of the final 3D instance prototype library, the instance labels of the same objects in the 2D instance masks of all views are uniformly updated to the instance ID of the group to which they belong, resulting in a consistent 2D supervision signal across views. The geometric association index is determined by both geometric overlap and semantic similarity. A learnable instance identity code is added to each 3D Gaussian volume, and a multidimensional loss function is constructed based on the 2D supervision signal, the spatial neighborhood relationship of the 3D Gaussian volume, and the cross-view consistent semantic features. The learnable parameters are optimized based on the multidimensional loss function, and the learnable parameters include at least the instance identity code. The similarity between Gaussian volumes is calculated based on the optimized instance identity encoding, and the instances of the 3D Gaussian volumes are grouped. The instance-level scene segmentation, local deletion and repair are completed based on the results of the instance grouping. The steps for constructing a multidimensional loss function based on the two-dimensional supervision signal, the spatial neighborhood relationship of the three-dimensional Gaussian volume, and the cross-view consistent semantic features include: The instance identity is encoded and rendered onto the two-dimensional view plane. The segmentation result is predicted by the linear classification layer. The cross-entropy loss is calculated based on the consistent two-dimensional supervision signal across the view to determine the two-dimensional group supervision loss. Calculate the symmetric KL divergence of the identity code distributions of each 3D Gaussian volume and its neighboring Gaussian volumes, and constrain the consistency of identity codes of neighboring Gaussian volumes to determine the 3D neighborhood consistency regularization loss: The instance identity is encoded and mapped to the semantic space through a learnable mapping head. The cosine similarity between the mapped encoding and the cross-view consistent semantic features is calculated. The cosine distance is minimized to constrain semantic consistency and determine the semantic consistency regularization loss. The multidimensional loss function is constructed based on the two-dimensional grouping supervision loss, the three-dimensional neighborhood consistency regularization loss, and the semantic consistency regularization loss.

2. The scene segmentation and editing method based on three-dimensional Gaussian sputtering according to claim 1, characterized in that, The steps for initializing and reconstructing a three-dimensional Gaussian volume set from the multi-view two-dimensional image set using three-dimensional Gaussian sputtering technology include: A 3D scene is captured by a camera from a sparse perspective, a multi-view 2D image sequence is acquired, and feature points are extracted and matched from the 2D image sequence using a motion reconstruction structure algorithm to restore the camera pose and generate a sparse 3D point cloud. Each point in the sparse 3D point cloud is initialized as a 3D Gaussian volume, and the parameters of the 3D Gaussian volume include the center point coordinates, covariance matrix, upper limit of opacity, and spherical harmonic coefficients. The three-dimensional Gaussian volume is projected onto each two-dimensional image plane by differentiable rendering. The difference between the reconstructed image and the input real image is calculated as the reconstruction loss. The parameters of the three-dimensional Gaussian volume are iteratively optimized using an optimization algorithm until the loss converges, thus obtaining the set of three-dimensional Gaussian volumes.

3. The scene segmentation and editing method based on three-dimensional Gaussian sputtering according to claim 1, characterized in that, Based on the set of 3D Gaussian volumes, the step of associating each 3D Gaussian volume with cross-view consistent semantic features extracted by a pre-trained visual language model includes: Obtain image patches of the projected region of each 3D Gaussian volume in all visible views; Each projection region image patch is input into a pre-trained visual language model to extract multimodal embedding features, while the parameters of the visual language model remain fixed. The aggregation operation combines multiple multimodal embedding features to form cross-view consistent semantic features of the three-dimensional Gaussian volume.

4. The scene segmentation and editing method based on three-dimensional Gaussian sputtering according to claim 1, characterized in that, The instance identity is encoded as a low-dimensional vector, initialized as a random vector, and dynamically adjusted during parameter optimization using a gradient descent algorithm. This vector is used to uniquely identify the instance identity of the three-dimensional Gaussian body, ensuring the consistency of the identity of the same instance from different perspectives.

5. The scene segmentation and editing method based on three-dimensional Gaussian sputtering according to claim 1, characterized in that, The steps for completing instance-level scene segmentation, partial deletion, and repair based on the instance grouping results include: Calculate the instance identity encoding similarity between any two 3D Gaussian volumes, construct an adjacency matrix and determine the connectivity between Gaussian volumes, and divide connected Gaussian volumes into the same instance group through connected component analysis; Based on the instance-level scene segmentation results, the set of pixels corresponding to the instance to be deleted is located, and then the initial set of three-dimensional Gaussian bodies projected onto the set of pixels is determined. After outlier removal and region integrity processing, the final set of Gaussian bodies to be deleted is removed to complete the deletion. The final set of Gaussian volumes to be deleted is projected onto a specified view to generate a deletion mask. Based on the repair loss function composed of reconstruction loss and perception loss, the Gaussian volume parameters corresponding to the repair area are optimized by gradient descent algorithm until the repaired content achieves visual coherence with the surrounding environment.

6. A scene segmentation and editing system based on three-dimensional Gaussian sputtering, characterized in that, The system for implementing the scene segmentation and editing method based on three-dimensional Gaussian sputtering as described in any one of claims 1 to 5 includes: The data processing module is used to perform two-dimensional instance segmentation on the multi-view two-dimensional image set of the three-dimensional scene to obtain two-dimensional instance masks, and to initialize three-dimensional reconstruction of the multi-view two-dimensional image set through three-dimensional Gaussian sputtering technology to obtain a three-dimensional Gaussian volume set. The prototype library construction module is used to associate each three-dimensional Gaussian volume with cross-view consistent semantic features extracted by a pre-trained visual language model based on the three-dimensional Gaussian volume set, and to construct and iteratively update a three-dimensional instance prototype library based on the two-dimensional instance mask, so as to transform the two-dimensional instance mask into a cross-view consistent two-dimensional supervision signal according to the three-dimensional instance prototype library. The parameter optimization module is used to add a learnable instance identity code to each 3D Gaussian volume, and construct a multidimensional loss function based on the 2D supervision signal, the spatial neighborhood relationship of the 3D Gaussian volume and the cross-view consistent semantic features, so as to optimize the learnable parameters based on the multidimensional loss function. The learnable parameters include at least the instance identity code. The scene processing module is used to calculate the similarity between Gaussian bodies based on the optimized instance identity encoding, realize the instance grouping of 3D Gaussian bodies, and complete instance-level scene segmentation, local deletion and repair based on the instance grouping results.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the scene segmentation and editing method based on three-dimensional Gaussian sputtering as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scene segmentation and editing method based on three-dimensional Gaussian sputtering as described in any one of claims 1-5.

Images