Open-vocabulary 3D segmentation method based on three-dimensional gaussian sputtering technique

Abstract

The application provides an open vocabulary 3D segmentation method based on a three-dimensional Gaussian sputtering technology, comprising the following steps: S1, acquiring a multi-view 2D image set and performing initial three-dimensional reconstruction; S2, using SAM to segment the multi-view and acquiring a 2D mask; S3, using a video tracking segmentation model (SAM2) to associate the masks of the same instance; S4, using a visual language model (CLIP) to extract open vocabulary features for each mask; S5, performing average pooling and dimension reduction on the CLIP features of the mask of each instance to form representative open vocabulary features of the instance; S7, training a model to supervise the embedding of the low-dimensional embedding features; and S8, obtaining Gaussian bodies with high relevance by comparing the CLIP features of a query text with the semantic features of the Gaussian bodies, and generating a mask of the query object through rasterization rendering. The method only needs 2D images of multi-view, and uses the three-dimensional Gaussian sputtering technology combined with the open vocabulary technology to segment three-dimensional objects and scenes, so as to solve the problems of difficult 3D data acquisition and high cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual language models, in particular, to an open-vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology. BACKGROUND

[0002] 3D segmentation technology is an important research direction in the field of computer vision, which aims to detect and segment objects in three-dimensional space, providing key technical support for automatic driving, robots, medical imaging and other fields. The current mainstream 3D segmentation method is divided into two categories: 3D segmentation method based on deep learning and open-vocabulary 3D segmentation method. Among them, the 3D segmentation method based on deep learning usually needs a large amount of artificial annotation data to train the model, and the recognition ability for unobserved classes is limited. The open-vocabulary 3D segmentation method can recognize and segment object classes that do not appear in the training set, and provide three-dimensional information such as physical shape, size and spatial position of the object, greatly enhancing the understanding of objects and scenes.

[0003] In the existing public technology, the paper "OpenScene: 3D Scene Understanding with OpenVocabularies" proposes a 3D scene segmentation technology for open vocabulary. This method relies on precisely aligned 2D images and 3D point cloud data, and in order to train the 3D segmentation model, it also needs to manually annotate the 3D segmentation mask.

[0004] The main problem existing in the prior art is that the existing open-vocabulary 3D segmentation technology usually relies on precisely aligned 2D images, depth maps and 3D point cloud data, which requires complex and expensive data collection and manual annotation, and at the same time limits the application flexibility and practicality of the model in unknown environments.

[0005] The difficulty of solving the above problems is that 3D segmentation data annotation is tedious and costly, and the open-source data set for 3D segmentation is relatively scarce, which limits the application flexibility and practicality of the model in unknown environments.

[0006] The significance of solving the above problems is to develop a suitable open-vocabulary 3D segmentation method to detect and segment objects and scenes in three-dimensional space, providing key technical support for automatic driving, robots, medical imaging and other fields. SUMMARY

[0007] The present application provides an open-vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology, which only needs multi-view 2D images, and uses three-dimensional Gaussian sputtering technology combined with open-vocabulary technology to segment three-dimensional objects and scenes, to solve the problems of difficult 3D data collection and high cost.

[0008] The technical solution of the present invention is as follows:

[0009] The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology of the present invention includes the following steps: S1. Obtaining multiple views Figure 2 S1. A set of images is generated and initialized with 3D Gaussian sputtering for 3D reconstruction; S2. Each image in the multi-view set is segmented using an image segmentation model (SAM) to obtain a 2D instance mask; S3. Masks of the same instance are associated using a video tracking segmentation model (SAM2); S4. Open vocabulary features are extracted from each mask using a visual language model (CLIP); S5. The CLIP features of each instance's mask are average-pooled and dimensionality-reduced to form representative open vocabulary features for that instance; S6. Low-dimensional embedding features are used... Initialize a 3D Gaussian volume and render the feature map L using a feature rasterizer; S7. Train the model by minimizing the feature distance between the rendered feature map L and the original CLIP features to supervise the low-dimensional embedding features. Embedding; S8. By comparing the CLIP features of the query text with the semantic features of the Gaussian body, a highly relevant Gaussian body is obtained, and a mask of the query object is generated by rasterization rendering.

[0010] Optionally, in the above-mentioned open-vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S1, the multi-view of the 3D scene or object is first obtained. Figure 2 A set of D images was then processed using Structure of Motion (SFM) for multiview. Figure 2 A sparse 3D point cloud is obtained from the D image, and these sparse 3D point clouds are used to initialize a 3D Gaussian volume set for initial 3D reconstruction.

[0011] Optionally, in the above-described open-vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S1, each Gaussian volume is composed of a mean µ and a 3D covariance matrix. Defined as follows: where R and S represent the rotation matrix and scaling matrix, and the center point µ of each Gaussian body corresponds to the three-dimensional coordinates of the feature point. The rotation matrix and scaling matrix are randomly initialized, and the shape and size of the Gaussian body are determined according to the distribution of feature points and disparity information.

[0012] Optionally, in the above-mentioned open-vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S3, the multi-view image of the 3D scene is regarded as a video sequence with gradually changing views, and a pre-trained video tracking segmentation model (SAM2) is used to track and associate the mask of the same instance to achieve 2D mask consistency across views.

[0013] Optionally, in the above open-vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology, in step S5, all CLIP features of the mask of each instance are fused together and passed through a learnable compression encoder The high-dimensional CLIP features are reduced in dimension to form the representative open-vocabulary features of the instance.

[0014] Optionally, in the above open-vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology, in step S5, the CLIP features of each instance mask are averaged pooled to obtain the average CLIP feature F, defined as:

[0015] (1)

[0016] wherein is the jth 2D mask CLIP feature of the ith instance, the high-dimensional CLIP feature F is compressed to a low-dimensional embedding feature , defined as: wherein is a learnable compression encoder, composed of two fully connected layers.

[0017] Optionally, in the above open-vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology, in step S6, the low-dimensional embedding feature is used to initialize the 3D Gaussian body, the CLIP feature is embedded into the 3D Gaussian body, and the low-dimensional embedding feature is taken as a parameter of the Gaussian body for rendering, and a feature rasterizer is used to render the feature map L, defined as:

[0018] (2)

[0019] wherein is the open-vocabulary feature embedded into the ith Gaussian body, is the open-vocabulary feature map rendered at pixel u.

[0020] Optionally, in the above open-vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology, in step S7, m pixels are randomly selected from each mask in the rendered feature map to represent the mask, then a trainable decoder Ψ is used to recover the latent CLIP feature, and loss calculation is performed with the CLIP feature obtained from the original RGB image, minimizing the difference between the latent low-dimensional feature and the CLIP feature.

[0021] Optionally, in the open-vocabulary 3D segmentation method based on the three-dimensional Gaussian sputtering technology, in step S7, the latent features of each 2D mask in the rendered feature map L are restored to the CLIP space, and then the distance between the restored latent CLIP features and the original CLIP features is minimized, and the specific operation is as follows: m pixels are randomly sampled in each mask, and then the latent CLIP features of all the sampled m pixels are restored through a trainable decoder composed of two fully connected layers The feature distillation loss between the restored latent CLIP features of all the sampled m pixels and the CLIP features is defined as:

[0022] (3).

[0023] Optionally, in the open-vocabulary 3D segmentation method based on the three-dimensional Gaussian sputtering technology, in step S8, after the training, the Gaussian body can express semantic features, and the object described by the language instruction is found through open-vocabulary query. First, the relevance score s of each Gaussian body text query is calculated, and is defined as:

[0024] (4)

[0025] wherein is the CLIP feature of the query text, is a set of canonical phrases, selected from "object", "thing", "stuff" and "texture", is a trainable decoder, and L is the feature map rendered by the Gaussian body; for each text query, a relevance heat map is obtained, wherein the Gaussian bodies with a relevance score lower than a predetermined threshold are filtered out, and the remaining areas are rendered by rasterization to form a mask for predicting the queried object.

[0026] According to the technical scheme of the present application, the beneficial effects are:

[0027] 1) Compared with the prior art, the pre-trained visual language model (CLIP) is used to obtain the text semantic features, form an open vocabulary, and combine the CLIP features with the Gaussian field through the three-dimensional Gaussian sputtering technology to form a Gaussian semantic field, so that the Gaussian field can respond to the text feature query, and the perception ability of the object and the real world scene is improved.

[0028] 2) The present application does not need additional manual annotation of 3D data, only needs multiple views Figure 2 ​D images, combined with existing visual language models and the most advanced 2D segmentation model, can realize open vocabulary segmentation of three-dimensional scenes, avoid complex and expensive manual 3D segmentation labeling, and improve the application flexibility and effect of three-dimensional scene understanding model in unknown environment.

[0029] 3) The present application can quickly update the three-dimensional Gaussian field by a small amount of new images when the object in the real scene is displaced or the scene changes, so as to realize real-time reconstruction of the dynamic scene, and maintain accurate capture and understanding of the scene.

[0030] In order to better understand and illustrate the concept, working principle and effect of the present application, the present application will be described in detail below by combining the drawings and specific examples. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the specific embodiment of the present application or the technical solution in the prior art, the drawings needed to be used in the specific embodiment or the prior art description will be briefly introduced as follows.

[0032] Figure 1 Flowchart of the open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology of the present application;

[0033] Figure 2 Framework diagram of the open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical method and advantages of the present application more clear, the present application will be further described in detail below by combining the drawings and specific examples. These examples are only illustrative, and are not limiting to the present application.

[0035] The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology of the present application uses three-dimensional Gaussian sputtering technology to reconstruct a three-dimensional scene, and combines CLIP text features to form a Gaussian semantic field, so that the Gaussian field can respond to text feature queries. The present application does not need additional manual 3D data labeling, only needs multi-view Figure 2 D images, combined with existing visual language models and the most advanced 2D segmentation model, can realize open vocabulary segmentation of three-dimensional scenes.

[0036] As shown in Figure 1 and Figure 2 , the open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology of the present application includes the following steps:

[0037] S1. Obtain a multi-view Figure 2 D image set, and initialize three-dimensional reconstruction by three-dimensional Gaussian sputtering technology;

[0038] In step S1, first obtain a multi-view image set of a three-dimensional scene or object Figure 2 D image set, and then process the multi-view image set using a motion recovery structure technology (SFM) to obtain a sparse three-dimensional point cloud representing feature points in the scene, and use the sparse three-dimensional point cloud to initialize a 3D Gaussian body set to perform an initial three-dimensional reconstruction. Figure 2 D image set, and then process the multi-view image set using a motion recovery structure technology (SFM) to obtain a sparse three-dimensional point cloud representing feature points in the scene, and use the sparse three-dimensional point cloud to initialize a 3D Gaussian body set to perform an initial three-dimensional reconstruction.

[0039] Specifically, the Gaussian bodies are initialized using the feature points, each Gaussian body being defined by a mean µ and a three-dimensional covariance matrix , where R and S represent a rotation matrix and a scaling matrix, the center point µ of each Gaussian body corresponding to the three-dimensional coordinates of a feature point, the rotation matrix and the scaling matrix being randomly initialized, and the shape and size of the Gaussian body being determined according to the distribution of the feature points and the parallax information.

[0040] S2. SAM model segmentation mask: using an image segmentation model (SAM) to segment each image in the multi-view set to obtain a 2D instance mask;

[0041] In step S2, first deploy the image segmentation model SAM to automatically generate a 2D instance mask for each image in the multi-view set. In order to give each 2D mask in the three-dimensional scene unique semantic information, it is necessary to associate the masks of the same instance in different views.

[0042] S3. SAM2 model associated mask: using a video tracking segmentation model (SAM2) to associate the masks of the same instance;

[0043] In step S3, the multi-view images of the 3D scene are regarded as a video sequence with gradually changing views, and in order to achieve consistency of the 2D masks across views, i.e. the semantic information represented by the 2D masks of the same instance in different views should be the same or similar, a pre-trained video tracking segmentation model (SAM2) is used to track and associate the masks of the same instance to achieve consistency of the 2D masks across views.

[0044] S4. Extracting open vocabulary features for each mask using a visual language model (CLIP);

[0045] In step S4, a pre-trained visual language model CLIP (i.e. the CLIP pre-training model in Figure 2 ) is used to extract features from each 2D instance mask to obtain corresponding CLIP features f.

[0046] S5. Obtaining representative open vocabulary features for each instance: performing average pooling and dimension reduction on the CLIP features of the mask of each instance to form representative open vocabulary features for the instance;

[0047] In step S5, all CLIP features of the mask for each instance are fused together and passed through a learnable compression encoder. Dimensionality reduction of high-dimensional CLIP features ( Figure 2 Feature compression in the middle), forming representative open vocabulary features for this instance ( Figure 2 (low-dimensional embedding features in the data).

[0048] Specifically, to maintain semantic consistency across multiple views, the CLIP features of the mask for each instance are averaged and pooled to obtain the average CLIP feature F, defined as:

[0049] (1)

[0050] in Let F represent the j-th 2D mask CLIP feature of the i-th instance. However, directly using the original CLIP features (over 500 dimensions) for Gaussian volume rendering would result in high space and time costs. Therefore, the high-dimensional CLIP feature F is compressed into a low-dimensional embedded feature. (Representative open vocabulary features), defined as: ,in This represents a learnable compression encoder, consisting of two fully connected layers.

[0051] S6. Open Lexical Feature Embedding: Using Low-Dimensional Embedding Features Initialize the 3D Gaussian volume and render the feature map L using a feature rasterizer;

[0052] In step S6, the representative CLIP feature is rendered as a parameter of the Gaussian volume. The rendering process is similar to the color optimization in 3D Gaussian sputtering, but focuses on the rendering of the CLIP feature.

[0053] Specifically, using low-dimensional embedding features Initialize a 3D Gaussian volume and embed CLIP features into it. Then, embed the low-dimensional features... It is rendered as a parameter of a Gaussian volume, and the rendering process is similar to color optimization in 3D Gaussian sputtering, but focuses on the rendering of CLIP features. This invention uses a feature rasterizer to render the feature map L, defined as:

[0054] (2)

[0055] in It is an open vocabulary feature embedded in the i-th Gaussian body. This represents the open vocabulary feature map rendered at pixel u.

[0056] S7. Train the model to minimize the feature distance between the rendered feature map L and the original CLIP feature to supervise the low-dimensional embedding feature of the embedding;

[0057] In step S7, m pixels are randomly selected from each mask in the rendered feature map to represent the mask, and then the latent CLIP feature is recovered through the trainable decoder Ψ, and the loss calculation is performed with the CLIP feature obtained from the original RGB image, and the difference between the latent low-dimensional feature and the CLIP feature is minimized.

[0058] Specifically, the latent feature of each 2D mask in the rendered feature map L is recovered to the CLIP space, and then the distance between the recovered latent CLIP feature and the original CLIP feature is minimized. The specific operation is as follows, m pixels are randomly sampled within each mask, and then the latent CLIP feature is recovered through a trainable decoder composed of two fully connected layers . Among them, the feature distillation loss between the recovered latent CLIP feature of all sampled m pixels and the CLIP feature is defined as:

[0059] (3)

[0060] S8. By comparing the CLIP feature of the query text with the semantic feature of the Gaussian body, the Gaussian body with high relevance is obtained, and the mask of the query object is generated by rasterization rendering.

[0061] In step S8, when the training is completed, by comparing the CLIP feature of the query text with the semantic feature of the Gaussian body, the corresponding Gaussian body (i.e., the Gaussian body with high relevance to the query text) is found, and then rendering is performed to obtain the object to be queried.

[0062] Specifically, after training, the Gaussian body can express the semantic feature, so that the object described by the language instruction can be found through the open vocabulary query. First, the relevance score s of each Gaussian body text query is calculated, which is defined as:

[0063] (4)

[0064] where is the CLIP feature of the query text, is a set of normative phrases, selected from "object", "thing", "thing" and "texture", ​For the trainable decoder, L is the feature map of the Gaussian volume rendering. For each text query, a relevance heat map is obtained, in which the Gaussian volume with a relevance score lower than a predetermined threshold will be filtered out. The remaining areas are rendered by rasterization to form a mask for predicting the queried object.

[0065] The following will briefly describe the specific application prediction process of the open vocabulary 3D segmentation method based on the three-dimensional Gaussian sputtering technology of the present application through a specific example.

[0066] First, input a multi-view 2D image set of a three-dimensional scene or object, then process the multi-view images using the structure from motion (SFM) technique to obtain a sparse three-dimensional point cloud, and use the sparse point cloud to initialize the 3D Gaussian volume for initial three-dimensional reconstruction.

[0067] Subsequently, the SAM segmentation model and the video tracking segmentation model SAM2 are used to segment and instance mask track each image in the multi-view set, obtain 2D masks, and fuse the masks of the same instance;

[0068] Then, the visual language model CLIP is used to extract open vocabulary features for each 2D mask, and the CLIP features of the masks of the same instance are fused to obtain the low-dimensional embedding open vocabulary features of the instance , and the low-dimensional embedding features are used to initialize the 3D Gaussian volume, and the trained Gaussian field obtains the ability to respond to text feature queries.

[0069] Finally, by comparing the CLIP features of the query text with the semantic features of the Gaussian volume, the Gaussian volume with high relevance to the query text can be efficiently found. Taking the query text "sofa" as an example, the CLIP model first converts the text into a feature vector, and then compares it with the semantic features of the Gaussian volume. The Gaussian volume with high relevance is selected, and through the rasterization rendering technique, the mask area of the queried object "sofa" can be obtained.

[0070] The effect of the present application will be further described below in combination with a simulation experiment:

[0071] (1) Simulation experiment conditions:

[0072] The hardware platform of the simulation experiment of the present application is: the processor is Intel(R) Xeon(R) Silver4310 CPU, the main frequency is 2.10 GHz, the memory is 128 GB, and the graphics card is NVIDIA-A100.

[0073] The software platform of the simulation experiment of the present application is: CentOS7.6 operating system, Pytorch2.0.1 framework, and Python3.9.

[0074] The data used in the simulation experiment is a 3D-OVS dataset developed for open-vocabulary 3D semantic segmentation, which is composed of objects photographed in various poses and different backgrounds, and provides a complete class list.

[0075] Under the above simulation experiment conditions, the present method and three baseline methods OV-Seg, ResNet50 and Vision-Transformer are respectively used for open-vocabulary segmentation of five object categories "bed", "bench", "room", "sofa" and "lawn" in the 3D-OVS dataset, and the mIoU scores of the respective methods are calculated, and the results are shown in Table 1.

[0076] Table 1. mIoU score comparison table

[0077] Method bed bench room sofa lawn OV-Seg 78.6 87.8 70.4 64.8 80.2 LERF 72.5 52.2 45.1 28.4 71.9 3D-OVS 87.5 87.9 90.2 73.5 85.2 The method of the invention 88.5 89.2 89.8 81.1 87.1

[0078] As can be seen from Table 1, the mIoU scores of the present application on the five categories are mostly higher than those of the baseline methods, and the present application has better open-vocabulary segmentation effect.

[0079] At the same time, since the three-dimensional Gaussian field is fully trained, when the object in the real scene is displaced or the scene changes,

[0080] By quickly updating the three-dimensional Gaussian field with a small amount of new images, real-time reconstruction of dynamic scenes can be achieved. This fast response capability makes the three-dimensional Gaussian field have a significant advantage in handling dynamic scenes.

[0081] The above description is based on the best embodiment of the concept and working principle of the application. The above embodiment should not be understood as limiting the scope of protection of the present application, and other embodiments and combinations of implementations according to the concept of the present application are within the scope of protection of the present application.

Claims

1. An open-vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technique, characterized in that, The method comprises the following steps: S1. Obtain a multi-view 2D image set and initialize a three-dimensional reconstruction using a three-dimensional Gaussian spray technique; S2. Segment each image in the multi-view set using an image segmentation model (SAM) to obtain a 2D instance mask; S3. Use a video tracking segmentation model (SAM2) to associate masks of the same instance; S4. Extract open vocabulary features for each mask using a visual language model (CLIP); S5. Average pool and reduce the dimension of the CLIP features of each instance mask to form representative open vocabulary features of the instance; In step S5, the CLIP features of each instance mask are averaged and pooled to obtain an average CLIP feature F, which is defined as: where f ij the jth 2D mask CLIP feature representing the ith instance, F i the average CLIP feature representing the ith instance, denoting the overall semantic representation of the instance in the open-vocabulary semantic space; subsequently, the high-dimensional CLIP feature F is compressed into a low-dimensional embedding feature l, defined as: l = Φ(F), where Φ represents a learnable compression encoder, consisting of two fully connected layers; S6. Initialize a 3D Gaussian body using a low-dimensional embedding feature l, and render a feature map L using a feature rasterizer; S7. Train the model to minimize the feature distance between the rendered feature map L and the original CLIP feature to supervise the embedding of the low-dimensional embedding feature l; S8. By comparing the CLIP feature of the query text with the semantic feature of the Gaussian body, the Gaussian body with high relevance is obtained, and the mask of the query object is generated by rasterization rendering.

2. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technique according to claim 1, wherein, In step S1, first obtain a multi-view 2D image set of a three-dimensional scene or object, then process the multi-view 2D image using a motion recovery structure technique (SFM) to obtain a sparse three-dimensional point cloud, and use the sparse three-dimensional point cloud to initialize a 3D Gaussian body set to perform the initialization three-dimensional reconstruction.

3. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technique according to claim 1, wherein, In step S1, each Gaussian body is defined by a mean μ and a three-dimensional covariance matrix ∑ = RSS T R T where R and S represent a rotation matrix and a scaling matrix, the center point μ of each Gaussian body corresponds to the three-dimensional coordinates of the feature points, the rotation matrix and the scaling matrix are randomly initialized, and the shape and size of the Gaussian body are determined according to the distribution of the feature points and the disparity information.

4. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technique according to claim 1, wherein, In step S3, the multi-view images of the three-dimensional scene are regarded as a video sequence with gradually changing views, and a pre-trained video tracking segmentation model (SAM2) is used to track and associate masks of the same instance to achieve cross-view 2D mask consistency.

5. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technique according to claim 1, wherein, In step S5, all CLIP features of each instance mask are fused, and a high-dimensional CLIP feature is reduced in dimension by a learnable compression encoder Φ to form representative open vocabulary features of the instance.

6. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technique according to claim 1, wherein, In step S6, the low-dimensional embedding feature l is used to initialize the 3D Gaussian body, the CLIP feature is embedded into the 3D Gaussian body, and the low-dimensional embedding feature l is used as a parameter of the Gaussian body for rendering. A feature rasterizer is used to render a feature map L, which is defined as: where l i is the embedding of the open vocabulary feature into the i-th Gaussian body, L(u) represents the rendered open vocabulary feature map at pixel u.

7. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technique according to claim 1, wherein, In step S7, m pixels are randomly selected from each mask in the rendered feature map to represent the mask, and then a trainable decoder Ψ is used to recover the latent CLIP feature, and the difference between the latent low-dimensional feature and the CLIP feature obtained from the original RGB image is calculated to minimize the difference.

8. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technique according to claim 1, wherein, In step S7, the latent features of each 2D mask in the rendered feature map L are recovered to the CLIP space, and then the distance between the recovered latent CLIP features and the original CLIP features is minimized, specifically: m pixels are randomly sampled within each mask, and then the latent CLIP features are recovered by a trainable decoder Ψ consisting of two fully connected layers, where the feature distillation loss L between the recovered latent CLIP features of all sampled m pixels and the original CLIP features is minimized. diss is defined as:

9. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technique according to claim 1, wherein, In step S8, after training, the Gaussian body can express semantic features, and objects described by language instructions are found by open vocabulary queries. First, calculate the relevance score s of each Gaussian body text query, which is defined as: where T q is the CLIP feature of the query text, is a set of canonical phrases, chosen from "object", "thing", "stuff", and "texture", Ψ is a trainable decoder, and L is the feature map of the Gaussian volume rendering; for each text query, a relevance heat map is obtained, where the Gaussian volume with a relevance score lower than a predetermined threshold will be filtered out, and the remaining area is rendered by rasterization to form a mask for predicting the queried object.

Images