Open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology

Abstract

The invention provides an open vocabulary 3D segmentation method based on a three-dimensional Gaussian sputtering technology, and the method comprises the steps: S1, obtaining a multi-view 2D image set, and carrying out the initialization three-dimensional reconstruction; s2, segmenting the multiple views by using SAM to obtain a 2D mask; s3, associating masks of the same instance by using a video tracking segmentation model (SAM2); s4, extracting open vocabulary features of each mask by using a visual language model (CLIP); s5, performing average pooling and dimension reduction on the CLIP feature of the mask of each instance to form a representative open vocabulary feature of the instance; s7, training the model, and supervising the embedding of a low-dimensional embedding feature # imgabs0 #; and S8, comparing the CLIP feature of the query text with the semantic feature of the Gaussian body to obtain the Gaussian body with high correlation, and generating a mask of the query object through rasterization rendering. According to the method, only a multi-view 2D image is needed, and the three-dimensional object and the scene are segmented by combining the three-dimensional Gaussian sputtering technology and the open vocabulary technology, so that the problems of difficulty in 3D data acquisition and high cost are solved.

Description

Technical Field

[0001] The present invention relates to the field of visual language models, and in particular to an open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology. Background Art

[0002] 3D segmentation technology is an important research direction in the field of computer vision. It aims to detect and segment objects in three-dimensional space, providing key technical support for multiple fields such as autonomous driving, robotics, and medical imaging. The current mainstream 3D segmentation methods are divided into deep learning-based 3D segmentation methods and open vocabulary 3D segmentation methods. Among them, deep learning-based 3D segmentation methods usually require a large amount of manually annotated data to train the model, and have limited ability to recognize unseen categories. The open vocabulary 3D segmentation method can identify object categories that have not appeared in the segmentation training set, and provide three-dimensional information such as the physical shape, size, and spatial position of the object, which greatly enhances the understanding of objects and scenes.

[0003] Among the existing public technologies, the paper "OpenScene: 3D Scene Understanding with OpenVocabularies" proposes a 3D scene segmentation technology for open vocabulary. This method relies on precisely aligned 2D images and 3D point cloud data, and in order to train the 3D segmentation model, manual annotation of the 3D segmentation mask is also required.

[0004] The main problem with the existing technology is that existing open vocabulary 3D segmentation techniques usually rely on precisely aligned 2D images, depth maps and 3D point cloud data, which requires complex and expensive data collection and manual annotation, while limiting the application flexibility and practicality of the model in unknown environments.

[0005] The difficulty in solving the above problems is that the 3D segmentation data annotation work is cumbersome and costly, and the open source datasets for 3D segmentation are relatively scarce, which limits the flexibility and practicality of the model in unknown environments.

[0006] The significance of solving the above problems is to develop a suitable open vocabulary 3D segmentation method for detecting and segmenting objects and scenes in three-dimensional space, providing key technical support for multiple fields such as autonomous driving, robotics, and medical imaging. Summary of the invention

[0007] The present invention provides an open vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, which only requires 2D images of multiple views and utilizes 3D Gaussian sputtering technology combined with open vocabulary technology to segment 3D objects and scenes to solve the problems of difficulty and high cost in 3D data collection.

[0008] The technical solution of the present invention is as follows: The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology of the present invention comprises the following steps: S1. Obtaining multi-view Figure 2 D image set, and initialize 3D reconstruction using 3D Gaussian sputtering technology; S2. Use image segmentation model (SAM) to segment each image in the multi-view set to obtain 2D instance masks; S3. Use video tracking segmentation model (SAM2) to associate masks of the same instance; S4. Use visual language model (CLIP) to extract open vocabulary features for each mask; S5. Average pool and reduce the dimension of CLIP features of each instance mask to form representative open vocabulary features of the instance; S6. Use low-dimensional embedding features Initialize the 3D Gaussian volume and use the feature rasterizer to render the feature map L; S7. Train the model to minimize the feature distance between the rendered feature map L and the original CLIP feature to supervise the low-dimensional embedding feature By comparing the CLIP features of the query text with the semantic features of the Gaussian body, a Gaussian body with high correlation is obtained, and a mask of the query object is generated through rasterization rendering.

[0009] Optionally, in the above open vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S1, a multi-view of a 3D scene or object is first obtained. Figure 2 D image collection, and then use the motion recovery structure (SFM) technique to process the multi-view Figure 2 D images to obtain sparse 3D point clouds, and use these sparse 3D point clouds to initialize a 3D Gaussian volume set for initial 3D reconstruction.

[0010] Optionally, in the above open vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S1, each Gaussian volume is composed of a mean µ and a 3D covariance matrix Definition, where R and S represent the rotation matrix and scaling matrix, the center point µ of each Gaussian body corresponds to the three-dimensional coordinates of the feature point, the rotation matrix and scaling matrix are randomly initialized, and the shape and size of the Gaussian body are determined according to the distribution of the feature points and the disparity information.

[0011] Optionally, in the above-mentioned open vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S3, the multi-view images of the three-dimensional scene are regarded as a video sequence with gradually changing views, and a pre-trained video tracking and segmentation model (SAM2) is used to track and associate masks of the same instance to achieve 2D mask consistency across views.

[0012] Optionally, in the above open vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S5, all CLIP features of the mask of each instance are fused and passed through a learnable compression encoder The high-dimensional CLIP features are reduced in dimension to form representative open vocabulary features for the instance.

[0013] Optionally, in the above open vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S5, the CLIP features of the mask of each instance are average pooled to obtain an average CLIP feature F, which is defined as: (1) in Represents the jth 2D mask CLIP feature of the i-th instance, compressing the high-dimensional CLIP feature F to a low-dimensional embedding feature , defined as: ,in It stands for learnable compression encoder and consists of two fully connected layers.

[0014] Optionally, in the above open vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S6, a low-dimensional embedding feature is used Initialize the 3D Gaussian, embed the CLIP features into the 3D Gaussian, and embed the low-dimensional features Rendered as a parameter of a Gaussian volume, the feature map L is rendered using the feature rasterizer, defined as: (2) in is the open vocabulary feature embedded into the i-th Gaussian, Represents the open vocabulary feature map rendered at pixel u.

[0015] Optionally, in the above-mentioned open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology, in step S7, m pixels are randomly selected from each mask in the rendered feature map to represent the mask, and then the potential CLIP features are restored through a trainable decoder Ψ, and the loss is calculated with the CLIP features obtained from the original RGB image to minimize the difference between the potential low-dimensional features and the CLIP features.

[0016] Optionally, in the above-mentioned open vocabulary 3D segmentation method based on 3D Gaussian sputtering technology, in step S7, the latent features of each 2D mask in the rendered feature map L are restored to the CLIP space, and then the distance between the restored latent CLIP features and the original CLIP features is minimized, specifically by randomly sampling m pixels in each mask, and then passing through a trainable decoder consisting of two fully connected layers Recover the latent CLIP features, where the feature distillation loss between the recovered latent CLIP features and CLIP features of all sampled m pixels is Defined as: (3).

[0017] Optionally, in the above-mentioned open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology, in step S8, after training, the Gaussian body can express semantic features, and find the object described by the language instruction through open vocabulary query. First, the relevance score s of each Gaussian body text query is calculated, which is defined as: (4) in is the CLIP feature of the query text, is a set of standard phrases, Choose from Objects, Things, Stuff, and Textures. is a trainable decoder, L is a feature map rendered by Gaussian volumes; for each text query, a relevance heat map is obtained, in which Gaussians with relevance scores below a predetermined threshold are filtered out, and the remaining area is rasterized to form a mask for predicting the queried object.

[0018] According to the technical solution of the present invention, the beneficial effects produced are: 1) Compared with the prior art, the present invention uses a pre-trained visual language model (CLIP) to obtain text semantic features and form an open vocabulary, and combines the CLIP features with the Gaussian field through the three-dimensional Gaussian sputtering technology to form a Gaussian semantic field, so that the Gaussian field can respond to text feature queries and improve the perception of objects and real-world scenes; 2) The present invention does not require additional manual annotation of 3D data, only multi-view Figure 2 By combining the existing visual language model with the most advanced 2D segmentation model, we can perform open vocabulary segmentation on the 3D scene, avoiding complex and expensive manual 3D segmentation annotation, and improving the application flexibility and effect of the 3D scene understanding model in unknown environments.

[0019] 3) In real scenes, when objects are displaced or scenes change, the present invention can quickly update the three-dimensional Gaussian field through a small number of new images, thereby achieving real-time reconstruction of dynamic scenes and maintaining accurate capture and understanding of the scenes.

[0020] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0022] Figure 1 is a flow chart of the open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology of the present invention; Figure 2 It is a framework diagram of the open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0024] The open vocabulary 3D segmentation method based on 3D Gaussian sputtering technology of the present invention uses 3D Gaussian sputtering technology to reconstruct the 3D scene and combines CLIP text features to form a Gaussian semantic field, so that the Gaussian field can respond to text feature queries. The present invention does not require additional manual annotation of 3D data, only multi-view Figure 2 3D images, combined with existing visual language models and state-of-the-art 2D segmentation models, can perform open vocabulary segmentation of 3D scenes.

[0025] like Figure 1 and Figure 2 As shown, the open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology of the present invention comprises the following steps: S1. Get multiple views Figure 2 D image collection, and use 3D Gaussian sputtering technology to initialize 3D reconstruction; In step S1, a multi-view image of a three-dimensional scene or object is first obtained. Figure 2 D image collection, and then use the motion recovery structure (SFM) technique to process the multi-view Figure 2 D images to obtain sparse 3D point clouds, which represent the feature points in the scene; and use these sparse 3D point clouds to initialize a 3D Gaussian volume set for initial 3D reconstruction.

[0026] Specifically, these feature points are used to initialize Gaussian volumes, each of which consists of a mean µ and a three-dimensional covariance matrix Definition, where R and S represent the rotation matrix and scaling matrix, the center point µ of each Gaussian body corresponds to the three-dimensional coordinates of the feature point, the rotation matrix and scaling matrix are randomly initialized, and the shape and size of the Gaussian body are determined according to the distribution of the feature points and the disparity information.

[0027] S2. SAM model segmentation mask: Use the image segmentation model (SAM) to segment each image in the multi-view set and obtain a 2D instance mask; In step S2, the image segmentation model SAM is first deployed to automatically generate a 2D instance mask for each image in the multi-view set. In order to give each 2D mask in the 3D scene unique semantic information, it is necessary to associate the masks of the same instance in different views.

[0028] S3. SAM2 model associated mask: Use the video tracking segmentation model (SAM2) to associate masks of the same instance; In step S3, the multi-view images of the 3D scene are regarded as video sequences with gradually changing views. In order to achieve 2D mask consistency across views, that is, the semantic information represented by the 2D mask of the same instance in different views should be the same or similar, a pre-trained video tracking and segmentation model (SAM2) is used to track and associate the masks of the same instance to achieve 2D mask consistency across views.

[0029] S4. Extract open vocabulary features for each mask using the visual language model (CLIP); In step S4, the pre-trained visual language model CLIP (i.e. Figure 2 The CLIP pre-trained model in is used to extract features from each 2D instance mask and obtain the corresponding CLIP feature f.

[0030] S5. Obtain the representative open vocabulary feature of each instance: Average pool the CLIP features of the mask of each instance and reduce the dimension to form the representative open vocabulary feature of the instance; In step S5, all CLIP features of the mask of each instance are fused and passed through a learnable compression encoder Reduce the dimension of high-dimensional CLIP features ( Figure 2 ), forming the representative open vocabulary feature of this instance ( Figure 2 low-dimensional embedding features in .

[0031] Specifically, in order to maintain semantic consistency under multiple views, the CLIP features of each instance mask are averaged and pooled to obtain the average CLIP feature F, which is defined as: (1) in Represents the jth 2D mask CLIP feature of the i-th instance. At the same time, since the original CLIP feature (more than 500 dimensions) is directly used for Gaussian volume rendering, it will result in high space and time costs. Therefore, the high-dimensional CLIP feature F is compressed into a low-dimensional embedding feature (representative open vocabulary features), defined as: ,in It stands for learnable compression encoder and consists of two fully connected layers.

[0032] S6. Open vocabulary feature embedding: using low-dimensional embedding features Initialize the 3D Gaussian volume and render the feature map L using the feature rasterizer; In step S6, the representative CLIP feature is rendered as a parameter of a Gaussian volume. The rendering process is similar to the color optimization in three-dimensional Gaussian sputtering, but focuses on the rendering of CLIP features.

[0033] Specifically, using low-dimensional embedding features Initialize the 3D Gaussian volume, embed the CLIP features into the Gaussian volume, and embed the low-dimensional features into the The rendering process is similar to the optimization of color in 3D Gaussian sputtering, but focuses on the rendering of CLIP features. The present invention uses a feature rasterizer to render the feature map L, which is defined as:

[0034] (2) in is the open vocabulary feature embedded into the i-th Gaussian, Represents the open vocabulary feature map rendered at pixel u.

[0035] S7. Train the model to minimize the feature distance between the rendered feature map L and the original CLIP feature to supervise the low-dimensional embedding feature Embedding In step S7, m pixels are randomly selected from each mask in the rendered feature map to represent the mask, and then the potential CLIP features are restored through a trainable decoder Ψ, and the loss is calculated with the CLIP features obtained from the original RGB image to minimize the difference between the potential low-dimensional features and the CLIP features.

[0036] Specifically, the latent features of each 2D mask in the rendered feature map L are restored to the CLIP space, and then the distance between the restored latent CLIP features and the original CLIP features is minimized. The specific operation is as follows: m pixels are randomly sampled in each mask, and then passed through a trainable decoder consisting of two fully connected layers. Restore the latent CLIP features. The feature distillation loss between the restored latent CLIP features and CLIP features of all sampled m pixels is Defined as:

[0037] (3) S8. By comparing the CLIP features of the query text with the semantic features of the Gaussian volume, a Gaussian volume with high correlation is obtained, and a mask of the query object is generated through rasterization rendering.

[0038] In step S8, when the training is completed, the CLIP features of the query text are compared with the semantic features of the Gaussian body to find the corresponding Gaussian body (that is, find the Gaussian body with high correlation with the query text), and then the object to be queried is obtained by rendering.

[0039] Specifically, after training, the Gaussian can express semantic features, so the object described by the language instruction can be found through open vocabulary query. First, the relevance score s of each Gaussian text query is calculated, which is defined as:

[0040] (4) in is the CLIP feature of the query text, is a set of standard phrases, Choose from Objects, Things, Stuff, and Textures. is a trainable decoder, L is a feature map rendered with Gaussian volumes. For each text query, a relevance heatmap is obtained, where Gaussian volumes with relevance scores below a predetermined threshold are filtered out. The remaining regions are rasterized to form a mask for predicting the queried object.

[0041] The specific application prediction process of the open vocabulary 3D segmentation method based on the three-dimensional Gaussian sputtering technology of the present invention is briefly described below through specific examples.

[0042] First, a set of multi-view 2D images of a 3D scene or object is input, and then the motion recovery structure technology (SFM) is used to process the multi-view images to obtain a sparse 3D point cloud. These sparse point clouds are used to initialize the 3D Gaussian volume for initial 3D reconstruction.

[0043] Then, the SAM segmentation model and the video tracking segmentation model SAM2 are used to segment and track instance masks for each image in the multi-view set, obtaining a 2D mask in parallel with the mask of the same instance; Then, we use the visual language model CLIP to extract open vocabulary features for each 2D mask, and fuse the CLIP features of the mask of the same instance to obtain the low-dimensional embedded open vocabulary features of the instance. , and embed features through low-dimensional Initialize the 3D Gaussian volume, and after training, the Gaussian field acquires the ability to respond to text feature queries.

[0044] Finally, by comparing the correlation between the CLIP features of the query text and the semantic features of the Gaussian body, the Gaussian body with high correlation with the query text can be efficiently found. Taking the query text "sofa" as an example, the CLIP model first converts the text into a feature vector and then compares it with the semantic features of the Gaussian body. After the Gaussian body with high correlation is selected, the mask area of ​​the query object "sofa" can be obtained through rasterization rendering technology.

[0045] The effect of the present invention is further described below in conjunction with simulation experiments: (1) Simulation experiment conditions: The hardware platform of the simulation experiment of the present invention is: the processor is Intel (R) Xeon (R) Silver 4310 CPU, the main frequency is 2.10 GHz, the memory is 128 GB, and the graphics card is NVIDIA -A100.

[0046] The software platform of the simulation experiment of the present invention is: CentOS7.6 operating system, Pytorch2.0.1 framework, Python3.9.

[0047] The data used in the simulation experiment is: 3D-OVS dataset, which is developed for open vocabulary 3D semantic segmentation. It consists of objects taken in various poses and different backgrounds and provides a complete list of categories.

[0048] Under the above simulation experimental conditions, the method of the present invention and three baseline methods OV-Seg, ResNet50, and Vision-Transformer are used to perform open vocabulary segmentation on five object categories such as "bed", "bench", "room", "sofa", and "lawn" in the 3D-OVS dataset, and the mIoU scores of each method are calculated. The results are shown in Table 1.

[0049] Table 1. mIoU score comparison table method bed bench room sofa lawn OV-Seg 78.6 87.8 70.4 64.8 80.2 LERF 72.5 52.2 45.1 28.4 71.9 3D-OVS 87.5 87.9 90.2 73.5 85.2 Method of the present invention 88.5 89.2 89.8 81.1 87.1 As can be seen from Table 1, the mIoU scores of the present invention on the five categories are mostly higher than those of the baseline method, and the present invention has a better open vocabulary segmentation effect.

[0050] At the same time, since the three-dimensional Gaussian field has been fully trained, when objects in the real scene are displaced or the scene changes, By quickly updating the 3D Gaussian field with a small amount of new images, real-time reconstruction of dynamic scenes can be achieved. This fast response capability gives the 3D Gaussian field a significant advantage in dealing with dynamically changing scenes.

[0051] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. An open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology, characterized in that: The following steps are involved: S1. Acquire a multi-view 2D image set and initialize 3D reconstruction using 3D Gaussian sputtering technology; S2. Use the image segmentation model (SAM) to segment each image in the multi-view set and obtain a 2D instance mask; S3. Associating masks of the same instance using the video tracking segmentation model (SAM2); S4. Extract open vocabulary features for each mask using the visual language model (CLIP); S5. Average pooling and dimension reduction of the CLIP features of each instance’s mask are performed to form a representative open vocabulary feature of the instance; S6. Using low-dimensional embedding features Initialize the 3D Gaussian volume and render the feature map L using the feature rasterizer; S7. Train the model to minimize the feature distance between the rendered feature map L and the original CLIP feature to supervise the low-dimensional embedding feature Embedding S8. By comparing the CLIP features of the query text with the semantic features of the Gaussian volume, a Gaussian volume with high correlation is obtained, and a mask of the query object is generated through rasterization rendering.

2. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: In step S1, the multi-view 2D image set of the three-dimensional scene or object is first obtained, and then the multi-view 2D images are processed using the motion recovery structure (SFM) technology to obtain a sparse three-dimensional point cloud, and these sparse three-dimensional point clouds are used to initialize a 3D Gaussian volume set to perform the initialized three-dimensional reconstruction.

3. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: In step S1, each Gaussian volume is represented by a mean µ and a three-dimensional covariance matrix Definition, where R and S represent the rotation matrix and scaling matrix, the center point µ of each Gaussian body corresponds to the three-dimensional coordinates of the feature point, the rotation matrix and scaling matrix are randomly initialized, and the shape and size of the Gaussian body are determined according to the distribution of the feature points and the disparity information.

4. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: In step S3, the multi-view images of the 3D scene are regarded as a video sequence with gradually changing views, and a pre-trained video tracking and segmentation model (SAM2) is used to track and associate masks of the same instance to achieve 2D mask consistency across views.

5. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: In step S5, all CLIP features of the mask of each instance are fused and passed through a learnable compression encoder The high-dimensional CLIP features are reduced in dimension to form representative open vocabulary features for the instance.

6. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: In step S5, the CLIP features of the mask of each instance are averaged and pooled to obtain the average CLIP feature F, which is defined as: (1) in Represents the jth 2D mask CLIP feature of the i-th instance, compressing the high-dimensional CLIP feature F to a low-dimensional embedding feature , defined as: ,in It stands for learnable compression encoder and consists of two fully connected layers.

7. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: In step S6, the low-dimensional embedding features are used Initialize a 3D Gaussian, embed the CLIP feature into the 3D Gaussian, and embed the low-dimensional feature Rendered as a parameter of a Gaussian volume, the feature map L is rendered using the feature rasterizer, defined as: (2) in is the open vocabulary feature embedded into the i-th Gaussian, Represents the open vocabulary feature map rendered at pixel u.

8. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: In step S7, m pixels are randomly selected from each mask in the rendered feature map to represent the mask, and then the potential CLIP features are restored through a trainable decoder Ψ, and the loss is calculated with the CLIP features obtained from the original RGB image to minimize the difference between the potential low-dimensional features and the CLIP features.

9. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: In step S7, the latent features of each 2D mask in the rendered feature map L are restored to the CLIP space, and then the distance between the restored latent CLIP features and the original CLIP features is minimized by randomly sampling m pixels in each mask and then passing through a trainable decoder consisting of two fully connected layers. Recover the latent CLIP features, where the feature distillation loss between the recovered latent CLIP features and CLIP features of all sampled m pixels is Defined as: (3)。 10. The open vocabulary 3D segmentation method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: In step S8, after training, the Gaussian can express semantic features and find the objects described by the language instructions through open vocabulary query. First, the relevance score s of each Gaussian text query is calculated, which is defined as: (4) in is the CLIP feature of the query text, is a set of standard phrases, Choose from Objects, Things, Stuff, and Textures. is a trainable decoder, L is a feature map rendered by Gaussian volumes; for each text query, a relevance heat map is obtained, in which Gaussian volumes with relevance scores below a predetermined threshold are filtered out, and the remaining area is rasterized to form a mask for predicting the queried object.

Images