An Open Semantic Segmentation Method for Indoor Point Cloud Scenes Based on Text Prompts

Abstract

This invention relates to an open semantic segmentation method for indoor point cloud scenes based on text prompts. Based on large-scale indoor scene point cloud data, this invention employs a 3D U-Net network structure based on Mamba blocks to extract local features of the point cloud scene, and extracts global features of the scene point cloud data through multi-level Mamba blocks and downsampling. Then, it recovers scene detail information through upsampling and skip connections to fuse the local-global features of the point cloud data. Text title prompts for multi-view perspectives of the scene are generated, and the projection matrix between the 2D view and the 3D scene is used to associate the text prompt features with the corresponding point cloud data features, aligning them. The weights of the category text embedding are then loaded into the scene segmentation head for semantic segmentation. During network training, a binary classification loss is added to the semantic segmentation loss to balance the scene semantic segmentation network's ability to recognize and understand basic and new categories. Finally, open semantic segmentation of indoor point cloud scenes is achieved.

Description

Technical Field

[0001] This invention belongs to the field of visual artificial intelligence technology and relates to a semantic segmentation method for indoor point cloud scenes based on text prompts. Background Technology

[0002] Point cloud scene semantic segmentation is a fundamental task in 3D scene understanding, which aims to distinguish the semantic categories of objects in a scene from 3D scan data, identify the location of objects, and infer their geometric properties. However, neural network models trained on manually annotated datasets can typically only understand the semantic categories of objects in that dataset—that is, they can identify the semantic categories of objects contained in the training set scene—but struggle to segment and understand the semantic categories of objects not visible in the training set scene. This significantly limits their applicability and generalization in various complex scenes.

[0003] Contrastive learning based on multimodal features offers a new approach to open semantic segmentation of point cloud scenes. By effectively utilizing scene text cues as auxiliary information for scene understanding, the semantic segmentation network can identify unknown semantic categories in the training dataset. Scene text cues, as a type of modal information that provides high-level scene semantics, can suggest prior knowledge such as scene object categories or scene semantics, thus compensating for the insufficiency of scene point cloud data containing only geometric semantic information. This invention utilizes a large-scale pre-trained language model to generate textual descriptions of high-level scene semantic information, adding additional semantic feature information to the scene point cloud and assisting the semantic segmentation network in learning fine-grained scene semantic feature representations. Furthermore, the introduction of scene text features helps reduce the segmentation network's dependence on large amounts of labeled data, improving the model's generalization ability.

[0004] To overcome the shortcomings of point cloud data irregularity, sparsity, and scene complexity in large-scale point cloud scene semantic segmentation, a text-based semantic segmentation method for indoor point cloud scenes is proposed. This method combines the aggregation of local information from scene point cloud data with global context modeling to effectively enhance the geometric features of scene point cloud data and facilitate open semantic segmentation of indoor point cloud scenes. Summary of the Invention

[0005] The purpose of this invention is to provide an open semantic segmentation method for indoor point cloud scenes based on text prompts. This method can achieve the capture of global context information of point cloud scenes with linear complexity. It combines the aggregation of local information of scene point cloud data with global context modeling and uses category text embedding to achieve open semantic segmentation of large-scale indoor point cloud scenes.

[0006] This invention is based on large-scale scene point cloud data input by users. It uses the SoftGroup method to voxelize the scene point cloud data and expand the voxel point cloud into a 1D sequence. Then, it employs Hilbert space curves and a recursive segmentation strategy to map the high-dimensional space to a 1D space. Next, in the network encoder, it combines 3D U-Net and Mamba structures, extracting global features of the scene point cloud data through multi-level Mamba blocks and downsampling. It then recovers scene detail information through upsampling and skip connections to fuse local-global point cloud features. Next, leveraging the projection correspondence between scene views and scene point clouds, it uses the ViT-GPT2 image captioning model to generate text title prompts for multiple views of the scene. Then, it uses a pre-trained CLIP text encoder to embed text features into the multi-view text title prompts, aligns the point cloud-text features using a point cloud text adapter, and loads the weights of the category text embedding into the point cloud semantic segmentation head, ultimately achieving open semantic segmentation of the point cloud scene.

[0007] Specifically, the following steps are included:

[0008] Step 1: Obtain 2D view data of the scene from the 3D scene, generate text title prompts for the multi-view scene, and then associate the scene point cloud with the scene text title.

[0009] Further, step one specifically involves: obtaining multi-view projection views from indoor scene point cloud data by utilizing the projection matrix between the 2D view and the 3D scene; then generating text title prompts for the multi-view images of the scene using the image captioning generation model ViT-GPT2; and finally associating the scene point cloud with the scene text title by using the projection matrix between the 2D view and the 3D scene.

[0010] Step 2: Embed the Mamba block into the U-Net framework to build a point cloud scene semantic segmentation network; use binary classification loss on the basis of semantic segmentation loss to balance the scene semantic segmentation network's ability to recognize and understand basic and new categories;

[0011] Step 3: Use Hilbert space curves to serialize the 3D structure of the point cloud and preserve its spatial relationships; the point cloud scene semantic segmentation network trained in Step 2 extracts and fuses the local-global features of the point cloud;

[0012] Specifically, the disordered voxel point cloud is converted into a 1D serialized structure suitable for Mamba modeling while preserving its 3D spatial relationships. The point cloud scene semantic segmentation network trained in step two uses a 3D U-Net network structure based on Mamba blocks to extract local features of the point cloud scene. It also extracts global features of the scene point cloud data through multi-level Mamba blocks and downsampling, and then restores scene detail information through upsampling and skip connections to fuse the local and global features of the point cloud.

[0013] Step 4: Extract text title prompts, generate text embeddings, and align point cloud-text features. Load the weights of the category text embeddings into the scene segmentation head for semantic segmentation tasks; ultimately achieving open semantic segmentation of large-scale indoor point cloud scenes.

[0014] Furthermore, a pre-trained CLIP text encoder Transformer is used to generate text embeddings from text title cues and point cloud-text features are aligned using a point cloud-text adapter.

[0015] Traditional point cloud scene semantic segmentation methods only use dataset semantic labels for network training and semantic segmentation. In contrast, the method in this invention trains the semantic segmentation network by comparing and learning features between scene point clouds and text prompts, ultimately achieving open semantic segmentation of large-scale indoor point cloud scenes.

[0016] Furthermore, the point cloud scene semantic segmentation network is based on the U-Net network framework, embedding Mamba blocks into two convolutional layers at each level of the U-Net framework downsampling process. This allows for the extraction of global features of the scene point cloud at different levels and the recovery of its local details through skip connections. The Mamba block includes a Hilbert curve layer, a linear projection layer, a 1D convolutional layer, a selective state space (SSM) block, skip connections, and a reshaping block.

[0017] Furthermore, in the process of converting disordered voxel point clouds into a 1D serialized structure suitable for Mamba modeling using Hilbert space curves, the scene point cloud data is first voxelized using the SoftGroup method and the voxel point cloud is expanded into a 1D sequence. Then, Hilbert space curves are used to map the high-dimensional space to the 1D space through a recursive segmentation strategy, which ensures that adjacent sampling points in the space remain adjacent in the expanded 1D sequence. Then, the Hilbert index is calculated for each sampling point, and the sampling points are sorted according to the index, finally obtaining a 1D sequence that preserves the original point cloud spatial structure.

[0018] In the open semantic segmentation network for indoor point cloud scenes proposed in this invention, a 3D U-Net network structure based on Mamba blocks is used to extract local features of the point cloud scene, and global features of the scene point cloud data are extracted through multi-level Mamba blocks and downsampling.

[0019] It can achieve linear complexity in capturing global contextual information of point cloud scenes. By combining the aggregation of local information from scene point cloud data with global context modeling, and using categorical text embedding, it achieves open semantic segmentation of indoor point cloud scenes. The method provided by this invention uses multimodal features such as scene point cloud data and text prompts, and enhances the semantic segmentation network's ability to understand open scenes through feature contrastive learning, thus achieving open semantic segmentation of point cloud scenes. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0021] Figure 2 This is an example process for converting scene point cloud data into text prompts in the embodiments;

[0022] Figure 3 A schematic diagram of the structure of a Mamba block in a point cloud scene semantic segmentation network;

[0023] Figure 4 A schematic diagram of a semantic segmentation network for a point cloud scene;

[0024] Figure 5 This is an example diagram illustrating the semantic segmentation effect of point cloud data in a conference room scenario in the embodiment.

[0025] Figure 6 This is an example diagram illustrating the semantic segmentation effect of point cloud data in an office scene, as shown in the embodiment.

[0026] Figure 7 This is a color illustration diagram for the 13 categories in the embodiment. Detailed Implementation

[0027] The technical method and semantic segmentation effect of the present invention will be further described and explained below with reference to the accompanying drawings.

[0028] like Figure 1 As shown, a text-based semantic segmentation method for indoor point cloud scenes consists of two branches: one branch extracts text features from text title prompts obtained from multi-view perspectives of the scene; the other branch performs voxelization on the input scene point cloud data and extracts point cloud features using a visual backbone network (point cloud scene semantic segmentation network). Then, through point cloud-text feature alignment, and using the category text embedding weights obtained by encoding the category names as the semantic segmentation head, open semantic segmentation of large-scale scene point cloud data is finally achieved. Specifically, the method includes the following steps:

[0029] Step 1: Obtain multi-view projection views from scene point cloud data using the projection matrix between the 2D view and the 3D scene. Then, generate text title prompts for the multi-view images of the scene using the image captioning generation model ViT-GPT2, thereby associating the scene point cloud with the scene text title.

[0030] In this embodiment, the first... A scenario The Using images as input, the pre-trained image captioning generation model ViT-GPT2 can generate their corresponding language descriptions. ,in A model for generating image captions. The generated results are as follows: Figure 2 As shown in the diagram, "a large room with tables and a chair in it" means "a large room with tables and chairs", "a room filled with board, desks, chairs, and a projector" means "a room with a blackboard, tables, chairs, and a projector", and "a room filled with board, chairs, and tables" means "a room with a blackboard, tables, and chairs".

[0031] Then, by using the projection matrix between the 2D view and the 3D scene to associate with the point cloud scene, the scene point cloud is then associated with the scene text title.

[0032] In this embodiment, the projection equation from world coordinates to image coordinates is: , where the projection matrix ,in K is the camera intrinsic parameter matrix. Let be a rotation matrix. It is a translation vector. 3D point coordinates These are planar coordinates.

[0033] Step 2: Embed the Mamba block into the U-Net framework to build a point cloud scene semantic segmentation network; use binary classification loss on the basis of semantic segmentation loss to balance the scene semantic segmentation network's ability to recognize and understand basic and new categories;

[0034] The loss function used for semantic segmentation of scene point clouds in this embodiment ;

[0035] Semantic segmentation loss The semantic loss for each sampling point is calculated using the semantic labels of the base categories, where For semantic score, Let σ be the semantic label, and σ be the Softmax function. For semantic segmentation header, Point cloud features are represented point by point;

[0036] Point cloud text contrast loss Then, the global features of the scene point cloud are obtained through pooling operations, and the contrast loss between the point cloud features and the matching text prompt features is calculated. This involves traversing point cloud-text paired samples and calculating point cloud features. Text features that match it similarity, This indicates iterating through all text features, for each fixed feature. Calculate point cloud features With all possible text features similarity, , It is a set of points associated with the text. It is a feature of the text association point set. It is a learnable temperature constant. This represents the number of given point cloud title cue pairs;

[0037] Binary cross-entropy loss Used to distinguish between the base category and the new category, where the predicted score This represents the probability that a sample point belongs to the new category. It is the predicted label (1 for belonging to the base category, 0 for belonging to the new category).

[0038] like Figure 3 As shown, the Mamba block includes Hilbert curve layers, linear projection layers, 1D convolutional layers, selective state space (SSM) blocks, skip connections, and reshape blocks. In Mamba, the state space (SSM) block applies a structured form to the state matrix and introduces specific algorithms. Specifically, it utilizes the High-Order Polynomial Projection Operator (HIPPO) to construct and initialize the state matrix, thereby building a deep sequence model with efficient long-range inference capabilities.

[0039] like Figure 4As shown, the point cloud scene semantic segmentation network takes point cloud data voxelized from scene point clouds as input. Based on the U-Net network structure, the network embeds Mamba blocks into two convolutional layers at each level of the U-Net architecture's downsampling process. This allows for the extraction of global point cloud features at different levels and the preservation of local details through skip connections. Traditional convolutional methods focus only on local regions, while the Mamba state-space model excels in processing long sequences, helping to capture global dependencies in large-scale point cloud data, and exhibits linear complexity compared to the Transformer.

[0040] Step 3: Use Hilbert space curves to serialize the 3D structure of the point cloud and preserve its spatial relationships; the point cloud scene semantic segmentation network trained in Step 2 extracts and fuses the local-global features of the point cloud;

[0041] Specifically, the disordered voxel point cloud is converted into a 1D serialized structure suitable for Mamba modeling using Hilbert space curves while preserving its 3D spatial relationships. The point cloud scene semantic segmentation network trained in step two uses a 3D U-Net network structure based on Mamba blocks to extract local features of the point cloud scene, and extracts global features of the scene point cloud data through multi-level Mamba blocks and downsampling. Then, it recovers scene detail information through upsampling and skip connections to fuse the local-global features of the point cloud.

[0042] The specific process of converting disordered voxel point clouds into a 1D serialized structure suitable for Mamba modeling is as follows: First, the scene point cloud data is voxelized using the SoftGroup method and the voxel point cloud is expanded into a 1D sequence. Then, Hilbert space curves are used to map the high-dimensional space to the 1D space through a recursive segmentation strategy, which ensures that adjacent sampling points in the space remain adjacent in the expanded 1D sequence. Next, the Hilbert index is calculated for each sampling point, and the sampling points are sorted according to the index, finally obtaining a 1D sequence that preserves the original point cloud spatial structure.

[0043] Step 4: Extract text title prompts, generate text embeddings, and align point cloud-text features. Load the weights of the category text embeddings into the scene segmentation head for semantic segmentation tasks; ultimately achieving open semantic segmentation of large-scale indoor point cloud scenes.

[0044] In this embodiment, the pre-trained CLIP text encoder Transformer is used to generate text feature embeddings based on text title cues, and a point cloud-text adapter is used to align scene point cloud-text features. Specifically, firstly, the pre-trained CLIP text encoder generates a 512-dimensional vector for the text title cues of each scene projection view. : , where t is the input text title prompt, The CLIP text encoder is then constructed; an MLP adapter is built to map 3D point cloud features to the text feature space to achieve feature alignment between point cloud and text modalities; finally, the weights of the category text feature embedding are loaded into the scene segmentation head for scene semantic segmentation.

[0045] like Figure 5 and Figure 7 As shown, the semantic segmentation results of point cloud data for a conference room scene in the S3DIS dataset are presented. Figure 5 In the middle, 'a' represents the input point cloud data of the conference room scene. Figure 5 In the middle, b represents the actual background category of the point cloud data for the conference room scene. Figure 5 Image c is an example of semantic segmentation of a point cloud scene in a conference room, achieved using the method described above. Figure 5 As can be seen, the semantic segmentation method for indoor point cloud scenes proposed in this invention can effectively segment point cloud data of conference room scenes. Its segmentation results can maintain the complete structure of the scene, and the boundaries between different objects obtained by segmentation are relatively clear. From the example image of the comparison between real background categories and semantic segmentation results, it can be seen that the categories such as chairs, tables, ceilings, floors, blackboards, walls, and pillars in the conference room point cloud scene can all be effectively segmented.

[0046] like Figure 6 and Figure 7 As shown, the semantic segmentation results of point cloud data for an office scene in the S3DIS dataset are presented. Figure 6 In the image, 'a' represents the input point cloud data of the office scene. Figure 6 In the middle, b represents the actual background category of the point cloud data for the office scene. Figure 6 Image c is an example of semantic segmentation of an office point cloud scene achieved using the method described above. Figure 6 As can be seen, the semantic segmentation method for indoor point cloud scenes proposed in this invention can effectively segment point cloud data of office scenes. From the example images comparing the real background categories and semantic segmentation results, it can be seen that most architectural elements in the office scene can be accurately segmented while maintaining their structural integrity. The method proposed in this invention can not only effectively segment visible categories during training but also accurately segment new invisible categories during training, ultimately achieving open semantic segmentation of large-scale indoor point cloud scenes.

Claims

1. An open semantic segmentation method for indoor point cloud scenes based on text prompts, characterized in that: Specifically, the following steps are included: Step 1: Obtain 2D view data of the scene from the 3D scene, generate text title prompts for the multi-view scene, and then associate the scene point cloud with the scene text title. Step 2: Embed the Mamba block into the U-Net framework to build a point cloud scene semantic segmentation network; use binary classification loss on the basis of semantic segmentation loss to balance the scene semantic segmentation network's ability to recognize and understand basic and new categories; Step 3: Use Hilbert space curves to serialize the 3D structure of the point cloud and preserve its spatial relationships; the point cloud scene semantic segmentation network trained in Step 2 extracts and fuses the local-global features of the point cloud; Specifically: The disordered voxel point cloud is converted into a 1D serialized structure suitable for Mamba modeling while preserving its 3D spatial relationships. The point cloud scene semantic segmentation network trained in step two uses a 3D U-Net network structure based on Mamba blocks to extract local features of the point cloud scene. It also extracts global features of the scene point cloud data through multi-level Mamba blocks and downsampling, and then restores scene detail information through upsampling and skip connections to fuse the local and global features of the point cloud. Step 4: Extract text title prompts, generate text embeddings, and align point cloud-text features. Load the weights of the category text embeddings into the scene segmentation head for semantic segmentation tasks; ultimately achieving open semantic segmentation of large-scale indoor point cloud scenes.

2. The open semantic segmentation method for indoor point cloud scenes based on text prompts as described in claim 1, characterized in that: Step one involves obtaining multi-view projection views from indoor scene point cloud data using the projection matrix between the 2D view and the 3D scene. Then, text title prompts for the multi-view images of the scene are generated using the image caption generation model ViT-GPT2. Finally, the projection matrix between the 2D view and the 3D scene is used to associate the point cloud scene with the text title, thereby associating the point cloud text title with the scene.

3. The open semantic segmentation method for indoor point cloud scenes based on text prompts as described in claim 1, characterized in that: The point cloud scene semantic segmentation network is based on the U-Net network framework. It embeds Mamba blocks into two convolutional layers at each level of the U-Net framework downsampling process, thereby extracting global features of the scene point cloud at different levels and restoring its local detail information through skip connections.

4. The open semantic segmentation method for indoor point cloud scenes based on text prompts as described in claim 1 or 3, characterized in that: The Mamba block includes Hilbert curve layers, linear projection layers, 1D convolutional layers, selective state space SSM blocks, skip connections, and reshape blocks.

5. The open semantic segmentation method for indoor point cloud scenes based on text prompts as described in claim 1, characterized in that: In step three, which uses Hilbert space curves to convert disordered voxel point clouds into a 1D serialized structure suitable for Mamba modeling, the scene point cloud data is first voxelized using the SoftGroup method and then expanded into a 1D sequence using Hilbert space curves through a recursive segmentation strategy. This ensures that adjacent sampling points remain adjacent in the expanded 1D sequence. Then, a Hilbert index is calculated for each sampling point, and the sampling points are sorted according to the index, ultimately resulting in a 1D sequence that preserves the original point cloud spatial structure.

6. The open semantic segmentation method for indoor point cloud scenes based on text prompts as described in claim 1, characterized in that: In step four, during the text embedding extraction process, the pre-trained CLIP text encoder Transformer is used to generate text embeddings from the text title prompts and aligns the point cloud-text features using a point cloud-text adapter. The weights of the category text embeddings are then loaded into the scene segmentation head for semantic segmentation tasks, ultimately achieving open semantic segmentation of large-scale indoor point cloud scenes.

Images