3D scene semantic segmentation system and method with open vocabulary

Abstract

The application discloses a 3D scene semantic segmentation system and method of an open vocabulary, and belongs to the technical field of vision and image processing, and comprises the following modules: a point cloud input module, which is used for inputting preprocessed point cloud data and forming a point cloud set; a multi-scale point cloud feature extraction module, which is used for extracting geometric local features of point cloud data in the point cloud set, further extracting context features of the geometric local features, and finally summarizing full-scale fusion features; a CLIP language embedding module, which is used for mapping a natural language sentence to a semantic embedding vector through a CLIP text encoder and performing normalization processing on the semantic embedding vector; and a vision-language semantic alignment module.The application can match natural language labels input by a user with 3D point cloud data, supports arbitrary semantic description, does not need to retrain a model, and thus realizes flexible and semantically rich three-dimensional space understanding.

Description

Technical Field

[0001] This invention relates to the field of vision and image processing technology, specifically to a 3D scene semantic segmentation system and method with open vocabulary. Background Technology

[0002] With the rapid development of artificial intelligence technology, 3D perception systems are playing an increasingly important role in fields such as robotics, autonomous driving, and augmented reality. Especially in home service robots and smart home systems, 3D scene semantic segmentation has become one of the core technologies. The task of semantic segmentation is to assign a semantic label to each point in the 3D point cloud data acquired by sensors, thereby helping the system understand the type and location of various objects in the scene.

[0003] However, traditional 3D semantic segmentation methods rely on closed-vocabulary classifiers, which predefine all possible semantic categories during the training phase, such as "sofa," "table," and "chair." During deployment, the system cannot adapt to new semantic labels proposed by users, such as "a suitable area for placing a remote control" or "a surface that can withstand a kettle," thus greatly limiting its practicality and scalability.

[0004] In recent years, the emergence of cross-modal vision-language models such as CLIP (Contrastive Language-Image Pretraining) has made open-vocabulary recognition possible. These models, trained on large-scale image-text pairing data, learn to project natural language and visual features into the same semantic space. Therefore, embedding CLIP into a 3D semantic segmentation system enables the 3D system to understand natural language descriptions and perform semantic segmentation.

[0005] However, several technical challenges remain in migrating the CLIP model from the image domain to the point cloud domain and achieving 3D semantic segmentation of open vocabularies. First, point cloud data differs significantly from images in structure; point clouds are sparse, irregular three-dimensional spatial distributions, while images are two-dimensional data on a regular grid. Second, point clouds lack high-dimensional semantic information such as texture and color, requiring new methods to construct semantic relationships. Third, single-point semantic labels cannot represent complex semantic relationships between objects, such as "a cup on the table" or "a sofa adjacent to the wall."

[0006] Based on this, the present invention designs an open-vocabulary 3D scene semantic segmentation system and method to solve the above problems. Summary of the Invention

[0007] To address the aforementioned shortcomings of existing technologies, this invention provides an open-vocabulary 3D scene semantic segmentation system and method.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] An open-vocabulary 3D scene semantic segmentation system, including:

[0010] Point cloud input module: used to input preprocessed point cloud data and form a point cloud set;

[0011] Multi-scale point cloud feature extraction module: used to extract the geometric local features of point cloud data within the point cloud set, then extract the context features of the geometric local features, and finally summarize them to obtain the full-scale fused features;

[0012] CLIP Language Embedding Module: Used to map natural language statements into semantic embedding vectors via CLIP text encoder, and to normalize the semantic embedding vectors;

[0013] The visual-language semantic alignment module is used to obtain point mapping features by fusing feature maps at all scales, and then combine them with normalized semantic vectors to obtain the probability prediction value of a point belonging to a natural language description semantic region.

[0014] Scene graph construction and graph neural network inference module: used to construct semantic scene graphs and determine node features;

[0015] Semantic prediction module: used to combine node features and probability prediction values ​​to obtain matching confidence.

[0016] A segmentation method for an open-vocabulary 3D scene semantic segmentation system includes the following steps:

[0017] Step 1: Input the preprocessed point cloud data into the point cloud input module to form a point cloud set. ;

[0018] Indicates the first The three-dimensional spatial coordinates of the points Optionally, its color information can be represented;

[0019] Step 2: Based on the point cloud set Full-scale fusion features are obtained from point cloud data. ;

[0020] Step 3: The CLIP text encoder maps the natural language sentence T to obtain the semantic embedding vector. The semantic embedding vector is then normalized to obtain a normalized semantic vector. ;

[0021] Step 4: Full-scale fusion features Mapping yields point mapping features Then, it is compared with the normalized semantic vector. The probability prediction value obtained by combining the points belongs to the semantic region described in natural language is as follows: ;

[0022] Step 5: Construct a semantic scene graph and determine node features;

[0023] Step 6: Node Features and probability prediction value Combine to obtain the matching confidence level .

[0024] Furthermore, the specific steps of step 2 are as follows:

[0025] Step 21: Extract the point cloud set Geometric local features of point cloud data ;

[0026] Step 22: Extract local geometric features Contextual features ;

[0027] Step 23: Based on contextual features Obtain full-scale fusion features .

[0028] Furthermore, the specific steps of step 22 are as follows:

[0029] Step 221: The feature of a point cloud in a certain layer is:

[0030] Step 222: Determine the different scales upsampling neighborhood :

[0031] Step 223: For each scale Extract contextual features .

[0032] Furthermore, the specific steps for step 4 are as follows:

[0033] Step 41: Fusion of full-scale features Mapping yields point mapping features ;

[0034] Step 42: Map features to points After normalization, we get ;

[0035] Step 43: semantic vectors with normalization Cosine similarity is obtained by combining ;

[0036] Step 44: Then, based on the cosine similarity... Predicted probability value of a point belonging to a semantic region of a natural language description .

[0037] Furthermore, The specific calculations are as follows:

[0038]

[0039] in, This is the temperature scaling factor.

[0040] Furthermore, the specific steps of step 5 are as follows:

[0041] Step 51: Construct a semantic scene graph Candidate semantic region nodes in the scene Semantic relation edges ;

[0042] Step 52: Calculate the center coordinates of each node and normal vector ;

[0043] Step 53: Calculate the relationship features of the edges between node i and node j. ;

[0044] Step 54: Relationship Characteristics ;

[0045] Step 55: Graph Neural Network Combined with Edge Representation The initial characteristics of the nodes are Semantic features of propagation nodes ;

[0046] Step 56: Through Node features output after layer propagation .

[0047] Furthermore, matching confidence level The specific calculations are as follows:

[0048]

[0049] in, For adjustable hyperparameters, It is the Sigmoid activation function. It is a linear mapping weight vector.

[0050] Beneficial effects: This invention can match user-input natural language labels with 3D point cloud data, enabling support for arbitrary semantic descriptions without retraining the model. This achieves flexible and semantically rich three-dimensional spatial understanding, breaking through the limitations of traditional closed-vocabulary label classification methods. It has significant improvements in semantic expression, generalization, and interactive response capabilities, and is of great practical and promotional value. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0052] Figure 1 This is a block diagram of the open-vocabulary 3D scene semantic segmentation system of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0054] The present invention will be further described below with reference to embodiments.

[0055] Example 1: Please refer to Figure 1 An open-vocabulary 3D scene semantic segmentation system, including:

[0056] Point cloud input module: used to input preprocessed point cloud data and form a point cloud set;

[0057] Preprocessing includes noise reduction, coordinate normalization, and format conversion;

[0058] Multi-scale point cloud feature extraction module: used to extract the geometric local features of point cloud data within the point cloud set, then extract the context features of the geometric local features, and finally summarize them to obtain the full-scale fused features;

[0059] CLIP Language Embedding Module: Used to map natural language statements into semantic embedding vectors via CLIP text encoder, and to normalize the semantic embedding vectors;

[0060] The visual-language semantic alignment module is used to obtain point mapping features by fusing feature maps at all scales, and then combine them with normalized semantic vectors to obtain the probability prediction value of a point belonging to a natural language description semantic region.

[0061] Scene graph construction and graph neural network inference module: used to construct semantic scene graphs and determine node features;

[0062] Semantic prediction module: used to combine node features and probability prediction values ​​to obtain matching confidence.

[0063] Match confidence is a comprehensive indicator of the degree of matching between the preprocessed point cloud data and the natural language statement in the final output of the system. It takes into account both the similarity of visual-language alignment and the reasoning results of semantic relationships in the scene graph, thereby achieving more accurate and robust open-vocabulary 3D semantic segmentation.

[0064] For the first time, the visual-language semantic alignment module and the CLIP language embedding module are combined to achieve cross-modal label reasoning by sharing a semantic space, breaking the limitation of fixed labels in traditional 3D semantic segmentation tasks.

[0065] The similarity between point clouds and natural language sentences is calculated, enabling support for arbitrary semantic descriptions without the need to retrain the model.

[0066] By introducing a scene graph and then strengthening semantic connections through GNN propagation, the system can infer implicit semantic regions.

[0067] It not only uses the local geometric information of point clouds, but also integrates contextual scene graph relationships, language embedding similarity, multi-scale feature pyramids and other methods to obtain the matching confidence.

[0068] Supports interactive interfaces for dynamic scenarios and multi-user needs: The system supports multiple input methods (voice, text, gesture) to describe semantic tags. Combined with CLIP language embedding and scene graph structure, it can realize real-time dynamic semantic understanding, which is especially suitable for human-computer interaction tasks in home service robots and smart furniture.

[0069] It can match user-input natural language labels with 3D point cloud data, supporting arbitrary semantic descriptions without retraining the model. This enables flexible and semantically rich 3D spatial understanding, breaking through the limitations of traditional closed-vocabulary label classification methods. It significantly improves semantic expression, generalization ability, and interactive response capabilities, making it highly practical and valuable for widespread application, especially suitable for:

[0070] Home service robots (such as robot vacuum cleaners and companion robots);

[0071] Smart furniture and home control (such as curtains, lighting, and desktop recognition);

[0072] Augmented reality and spatial interaction experience system;

[0073] Semantic region-based assisted navigation in smart healthcare and wellness scenarios.

[0074] Example 2: A segmentation method for an open-vocabulary 3D scene semantic segmentation system, comprising the following steps:

[0075] Step 1: Input the preprocessed point cloud data into the point cloud input module to form a point cloud set. ;

[0076] Indicates the first The three-dimensional spatial coordinates of the points Optionally, its color information can be represented;

[0077] Step 2: Based on the point cloud set Full-scale fusion features are obtained from point cloud data. ;

[0078] Step 21: Extract the point cloud set Geometric local features of point cloud data ;

[0079] ;

[0080] For the point The set of points in the neighborhood of the center, For feature dimensions;

[0081] 128 or 256 dimensions;

[0082]

[0083] Indicates the first The three-dimensional spatial coordinates of the points This indicates its color information;

[0084] Step 22: Extract local geometric features Contextual features ;

[0085] Step 221: The feature of a point cloud in a certain layer is:

[0086] Step 222: Determine the different scales upsampling neighborhood :

[0087]

[0088] Step 223: For each scale Extract contextual features :

[0089] The specific calculations are as follows:

[0090]

[0091] Step 23: Based on contextual features Obtain full-scale fusion features ;

[0092] The specific calculations are as follows:

[0093]

[0094] This is expressed as the number of branches;

[0095] Represented as the first Characteristic transformation function of each branch;

[0096] Step 3: The CLIP text encoder maps the natural language sentence T to obtain the semantic embedding vector. The semantic embedding vector is then normalized to obtain a normalized semantic vector. ;

[0097] The specific calculations are as follows:

[0098]

[0099]

[0100] and Same dimension;

[0101] Step 4: Full-scale fusion features Mapping yields point mapping features Then, it is compared with the normalized semantic vector. The probability prediction value obtained by combining the points belongs to the semantic region described in natural language is as follows: ;

[0102] The specific steps for step 4 are as follows:

[0103] Step 41: Fusion of full-scale features Mapping yields point mapping features ;

[0104] The specific calculations are as follows:

[0105]

[0106] Step 42: Map features to points After normalization, we get ;

[0107] The specific calculations are as follows:

[0108]

[0109] Step 43: semantic vectors with normalization Cosine similarity is obtained by combining ;

[0110] The specific calculations are as follows:

[0111]

[0112] Step 44: Then, based on the cosine similarity... Predicted probability value of a point belonging to a semantic region of a natural language description ;

[0113] The specific calculations are as follows:

[0114]

[0115] in, This is a temperature scaling factor used to adjust the smoothness of the activation function's distribution.

[0116] Output That is, to represent a point The probability of belonging to a semantic region described in natural language;

[0117] Step 5: Construct a semantic scene graph and determine node features;

[0118] The specific steps for step 5 are as follows:

[0119] Step 51: Construct a semantic scene graph Candidate semantic region nodes in the scene Semantic relation edges ;

[0120] Step 52: Calculate the center coordinates of each node and normal vector ;

[0121] Node center coordinates and normal vector The specific calculations are as follows:

[0122]

[0123] Step 53: Calculate the relationship features of the edges between node i and node j. ;

[0124] The specific calculations are as follows:

[0125]

[0126] Step 54: Relationship Characteristics ;

[0127] The specific calculations are as follows:

[0128]

[0129] Represented as a relational network;

[0130] Dimensions of edge features

[0131] Step 55: Graph Neural Network Combined with Edge Representation The initial characteristics of the nodes are Semantic features of propagation nodes ;

[0132] The specific calculations are as follows:

[0133]

[0134] This represents the updated layer number. For message functions;

[0135] Step 56: Through Node features output after layer propagation ;

[0136] The specific calculations are as follows:

[0137]

[0138] For gating update unit;

[0139] Step 6: Node Features and probability prediction value Combine to obtain the matching confidence level .

[0140] Matching confidence The specific calculations are as follows:

[0141]

[0142] in, For adjustable hyperparameters, It is the Sigmoid activation function. It is a linear mapping weight vector.

[0143] , Node features Dimensions.

[0144] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An open-vocabulary 3D scene semantic segmentation system, characterized by: include: Point cloud input module: used to input preprocessed point cloud data and form a point cloud set; Multi-scale point cloud feature extraction module: used to extract the geometric local features of point cloud data within the point cloud set, then extract the context features of the geometric local features, and finally summarize them to obtain the full-scale fused features; CLIP Language Embedding Module: Used to map natural language statements into semantic embedding vectors via CLIP text encoder, and to normalize the semantic embedding vectors; The visual-language semantic alignment module is used to obtain point mapping features by fusing feature maps across the entire scale, and then combine them with normalized semantic vectors to obtain the probability prediction value of a point belonging to a natural language description semantic region. Scene graph construction and graph neural network inference module: used to construct semantic scene graphs and determine node features; specific steps are as follows: Step 51: Construct a semantic scene graph Candidate semantic region nodes in the scene Semantic relation edges ; Step 52: Calculate the center coordinates of each node and normal vector ; Step 53: Calculate the relationship features of the edges between node i and node j. ; Step 54: Relationship Characteristics ; Step 55: Graph Neural Network Combined with Edge Representation The initial characteristics of the nodes are Semantic features of propagation nodes ; Step 56: Through Node features output after layer propagation ; The semantic prediction module combines node features and probability prediction values ​​to obtain the matching confidence score. Matching confidence The specific calculations are as follows: ； in, For adjustable hyperparameters, It is the Sigmoid activation function. It is a linear mapping weight vector.

2. A segmentation method for an open-vocabulary 3D scene semantic segmentation system according to claim 1, characterized in that, Includes the following steps: Step 1: Input the preprocessed point cloud data into the point cloud input module to form a point cloud set. ; Step 2: Based on the point cloud set Full-scale fusion features are obtained from point cloud data. ; Step 3: The CLIP text encoder maps the natural language sentence T to obtain the semantic embedding vector. The semantic embedding vector is then normalized to obtain a normalized semantic vector. ; Step 4: Full-scale fusion features Mapping yields point mapping features Then, with the normalized semantic vector The probability prediction value obtained by combining the points belongs to the semantic region described in natural language is as follows: ; Step 5: Construct a semantic scene graph and determine node features; the specific steps are as follows: Step 51: Construct a semantic scene graph Candidate semantic region nodes in the scene Semantic relation edges ; Step 52: Calculate the center coordinates of each node and normal vector ; Step 53: Calculate the relationship features of the edges between node i and node j. ; Step 54: Relationship Characteristics ; Step 55: Graph Neural Network Combined with Edge Representation The initial characteristics of the nodes are Semantic features of propagation nodes ; Step 56: Through Node features output after layer propagation ; Step 6: Node Features and probability prediction value Combined to obtain the matching confidence level ; Matching confidence The specific calculations are as follows: ； in, For adjustable hyperparameters, It is the Sigmoid activation function. It is a linear mapping weight vector.

3. The segmentation method according to claim 2, characterized in that, The specific steps for step 2 are as follows: Step 21: Extract the point cloud set Geometric local features of point cloud data ; Step 22: Extract local geometric features Contextual features ; Step 23: Based on contextual features Obtain full-scale fusion features .

4. The segmentation method according to claim 3, characterized in that, The specific steps of step 22 are as follows: Step 221: The feature of a point cloud in a certain layer is: ; Step 222: Determine the different scales upsampling neighborhood : Step 223: For each scale Extract contextual features .

5. The segmentation method according to claim 4, characterized in that, The specific steps for step 4 are as follows: Step 41: Fusion of full-scale features Mapping yields point mapping features ; Step 42: Map features to points After normalization, we get ; Step 43: semantic vectors with normalization Cosine similarity is obtained by combining ; Step 44: Then, based on the cosine similarity... The predicted probability that a point belongs to a semantic region described in natural language. .

6. The segmentation method according to claim 5, characterized in that, The specific calculations are as follows: ； in, This is the temperature scaling factor.

Images