A confidence-guided multi-expert fusion open-vocabulary 3D point cloud segmentation method

Abstract

The present application relates to point cloud semantic segmentation technology, especially to a confidence-guided multi-expert fusion open vocabulary three-dimensional point cloud segmentation method, comprising obtaining three-dimensional geometric features of to-be-segmented point cloud data from the to-be-segmented point cloud data by using a pre-trained student encoder, obtaining an adaptive text feature dictionary by using a CLIP text encoder, multiplying the obtained three-dimensional geometric features and the adaptive text feature dictionary, and obtaining a final segmentation result; the present application combines multi-model fusion, confidence-guided distillation and learnable prompt optimization, solves the problem of insufficient semantic supervision robustness and limited alignment accuracy of fixed text features and three-dimensional features caused by the dependence of the existing open vocabulary three-dimensional point cloud segmentation method on a single two-dimensional model, proposes a confidence-guided multi-model feature fusion strategy and a scene-adaptive text prompt optimization scheme, and solves the problems of multi-source model feature heterogeneity and poor scene adaptability of text features.

Description

Technical Field

[0001] This invention relates to point cloud semantic segmentation technology, and in particular to a confidence-guided multi-expert fusion open vocabulary 3D point cloud segmentation method. Background Technology

[0002] 3D point cloud semantic segmentation is a fundamental task in 3D scene understanding, assigning semantic category labels to each 3D point and finding wide applications in autonomous driving, robot navigation, and augmented reality. The development of deep learning has driven advancements in closed-set 3D point cloud semantic segmentation methods. These methods can achieve high-precision semantic segmentation, but they can only recognize fixed categories predefined during the training phase, failing to meet the segmentation needs of arbitrary text descriptions in open scenes. This limitation severely restricts their practical application.

[0003] The emergence of vision-language pre-trained models has driven the rapid development of open-vocabulary 2D image semantic segmentation. Strategies such as pixel-level alignment, two-stage segmentation, and multi-task unified modeling have been proposed, successfully achieving zero-shot segmentation of unseen categories in 2D images. Inspired by this, open-vocabulary 3D point cloud semantic segmentation has become a research hotspot. Existing methods all rely on camera parameters to establish the association between 2D images and 3D point clouds, transferring the semantic knowledge of 2D vision-language models to 3D space. Based on the supervision signal, they can be divided into three categories: text supervision, image supervision, and joint text-image supervision.

[0004] Existing open-vocabulary 3D point cloud semantic segmentation methods still suffer from two major unresolved technical limitations: First, most methods rely on a single 2D open-vocabulary model or single-modal supervision, failing to leverage the complementary advantages of multi-source models for integrated learning at the feature and prediction levels. This results in insufficient robustness of semantic supervision and the reliability of feature transfer is easily affected by scene interference. Second, fixed text features cannot adapt to the point cloud distribution features of different scenes, exhibiting alignment deviations with 3D point features, thus limiting semantic matching accuracy and cross-scene generalization ability. Summary of the Invention

[0005] To address the shortcomings of existing open-vocabulary 3D point cloud semantic segmentation techniques, which rely on a single 2D open-vocabulary model leading to insufficient robustness of semantic supervision and limited alignment accuracy between fixed text features and 3D point features, this invention proposes a confidence-guided multi-expert fusion open-vocabulary 3D point cloud segmentation method. This method utilizes a pre-trained student encoder to obtain the 3D geometric features from the point cloud data to be segmented, and a CLIP text encoder to obtain its adapted text feature dictionary. The obtained 3D geometric features are multiplied by the adapted text feature dictionary to obtain the final segmentation result. The training process of the student encoder and the CLIP text encoder includes:

[0006] Construct training data, where each training sample consists of 3D point cloud data and its corresponding multi-view RGB image;

[0007] Two different two-dimensional segmentation models are used to obtain two-dimensional image features from multi-view RGB images, and the segmentation results are fused based on the confidence of the two-dimensional image features to obtain two-dimensional fused features;

[0008] A point cloud encoder is used to extract 3D point cloud features from 3D point cloud data. Based on the camera intrinsic and extrinsic parameter matrices, the 3D point cloud features are matched with the 2D fusion features. The difference between the 3D point cloud features and the 2D fusion features at each matching point is calculated. The point cloud encoder is trained with the goal of minimizing this difference.

[0009] The point cloud encoder was used as the teacher model, and the network parameters of the teacher model were frozen. The student model and CLIP text encoder were further trained under the guidance of the teacher model.

[0010] The point cloud data to be segmented is input into the trained student model to obtain its 3D geometric features, and input into the trained CLIP text encoder to obtain its adapted text feature dictionary. The 3D geometric features and the adapted text feature dictionary are multiplied together to obtain the final segmentation result.

[0011] Preferably, the two different two-dimensional segmentation models of the present invention are the LSeg segmentation model (from "LANGUAGE-DRIVEN SEMANTIC SEGMENTATION") and the SEEM segmentation model (from "Segment Everything Everywhere All at Once").

[0012] This invention combines multi-model fusion, confidence-guided distillation, and learnable prompt optimization to address the shortcomings of existing open-vocabulary 3D point cloud segmentation methods, which rely on a single 2D model and suffer from insufficient semantic supervision robustness and limited alignment accuracy between fixed text features and 3D features. It proposes a confidence-guided multi-model feature fusion strategy and a scene-adaptive text prompt optimization scheme to solve the problems of heterogeneous features from multiple sources and poor scene adaptability of text features. This invention mines complementary semantic information through multi-model fusion, improves knowledge transfer accuracy with dynamic confidence weights, and enhances text-3D feature alignment with learnable prompts, achieving open-vocabulary 3D point cloud segmentation. Furthermore, this invention eliminates the need for RGB images during the inference stage, requiring only the original point cloud as input for segmentation, simplifying the application process, reducing practical deployment difficulty, and enhancing the method's practical value. Attached Figure Description

[0013] Figure 1 This is an overall structural diagram of a confidence-guided multi-expert fusion open vocabulary 3D point cloud segmentation method according to the present invention;

[0014] Figure 2 This is a schematic diagram of the training phase process of the present invention;

[0015] Figure 3 This is a schematic diagram of the confidence-guided distillation process during the training phase of this invention;

[0016] Figure 4 This is a schematic diagram comparing the segmentation results of the present invention with those of existing segmentation methods. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This invention proposes a confidence-guided multi-expert fusion open-vocabulary 3D point cloud segmentation method. It utilizes a pre-trained student encoder to obtain the 3D geometric features from the point cloud data to be segmented, and a CLIP text encoder to obtain its adapted text feature dictionary. The obtained 3D geometric features are multiplied by the adapted text feature dictionary to obtain the final segmentation result. The training process of the student encoder and the CLIP text encoder includes:

[0019] Construct training data, where each training sample consists of 3D point cloud data and its corresponding multi-view RGB image;

[0020] Two different two-dimensional segmentation models are used to obtain two-dimensional image features from multi-view RGB images, and the segmentation results are fused based on the confidence of the two-dimensional image features to obtain two-dimensional fused features;

[0021] A point cloud encoder is used to extract 3D point cloud features from 3D point cloud data. Based on the camera intrinsic and extrinsic parameter matrices, the 3D point cloud features are matched with the 2D fusion features. The difference between the 3D point cloud features and the 2D fusion features at each matching point is calculated. The point cloud encoder is trained with the goal of minimizing this difference.

[0022] The point cloud encoder was used as the teacher model, and the network parameters of the teacher model were frozen. The student model and CLIP text encoder were further trained under the guidance of the teacher model.

[0023] The point cloud data to be segmented is input into the trained student model to obtain its 3D geometric features, and input into the trained CLIP text encoder to obtain its adapted text feature dictionary. The 3D geometric features and the adapted text feature dictionary are multiplied together to obtain the final segmentation result.

[0024] To address the shortcomings of existing open-vocabulary 3D point cloud semantic segmentation techniques, which rely on a single 2D open-vocabulary model leading to insufficient robustness of semantic supervision and limited alignment accuracy between fixed text features and 3D point features, this invention integrates the complementary features of multiple pre-trained 2D open-vocabulary models. Combined with confidence-guided feature fusion and knowledge distillation strategies, it achieves reliable 2D-to-3D semantic knowledge transfer. Simultaneously, a learnable hint module dynamically adapts text features to point cloud scene distribution features. This effectively improves the reliability of semantic supervision and the alignment accuracy of text-3D features in open-vocabulary 3D point cloud semantic segmentation, alleviates feature redundancy and missing effective information, and ultimately enhances the semantic segmentation performance and cross-scene generalization ability of 3D point clouds in open-vocabulary scenarios.

[0025] The overall approach of this invention is to first fuse features from multiple two-dimensional open vocabulary models to generate robust semantic features, then transfer these features to a three-dimensional backbone network, and finally optimize the alignment between the text and the three-dimensional features. Specifically, as... Figure 1 First, pixel-by-pixel features of the image are extracted using multiple pre-trained two-dimensional open vocabulary models. After feature alignment and confidence-weighted fusion, the features are mapped to a three-dimensional point cloud through camera parameters to obtain three-dimensional features with two-dimensional semantic information. Then, these semantic information are transferred to the three-dimensional backbone network through a teacher-student architecture to complete the model initialization. Finally, the text features are adjusted through learnable text prompts to further optimize the model performance.

[0026] Existing open-vocabulary 3D point cloud segmentation methods, when transferring 2D knowledge to 3D, either directly project the features of a single 2D model to 3D or employ a simple distillation strategy, failing to consider the reliability differences of features in different regions, resulting in insufficient accuracy of the transferred semantic information. This invention improves upon this by introducing confidence-guided dynamic weights during the transfer process. First, the prediction confidence of the teacher signal (fused 3D features) and the student model (3D backbone network) is calculated. The difference between these two is used to adjust the learning weights for different regions, allowing the student model to prioritize learning features from high-reliability regions and reduce interference from low-reliability regions. Specifically, a Minkowski network is used as the 3D backbone network, and the fused 3D features are used as the teacher signal. Weighted cosine similarity loss is used to ensure consistency between teacher and student features, thereby completing the initial knowledge transfer.

[0027] Meanwhile, previous methods lacked scene adaptation design for text features, and fixed text features could not match the scene differences of different datasets. To address this, we added a learnable prompt module, incorporating a linear adaptation layer at the input of the CLIP text encoder. This layer generates scene-adapted prompt words from trainable vectors, which are then added to the original text features to obtain the adapted text features. To ensure training stability, this invention reuses the initially trained 3D model as the teacher model and freezes its parameters. A trainable student model with the same structure is then reinitialized, and its parameters are updated using an exponential moving average strategy. The total loss includes a corrected confidence-guided distillation loss and a text-3D alignment loss. Joint training enhances the semantic matching ability of the student model with the adapted text features, resolving the alignment deviation between text and 3D features.

[0028] This method eliminates the need for RGB images during the inference phase, requiring only the original point cloud as input for semantic segmentation. It extracts 3D features from each point using a trained student model, encodes the target category text using CLIP, and combines this with trained prompt word vectors to obtain suitable text features. The cosine similarity between the 3D features and the suitable text features for each category is calculated, and the category corresponding to the highest value is used as the final prediction result. By fusing multiple models to mine complementary semantic information, leveraging confidence-based guidance to improve knowledge transfer accuracy, and incorporating learnable prompts to optimize text-3D alignment, this method ultimately enhances the performance and cross-scene generalization ability of open-vocabulary 3D point cloud segmentation, while simplifying the inference process and improving the convenience of practical applications.

[0029] As shown in the figure, this method is divided into three core stages, each corresponding to a different module in the figure: the 2D integration stage, the 2D-3D distillation stage, and the self-distillation stage. The core functions of each stage and its corresponding module are as follows:

[0030] 1. 2D integration stage.

[0031] like Figure 2 #Phase 0: The core of this phase is to integrate the features of multiple two-dimensional models. In this embodiment, LSeg and SEEM are selected as representative two-dimensional models. The core of the two-dimensional open vocabulary model inference is to achieve category prediction through similarity matching of text-image features. For each pixel, the prediction confidence of LSeg and SEEM is calculated (in this embodiment, the prediction confidence is defined as the set of categories). superior The maximum output value is calculated using the following formula:

[0032]

[0033]

[0034] in, The prediction confidence of the LSeg segmentation model; This indicates finding the maximum value; express function; CLIP-encoded text features representing all target categories Represents the set of real numbers. This represents the total number of target categories. Indicates feature dimension; This indicates the transpose; This represents the prediction confidence of the SEEM segmentation model.

[0035] To highlight the reliability of high-confidence predictions, this embodiment employs a confidence-weighted strategy to fuse pixel-by-pixel features from the two models, as shown in the formula:

[0036]

[0037] in, It is a two-dimensional fusion feature; The segmentation results of the LSeg segmentation model; The segmentation results are from the SEEM segmentation model.

[0038] This embodiment then utilizes the camera's intrinsic and extrinsic parameter matrices to project the 3D point cloud onto a 2D image plane, establishing a mapping relationship between 3D points and 2D pixels. Subsequently, 2D features at the corresponding pixel locations are extracted and assigned to the corresponding 3D points, effectively transforming the 2D fused features into the feature space of the point cloud. Then, through a multi-view fusion strategy, the fused pixel-by-pixel features are mapped to the 3D geometric features of each point data point. , This represents the number of points in the point cloud.

[0039] 2. 2D-3D distillation stage.

[0040] This stage is a confidence-guided distillation from two-dimensional to three-dimensional. The aim of this stage is to transfer the rich semantic knowledge carried by the fusion two-dimensional features to the three-dimensional domain and complete the initial training of the three-dimensional backbone network. The core is to build a confidence-guided distillation mechanism.

[0041] The three-dimensional geometric features obtained in the previous stage Robust semantic information from multiple pre-trained 2D models is aggregated and can serve as a reliable supervision signal. In this embodiment, the student model employs a sparse convolution-based 3D backbone network, the Minkowski network, denoted as […]. Input the original point cloud coordinates Input the network and output the corresponding 3D geometric features. Since the student model parameters are randomly initialized during the initial training phase, their output features differ significantly from the teacher's signal. To guide the student model to prioritize learning high-reliability regions, this embodiment designs dynamic distillation weights using the teacher-student confidence difference, as shown in the following formula:

[0042]

[0043] in, Indicates the first Dynamic distillation weights at each point; express function; The teacher model is represented by the first... The confidence level of the three-dimensional geometric features at each point is expressed as follows: Since the encoder trained in the freezing phase 1 of this invention is used as the teacher model, in this embodiment... ; The student model is represented by the first... The confidence level of the three-dimensional geometric features at each point is expressed as follows: In this embodiment, when the teacher's confidence level is significantly higher than the student's confidence level, the dynamic distillation weight approaches 1, strengthening the learning weight of that region; conversely, it approaches 0, reducing interference in low-reliability regions.

[0044] We employ weighted cosine similarity loss to constrain the consistency between teacher and student features, ensuring effective transfer of two-dimensional semantic knowledge. We also use confidence-guided distillation loss between the teacher and student models. Represented as:

[0045]

[0046] in, This indicates the calculation of cosine similarity. The first output of the student model represents the... Three-dimensional geometric features at each point; The output of the teacher model represents the first... Three-dimensional geometric features at each point.

[0047] 3. Self-distillation stage.

[0048] This stage aims to optimize model performance by incorporating learnable cue words through self-distillation, achieving joint training of text feature adaptation and 3D model optimization. In this embodiment, the 3D model trained in the previous stage is used as the teacher model, all parameters are frozen, and parameters are updated via EMA. For the student model, a model with the exact same structure as the teacher model is initialized, with trainable parameters. A linear adaptation layer is added to the input of the CLIP text encoder to generate scene-adaptive cue word vectors. The linear adaptation layer is represented as follows:

[0049]

[0050] Among them, the output is the result of adding a linear adaptation layer at the input of the CLIP text encoder; Let be a trainable weight matrix, and D be the dimension of the text features; This is a trainable initialization vector.

[0051] Subsequently, the generated prompt word vectors are compared with the original CLIP text features. By adding them together, we obtain the adapted, class-specific text features. , represented as:

[0052]

[0053] The above design enables the model to maintain CLIP's general semantic representation capabilities while fine-tuning semantic cluster centers through a residual injection mechanism. The adapted text features are dynamically updated during training, effectively capturing the scene semantic characteristics of a specific dataset, thereby improving the cross-modal alignment accuracy between text and 3D point features.

[0054] In this embodiment, the alignment loss between text features and 3D features The core objective is to enhance the semantic matching between the geometric features extracted by the student model and the text features adapted to the current scene. The loss function is expressed as:

[0055]

[0056] in, This indicates finding all categories. The maximum value in; This represents the three-dimensional geometric features of the i-th point data obtained by the student model after training; This represents calculating the magnitude of the vector. This loss term calculates the similarity between point features and text features of each category along the feature dimension, and maximizes the highest response value. This prompts the three-dimensional geometric feature distribution to fit the scene semantic clustering center after the introduction of prompt words. Through this constraint, the model achieves adaptation from general open vocabulary to scene-specific open vocabulary within a unified metric space.

[0057] The loss function used when further training the student model and the CLIP text encoder. Represented as:

[0058]

[0059] in, This is a balancing factor. In this embodiment, the loss... To guide distillation loss with the corrected confidence level, such as Figure 3It achieves this by using fixed text features in the original formula. Replace with adapted text features Constraining teacher and student characteristics to maintain consistency within the updated semantic space; loss The text-3D alignment loss aims to enhance the semantic matching between the geometric features extracted by the student model and the text features adapted to the current scene.

[0060] Through the above three stages, this paper can effectively train a 3D point cloud network without 3D point cloud labels and has open vocabulary capabilities.

[0061] Figure 4 To compare the results generated by different methods, the 3D Scene column represents the input point cloud image, the GT column represents the segmentation results of manually annotated datasets, the SAS and OpenScene columns represent the segmentation results of two representative works from 2023 and 2025, and the Ours column represents the segmentation results of this invention. Each row represents a segmentation instance, with a total of 21 class labels for the three instances. The segmentation results show that the method of this invention is effective for fine-grained objects (with...). Figure 4 Taking the red dashed box in the middle as an example, such as shower curtains and curtains, the segmentation results are even better than in the past.

[0062] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A confidence-guided multi-expert fusion open-vocabulary 3D point cloud segmentation method, characterized in that, The pre-trained student encoder extracts the 3D geometric features from the point cloud data to be segmented, and the CLIP text encoder obtains its adapted text feature dictionary. The obtained 3D geometric features are multiplied by the adapted text feature dictionary to obtain the final segmentation result. The training process of the student encoder and CLIP text encoder includes: Construct training data, where each training sample consists of 3D point cloud data and its corresponding multi-view RGB image; Two different two-dimensional segmentation models are used to obtain two-dimensional image features from multi-view RGB images, and the segmentation results are fused based on the confidence of the two-dimensional image features to obtain two-dimensional fused features; A point cloud encoder is used to extract 3D point cloud features from 3D point cloud data. Based on the camera intrinsic and extrinsic parameter matrices, the 3D point cloud features are matched with the 2D fusion features. The difference between the 3D point cloud features and the 2D fusion features at each matching point is calculated. The point cloud encoder is trained with the goal of minimizing this difference. The point cloud encoder was used as the teacher model, and the network parameters of the teacher model were frozen. The student model and CLIP text encoder were further trained under the guidance of the teacher model. The point cloud data to be segmented is input into the trained student model to extract the three-dimensional geometric features of each point. The target category text is input into the trained CLIP text encoder and combined with the trained prompt word vectors to obtain the adapted text features of each category. The cosine similarity between the three-dimensional geometric features and the adapted text features of each category is calculated, and the category corresponding to the maximum value is taken as the final segmentation result.

2. The confidence-guided multi-expert fusion open vocabulary 3D point cloud segmentation method according to claim 1, characterized in that, Two different two-dimensional segmentation models were chosen: the LSeg segmentation model and the SEEM segmentation model.

3. The confidence-guided multi-expert fusion open vocabulary 3D point cloud segmentation method according to claim 2, characterized in that, The acquisition of two-dimensional fusion features includes: in, It is a two-dimensional fusion feature; The confidence level predicted by the LSeg segmentation model; The segmentation results of the LSeg segmentation model; The prediction confidence of the SEEM segmentation model; The segmentation results are from the SEEM segmentation model.

4. The confidence-guided multi-expert fusion open vocabulary 3D point cloud segmentation method according to claim 3, characterized in that, Apply the segmentation model to the set of categories. The maximum value of the output is used as the confidence score of the model, that is: in, This indicates finding the maximum value; express function; CLIP-encoded text features representing all target categories; This indicates a transpose expression.

5. The confidence-guided multi-expert fusion open vocabulary 3D point cloud segmentation method according to claim 1, characterized in that, The student model uses a 3D backbone network based on sparse convolution, the Minkowski network, and the network parameters of the student model are randomly initialized at the start of training.

6. The confidence-guided multi-expert fusion open vocabulary 3D point cloud segmentation method according to claim 1, characterized in that, The loss function used for further training of the student model and CLIP text encoder is: in, The loss function used for further training of the student model and CLIP text encoder; Distillation loss guided by confidence between teacher and student models; The alignment loss between text features and 3D features. It is a balancing factor.

7. A confidence-guided multi-expert fusion open vocabulary 3D point cloud segmentation method according to claim 6, characterized in that, Confidence-guided distillation loss between teacher and student models Represented as: Where N is the number of points in the point cloud data; Indicates the first Dynamic distillation weights at each point; This indicates the calculation of cosine similarity. The first output of the student model represents the... Three-dimensional geometric features at each point; The output of the teacher model represents the first... Three-dimensional geometric features at each point; express function; The teacher model is represented by the first... Confidence level of three-dimensional geometric features at each point; The student model is represented by the first... Confidence level of three-dimensional geometric features at each point.

8. A confidence-guided multi-expert fusion open vocabulary 3D point cloud segmentation method according to claim 6, characterized in that, Alignment loss between text features and 3D features Represented as: Where N is the number of points in the point cloud data; This indicates finding all categories. The maximum value in; This represents the three-dimensional geometric features of the i-th point data obtained by the student model after training; Represents the class-specific text features after adaptation; This indicates finding the magnitude of a vector; This indicates a transpose expression.

9. A confidence-guided multi-expert fusion open-vocabulary 3D point cloud segmentation method according to claim 1 or 8, characterized in that, The adapted, class-by-class text features are represented as follows: in, The raw text features output by the CLIP text encoder; To add a linear adaptation layer to the output of the CLIP text encoder input; For a trainable weight matrix, This is a trainable initialization vector.

Images