An open-vocabulary few-shot 3D part segmentation method and system

Abstract

The application provides an open-vocabulary few-shot three-dimensional part segmentation method and system, and relates to the technical field of image processing. A 2D teacher network is used to process a three-dimensional point cloud sample to generate a 3D knowledge unit containing part semantic information; the three-dimensional point cloud sample and the 3D knowledge unit are input into a 3D student network including an encoder, an adapter and a segmentation head, and training is carried out based on knowledge distillation by using a double-layer circulation mechanism; the double-layer circulation mechanism is respectively used for enabling the network to learn task-specific prior knowledge and cross-class universal prior knowledge; in the reasoning stage, a search enhancement strategy is used to dynamically select a support set for a target point cloud, and a segmentation head is fine-tuned through an inner-layer circulation mechanism to obtain a part segmentation result of the target point cloud. The application effectively bridges the representation difference between a two-dimensional visual language model and three-dimensional point cloud processing, breaks through the limitation of a traditional single-class model, and realizes rapid adaptation to unknown object classes and high-quality open-vocabulary few-shot three-dimensional part segmentation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an open-vocabulary, few-sample 3D part segmentation method and system. Background Technology

[0002] 3D component semantic segmentation refers to the process of decomposing a 3D object into semantically meaningful and functionally distinct components. Open-lexicon 3D component segmentation, on the other hand, is the process of decomposing any 3D object into several constituent parts based on its geometric shape and semantic features, using any lexicon to describe it.

[0003] Most existing methods rely on various labeled component segmentation datasets and design different network architectures to extract geometric and semantic features. However, due to the significant differences in component semantics, shape, size, and function among different object categories, the generalization ability of current methods is generally poor. For example, the invention application entitled "A Method and System for Instance Segmentation of 3D Point Cloud Data in Autonomous Driving Scenarios" discloses an instance segmentation method based on point cloud view column extraction and focus loss function. However, because it relies on labeled data for specific autonomous driving scenarios and fixed category settings, its generalization ability for unknown object categories is insufficient, and it cannot achieve fine-grained component segmentation under open vocabulary.

[0004] Recent research has begun to utilize pre-trained large-scale visual language models (VLMs) to address the generalization problem. On one hand, some works render 3D objects as multi-view images, use VLMs on 2D images for part localization or segmentation, and then obtain 3D part segmentation results through projection and aggregation. These methods can achieve part segmentation without 3D semantic annotation; however, inconsistencies between multi-view images often lead to missing small parts and inaccurate segmentation results. On the other hand, there is a lack of reasonable decoupling learning between cross-category general prior knowledge and task-specific prior knowledge. For example, an invention application titled "A Point Cloud Semantic Segmentation Method and System Based on Adversarial Learning and Multimodal Learning" discloses using adversarial learning and multimodal alignment to improve the domain adaptability of point cloud semantic segmentation. However, due to the need for accurate 2D-3D data correspondence and the lack of optimization for part-level segmentation, the model adaptation efficiency is low and the accuracy of detailed part segmentation is insufficient in low-sample scenarios. Therefore, existing technologies still suffer from slow model adaptation to new tasks, susceptibility to overfitting to known categories, and inability to efficiently complete open-vocabulary 3D part segmentation tasks with limited samples. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes an open-vocabulary few-shot 3D component segmentation method and system. By integrating cross-modal knowledge distillation and meta-learning optimization, and introducing retrieval enhancement strategies, it effectively bridges the representational differences between 2D visual language models and 3D point cloud processing, breaks through the limitations of traditional single-class models, and achieves rapid adaptation to unknown object categories and high-quality open-vocabulary few-shot 3D component segmentation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides an open-vocabulary few-sample 3D component segmentation method, comprising:

[0008] A 2D teacher network is used to process multi-view rendered images of 3D point cloud samples to generate 3D knowledge units containing semantic information of components.

[0009] A 3D student network comprising an encoder, adapter, and segmentation head is constructed. The 3D point cloud samples and 3D knowledge units are input into the 3D student network, and training is performed using a two-layer loop mechanism based on knowledge distillation. In the two-layer loop mechanism, the inner loop is used to update the parameters of the segmentation head based on the task support set and the 3D knowledge units for a single segmentation task, enabling the network to learn task-specific prior knowledge. The outer loop is used to aggregate the query set loss across multiple tasks and perform meta-learning updates on the initial parameters of the adapter and segmentation head, enabling the network to learn cross-category general prior knowledge.

[0010] During the inference phase, a retrieval enhancement strategy is used to dynamically select the support set for the target point cloud, and the segmentation head is fine-tuned through an inner loop mechanism to obtain the component segmentation results of the target point cloud.

[0011] Secondly, the present invention provides an open-vocabulary few-sample 3D component segmentation system, comprising:

[0012] The 3D knowledge acquisition unit is used to process multi-view rendered images of 3D point cloud samples using a 2D teacher network to generate 3D knowledge units containing semantic information of components.

[0013] A student network training unit is used to construct a 3D student network including an encoder, adapter, and segmentation head. The 3D point cloud samples and 3D knowledge units are input into the 3D student network, and training is performed using a two-layer loop mechanism based on knowledge distillation. In the two-layer loop mechanism, the inner loop is used to update the parameters of the segmentation head based on the task support set and the 3D knowledge units for a single segmentation task, enabling the network to learn task-specific prior knowledge. The outer loop is used to aggregate the query set loss across multiple tasks and perform meta-learning to update the initial parameters of the adapter and segmentation head, enabling the network to learn cross-category general prior knowledge.

[0014] The inference unit is used to dynamically select a support set for the target point cloud using a retrieval enhancement strategy during the inference phase, and to fine-tune the segmentation head through an inner loop mechanism to obtain the component segmentation results of the target point cloud.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the open-vocabulary few-sample three-dimensional component segmentation method described in the first aspect.

[0016] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the open vocabulary few-sample three-dimensional part segmentation method described in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0018] This invention addresses key challenges in open-vocabulary 3D component segmentation, such as the modal gap and insufficient accuracy in fine-grained component segmentation. It proposes an innovative method integrating cross-modal knowledge distillation and meta-learning optimization. By establishing a deep association between geometric features and semantic representations, it effectively bridges the representational differences between 2D visual language models and 3D point cloud processing. Furthermore, the proposed retrieval enhancement strategy provides a new optimization path for 3D component segmentation in few-shot scenarios, overcoming the limitations of traditional single-class models and offering new insights into validating the feasibility of open-vocabulary 3D understanding tasks.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0021] Figure 1 The main flowchart of an open-vocabulary few-sample 3D component segmentation method provided in this embodiment of the invention;

[0022] Figure 2 This is a framework diagram of an open-vocabulary, few-sample 3D component segmentation method provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram illustrating four retrieval enhancement strategies provided in embodiments of the present invention;

[0024] Figure 4Qualitative experimental results on all categories of the ShapeNetPart dataset provided in this embodiment of the invention, in the joint training (J) experimental configuration.

[0025] Figure 5 Qualitative experimental results of invisible objects in the generalization experiment (G) configuration of the ShapeNetPart dataset provided in the embodiments of the present invention;

[0026] Figure 6 This is a qualitative experimental result of the ablation experiment on the ShapeNetPart dataset provided in the embodiments of the present invention using retrieval enhancement strategies;

[0027] Figure 7 Examples of support set shapes retrieved by the four retrieval enhancement strategies provided in embodiments of the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] Example 1

[0030] like Figure 1 As shown, this embodiment discloses an open-vocabulary few-shot 3D component segmentation method, including the following steps:

[0031] S1: Use a 2D teacher network to process multi-view rendered images of 3D point cloud samples to generate 3D knowledge units containing semantic information of components.

[0032] S2: Construct a 3D student network including an encoder, adapter, and segmentation head. Input the 3D point cloud samples and 3D knowledge units into the 3D student network and train it using a two-layer loop mechanism based on knowledge distillation. In the two-layer loop mechanism, the inner loop is used to update the parameters of the segmentation head based on the task support set and the 3D knowledge units for a single segmentation task, so that the network learns task-specific prior knowledge. The outer loop is used to aggregate the query set loss across multiple tasks and perform meta-learning to update the initial parameters of the adapter and segmentation head, so that the network learns cross-category general prior knowledge.

[0033] S3: During the inference phase, a retrieval enhancement strategy is used to dynamically select the support set for the target point cloud, and the segmentation head is fine-tuned through an inner loop mechanism to obtain the component segmentation results of the target point cloud.

[0034] Next, combined Figure 2 This embodiment provides a detailed description of an open-vocabulary, few-sample 3D component segmentation method.

[0035] In S1, firstly, the point cloud samples are processed using a 2D teacher network to obtain 2D knowledge of the point cloud, which is then used for distillation of the 3D student network.

[0036] As one implementation, the 2D teacher network includes a pre-trained visual language module (VLM) and a knowledge extraction module. The visual language module, built on the GLIP model, performs object detection and classification of open vocabulary; the knowledge extraction module, built on a 2D-3D backprojection algorithm and a probabilistic aggregation strategy, generates 3D knowledge units.

[0037] Specifically, given a point cloud sample P and its corresponding R parts with text prompts. For example, this embodiment obtains a point cloud sample P of a chair, where the R parts include the backrest, seat, legs, and armrests.

[0038] First, render V images of the point cloud P from multiple perspectives. Where H and W are the height and width of the image. It should be understood that rendering can be performed using the multi-view rendering methods in the PyTorch3D library.

[0039] The multi-view rendered image V and the text prompts for the R components are fed together into a pre-trained visual language module.

[0040] For each view v, the visual language module generates a set of bounding boxes. , This represents the bounding box coordinates predicted by the visual language module on view v and the probability that the bounding box belongs to one of R parts. β represents the number of bounding boxes predicted by the VLM on view v.

[0041] For V views, all bounding boxes were obtained. , where D is the number of bounding boxes obtained after V multi-view rendered images and R text prompts are input into the VLM.

[0042] Then, the bounding box B is input into the knowledge extraction module, and knowledge extraction is applied to all bounding boxes B. That is, the 2D to 3D back projection technique is used to transform the bounding box B into easily distilled 3D knowledge units. Each pair Represents a knowledge unit, where Indicates whether each point of the 3D shape is bounded by the d-th bounding box. Surround, Indicated in the bounding box The probability of each point in the point cloud on each component, where N represents the number of points in the point cloud.

[0043] It should be understood that the back projection technique is something that those skilled in the art can choose according to the actual situation, such as the perspective projection back calculation method of OpenGL, the ray projection back projection method based on camera intrinsic and extrinsic parameters, etc., and is not limited here.

[0044] In S2, a 3D student network is constructed, taking the point cloud sample P and the maximum number of parts in all categories of the dataset as input. This network is used to fuse the 3D geometric features of the point cloud itself with the 2D knowledge extracted from the 2D teacher network, and finally outputs the probability distribution of each point in the point cloud in each part category.

[0045] As one implementation, the 3D student network consists of an encoder, an adapter, and a segmentation head.

[0046] The encoder is a pre-trained model trained on a large-scale point cloud dataset to extract rich local features for each point in the input point cloud. Throughout the distillation process, the encoder's parameters remain frozen to provide a stable and robust representation of the underlying features.

[0047] The adapter is implemented by two consecutive Point Transformer blocks, employing a vector self-attention mechanism based on local 16 nearest neighbors, equipped with learnable relative position encoding, and optimized for fusing geometric and semantic features through residual connections and a pointwise multilayer perceptron (MLP). The adapter is wrapped by two linear layers before and after it, primarily because the encoder output has a 681-dimensional dimension, while the Point Transformer block has a 512-dimensional dimension. Therefore, the two linear layers mainly reduce and increase the dimensionality to ensure the overall dimensionality consistency of the network. Furthermore, the adapter is designed to be category-agnostic, giving it strong cross-class generalization capabilities.

[0048] The segmentation head consists of a lightweight two-layer multilayer perceptron (MLP), both fully connected layers. It receives features processed by the adapter and predicts the probability distribution of each point in the point cloud for that category of part. The segmentation head is category-specific, and its output dimension is determined by the maximum number of parts in the dataset. The first layer has 681 dimensions for both input and output, and the second layer has 681 dimensions for input and output dimensions that are the maximum number of part categories in the dataset. For example, in the ShapeNetPart dataset, the maximum number of parts is 6, so the final output dimension of the network is 6, corresponding to the part probability distribution of each point.

[0049] Existing 3D part segmentation networks typically output a dimension equal to the number of part labels in a single category dataset. This hinders their ability to generalize to unseen object categories because the network cannot dynamically expand the predefined number of part labels during inference. Therefore, this embodiment sets the output dimension of the segmentation head to the maximum number of parts in the dataset, and the semantic part label corresponding to each output dimension varies based on each specific instance.

[0050] When the input point cloud belongs to the chair category, the first four dimensions of the output probability vector represent the backrest, seat, legs, and armrests, respectively, while the output of the remaining dimensions is considered invalid. For example, different categories in the dataset have different numbers of parts (e.g., chairs have 4 parts, tables have 3 parts, and other categories have a maximum of 6 parts), and the semantic labels of the parts in each category are also different. The segmentation head output dimension is fixed to the maximum number of parts in the dataset to ensure compatibility with all categories. When processing a specific chair category point cloud, only the first four dimensions of the output vector corresponding to the chair parts (backrest, seat, legs, armrests, a total of 4) are used. The remaining dimensions (because the maximum number of parts is greater than 4) do not correspond to valid chair parts and are therefore considered invalid. That is, different instances (categories) will use different dimensions in the output vector to represent their own parts.

[0051] The 3D student network is trained in a supervised manner on a dataset processed by a Visual Language Model (VLM) without requiring manual part labeling. During training, the loss function employed is the distillation loss function proposed by PartDistill, to improve the student network's predictions. Knowledge Units with Teacher Networks Alignment, the loss function is defined as follows:

[0052]

[0053] in, express The number of points covered. Represents bounding box The highest probability on R components at point n. Indicates whether component r belongs to the bounding box. The highest probability at point n is 1 if it is true and 0 otherwise. Although the labels generated by VLM are noisy and inconsistent, this distillation loss guides the model to effectively learn prior knowledge of 3D part segmentation from large-scale object datasets by focusing on high-confidence labels.

[0054] To improve the model's generalization ability, this embodiment uses the meta-learning algorithm MAML for training. Below are the key terms related to the MAML algorithm:

[0055] (1) Task: In meta-learning, a “task” corresponds to a specific learning problem and contains its own small-scale dataset. Each task is divided into a support set and a query set. The support set is used for the model to quickly adapt within the task (inner loop update), while the query set is used to evaluate the performance of the adapted model on the task and to calculate the meta-gradient for updating the meta-model based on this.

[0056] (2) Inner-Loop: refers to the fast parameter adaptation process performed on a single task. For each task, the algorithm performs a few steps of gradient descent based on the current initial parameters of the model and the support set of the task to obtain the adapted parameters for the task.

[0057] (3) Outer-Loop: refers to the meta-learning process of optimizing the initial parameters of the model across tasks. It evaluates the model adapted by the inner loop on the query set of each task, summarizes the loss on all tasks, and performs a primary gradient update on the initial parameters of the model based on the comprehensive loss. Its goal is to learn a universal initialization parameter so that the model can quickly adapt to new tasks with only a few gradient steps.

[0058] The training phase of the method in this embodiment is based on the meta-learning algorithm MAML, which aims to enable the 3D student network to acquire strong cross-class generalization ability through cross-task learning.

[0059] First, the parameters of each module of the 3D student network are initialized. The encoder module uses a pre-trained multi-scale masked autoencoder Point-M2AE, and its parameters are frozen throughout the training phase; the adapter and segmentation head modules are randomly initialized.

[0060] The training process optimizes the network parameters of the adapter and segmentation head through the outer loop to obtain good initial network parameters; and optimizes the network parameters of the segmentation head through the inner loop to enable it to quickly adapt to unseen instances.

[0061] During the task construction phase, each task consists of a support set and a query set. For example, the query set contains one shape for evaluating task adaptability; the support set contains eight shapes, from which samples related to the query set shapes are retrieved from the training set using a retrieval enhancement strategy. This retrieval enhancement strategy is based on semantic and geometric similarity for sample retrieval, and its specific implementation will be described in detail later.

[0062] (1) Inner loop training

[0063] During training, for each task, the inner loop first freezes the parameters of the encoder and adapter, optimizing only the segmentation head. Specifically, the distillation loss is calculated using eight shapes from the support set, and the segmentation head parameters are adjusted through five gradient updates to quickly adapt to the current task.

[0064] Since the support set contains samples related to the shape of the query set, using these samples to calculate the distillation loss allows the segmentation head to quickly focus on the features and patterns of the current task during training, such as the segmentation of parts of a specific object. Through five gradient updates, the parameters of the segmentation head can be specifically adjusted to quickly adapt to the segmentation requirements of the current task.

[0065] (2) Outer loop training

[0066] After the inner loop adaptation is completed, the outer loop optimization phase begins. This phase first uses the adapted segmentation head to predict the query set shape and calculates the distillation loss. Then, it aggregates the query set losses for all tasks in the current batch and calculates their average loss as the meta-objective function. Finally, it updates the initial parameters of both the adapter and the segmentation head simultaneously using gradient descent, allowing the model to learn general prior knowledge across tasks. This enables the model to quickly adapt to new tasks (unseen categories or segmentation scenarios) based on this learned general knowledge. It's not simply about learning task-specific knowledge, but rather learning a general ability that allows the model to quickly adapt to various new tasks.

[0067] Through several iterations of inner and outer loops, the method can gradually learn model parameters with strong generalization ability, enabling it to quickly adapt to new object categories and their component segmentation tasks with a small number of samples.

[0068] In this embodiment, the segmentation head is designed to be category-dependent, performing fast adaptation only for objects of the current category in the inner loop. Its initial parameter values ​​are jointly optimized in the outer loop and adjusted in the inner loop with a small number of gradient steps to fit the label distribution of the current support set, thereby achieving category-specific inference based on the query set. The adapter, on the other hand, is designed to be category-independent, shared by all categories throughout training. Its optimization relies on multi-sample aggregated gradients in the outer loop. This allows it to acquire geometric knowledge from the features obtained from the encoder and utilize gradient updates to fuse semantic knowledge in 3D knowledge units. The parameter update direction is guided by multiple categories, tending towards generalization.

[0069] By using inner loop training, the encoder and adapter parameters are frozen first, and only the segmentation head is fine-tuned, allowing it to quickly focus and adapt to the component segmentation requirements of the current specific task. In outer loop training, based on the segmentation head adapted after inner loop training, the initial parameters of the adapter and segmentation head are further updated, allowing the model to learn cross-task general prior knowledge. This cleverly advances the learning of the segmentation head's rapid adaptation capability for a single task and the learning of cross-task general knowledge of the adapter and segmentation head's initial parameters in stages, achieving decoupling between task-specific prior knowledge learning and cross-category general prior knowledge learning. This ensures the model's rapid adaptation to a single new task while also giving the model the ability to generalize and quickly adjust when facing various new tasks, helping the model to efficiently complete open-vocabulary 3D component segmentation in scenarios with few samples.

[0070] In S3, during the inference phase, the method of this embodiment can quickly adapt to unseen object categories and complete 3D component segmentation.

[0071] Specifically, given a query set point cloud to be segmented, firstly, eight samples related to the shape of the query set are automatically retrieved from the training set using a retrieval enhancement strategy to serve as the support set for this task. Secondly, following the inner loop optimization mechanism consistent with the training phase, the segmentation head is adapted on the support set; the difference is that 10 gradient updates are performed during the inference phase to more fully adjust the segmentation head parameters. The objective function used in the optimization process remains the distillation loss function proposed by PartDistill, ensuring consistency with the training objective.

[0072] After the inner loop adaptation is completed, the optimized network is used to infer the query set point cloud, outputting the probability distribution of each point in each component category, and thus generating the final segmentation result. The entire inference process requires no manual annotation information and can achieve rapid generalization to new categories with only a small number of support samples.

[0073] like Figure 3 As shown, the retrieval enhancement strategy of this embodiment provides four support set selection methods for retrieving samples related to the query set point cloud from the training set during the meta-learning task construction stage: random selection strategy, similarity selection strategy, component completeness selection strategy, and joint selection strategy.

[0074] (1) Random selection strategy. For each task, eight 3D shapes are randomly selected directly from the training set as the support set.

[0075] (2) Similarity selection strategy. Extract the basic point set features of the target point cloud and the samples in the training set, calculate the cosine similarity between the features, and select the preset number of samples with the highest similarity as the support set.

[0076] For example, for each task, the basic point set features of the query set point cloud are first extracted, and the cosine similarity between each feature and the BPS features of each shape in the training set is calculated. Then, the eight shapes with the lowest similarity are selected as the support set. BPS is a 3D point cloud feature representation method based on a predefined basic point set. Its core idea is to predefine a fixed set of basic points. Calculate the point cloud from each base point. The minimum Euclidean distance between the nearest neighbors in the vector is used to generate a fixed-dimensional feature vector, as shown in the following formula:

[0077]

[0078] (3) Component completeness selection strategy. First, select samples with complete components in the training set (by calculating the sum of the probabilities of each component in the sample, it is determined whether the proportion of each component meets the preset threshold) to form a candidate set. Then, calculate the BPS feature similarity between the target point cloud and the candidate set samples, and select the preset number of samples with the highest similarity as the support set. If the candidate set samples are insufficient, supplement them from the training set.

[0079] For example, this strategy considers both semantic completeness and geometric similarity. Its core idea is to group relatively complete shapes from the training set together, and the support set for all tasks is selected only from this group of shapes. Specifically, first, all "complete" shapes are selected from the training set to form a candidate set M. The criteria for determining whether a shape is "complete" are as follows:

[0080]

[0081] in, This represents the sum of the bounding box probabilities of point n of shape on component r. This represents the sum of probabilities of a point n of shape falling on the bounding box of component k. The brackets represent Iverson brackets, where a value of 1 indicates a condition met and a value of 0 indicates a condition not met. The ratio represents the sum of the probabilities of the largest component on each part of the shape. After obtaining the ratio for each shape, it is first normalized and component-filtered; that is, if the proportion of a component in the ratio is lower than 0.01, the proportion of that component is set to 0. Then, it is determined whether the value of each component in the ratio is greater than 0. If the condition is met, the shape is considered "component-complete" and is included in set M. Subsequently, for each task, the cosine similarity between the BPS features of the query set point cloud and the BPS features of each shape in set M is calculated, and the 8 shapes with the lowest similarity are selected as the support set. If the number of shapes in the candidate set M is less than 8, all shapes in M ​​are used first, and the remaining shapes are supplemented from the entire training set according to the similarity selection strategy.

[0082] (4) Joint selection strategy. The similarity selection strategy and the component completeness selection strategy are combined. Specifically, half of the samples are selected by the similarity selection strategy, and the other half of the samples are selected by the component completeness selection strategy. If the component completeness candidate set is insufficient, the remaining samples are supplemented by the similarity selection strategy.

[0083] For example, this strategy combines the two retrieval methods described above. First, it selects four shapes using a similarity selection strategy, and then selects another four shapes using a component completeness selection strategy. If the number of component completeness candidate sets M is less than four, the remaining parts are supplemented by the similarity selection strategy.

[0084] Example 1

[0085] This embodiment uses the following two 3D object part segmentation datasets to train and validate the proposed method: ShapeNetPart and PartNetE.

[0086] The ShapeNetPart dataset is a benchmark dataset widely used for 3D shape part segmentation tasks, containing 16 object categories and 50 part categories. The dataset provides the location information, normal information, and part category labels for each point cloud, totaling 14,775 3D shapes. Following common segmentation criteria, 11,955 shapes were used for training, and 2,820 shapes were used for testing.

[0087] The PartNetE dataset, proposed by PartSLIP, is built upon existing datasets PartNet and PartNet-Mobility and is specifically designed for single-class few-shot semantic segmentation tasks. The original dataset contains 45 classes and 2266 shapes, with 8 shapes provided for each class as few-shot training samples, and the remainder used for testing. To accommodate the requirement of dynamically constructing the support set based on query shapes in this embodiment, the original 8 few-shot training samples and test samples are merged and re-divided into training and test sets in a 7:3 ratio. After the division, the training set contains 1562 shapes, and the test set contains 704 shapes.

[0088] In this embodiment, the mean intersection-over-union ratio (mIoU) is used to evaluate the accuracy of the model's predictions, as shown in the following formula:

[0089]

[0090] Where k represents the total number of categories; Represents the intersection-union ratio of category i; This represents the number of true cases, i.e., the number of points whose true label is class i and which the model correctly predicts to be class i. This represents the number of false positives, i.e., the number of points whose true label is not class i, but which the model incorrectly predicts as class i. This represents the number of false negatives, where the true label is class i, but the model incorrectly predicts it as another class.

[0091] This embodiment verifies the effectiveness of the proposed method through detailed comparative and ablation experiments. The experiments are divided into two main categories: comparative experiments and ablation experiments. These two types of experiments will be described below.

[0092] (a) Comparative experiment.

[0093] This experiment was used to evaluate the performance advantages of the method in this embodiment compared to existing technologies, and was specifically divided into zero-sample comparison experiments and few-sample comparison experiments. All comparison experiments were comprehensively evaluated under the following three experimental configurations:

[0094] 1. Training by category (C): The model is trained and tested separately for each category;

[0095] 2. Joint training across all categories (J): The model is trained uniformly on data from all categories and then tested;

[0096] 3. Generalization Experiment (G): The model is trained only on some categories (visible categories) and then tested on all categories (visible and invisible categories) to evaluate its cross-category generalization ability.

[0097] The method proposed in this embodiment is compared with current state-of-the-art methods. The methods involved in the comparison are divided into three categories:

[0098] 1. PointCLIPv2, PartSLIP, and PartSLIP++: These are based on pre-trained visual language models and directly perform zero-shot or few-shot segmentation.

[0099] 2. Find3D: Proposes a data engine based on a pre-trained visual language model to annotate a large-scale 3D shape dataset and train a segmentation model on the annotated dataset. This experiment uses its publicly released open-source model.

[0100] 3. PartDistill: Distills 2D prior knowledge from a pre-trained visual language model and combines it with 3D geometric information for category-level part segmentation. This experiment retrains the model on the same training set as the method in this embodiment, based on its publicly available configuration, and extends it to the three experimental configurations mentioned above to ensure the fairness and comprehensiveness of the comparison.

[0101] The following are the quantitative and qualitative analyses of the comparative experiment.

[0102] (1) Quantitative analysis of comparative experiments.

[0103] Table 1. Quantitative analysis of zero-shot comparison experiments on the ShapeNetPart dataset;

[0104]

[0105] Table 2. Quantitative analysis of zero-sample control experiments on the PartNetE dataset;

[0106]

[0107] 1) Quantitative analysis of zero-shot comparison experiments. The quantitative results of the zero-shot comparison experiments are shown in Tables 1 and 2, representing the performance evaluations on the ShapeNetPart and PartNetE datasets, respectively.

[0108] In the ShapeNetPart dataset (Table 1), the method of this embodiment significantly outperforms existing methods in all three configurations: On configurations C and J, the method of this embodiment achieves mIoU of 65.15% and 62.9% respectively, which is 8.16% and 19.91% higher than the current state-of-the-art method PartDistill. It can be seen that the results of this embodiment on these two experimental configurations differ by only 2.25%, which is much lower than PartDistill's 14%, demonstrating the strong generalization and stability of this embodiment. On configuration G, to ensure fairness, both this embodiment and PartDistill are trained using only five categories: airplane, car, chair, lamp, and table (the ShapeNetPart dataset has 16 categories). However, in terms of results, this embodiment not only performs well in visible shapes but also achieves mIoU of 60.47% in invisible shapes, which is 44.16% better than PartDistill's performance in invisible shapes, proving the strong cross-class generalization ability of this embodiment.

[0109] In the PartNetE dataset (Table 2), the method of this embodiment also maintains its leading position in most implementation configurations. In particular, it achieves a 30.52% mIoU on the invisible categories of configuration G, which is 7.88% higher than PartDistill, further validating the generalization ability of the method of this embodiment. In addition, the performance fluctuations across the three experimental configurations are small, demonstrating the excellent stability of the method of this embodiment.

[0110] Table 3. Quantitative analysis of the few-sample comparison experiment on the ShapeNetPart dataset;

[0111]

[0112] Table 4. Quantitative analysis of the few-sample comparison experiment on the PartNetE dataset;

[0113]

[0114] 2) Quantitative analysis of small sample size control experiments. In small sample size experiments, the small sample size setting is mainly reflected in two aspects:

[0115] 1.2 Acquisition of Distillation Knowledge: On the PartNetE dataset, the PartSLIP method provides a GLIP model finely tuned based on 8 labeled samples of this dataset, which is adopted in this experiment; however, on the ShapeNetPart dataset, since PartSLIP does not provide a corresponding finely tuned model and training code, the method in this embodiment still uses the untuned original GLIP model as the source of 2D knowledge.

[0116] 2. Loss function design: When training the 3D student network, the method in this embodiment simultaneously minimizes the distillation loss proposed by PartDistill and the cross-entropy loss between the predicted segmentation result and the real annotation, so as to achieve effective supervision with a small amount of annotation information.

[0117] The quantitative results of the few-shot comparison experiments are shown in Tables 3 and 4, representing the performance evaluations on the ShapeNetPart and PartNetE datasets, respectively. On the ShapeNetPart dataset, the method of this embodiment outperforms existing methods in all three configurations, particularly in the invisible categories of configuration G, achieving a mIoU of 70.09%, a 50.52% improvement over PartDistill, demonstrating the powerful cross-class generalization ability of the method of this embodiment. On the PartNetE dataset, the method of this embodiment also maintains a leading position in most implementation configurations.

[0118] (2) Qualitative analysis of comparative experiments.

[0119] Figure 4This document presents the visualization results of various methods for segmenting components of objects such as airplanes, chairs, and tables using a joint training (J) configuration across all categories on the ShapeNetPart dataset. As can be seen from the figures, the method in this embodiment demonstrates clearer and more accurate segmentation boundaries in structurally complex regions (such as the junction of an airplane wing and fuselage, and the boundary between a chair armrest and seat), with significantly better component integrity than the comparative methods. Particularly in the segmentation of small components, such as airplane engines and chair armrests, this method still achieves accurate segmentation, demonstrating excellent local perception capabilities. In contrast, benchmark methods such as PartDistill exhibit significant over-segmentation or under-segmentation in detailed areas. Taking the chair back area as an example, PartDistill suffers from poor segmentation results in this region due to interference from other category shapes during joint training; while the method in this embodiment, through the proposed category-independent adapter, effectively learns cross-category geometric and semantic features, thus demonstrating stronger shape understanding and component discrimination capabilities across different object structures.

[0120] Figure 5 The paper further demonstrates the comparison of segmentation performance for invisible categories during the training phase under the generalization experiment (G) configuration. The results show that the method in this embodiment can still maintain stable part segmentation performance on invisible categories, significantly reduce mislabeling and boundary blurring, and verify the cross-class generalization ability obtained through the double-layer loop mechanism.

[0121] (ii) Ablation experiment.

[0122] To verify the effectiveness of the key design components in this embodiment, the method was evaluated through ablation experiments to assess the contributions of the two-layer loop mechanism and the retrieval enhancement strategy. All ablation experiment results were obtained under a generalization experiment (G) configuration. The following is a description of the two types of ablation experiments: the two-layer loop mechanism and the retrieval enhancement strategy.

[0123] (1) Ablation experiments on a two-layer circulation mechanism.

[0124] Table 5 Ablation experiments based on the two-layer circulation mechanism;

[0125]

[0126] The quantitative results of the ablation experiment on the two-layer recurrent mechanism are shown in Table 5, which serves as a performance evaluation of the ShapeNetPart dataset. This experiment mainly explores the performance of the two-layer recurrent mechanism, the retrieval enhancement strategy (the part completeness selection strategy used in this experiment), and the adapter. Experiment 2 indicates that the MAML algorithm was not used for training, but the retrieval enhancement strategy proposed in this embodiment was used during the inference stage to select the support set shape for fine-tuning the segmentation head network in the inner recurrent layer.

[0127] By comparing the four experimental settings in the table, the following conclusions can be drawn:

[0128] 1. Comparing Experiment 1 and Experiment 2, the model's performance on the invisible category was significantly improved by 24.33% after adopting the retrieval enhancement strategy during inference, indicating that the strategy can effectively utilize the retrieved support set samples to enhance the model's generalization performance.

[0129] 2. Comparing Experiment 2 and Experiment 3, when a double-layer loop mechanism was introduced during training, the model's performance on both visible and invisible categories was effectively improved. The overall performance increased from 47.67% to 60.93%, an improvement of 13.26%, proving that the mechanism significantly improved the model's generalization ability.

[0130] 3. Comparing Experiments 3 and 4, when the model incorporates the category-independent adapter proposed in this embodiment, the overall performance of the model is further improved by 0.9%, reaching 61.83%, demonstrating that this module can effectively optimize the component segmentation effect by fusing geometric and semantic features.

[0131] (2) Ablation experiments on retrieval enhancement strategies.

[0132] Table 6 Ablation experiments using retrieval enhancement strategies;

[0133]

[0134] The quantitative results of the ablation experiments on the retrieval enhancement strategy are shown in Table 6, which evaluates the performance of the ShapeNetPart dataset. The experimental results show that the performance of different retrieval strategies decreases in the following order: component completeness selection strategy - joint selection strategy - similarity selection strategy - random selection strategy. This result indicates that relying solely on geometric similarity (similarity selection strategy) is insufficient to fully guarantee knowledge transfer effectiveness. In contrast, the component completeness selection strategy proposed in this embodiment, under the combined constraints of semantic completeness and geometric relevance, can more effectively provide supporting samples, thereby significantly improving the model's generalization performance.

[0135] Qualitative experimental results of ablation experiments based on retrieval enhancement strategies are as follows: Figure 6 As shown in the visualization results, when the component completeness selection strategy is adopted, the model is more refined in detail processing, and the segmentation results on objects with special geometric structures (such as the chair in the fifth column) are more accurate and complete, further verifying the superiority and robustness of the strategy. Figure 7 Examples of support set shapes retrieved by four retrieval enhancement strategies are shown. It can be seen that the component completeness selection strategy, while maintaining geometric similarity, pays more attention to semantic integrity. For example, when the query shape has no handrail, the retrieved support set shapes can still provide complete component samples with handrails, thereby enhancing the model's structural recognition ability.

[0136] This specific embodiment generates 3D knowledge units by processing multi-view rendered images of 3D point clouds through a 2D teacher network, providing accurate semantic and geometric association information for the 3D student network and bridging the gap between 2D and 3D representations. A 3D student network containing an encoder, adapter, and segmenter is constructed. A two-layer training mechanism is used: an inner loop optimizes the segmenter head to learn task priors, and an outer loop updates initial parameters to learn cross-class general priors, improving the model's generalization and adaptability. During inference, a retrieval enhancement strategy is used to select support sets and fine-tune the segmenter head, helping the model quickly adapt to unknown categories, breaking through the limitations of traditional single-class models, and providing an efficient and feasible path for open-vocabulary 3D understanding.

[0137] Example 2

[0138] This embodiment provides an open-vocabulary few-shot 3D component segmentation system, including:

[0139] The 3D knowledge acquisition unit is used to process multi-view rendered images of 3D point cloud samples using a 2D teacher network to generate 3D knowledge units containing semantic information of components.

[0140] A student network training unit is used to construct a 3D student network including an encoder, adapter, and segmentation head. The 3D point cloud samples and 3D knowledge units are input into the 3D student network, and training is performed using a two-layer loop mechanism based on knowledge distillation. In the two-layer loop mechanism, the inner loop is used to update the parameters of the segmentation head based on the task support set and the 3D knowledge units for a single segmentation task, enabling the network to learn task-specific prior knowledge. The outer loop is used to aggregate the query set loss across multiple tasks and perform meta-learning to update the initial parameters of the adapter and segmentation head, enabling the network to learn cross-category general prior knowledge.

[0141] The inference unit is used to dynamically select a support set for the target point cloud using a retrieval enhancement strategy during the inference phase, and to fine-tune the segmentation head through an inner loop mechanism to obtain the component segmentation results of the target point cloud.

[0142] Example 3

[0143] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the open-vocabulary few-sample 3D component segmentation method described in Embodiment 1 above.

[0144] Example 4

[0145] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the open vocabulary few-sample 3D part segmentation method described in Embodiment 1 above.

[0146] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0147] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for segmenting three-dimensional components with few samples using an open vocabulary, characterized in that, include: A 2D teacher network is used to process multi-view rendered images of 3D point cloud samples to generate 3D knowledge units containing semantic information of components. A 3D student network comprising an encoder, adapter, and segmentation head is constructed. The 3D point cloud samples and 3D knowledge units are input into the 3D student network, and training is performed using a two-layer loop mechanism based on knowledge distillation. In the two-layer loop mechanism, the inner loop is used to update the parameters of the segmentation head based on the task support set and the 3D knowledge units for a single segmentation task, enabling the network to learn task-specific prior knowledge. The outer loop is used to aggregate the query set loss across multiple tasks and perform meta-learning updates on the initial parameters of the adapter and segmentation head, enabling the network to learn cross-category general prior knowledge. During the inference phase, a retrieval enhancement strategy is used to dynamically select the support set for the target point cloud, and the segmentation head is fine-tuned through an inner loop mechanism to obtain the component segmentation results of the target point cloud. The process of using a 2D teacher network to process multi-view rendered images of 3D point cloud samples to generate 3D knowledge units containing component semantic information specifically includes: Obtain 3D point cloud samples and corresponding component text prompts, and render multiple images of the 3D point cloud samples from multiple perspectives to obtain multi-view rendered images; Multi-view rendered images and component text prompts are input into a pre-trained 2D teacher network to generate 2D bounding boxes and the probability that the bounding boxes belong to components, and 3D knowledge units are obtained by back projection. The 2D teacher network includes a visual language module and a knowledge extraction module; The visual language module is used to extract component features from the point cloud and generate two-dimensional bounding boxes and the probability that the bounding boxes belong to components. The knowledge extraction module is built based on a 2D-3D back-projection algorithm and a probabilistic aggregation strategy, and is used to generate 3D knowledge units. ; Among them, each pair Represents a knowledge unit. Indicates whether each point of the shape of the 3D point cloud sample is bounded by the d-th bounding box. Surround, Indicated in the bounding box The probability of each point in the point cloud on each component; N represents the number of points in the point cloud, R represents the number of components; D represents the total number of bounding boxes obtained after multiple multi-view rendered images and R text prompts are input into the visual language module.

2. The open-vocabulary few-sample 3D component segmentation method as described in claim 1, characterized in that, The inner loop is used to update the parameters of the segmentation head based on the task support set and the 3D knowledge unit for a single segmentation task, enabling the network to learn task-specific prior knowledge, specifically including: The 3D point cloud of the task support set is input into the 3D student network, processed by the encoder and adapter, and then transmitted to the segmentation head. Based on the 3D knowledge unit, calculate the distillation loss between the segmentation head output and the 3D knowledge unit; Based on the distillation loss, the segmentation head parameters are updated using the gradient descent algorithm to complete the segmentation head parameter adjustment for a single task.

3. The open-vocabulary few-sample 3D component segmentation method as described in claim 1, characterized in that, The outer loop is used to aggregate query set loss across multiple tasks and perform meta-learning updates on the initial parameters of the adapter and segmentation head, enabling the network to learn cross-category general prior knowledge, specifically including: Using the segmentation head trained by the inner loop, the 3D point cloud shape of each task query set is used to perform component segmentation prediction and calculate the corresponding distillation loss. Summarize the query set loss of all tasks in the current batch and take the average loss as the meta-objective function; Based on the aforementioned meta-objective function, the initial parameters of the adapter and segmentation head are updated synchronously using the gradient descent algorithm, enabling the network to learn cross-category general prior knowledge.

4. The open-vocabulary few-sample 3D component segmentation method as described in claim 1, characterized in that, The dynamic selection of the support set using retrieval enhancement strategies includes random selection strategy, similarity selection strategy, component completeness selection strategy, and joint selection strategy. Specifically, the similarity selection strategy is as follows: extract the basic point set features of the target point cloud and the samples in the training set, calculate the cosine similarity between the features, and select a preset number of samples with the highest similarity as the support set; The component completeness selection strategy is as follows: first, samples with complete components in the training set are selected to form a candidate set; then, the similarity between the target point cloud and the basic point set features of the candidate set samples is calculated; the preset number of samples with the highest similarity are selected as the support set; if the candidate set samples are insufficient, they are supplemented from the training set. The joint selection strategy integrates the similarity selection strategy and the component completeness selection strategy. Specifically, half of the samples are selected by the similarity selection strategy, and the other half are selected by the component completeness selection strategy. If the component completeness candidate set is insufficient, the remaining samples are supplemented by the similarity selection strategy.

5. An open-vocabulary, few-sample 3D component segmentation system, characterized in that, include: The 3D knowledge acquisition unit is used to process multi-view rendered images of 3D point cloud samples using a 2D teacher network to generate 3D knowledge units containing semantic information of components. A student network training unit is used to construct a 3D student network including an encoder, adapter, and segmentation head. The 3D point cloud samples and 3D knowledge units are input into the 3D student network, and training is performed using a two-layer loop mechanism based on knowledge distillation. In the two-layer loop mechanism, the inner loop is used to update the parameters of the segmentation head based on the task support set and the 3D knowledge units for a single segmentation task, enabling the network to learn task-specific prior knowledge. The outer loop is used to aggregate the query set loss across multiple tasks and perform meta-learning to update the initial parameters of the adapter and segmentation head, enabling the network to learn cross-category general prior knowledge. The inference unit is used to dynamically select a support set for the target point cloud using a retrieval enhancement strategy during the inference phase, and to fine-tune the segmentation head through an inner loop mechanism to obtain the component segmentation results of the target point cloud. The process of using a 2D teacher network to process multi-view rendered images of 3D point cloud samples to generate 3D knowledge units containing component semantic information specifically includes: Obtain 3D point cloud samples and corresponding component text prompts, and render multiple images of the 3D point cloud samples from multiple perspectives to obtain multi-view rendered images; Multi-view rendered images and component text prompts are input into a pre-trained 2D teacher network to generate 2D bounding boxes and the probability that the bounding boxes belong to components, and 3D knowledge units are obtained by back projection. The 2D teacher network includes a visual language module and a knowledge extraction module; The visual language module is used to extract component features from the point cloud and generate two-dimensional bounding boxes and the probability that the bounding boxes belong to components. The knowledge extraction module is built based on a 2D-3D back-projection algorithm and a probabilistic aggregation strategy, and is used to generate 3D knowledge units. ; Among them, each pair Represents a knowledge unit. Indicates whether each point of the shape of the 3D point cloud sample is bounded by the d-th bounding box. Surround, Indicated in the bounding box The probability of each point in the point cloud on each component; N represents the number of points in the point cloud, R represents the number of components; D represents the total number of bounding boxes obtained after multiple multi-view rendered images and R text prompts are input into the visual language module.

6. The open-vocabulary few-sample 3D component segmentation system as described in claim 5, characterized in that, The process of using a 2D teacher network to process multi-view rendered images of 3D point cloud samples to generate 3D knowledge units containing component semantic information specifically includes: Obtain 3D point cloud samples and corresponding component text prompts, and render multiple images of the 3D point cloud samples from multiple perspectives to obtain multi-view rendered images; Multi-view rendered images and component text prompts are input into a pre-trained 2D teacher network to generate 2D bounding boxes and the probability that the bounding boxes belong to components, and 3D knowledge units are obtained by back projection.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the open vocabulary few-sample 3D component segmentation method as described in any one of claims 1-4.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the open vocabulary few-sample 3D component segmentation method as described in any one of claims 1-4.

Images