A method and system for few-shot point cloud semantic segmentation based on cross-modal training-free operation

Abstract

This invention discloses a cross-modal, training-free method and system for few-shot point cloud semantic segmentation. The method includes the following steps: extracting point cloud features from samples in a support set and a query set containing only a small number of samples; generating text descriptions corresponding to semantic category labels using a specified large language model, mapping the text descriptions to a given 3D scene point cloud space to obtain text-mapped point cloud features; aggregating the point cloud features of the support set to obtain initial point cloud prototype features of the support set; updating the point cloud prototype features of the support set according to the text-point cloud similarity weight; fusing the point cloud features of the query set and the point cloud prototype features of the support set according to the similarity weight to obtain corrected point cloud prototype features; calculating the probability distribution of each point cloud in the query set on each semantic category label to determine the semantic category corresponding to the point cloud, and performing semantic segmentation. This invention significantly improves the performance of few-shot point cloud semantic segmentation without additional training.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and in particular to a method and system for semantic segmentation of few-sample point clouds based on cross-modal training-free methods. Background Technology

[0002] With the development of applications such as autonomous driving, robot navigation, and virtual reality, 3D point cloud data has become an important data form for scene perception. Point cloud semantic segmentation aims to assign correct semantic labels to each point cloud data point, but traditional point cloud segmentation methods face significant challenges due to the sparsity, irregularity, and disorder of point clouds. Although existing methods such as voxel-based convolutional networks and point cloud-based networks (such as PointNet and Seg-NN) have achieved some success, they still suffer from problems such as insufficient prototype representation, poor point-level matching of edge regions, sensitivity to noise, and large information gaps between classes when handling few-shot segmentation tasks. Summary of the Invention

[0003] The main objective of this invention is to provide a cross-modal, training-free method and system for semantic segmentation of few-sample point clouds, which can significantly improve the performance of semantic segmentation of few-sample point clouds without additional training.

[0004] The technical solution adopted in this invention is:

[0005] A few-shot point cloud semantic segmentation method based on cross-modal training-free implementation is provided, comprising the following steps:

[0006] S1. Extract point cloud features from the support set and query set of the given 3D scene point cloud space; wherein the support set includes multiple point cloud samples with semantic category labels, the number of semantic category labels is less than or equal to 3, and the number of samples with the same semantic category label is less than or equal to 5; the query set contains only multiple point cloud samples to be segmented.

[0007] S2. Based on semantic category labels and a given 3D scene, use a specified large language model to generate text descriptions corresponding to the semantic category labels, and map the text descriptions to the given 3D scene point cloud space to obtain text-mapped point cloud features.

[0008] S3. Aggregate the point cloud features of the support set to obtain the initial point cloud prototype features of the support set; where the initial point cloud prototype features are the representative center representations of a certain semantic category in the point cloud feature space.

[0009] S4. Align the feature dimensions of the text mapping point cloud features and the initial point cloud prototype features of the support set, calculate the correlation between the two and calculate the text-point cloud similarity weight accordingly, and update the point cloud prototype features of the support set based on the text-point cloud similarity weight.

[0010] S5. Based on the point cloud features of the query set and the updated point cloud prototype features of the support set, calculate the similarity distribution between the two and calculate the similarity weight accordingly. Then, based on the similarity weight, fuse the point cloud features of the query set and the point cloud prototype features of the support set to obtain the corrected point cloud prototype features.

[0011] S6. Calculate the similarity between the features of each point cloud in the query set and the modified point cloud prototype features to obtain the probability distribution of each point cloud in the query set on each semantic category label, determine the semantic category corresponding to each query point cloud, and perform semantic segmentation.

[0012] Following the above technical solution, the large language model specified in step S2 is DeepSeek.

[0013] Following the above technical solution, the aggregation process of the initial point cloud prototype features in step S3 is as follows: the point cloud features of the support set are subjected to element-wise masking constraints through an element-wise masking average pooling layer, and only the effective point features are aggregated to generate the initial point cloud prototype features.

[0014] Following the above technical solution, the pre-trained point cloud generative model Shap-E is used to map the text description to the point cloud space, and the text-mapped point cloud features are obtained through the point cloud encoder TaylorSeg.

[0015] Following the above technical solution, the specific calculation process of the text-point cloud similarity weight in step S4 is as follows: Align the text mapping point cloud features and the initial point cloud prototype features to the same feature dimension, calculate the category correlation between the two, combine the Softmax normalization result with the identity matrix for weighted fusion to obtain the similarity weight, and obtain the updated point cloud prototype features by multiplying the initial point cloud prototype features and the similarity weight.

[0016] Following the above technical solution, the inner product is specifically used to calculate the class correlation between the text mapping point cloud features and the initial point cloud prototype features.

[0017] Following the above technical solution, in step S5, when fusing the point cloud features of the query set and the point cloud prototype features of the support set according to the similarity weight, a dynamic weighted average strategy is specifically adopted for weighted fusion.

[0018] Following the above technical solution, step S6 specifically involves: calculating the cosine similarity between the features of each point cloud in the query set and the modified point cloud prototype features to obtain the response distribution of each point cloud across all semantic category labels; normalizing the response distribution and determining the semantic category corresponding to each point cloud based on the maximum response principle; segmenting the point cloud according to the semantic category to obtain the semantic segmentation result of the point cloud.

[0019] This invention also provides a cross-modal, training-free few-shot point cloud semantic segmentation system, comprising:

[0020] The point cloud feature extraction module is used to extract point cloud features from samples in the support set and query set of a given 3D scene point cloud space. The support set includes multiple point cloud samples with semantic category labels, the number of semantic category labels is less than or equal to 3, and the number of samples with the same semantic category label is less than or equal to 5. The query set contains only multiple point cloud samples to be segmented.

[0021] The text mapping module is used to generate text descriptions corresponding to semantic category labels based on semantic category labels and a given 3D scene using a specified large language model, and then map the text descriptions to the given 3D scene point cloud space to obtain text-mapped point cloud features.

[0022] The initial point cloud prototype feature acquisition module is used to aggregate the point cloud features of the support set to obtain the initial point cloud prototype features of the support set, wherein the initial point cloud prototype features are representative center representations of a certain semantic category in the point cloud feature space.

[0023] The point cloud prototype feature update module is used to align the feature dimensions of the text-mapped point cloud features and the initial point cloud prototype features of the support set, calculate the correlation between the two and calculate the text-point cloud similarity weight accordingly, and update the point cloud prototype features of the support set based on the text-point cloud similarity weight.

[0024] The point cloud prototype feature correction module is used to calculate the similarity distribution between the point cloud features of the query set and the updated point cloud prototype features of the support set, and calculate the similarity weight accordingly. Then, based on the similarity weight, the point cloud features of the query set and the point cloud prototype features of the support set are fused to obtain the corrected point cloud prototype features.

[0025] The semantic segmentation module is used to calculate the similarity between the features of each point cloud in the query set and the modified point cloud prototype features, obtain the probability distribution of each point cloud in the query set on each semantic category label, determine the semantic category corresponding to each query point cloud, and perform semantic segmentation.

[0026] The present invention also provides a computer storage medium storing a computer program executable by a processor, the computer program being used to implement the cross-modal training-free few-sample point cloud semantic segmentation method described in the above technical solution.

[0027] The beneficial effects of this invention are as follows: When performing few-shot segmentation tasks, this invention maps text descriptions to point cloud space and extracts features. Text semantics are explicitly projected into a geometric feature space consistent with the point cloud prototype features. Thus, without additional training or cross-modal alignment learning, it achieves effective fusion between text semantics and point cloud prototype features. This allows the point cloud prototype features to not only retain their original geometric discrimination capabilities but also possess stronger semantic expression capabilities and category generalization capabilities. At the same time, it significantly improves the performance of point cloud segmentation.

[0028] Furthermore, in constructing the initial point cloud prototype features, to avoid the loss of semantic information caused by traditional average pooling, an element-wise masked average pooling layer is proposed to aggregate the support set point cloud features element-wise in each channel dimension, so as to retain more fine-grained discriminative information and thus obtain more stable and more representative initial prototype features.

[0029] Furthermore, to ensure that the initial point cloud prototype features learn more semantic information, a prototype feature update method based on text-point cloud alignment is proposed. This method calculates the similarity matrix between the text-mapped point cloud features and the initial point cloud prototype features, and uses this similarity matrix to perform weighted fusion updates on the initial point cloud prototype features, thereby achieving semantic supplementation. In addition, based on the similarity distribution between the query features and the point cloud prototype features, adaptive refinement weights are generated to refine the point cloud prototype features, achieving local control of the point cloud prototype features by the query features, ultimately yielding the refined point cloud prototype features.

[0030] In summary, this method has good generalization ability, noise resistance and feature representation ability, and can be widely used in fields such as few-sample 3D point cloud recognition and segmentation.

[0031] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of a few-shot point cloud semantic segmentation method based on cross-modal training-free implementation according to an embodiment of the present invention;

[0034] Figure 2 This is a flowchart illustrating the main process of the few-sample point cloud semantic segmentation method based on cross-modal training-free implementation in this invention.

[0035] Figure 3 This is a schematic diagram of the experimental results of Embodiment 1 of the present invention;

[0036] Figure 4 This is a schematic diagram of the experimental results of Embodiment 2 of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0038] It should be noted that the illustrations provided in the embodiments of the present invention are only schematic representations of the basic concept of the present invention. Therefore, the illustrations only show the components related to the present invention and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0039] In this invention, it should also be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" are used only for descriptive and distinguishing purposes and should not be construed as indicating or implying relative importance.

[0040] Furthermore, it should be noted that the features of the various embodiments of the present invention can be combined or integrated in whole or in part, and as those skilled in the art will understand, they can interact and operate in different ways. Each embodiment can be implemented independently of each other or in association with one another.

[0041] like Figure 1 , Figure 2 As shown, the few-sample point cloud semantic segmentation method based on cross-modal training-free implementation in this embodiment of the invention mainly includes the following steps:

[0042] S1. Extract point cloud features from the support set and query set of the given 3D scene point cloud space; wherein the support set includes multiple point cloud samples with semantic category labels, the number of semantic category labels is less than or equal to 3, and the number of samples with the same semantic category label is less than or equal to 5; the query set contains only multiple point cloud samples to be segmented.

[0043] S2. Based on semantic category labels and a given 3D scene, use a specified large language model to generate text descriptions corresponding to the semantic category labels, and map the text descriptions to the given 3D scene point cloud space to obtain text-mapped point cloud features.

[0044] S3. Aggregate the point cloud features of the support set to obtain the initial point cloud prototype features of the support set; where the initial point cloud prototype features are the representative center representations of a certain semantic category in the point cloud feature space.

[0045] S4. Align the feature dimensions of the text mapping point cloud features and the initial point cloud prototype features of the support set, calculate the correlation between the two and calculate the text-point cloud similarity weight accordingly, and update the point cloud prototype features of the support set based on the text-point cloud similarity weight.

[0046] S5. Based on the point cloud features of the query set and the updated point cloud prototype features of the support set, calculate the similarity distribution between the two and calculate the similarity weight accordingly. Then, based on the similarity weight, fuse the point cloud features of the query set and the point cloud prototype features of the support set to obtain the corrected point cloud prototype features.

[0047] S6. Calculate the similarity between the features of each point cloud in the query set and the modified point cloud prototype features to obtain the probability distribution of each point cloud in the query set on each semantic category label, determine the semantic category corresponding to each query point cloud, and perform semantic segmentation.

[0048] In this embodiment of the invention, the large language model specified in step S2 is DeepSeek, but it can also be other large language models, such as Hatgpt, Gemini, Qianwen, etc.

[0049] Furthermore, the aggregation process of the initial point cloud prototype features in step S3 is as follows: the point cloud features of the support set are subjected to element-wise masking constraints through an element-wise masked average pooling layer, and only the effective point features are aggregated to generate the initial point cloud prototype features.

[0050] In step S2, the pre-trained point cloud generative model Shap-E is used to map the text description to the point cloud space, and the text-mapped point cloud features are obtained through the point cloud encoder TaylorSeg.

[0051] The specific calculation process for the text-point cloud similarity weight in step S4 is as follows: Align the text-mapped point cloud features and the initial point cloud prototype features to the same feature dimension, calculate the class correlation between them, and combine the Softmax normalization result with the identity matrix for weighted fusion to obtain the similarity weight. The updated point cloud prototype features are obtained by calculating the dot product of the initial point cloud prototype features and the similarity weight. The class correlation between the text-mapped point cloud features and the initial point cloud prototype features can be calculated using the inner product.

[0052] In step S5, when fusing the point cloud features of the query set and the point cloud prototype features of the support set according to the similarity weight, a dynamic weighted average strategy is specifically adopted for weighted fusion.

[0053] Step S6 specifically involves: calculating the cosine similarity between the features of each point cloud in the query set and the modified point cloud prototype features to obtain the response distribution of each point cloud across all semantic category labels; normalizing the response distribution and determining the semantic category corresponding to each point cloud based on the maximum response principle; segmenting the point cloud according to the semantic category to obtain the semantic segmentation result of the point cloud.

[0054] This invention explicitly projects text semantics into a geometric feature space consistent with the point cloud prototype features, thereby achieving effective fusion between text semantics and point cloud prototype features without additional training or cross-modal alignment learning. This allows the prototype to not only retain its original geometric discrimination ability but also to have stronger semantic expression and category generalization ability, while significantly improving the performance of point cloud segmentation.

[0055] Example 1

[0056] This embodiment uses the S3DIS dataset as an example to achieve cross-modal, training-free, few-sample point cloud semantic segmentation. S3DIS is a 3D point cloud dataset for indoor scenes, primarily used for indoor semantic segmentation tasks. Released by the Computer Science and Artificial Intelligence Laboratory (CSAIL) of a renowned international university, it focuses on modeling and analyzing large-scale indoor environments. The S3DIS dataset is an important dataset in indoor scene research, widely used as a benchmark dataset for 3D point cloud semantic segmentation and scene understanding. It mainly consists of 3D RGB point cloud datasets collected from 272 rooms across six indoor environments, containing a total of 13 categories. In existing technologies, these 13 categories are typically divided into two non-overlapping groups (S0 and S1). In this training-free implementation, a small number of point cloud samples with point-by-point semantic annotations can be selected from one of the groups as the support set; simultaneously, other point cloud samples from the same group that do not overlap with the support set are selected as the query set, and the samples in the query set do not have semantic annotations.

[0057] The few-shot point cloud semantic segmentation method based on the S3DIS dataset and requiring no training across modalities mainly includes the following steps:

[0058] S10. In the N-way K-shot few-shot semantic segmentation task, point cloud samples from the support set are first obtained. (including labels), among which Represents the number of K-shots. Indicates the number of points. This represents the original feature dimension of the point cloud. For each target category, a one-vs-rest modeling approach is used to construct a foreground mask based on the support set labels. The mask labels are represented as follows: Points belonging to the current category are considered foreground, while all other points are uniformly considered background. This is first achieved using a frozen TaylorSeg point cloud encoder. Extracting stable feature representations:

[0059]

[0060] here, This indicates that the TaylorSeg point cloud encoder supports set features, taking into account both local geometric relationships and global semantic context.

[0061] For point cloud samples in the query set (excluding labels), among which Indicates batch size, Indicates the number of points. This represents the original feature dimension of the point cloud. The frozen TaylorSeg point cloud encoder is also used. Extracting stable feature representations:

[0062]

[0063] here, Indicates query characteristics.

[0064] For each category name ("ceiling", "table"), a rough prompt template for the sample category is first generated using the DeepSeek large language model based on the indoor point cloud scene. Then, through pruning and fine-tuning (simplifying sentence structure, constraining the focus of description, etc.), the template is refined to: "Describe the indoor object {name} in simple and clear English. Focus on its basic shape, position, and relation to the room. The description should be concise and easy to use for semantic segmentation in the S3DIS dataset." Inputting a given specific sample category name (e.g., "ceiling"), the DeepSeek Large Language Model API is invoked, combined with prompt word templates, to generate a text description corresponding to that sample category: "The ceiling is the flat, horizontal surface at the top of a room. It covers the upper boundary, parallel to the floor. Typically smooth and uniform, it connects to walls at the edges. It shields the room from the structure above, like roofs or higher floors." This generated text description is then used as input to a pre-trained Shap-E point cloud generative model to generate corresponding point cloud samples. Then, the frozen TaylorSeg point cloud encoder is used to extract text mapping point cloud features.

[0065]

[0066] S20. For support set point clouds, it is necessary to obtain the support set point cloud prototype features (point cloud prototype features are representative center representations of a certain semantic category in the point cloud feature space, used to summarize the overall geometric and semantic features of the point cloud of that category) as a standard for final segmentation prediction. To generate better initial point cloud prototype features, first... and Flatten them separately, , Based on the above formula, flattened support set features are extracted from the flattened support set samples. After element-wise masked average pooling, only the foreground features are weighted and averaged to suppress background noise and alleviate the instability caused by the sparsity of foreground features under conditions of few samples. Simultaneously, a global background prototype feature is constructed using background points from all support samples to enhance the model's ability to distinguish non-target regions, resulting in the initial point cloud prototype features of the support set foreground and background, as shown in the following formula:

[0067]

[0068]

[0069] After obtaining the foreground and background prototype features, they are further fused into a unified initial point cloud prototype feature. This balances the foreground's distinguishability with the contextual information of the background. Specifically, it uses a direct splicing method to merge foreground and background information, with the following mathematical formula:

[0070]

[0071] S30. The initial point cloud prototype features obtained so far do not have semantic learning capabilities. In order to enhance the representation ability in few-shot segmentation, the obtained text-mapped point cloud features and the initial point cloud prototype features are fused across modally. First, the two need to be mapped to the same feature dimension. Then, the inner product is used to enable the initial point cloud prototype features to learn more semantic information from the text-mapped point cloud features. The specific operation is as follows:

[0072]

[0073] Where S represents the class correlation between the text-mapped point cloud features and the initial point cloud prototype features, used to construct a cross-modal similarity matrix to measure the consistency between the two features. Since the text-mapped point cloud features should pay more attention to their corresponding initial point cloud prototype features, the correlation results are processed in two steps: (1) normalization; (2) diagonal enhancement. The formula is as follows:

[0074]

[0075]

[0076] in, It is an identity matrix, and its diagonal elements correspond to "subcategories". It is the balance factor, which is set to 0.7 in the experiment. This represents the processed similarity matrix. By applying the similarity matrix... The initial point cloud prototype features are compared with the similarity matrix. After updating through multiplication, the final prototype features updated by the text-mapped point cloud features are obtained:

[0077]

[0078] S40. To enhance the generalization ability of prototype features, enabling them to learn the feature distribution in the query set while relying on support set features, this embodiment proposes a method for prototype feature refinement. This method dynamically adjusts the refinement process of prototype features by introducing a weighted and similarity matching mechanism, thereby accurately optimizing the prototype features for each category.

[0079] The specific steps are as follows:

[0080] First, the similarity between the query feature and the prototype feature is calculated using the dot product. Each query feature is a D-dimensional vector, and each category prototype feature is also a D-dimensional vector. The dot product is used to measure the similarity between two vectors, and its calculation formula is as follows:

[0081]

[0082] in, This represents the transpose of the query feature vector. Indicates category The prototype feature vector. The result is a similarity matrix. ,in This represents the total number of categories (including the background class). Each element in the matrix... This indicates the similarity between the query feature and the prototype feature.

[0083] After obtaining the similarity between the query features and the prototype features, through The normalization process converts this similarity into a probability distribution, as shown in the following formula:

[0084]

[0085] The numerator represents the exponential form of similarity, and the denominator is the sum of the exponents of similarity. In this way, The function normalizes the similarity values ​​to a probability distribution, ensuring that the sum of similarities across all categories is 1. The final similarity weights are then obtained. Indicate query characteristics Category The degree of.

[0086] Next, a dynamic weighted average strategy is adopted for each query feature. Its corresponding category Similarity weight Multiply. Then, sum the results of multiplying all categories and divide by the sum of all weights to obtain the refinement direction of the prototype features for that category. The formula is as follows:

[0087]

[0088] in, It is a smoothing term used to prevent division by zero errors, and is usually set to a very small value.

[0089] To preserve the updated prototype feature information, a smoothing strategy is introduced to fuse the old and new prototype features, resulting in refined prototype features. This smoothing update mechanism prevents the new prototype features from changing too rapidly, thus avoiding overfitting and the loss of inherent feature information. Subsequently, the refined prototype features are normalized to maintain a consistent representation in the feature space, as shown in the following formula:

[0090]

[0091]

[0092] in For the updated point cloud prototype features, This indicates that the step size coefficient is being updated. Experimental setup It is 0.2, while This represents the final refined prototype features.

[0093] Finally, the background prototype features and the prototype features of each category are combined into a new prototype matrix, which is then used for subsequent downstream segmentation prediction tasks. The specific steps are as follows:

[0094]

[0095] in, This represents the feature dimension of the point cloud, while This indicates the total number of categories, including the background category.

[0096] S50. In the final prediction stage, calculate the similarity between the query features and the refined prototype features in the point cloud feature space.

[0097]

[0098] The obtained similarity is converted into a positive response strength, so that the query feature that is closer to the prototype feature has a larger response in the corresponding category.

[0099]

[0100] in The scaling factor is used to enhance high-similarity responses. Subsequently, the response results are normalized, and the class probability distribution of the query samples is output as the final semantic segmentation prediction result.

[0101]

[0102] In this embodiment, the hyperparameters are modulated. and To achieve optimal segmentation performance, such as Figure 3 As shown.

[0103] Example 2

[0104] This embodiment uses the ScanNet dataset as an example. The few-sample point cloud semantic segmentation method based on cross-modal point cloud prototype update mainly includes the following steps:

[0105] ScanNet is a large-scale indoor 3D point cloud dataset designed to advance research in tasks such as indoor scene understanding, semantic segmentation, and 3D reconstruction. Released jointly by the Computer Science and Artificial Intelligence Laboratory (CSAIL) of a renowned university and other academic institutions, it focuses on 3D modeling of indoor environments. ScanNet is one of the most well-known and widely used indoor point cloud datasets. The ScanNet indoor scene dataset contains 305 data points collected by RGB-D cameras and structured light sensors, covering 20 categories across 1,513 indoor scenes.

[0106] S10. In the N-way K-shot few-shot semantic segmentation task, point cloud samples from the support set are first obtained. (including labels), among which Represents the number of K-shots. Indicates the number of points. This represents the original feature dimension of the point cloud. For each target category, a one-vs-rest modeling approach is used to construct a foreground mask based on the support set labels. The mask labels are represented as follows: Points belonging to the current category are considered foreground, while all other points are uniformly considered background. This is first achieved using a frozen TaylorSeg point cloud encoder. Extracting stable feature representations:

[0107]

[0108] here, This indicates that the support set features, and the point cloud encoder takes into account both local geometric relationships and global semantic context.

[0109] For point cloud samples in the query set (excluding labels), among which Indicates batch size, Indicates the number of points. This represents the original feature dimension of the point cloud. The frozen TaylorSeg point cloud encoder is also used. Extracting stable feature representations:

[0110]

[0111] here, Indicates query characteristics.

[0112] For each category name ("wall", "floor"), a rough prompt template for the sample category is first generated using the DeepSeek large language model based on the point cloud scene. Then, through pruning and fine-tuning (simplifying sentence structure, constraining the focus of description, etc.), the template is refined to: "Describe the indoor object {name} in simple and clear English. Focus on its basic shape, position, and relation to the room. The description should be concise and easy to use for semantic segmentation in the ScanNet dataset." Inputting a given specific sample category name (e.g., "wall"), the DeepSeek Large Language Model API is invoked, combined with prompt word templates, to generate a text description corresponding to that sample category: "A wall is a large, flat, vertical surface. It forms the perimeter of a room, connecting the floor to the ceiling. Walls are typically rectangular and stand upright. They define the room's boundaries and separate it from other spaces." This generated text description is then used as input to a pre-trained Shap-E point cloud generative model to generate corresponding point cloud samples. Then, the frozen TaylorSeg point cloud encoder is used to extract text mapping point cloud features.

[0113]

[0114] S20. For support set point clouds, it is necessary to obtain the support set point cloud prototype features (point cloud prototype features are representative center representations of a certain semantic category in the point cloud feature space, used to summarize the overall geometric and semantic features of the point cloud of that category) as a standard for final segmentation prediction. To generate better initial point cloud prototype features, first... and Flatten them separately, , Based on the above formula, flattened support set features are extracted from the flattened support set samples. After element-wise masked average pooling, only the foreground features are weighted and averaged to suppress background noise and alleviate the instability caused by the sparsity of foreground features under conditions of few samples. Simultaneously, a global background prototype feature is constructed using background points from all support samples to enhance the model's ability to distinguish non-target regions. The initial point cloud prototype features of the foreground and background of the support set are obtained. The formula is as follows:

[0115]

[0116]

[0117] After obtaining the foreground and background prototype features, they are further fused into a unified initial point cloud prototype feature. This method aims to balance the distinguishability of the foreground with the contextual information of the background. Specifically, it uses a direct splicing method to merge foreground and background information. The mathematical formula is as follows:

[0118]

[0119] S30. However, the initial point cloud prototype features obtained so far do not have semantic learning capabilities. In order to enhance the representation ability in few-shot segmentation, the obtained text-mapped point cloud features and the initial point cloud prototype features are fused across modally. First, the two need to be mapped to the same feature dimension. Then, inner product calculation is used to enable the initial point cloud prototype features to learn more semantic information from the text-mapped point cloud features. The specific operation is as follows:

[0120]

[0121] Where S represents the class correlation between the text-mapped point cloud features and the initial point cloud prototype features, used to construct a cross-modal similarity matrix to measure the consistency between the two features. Since the text-mapped point cloud features should pay more attention to their corresponding initial point cloud prototype features, the correlation results are processed in two steps: (1) normalization; (2) diagonal enhancement. The formula is as follows:

[0122]

[0123]

[0124] in, It is an identity matrix, and its diagonal elements correspond to "subcategories". It is the balance factor, which is set to 0.7 in the experiment. This represents the processed similarity matrix. By applying the similarity matrix... The initial point cloud prototype features are compared with the similarity matrix. After updating through multiplication, the final prototype features updated by the text-mapped point cloud features are obtained:

[0125]

[0126] S40. To enhance the generalization ability of prototype features, enabling them to learn the feature distribution in the query set while relying on support set features, this study proposes a method for prototype feature refinement. This method dynamically adjusts the refinement process of prototype features by introducing a weighted and similarity-based matching mechanism, thereby accurately optimizing the prototype features for each category. The specific steps are as follows.

[0127] First, the similarity between the query feature and the prototype feature is calculated using the dot product. Each query feature is a D-dimensional vector, and each category prototype feature is also a D-dimensional vector. The dot product is used to measure the similarity between the two vectors. The formula is as follows:

[0128]

[0129] in, This represents the transpose of the query feature vector. Indicates category The prototype feature vector. The result is a similarity matrix. ,in This represents the total number of categories (including the background class). Each element in the matrix... This indicates the similarity between the query feature and the prototype feature.

[0130] After obtaining the similarity between the query features and the prototype features, through The normalization process converts this similarity into a probability distribution, as shown in the following formula:

[0131]

[0132] The numerator represents the exponential form of similarity, and the denominator is the sum of the exponents of similarity. In this way, The function normalizes the similarity values ​​to a probability distribution, ensuring that the sum of similarities across all categories is 1. The final similarity weights are then obtained. Indicate query characteristics Category The degree of.

[0133] Next, a dynamic weighted average strategy is adopted for each query feature. Its corresponding category Similarity weight Multiply. Then, sum the results of multiplying all categories and divide by the sum of all weights to obtain the refinement direction of the prototype features for that category. The formula is as follows:

[0134]

[0135] in, It is a smoothing term used to prevent division by zero errors, and is usually set to a very small value.

[0136] To preserve the updated prototype feature information, a smoothing strategy is introduced to fuse the old and new prototype features, resulting in refined prototype features. This smoothing update mechanism prevents the new prototype features from changing too rapidly, thus avoiding overfitting and the loss of inherent feature information. Subsequently, the refined prototype features are normalized to maintain a consistent representation in the feature space. The formula is as follows:

[0137]

[0138]

[0139] in For the updated point cloud prototype features, This indicates that the step size coefficient is being updated. Experimental setup It is 0.2, while This represents the final refined prototype features.

[0140] Finally, the background prototype features and the prototype features of each category are combined into a new prototype matrix, which is then used for subsequent downstream segmentation prediction tasks. The specific steps are as follows:

[0141]

[0142] in, This represents the feature dimension of the point cloud, while This indicates the total number of categories, including the background category.

[0143] S50. In the final prediction stage, calculate the similarity between the query features and the refined prototype features in the point cloud feature space.

[0144]

[0145] The obtained similarity is converted into a positive response strength, so that the query feature that is closer to the prototype feature has a larger response in the corresponding category.

[0146]

[0147] in The scaling factor is used to enhance high-similarity responses. Subsequently, the response results are normalized, and the class probability distribution of the query samples is output as the final semantic segmentation prediction result.

[0148]

[0149] In this embodiment, the hyperparameters are modulated. and To achieve optimal segmentation performance, such as Figure 4 As shown.

[0150] As shown in Tables 1 and 2, where N-way represents N semantic categories, K-shot represents K samples per semantic category, and s0 and s1 represent dataset category groupings, compared with training-free methods such as Point-NN, Seg-NN, and TaylorSeg-NN, the proposed method achieves good segmentation performance under training-free cross-modal fusion. The following results demonstrate that the proposed cross-modal training-free few-shot point cloud semantic segmentation method, while extracting point cloud features using the TaylorSeg point cloud encoder, utilizes text-generated point clouds to guide prototype updates and element-wise masked average pooling prototype enhancement. This effectively addresses the problems of insufficient 3D spatial semantic representation and high computational overhead of cross-modal fusion, reduces the impact of noise on prototype updates and segmentation tasks, and compensates for the limitation of prototypes relying solely on the support set by query-guided prototype correction, effectively improving the accuracy of point cloud segmentation. The results show that this method is superior to other methods.

[0151] Table 1. Schematic diagram of experimental results in Example 1

[0152]

[0153] Note: TaylorSeg-NN originates from [Wang et al., 2025] Changshuo Wang, Shuting He, Xiang Fang, Meiqing Wu, Siew-Kei Lam, and Prayag Tiwari. Taylor series-inspired local structure fitting network for few-shot point cloud semanticsegmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7527–7535, 2025. (i.e., [Wang Changshuo et al., 2025] Wang Changshuo, He Shuting, Fang Xiang, Wu Meiqing, Lin Shouji, Prayag Tiwari, Taylor series-inspired local structure fitting network for few-shot point cloud semanticsegmentation, included in Proceedings of the AAAI Conference on Artificial Intelligence, Volume 39, pp. 7527–7535, 2025.)

[0154] Table 2. Schematic diagram of experimental results in Example 2

[0155]

[0156] Classification performance on the S3DIS and ScanNet datasets, all figures are expressed as percentages.

[0157] In summary, the point cloud segmentation method of the present invention does not require additional training, and the segmentation is efficient and accurate. Compared with other point cloud segmentation methods that do not require training, the segmentation performance of the present invention has achieved a certain degree of superiority.

[0158] To implement the above method embodiments, the present invention also provides a few-sample point cloud semantic segmentation system based on cross-modal training-free operation, comprising:

[0159] The point cloud feature extraction module is used to extract point cloud features from samples in the support set and query set of a given 3D scene point cloud space. The support set includes multiple point cloud samples with semantic category labels, the number of semantic category labels is less than or equal to 3, and the number of samples with the same semantic category label is less than or equal to 5. The query set contains only multiple point cloud samples to be segmented.

[0160] The text mapping module is used to generate text descriptions corresponding to semantic category labels based on semantic category labels and a given 3D scene using a specified large language model, and then map the text descriptions to the given 3D scene point cloud space to obtain text-mapped point cloud features.

[0161] The initial point cloud prototype feature acquisition module is used to aggregate the point cloud features of the support set to obtain the initial point cloud prototype features of the support set; wherein the initial point cloud prototype features are representative center representations of a certain semantic category in the point cloud feature space.

[0162] The point cloud prototype feature update module is used to align the feature dimensions of the text-mapped point cloud features and the initial point cloud prototype features of the support set, calculate the correlation between the two and calculate the text-point cloud similarity weight accordingly, and update the point cloud prototype features of the support set based on the text-point cloud similarity weight.

[0163] The point cloud prototype feature correction module is used to calculate the similarity distribution between the point cloud features of the query set and the updated point cloud prototype features of the support set, and calculate the similarity weight accordingly. Then, based on the similarity weight, the point cloud features of the query set and the point cloud prototype features of the support set are fused to obtain the corrected point cloud prototype features.

[0164] The semantic segmentation module is used to calculate the similarity between the features of each point cloud in the query set and the modified point cloud prototype features, obtain the probability distribution of each point cloud in the query set on each semantic category label, determine the semantic category corresponding to each query point cloud, and perform semantic segmentation.

[0165] Each module is mainly used to implement the above methods and steps, and will not be described in detail here.

[0166] This application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, app store, etc., which stores a computer program, and the program implements corresponding functions when executed by a processor. The computer-readable storage medium of this embodiment implements the method of cross-modal training-free few-sample point cloud semantic segmentation based on this embodiment when executed by a processor.

[0167] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0168] The order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0169] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A few-sample point cloud semantic segmentation method based on cross-modal training-free operation, characterized in that, Includes the following steps: S1. Extract point cloud features from the support set and query set of a given 3D scene point cloud space; The support set includes multiple point cloud samples with semantic category labels, the number of semantic category labels is less than or equal to 3, and the number of samples with the same semantic category label is less than or equal to 5; the query set contains only multiple point cloud samples to be segmented. S2. Based on semantic category labels and a given 3D scene, use a specified large language model to generate text descriptions corresponding to the semantic category labels, and map the text descriptions to the given 3D scene point cloud space to obtain text-mapped point cloud features. S3. Aggregate the point cloud features of the support set to obtain the initial point cloud prototype features of the support set, where the initial point cloud prototype features are the representative center representations of a certain semantic category in the point cloud feature space. S4. Align the feature dimensions of the text mapping point cloud features and the initial point cloud prototype features of the support set, calculate the correlation between the two and calculate the text-point cloud similarity weight accordingly, and update the point cloud prototype features of the support set based on the text-point cloud similarity weight. S5. Based on the point cloud features of the query set and the updated point cloud prototype features of the support set, calculate the similarity distribution between the two and calculate the similarity weight accordingly. Then, based on the similarity weight, fuse the point cloud features of the query set and the point cloud prototype features of the support set to obtain the corrected point cloud prototype features. S6. Calculate the similarity between the features of each point cloud in the query set and the modified point cloud prototype features to obtain the probability distribution of each point cloud in the query set on each semantic category label, determine the semantic category corresponding to each query point cloud, and perform semantic segmentation.

2. The few-shot point cloud semantic segmentation method based on cross-modal training-free operation according to claim 1, characterized in that, The large language model specified in step S2 is DeepSeek.

3. The few-shot point cloud semantic segmentation method based on cross-modal training-free operation according to claim 1, characterized in that, The aggregation process of the initial point cloud prototype features in step S3 is as follows: the point cloud features of the support set are subjected to element-wise masking constraints through the element-wise masked average pooling layer, and only the effective point features are aggregated to generate the initial point cloud prototype features.

4. The few-shot point cloud semantic segmentation method based on cross-modal training-free operation according to claim 1, characterized in that, Specifically, the pre-trained point cloud generative model Shap-E is used to map text descriptions to point cloud space, and the text-mapped point cloud features are obtained through the point cloud encoder TaylorSeg.

5. The few-shot point cloud semantic segmentation method based on cross-modal training-free operation according to claim 1, characterized in that, The specific calculation process of text-point cloud similarity weight in step S4 is as follows: Align the text mapping point cloud features and the initial point cloud prototype features to the same feature dimension, calculate the class correlation between the two, combine the Softmax normalization result with the identity matrix for weighted fusion to obtain the similarity weight, and obtain the updated point cloud prototype features by multiplying the initial point cloud prototype features and the similarity weight.

6. The few-shot point cloud semantic segmentation method based on cross-modal training-free operation according to claim 5, characterized in that, Specifically, the inner product is used to calculate the class correlation between the text mapping point cloud features and the initial point cloud prototype features.

7. The few-shot point cloud semantic segmentation method based on cross-modal training-free operation according to claim 1, characterized in that, In step S5, when fusing the point cloud features of the query set and the point cloud prototype features of the support set according to the similarity weight, a dynamic weighted average strategy is specifically adopted for weighted fusion.

8. The few-shot point cloud semantic segmentation method based on cross-modal training-free operation according to claim 1, characterized in that, Step S6 specifically involves: calculating the cosine similarity between the features of each point cloud in the query set and the modified point cloud prototype features to obtain the response distribution of each point cloud across all semantic category labels; normalizing the response distribution and determining the semantic category corresponding to each point cloud based on the maximum response principle; segmenting the point cloud according to the semantic category to obtain the semantic segmentation result of the point cloud.

9. A few-shot point cloud semantic segmentation system based on cross-modal training-free operation, characterized in that, include: The point cloud feature extraction module is used to extract point cloud features from samples in the support set and query set of a given 3D scene point cloud space. The support set includes multiple point cloud samples with semantic category labels, the number of semantic category labels is less than or equal to 3, and the number of samples with the same semantic category label is less than or equal to 5; the query set contains only multiple point cloud samples to be segmented. The text mapping module is used to generate text descriptions corresponding to semantic category labels based on semantic category labels and a given 3D scene using a specified large language model, and then map the text descriptions to the given 3D scene point cloud space to obtain text-mapped point cloud features. The initial point cloud prototype feature acquisition module is used to aggregate the point cloud features of the support set to obtain the initial point cloud prototype features of the support set. The initial point cloud prototype feature is a representative center representation of a certain semantic category in the point cloud feature space; The point cloud prototype feature update module is used to align the feature dimensions of the text-mapped point cloud features and the initial point cloud prototype features of the support set, calculate the correlation between the two and calculate the text-point cloud similarity weight accordingly, and update the point cloud prototype features of the support set based on the text-point cloud similarity weight. The point cloud prototype feature correction module is used to calculate the similarity distribution between the point cloud features of the query set and the updated point cloud prototype features of the support set, and calculate the similarity weight accordingly. Then, based on the similarity weight, the point cloud features of the query set and the point cloud prototype features of the support set are fused to obtain the corrected point cloud prototype features. The semantic segmentation module is used to calculate the similarity between the features of each point cloud in the query set and the modified point cloud prototype features, obtain the probability distribution of each point cloud in the query set on each semantic category label, determine the semantic category corresponding to each query point cloud, and perform semantic segmentation.

10. A computer storage medium, characterized in that, It contains a computer program that can be executed by a processor, which is used to implement the few-sample point cloud semantic segmentation method based on cross-modal training-free method as described in any one of claims 1-8.

Images