An uncertainty-guided few-shot 3D indoor scene point cloud semantic segmentation method and system

Abstract

The application discloses a few-shot three-dimensional indoor scene point cloud semantic segmentation method and system based on uncertainty guidance, aiming at the problems of new class label scarcity, high pseudo-label noise, few-shot adaptation easily leading to base class knowledge forgetting and the like, and constructs a double reliability control framework of "front pseudo-label uncertainty purification + rear gradient uncertainty modulation". A three-dimensional visual language model is used to generate point-by-point pseudo-labels, class responses and point features, and the pseudo-labels are filtered by quantile, and high-uncertainty points are removed; meanwhile, reliable ignored points are adaptively completed in combination with a new class visual prototype. In the model training stage, the system constructs a base class gradient orthogonal basis offline, estimates batch uncertainty online, orthogonally decomposes and adaptively modulates original gradients, enhances new class learning in reliable batches, and suppresses semantic drift in unreliable batches. The application is suitable for indoor robot navigation, intelligent building digital twinning, augmented reality and three-dimensional scene understanding and the like.

Description

Technical Field

[0001] This invention relates to the technical field of cross-fusion of artificial intelligence, 3D computer vision, few-shot learning and point cloud semantic segmentation, specifically to a few-shot 3D indoor scene point cloud semantic segmentation method and system based on uncertainty guidance. Background Technology

[0002] 3D point cloud semantic segmentation is a fundamental task in 3D scene understanding, aiming to assign a corresponding semantic category label to each point in an indoor 3D point cloud. This technology is widely used in scenarios such as indoor robot navigation, augmented reality, virtual reality, digital twins of smart buildings, spatial interaction, indoor security sensing, and service robot grasping and path planning.

[0003] With the development of deep learning technology, fully supervised 3D point cloud semantic segmentation methods based on point cloud neural networks, sparse convolutional networks, point cloud attention networks, and 3D Transformer networks have made significant progress. However, fully supervised methods usually require point-by-point annotation of a large number of 3D point clouds, while indoor 3D scenes often have a large number of points, high granularity of categories, and widespread occlusion and scan gaps, making point-by-point annotation extremely costly.

[0004] In practical applications, 3D scene segmentation systems often face the problem of continuously expanding categories. For example, during the initial training phase, the system may only learn common categories such as walls, floors, tables, and chairs, while in the subsequent deployment phase, it needs to quickly identify new indoor object categories. For these newly added categories, only a very small number of labeled samples for each category are usually available. Therefore, few-sample 3D point cloud semantic segmentation has become an important technical approach to solve this problem.

[0005] Semantic segmentation of few-shot 3D point clouds typically employs a two-stage training paradigm: the first stage trains a basic segmentation model using well-annotated base class samples; the second stage performs adaptation training on a small number of newly annotated class samples, enabling the model to learn new classes while maintaining its segmentation capabilities for the base classes. However, this process presents the following technical challenges.

[0006] First, the few newly labeled samples exhibit sparsity and incompleteness. Indoor point clouds are formed by sensor data collection and are often affected by factors such as viewing angle, occlusion, reflection, object stacking, and uneven scan density. A single sample in a few labeled samples may only be a local fragment of a new object class and cannot represent the complete class morphology. Directly using such sparse labels for model updates can easily lead to the model learning random geometric patterns or local noise.

[0007] Second, while 3D visual language models can provide dense pseudo-supervision for few-shot segmentation, their output pseudo-labels still contain errors. 3D visual language models typically obtain open-class responses by matching visual features with language category features. However, in complex indoor scenes, due to sparse point clouds, occlusion, category similarity, projection errors, or semantic ambiguity, pseudo-labels may result in false positives, false negatives, or category confusion. Directly using these pseudo-labels as training supervision would inject erroneous supervisory signals into the model training process.

[0008] Third, existing pseudo-label filtering methods mostly employ block-level or region-level hard filtering strategies. For example, they might retain or discard an entire pseudo-label region. While these methods are simple, they cannot handle the problem of inconsistent reliability among different points within the same region. When a region contains both high-confidence and low-confidence points, block-level hard filtering can lead to two types of problems: on the one hand, it may discard valuable reliable points, and on the other hand, it may retain unreliable points, thus affecting the learning of new classes.

[0009] Fourth, the limited sample adaptation stage can easily lead to the forgetting of base class knowledge. The model has already learned stable representations of the base classes in the first stage, but in the second stage, there are few new class samples and the pseudo-labels are noisy. If the same gradient intensity is used to update all training batches, low-quality batches may cause the model parameters to change in the wrong direction, destroying the original base class feature space and resulting in a decline in base class performance.

[0010] Fifth, existing gradient constraint methods in continuous learning or few-shot adaptation often lack awareness of the reliability of the current training batch. Some methods employ fixed projection, fixed freezing, or fixed regularization strategies to protect old knowledge, but these methods are usually static and cannot automatically adjust the learning intensity based on the quality of new class samples in the current batch. When the supervision signal of the current batch is reliable, overly strong constraints will hinder the learning of new classes; when the supervision signal of the current batch is unreliable, overly weak constraints will cause semantic drift.

[0011] Sixth, the utilization of uncertainty signals in existing 3D point cloud few-shot segmentation is still insufficient. Even when uncertainty is used in existing methods, it is mostly used for sample selection or loss weighting, and rarely as a control signal at the training and update level. Specifically, existing methods lack a unified mechanism that can simultaneously clean up the supervision content at the pseudo-label input and adjust the optimization direction and update intensity at the gradient update level.

[0012] Therefore, there is an urgent need for a new semantic segmentation method and system for point clouds in 3D indoor scenes with few samples. After generating pseudo-labels, uncertainty is first used for fine-grained purification, and after backpropagation of the model, round-level uncertainty is used to adaptively modulate the gradient, thereby achieving a better balance between new class learning ability and base class knowledge retention. Summary of the Invention

[0013] This invention addresses the aforementioned problems by providing a method and system for semantic segmentation of point clouds in 3D indoor scenes based on uncertainty guidance. The method takes point clouds of indoor 3D scenes and a small number of newly labeled samples as input. Through a process of "pseudo-label generation and uncertainty cleanup—calibration supervision target construction—backbone network few-sample adaptation—uncertainty-guided gradient modulation—multi-registration set aggregation and evaluation," it achieves point-by-point semantic segmentation of scenes where base and new classes coexist.

[0014] This invention provides a semantic segmentation method for point clouds in 3D indoor scenes based on uncertainty guidance, characterized by the following steps: Step 1, acquiring indoor 3D scene point cloud data and a small number of new class labeled samples and preprocessing them to obtain training data; Step 2, inputting the preprocessed training data into a pre-trained 3D visual language model to obtain initial pseudo-labels, constructing new class visual prototypes based on a small number of new class labeled samples, performing similarity matching between the initial pseudo-labels and the corresponding new class visual prototypes, removing pseudo-label points or pseudo-label regions with large differences to obtain preliminary pseudo-labels; Step 3, performing uncertainty point-level filtering on the preliminary pseudo-labels, setting pseudo-labels with high uncertainty points as ignore markers to obtain purified pseudo-labels; Step 4, further refining the purified pseudo-labels... Step 5: Merge pseudo-labels with a small number of new class real labels to form the calibration supervision target for the current training batch; Step 6: Input the 3D point cloud of the current training batch into the backbone network segmentation model, calculate the point-by-point segmentation loss according to the calibration supervision target, and perform few-shot adaptation training on the backbone network; Step 7: During the few-shot adaptation training, estimate the round-level uncertainty based on the new class supervision signal of the current training batch, modulate the vertical component of the current gradient according to the round-level uncertainty, obtain the adjusted gradient, and use the modulated gradient as the basis for updating the model parameters, and the optimizer performs the parameter update; Step 8: Repeat steps 5 to 6 until all registration sets are trained, summarize the evaluation metrics of multiple registration sets, and obtain the semantic segmentation result.

[0015] Preferably, in step 1, the indoor 3D scene point cloud data includes one or more of the following: 3D coordinates, color, normal vector, reflection intensity, and timestamp; a small number of new class labeled samples are used for new class adaptation training based on the existing segmentation capabilities of the base class model. Preprocessing includes point cloud parsing, coordinate normalization, sampling, voxelization, feature processing, data augmentation, and training batch construction of the indoor 3D scene point cloud data.

[0016] Preferably, in step 2, the preprocessed training data is input into the pre-trained 3D visual language model to obtain point-by-point category-related responses, point-by-point category logical scores, and point-by-point visual features. The 3D visual language model is any model capable of outputting point-level, region-level, or scene-level category-related responses based on 3D point clouds and category text information.

[0017] Preferably, step 2 further includes: constructing a new class visual prototype based on a small number of new class labeled samples, point-by-point visual features or point-by-point category-related responses output by the 3D visual language model.

[0018] Preferably, in step 3, the uncertainty-level filtering includes: normalizing the point-by-point category logical scores output by the 3D visual language model to obtain a point-by-point category probability distribution; calculating the uncertainty of each point based on the point-by-point category probability distribution; and performing point-level filtering on the initially screened pseudo-labels according to a preset filtering granularity and quantile threshold strategy, setting pseudo-labels with high uncertainty points as ignore tags to obtain purified pseudo-labels. Uncertainty can be one or more of the following: information entropy, category margin, maximum probability, category response energy, prediction variance, consistency across multiple perturbations, or model ensemble differences. The filtering granularity includes intra-class filtering granularity and global filtering granularity.

[0019] Preferably, the method further includes an adaptive prototype completion step: for points that are set as ignored, a completion judgment is made based on the similarity between their point features and the visual prototype of the new class. When the similarity is higher than the preset completion threshold, the point is completed as the corresponding new class label; otherwise, the ignored label is maintained.

[0020] Preferably, the method further includes an offline gradient basis construction step: using the model after the base class training is completed, parameter gradients corresponding to multiple training batches are collected on the base class data or base class validation data. For a preset parameter set, the parameter gradients obtained by backpropagation of each batch are organized into gradient vectors according to the parameter tensors, and the gradient vectors of multiple batches are combined into a gradient data matrix. Orthogonalization or low-rank subspace estimation is performed on the gradient data matrix to obtain the parameter-level orthogonal gradient basis. The parameter-level orthogonal gradient basis is used to characterize the stable optimization direction formed during the base class training phase, which is the gradient subspace of the base class task. The gradient basis construction method can be any one or more of QR decomposition, singular value decomposition, principal component analysis, Gram-Schmidt orthogonalization, eigenvalue decomposition, and stochastic low-rank approximation.

[0021] Preferably, in step 6, the method for estimating round-level uncertainty is as follows: Point-by-point features of the current training batch are extracted using the backbone network. For each new class in the current training batch, the average distance, variance, or dispersion between the feature points of that class and its class center is calculated. Then, the intra-class dispersion of multiple new classes is averaged to obtain the original round-level uncertainty. The original round-level uncertainty is then subjected to runtime statistical normalization. The gradient modulation coefficient is calculated based on the normalized round-level uncertainty. The gradient modulation coefficient is negatively correlated with the round-level uncertainty. The gradient modulation method is as follows: Using the parameter-level orthogonal gradient basis constructed offline, the original gradient of the current gradient is orthogonally decomposed to obtain parallel gradient components and vertical gradient components. The vertical gradient component is multiplied by the gradient modulation coefficient and then combined with the parallel gradient components to synthesize the modulated gradient.

[0022] Preferably, in step 7, the evaluation metrics include one or more of the following: base class segmentation metric, new class segmentation metric, overall segmentation metric, harmonic mean of base class and new class segmentation metric, category-level intersection-union ratio, category-level precision, category-level recall, and overall accuracy.

[0023] This invention provides a semantic segmentation system for few-sample 3D indoor scene point clouds based on uncertainty guidance, comprising: a data preparation and preprocessing module for acquiring indoor 3D scene point cloud data and a small number of new class labeled samples and preprocessing them to obtain training data; a 3D visual language model inference module, connected to the data preparation and preprocessing module, for inputting the preprocessed training data into a pre-trained 3D visual language model to obtain initial pseudo-labels, constructing new class visual prototypes based on a small number of new class labeled samples, performing similarity matching between the initial pseudo-labels and the corresponding new class visual prototypes, removing pseudo-label points or pseudo-label regions with large differences, and obtaining preliminary pseudo-labels; an uncertainty pseudo-label filtering module, connected to the 3D visual language model inference module, for performing uncertainty point-level filtering on the preliminary pseudo-labels, setting pseudo-labels with high uncertainty points as ignore markers, and obtaining purified pseudo-labels; and a backbone network segmentation module, connected to the uncertainty pseudo-label... The module is connected to the label filtering module, which merges cleaned pseudo-labels with a small number of new class real labels to form the calibration supervision target for the current training batch. It inputs the 3D point cloud of the current training batch into the backbone network segmentation model, calculates the point-by-point segmentation loss based on the calibration supervision target, and performs few-shot adaptation training on the backbone network. The uncertainty-guided gradient modulation module, connected to the backbone network segmentation module, estimates round-level uncertainty based on the new class supervision signal of the current training batch during few-shot adaptation training. It modulates the vertical component of the current gradient based on the round-level uncertainty to obtain the adjusted gradient, which is then used as the basis for updating model parameters, and the optimizer performs the parameter update. The multi-registration set training and summary evaluation module, connected to the uncertainty-guided gradient modulation module, repeats the operations of the backbone network segmentation module and the uncertainty-guided gradient modulation module until all registration sets are trained, obtaining the semantic segmentation result.

[0024] Technical effect According to the uncertainty-guided few-shot 3D indoor scene point cloud semantic segmentation method and system of the present invention, the system performs normalization processing on the input indoor point cloud through a data preparation and preprocessing module, generates dense pseudo-labels and category-related responses through a 3D visual language model inference module, performs point-level purification of high-uncertainty pseudo-labels through an uncertainty pseudo-label filtering module, completes recoverable ignored points through an adaptive prototype completion module, performs point-by-point segmentation training through a backbone network segmentation module, adaptively adjusts gradient updates according to the reliability of the current batch through an uncertainty-guided gradient modulation module, improves the stability and reproducibility of few-shot experiments through a multi-registration set training and summary evaluation module, and records the training status and analysis results through a visualization and log output module.

[0025] Compared with the prior art, the present invention has at least the following beneficial effects.

[0026] First, this invention elevates the screening of pseudo-labels in 3D visual language models from region-level or block-level hard screening to point-level uncertainty screening. By using metrics such as information entropy, class interval, and maximum probability, low-reliability points in pseudo-labels can be identified more precisely, thereby improving the quality of supervisory signals entering the training process.

[0027] Second, this invention provides two filtering granularities: intra-class filtering and global filtering. Intra-class filtering can avoid the category-dominated threshold estimation with a large spatial area or a large number of points, so that each new class can obtain relatively balanced high-confidence pseudo-supervision; global filtering is suitable for scenarios with relatively uniform category distribution or those requiring uniform threshold control.

[0028] Third, the present invention introduces an adaptive prototype completion mechanism after pseudo-label filtering, which can reduce high uncertainty noise points while restoring reliable points that are highly consistent with the prototype of the new class, thus avoiding insufficient supervision signals for the new class due to over-filtering.

[0029] Fourth, this invention proposes an uncertainty-guided gradient modulation mechanism, which elevates uncertainty from the supervisory signal level to the gradient update level. This mechanism no longer merely changes sample weights or loss weights, but directly adjusts the update direction and intensity of model parameters, enabling more effective control of semantic drift in few-sample fitting.

[0030] Fifth, this invention preserves the optimized geometric information of the base class training phase through offline gradient basis construction, distinguishes the base class-related update direction from the new class adaptation update direction through online gradient orthogonal decomposition, and adaptively adjusts the step size of the new class adaptation direction according to the uncertainty of the current batch, thereby achieving a continuous balance between stability and plasticity.

[0031] Sixth, this invention is more adaptive than fixed gradient projection, fixed parameter freezing, or fixed regularization methods. When the current training batch is reliable, the system allows for more comprehensive learning of new classes; when the current training batch is unreliable, the system automatically shrinks unstable update directions to reduce the risk of catastrophic forgetting.

[0032] Seventh, this invention does not rely on additional large-scale base class sample replay, nor does it require storing all historical samples. It only needs to save the low-rank gradient basis constructed offline, thus having better engineering deployment efficiency.

[0033] Eighth, the two core mechanisms of this invention can be used in conjunction or deployed independently. The uncertainty pseudo-label filtering method can be used as a preprocessing plugin for any 3D visual language model pseudo-label generation system; the uncertainty guided gradient modulation method can be used as an optimizer proxy module in the training process of any neural network.

[0034] Ninth, this invention has strong generalization ability and is not limited to specific 3D visual language models, specific point cloud segmentation backbone networks, specific datasets, specific training frameworks or specific optimizers. It is applicable to various indoor 3D scene point cloud semantic segmentation tasks. Attached Figure Description

[0035] Figure 1 This is a flowchart of the uncertainty-guided few-sample 3D indoor scene point cloud semantic segmentation method in Embodiment 1 of the present invention; Figure 2 This is a detailed diagram illustrating the principle of the uncertainty pseudo-label filtering mechanism in Embodiment 1 of the present invention; Figure 3 This is a detailed schematic diagram illustrating the principle of the uncertainty-guided gradient modulation mechanism in Embodiment 1 of the present invention; Figure 4 This is a flowchart of the multi-registration set training and summary evaluation process in Embodiment 1 of the present invention; Figure 5 This is a block diagram of the uncertainty-guided few-sample 3D indoor scene point cloud semantic segmentation system in Embodiment 2 of the present invention; Figure 6 This is a schematic diagram illustrating the working process of the uncertainty-guided few-sample 3D indoor scene point cloud semantic segmentation system in Embodiment 2 of the present invention. Detailed Implementation

[0036] To make the technical solution, implementation process, and technical effects of the present invention clearer, the specific embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the following embodiments are only used to illustrate possible implementations of the present invention and are not intended to limit the scope of protection of the present invention.

[0037] Example 1 This embodiment provides a semantic segmentation method for point clouds in a few-sample 3D indoor scene based on uncertainty guidance.

[0038] Figure 1 This is a flowchart of a few-sample 3D indoor scene point cloud semantic segmentation method based on uncertainty guidance in an embodiment of the present invention.

[0039] like Figure 1 As shown, the uncertainty-guided few-sample 3D indoor scene point cloud semantic segmentation method in this embodiment includes the following steps: Step S1: Obtain indoor 3D scene point cloud data and a small number of new class labeled samples. Perform point cloud parsing, coordinate normalization, sampling, voxelization, feature processing, data augmentation, and training batch construction on the 3D point cloud data to obtain training data suitable for 3D point cloud semantic segmentation model processing.

[0040] In this embodiment, the indoor 3D scene point cloud data includes 3D coordinates, color, normal vector, reflection intensity, timestamp, and other additional features. A small number of newly labeled samples are used for new class adaptation training based on the existing segmentation capabilities of the base class model.

[0041] Step S2: Input the preprocessed training data into the pre-trained 3D visual language model to obtain point-by-point category-related responses, point-by-point category logical scores, point-by-point visual features, and initial pseudo-labels. Based on a small number of new class labeled samples and the point-by-point visual features or category-related responses output by the 3D visual language model, construct a new class visual prototype. Then, perform similarity matching between the initial pseudo-label points or pseudo-label regions generated by the 3D visual language model and the corresponding new class visual prototypes to initially remove pseudo-label points or pseudo-label regions that are significantly different from the new class visual prototypes, thus obtaining preliminary pseudo-labels.

[0042] In this embodiment, the 3D visual language model is not limited to a specific model name or specific network structure, but can be any model that can output point-level, region-level, or scene-level category-related responses based on 3D point cloud and category text information. Figure 2 This is a detailed diagram illustrating the principle of the uncertainty pseudo-label filtering mechanism in this embodiment of the invention.

[0043] Step S3, after obtaining the initial screening of pseudo-labels, further performs uncertain pseudo-label filtering, such as... Figure 2 As shown, the point-by-point category logical scores output by the 3D visual language model are normalized to obtain the point-by-point category probability distribution. Based on the point-by-point category probability distribution, the uncertainty of each point is calculated, and according to the preset filtering granularity and quantile threshold strategy, the initial screening pseudo-labels are filtered at the point level, and the pseudo-labels with high uncertainty are set as ignore markers to obtain purified pseudo-labels.

[0044] In this embodiment, point-by-point uncertainty can be expressed as information entropy, class interval, maximum probability, class response energy, prediction variance, consistency of multiple perturbations, model ensemble differences, or other indicators that can reflect the reliability of point-level prediction.

[0045] Filtering granularity includes intra-class filtering granularity and global filtering granularity. Intra-class filtering granularity refers to calculating the uncertainty distribution of pseudo-label points for each new category and calculating the quantile threshold for each category. Global filtering granularity refers to calculating the uncertainty distribution of pseudo-label points for all new categories as a whole and calculating a uniform quantile threshold.

[0046] Step S4: To avoid insufficient new class supervision signals due to over-filtering, the system performs adaptive prototype completion after uncertainty filtering. Specifically, for points marked as ignored, completion judgment is made based on the similarity between their point features and the new class visual prototype; when the maximum similarity between an ignored point and a new class visual prototype is higher than a preset completion threshold, the ignored point is completed with the corresponding new class label; when the maximum similarity is lower than or equal to the completion threshold, the point continues to be marked as ignored.

[0047] Step S5: Merge the cleaned pseudo-labels with a small number of new class real labels to form the calibration supervision target for the current training batch; Step S6: After obtaining the calibration supervision target, input the 3D point cloud of the current training batch into the backbone network segmentation model, output the point-by-point category prediction result, and calculate the point-by-point segmentation loss according to the calibration supervision target. Points marked as ignored are not included in the loss calculation. The backbone network segmentation model is not limited to a specific network structure and can employ point cloud multilayer perceptrons, point cloud convolutional networks, sparse convolutional networks, 3D attention networks, point cloud Transformers, sparse convolutional Transformers, graph neural networks, hybrid convolutional attention networks, or other neural network structures capable of performing point-by-point semantic segmentation of 3D point clouds.

[0048] Figure 3 This is a detailed diagram illustrating the principle of the uncertainty-guided gradient modulation mechanism in an embodiment of the present invention.

[0049] In step S7, before the few-shot adaptation training begins, the system also performs offline gradient basis construction in advance. For example... Figure 3 As shown in the offline stage, the parameter gradients corresponding to multiple training batches are collected on the base class data or base class validation data after the model is trained using the base class data. For the preset parameter set, the parameter gradients obtained by backpropagation of each batch are organized into gradient vectors according to the parameter tensor, and the gradient vectors of multiple batches are combined into a gradient data matrix. Orthogonalization or low-rank subspace estimation is performed on the gradient data matrix to obtain the parameter-level orthogonal gradient basis.

[0050] In this embodiment, the parametric orthogonal gradient basis is used to characterize the stable optimization direction formed during the training phase of the base class, and can also be understood as the gradient subspace of the base class task.

[0051] Gradient basis construction methods can employ QR decomposition, singular value decomposition, principal component analysis, Gram-Schmidt orthogonalization, eigenvalue decomposition, stochastic low-rank approximation or other orthogonalization and subspace estimation methods.

[0052] Step S8: In each training batch of few-shot adaptation training, the system estimates round-level uncertainty based on the new class of supervision signals for the current training batch. For example... Figure 3 As shown in the online phase, the backbone network is used to extract point-by-point features of the current training batch. For each new class in the current training batch, the average distance, variance, or dispersion between the feature points of that class and its class center is calculated. Then, the intra-class dispersion of multiple new classes is averaged to obtain the original round-level uncertainty. When the feature points of the new class are relatively compact within the same class, it indicates that the new class supervision signal of the current training batch has high consistency and strong reliability. When the feature points of the new class are relatively dispersed within the same class, it indicates that the current training batch may have occlusion, local fragments, incorrect pseudo-labels, or abnormal samples, corresponding to higher round-level uncertainty.

[0053] To make the uncertainty scales comparable across different training stages and registration sets, the system performs runtime statistical normalization on the original round-level uncertainty to obtain normalized round-level uncertainty. Runtime statistical normalization can employ moving averages, exponential moving averages, historical mean-variance statistics, historical quantile statistics, or other online statistical methods. Subsequently, gradient modulation coefficients are calculated based on the normalized round-level uncertainty. The gradient modulation coefficients are negatively correlated with round-level uncertainty; that is, the higher the uncertainty of the current training batch, the smaller the modulation coefficients; conversely, the lower the uncertainty of the current training batch, the larger the modulation coefficients. The modulation function can be a sigmoid function, an exponentially decaying function, a piecewise linear function, an inverse proportional function, a threshold function, a soft-gated function, or a learnable gated function.

[0054] After backpropagation of the segmentation loss, the system obtains the original gradient corresponding to the current training batch. Using the offline-constructed parametric orthogonal gradient basis, the original gradient is orthogonally decomposed to obtain parallel gradient components and vertical gradient components. The parallel gradient components are the projections of the original gradient onto the base class gradient subspace, representing update directions consistent with or similar to the stable optimization directions during the base class training phase. The vertical gradient components are the components of the original gradient located outside the base class gradient subspace, representing the new update directions introduced by the new class-labeled samples and pseudo-label supervision during the few-sample adaptation phase. The system scales the vertical gradient components according to the gradient modulation coefficient and synthesizes the modulated gradient with the parallel gradient components. Preferably, the parallel gradient components are kept intact, and the vertical gradient components are multiplied by the modulation coefficient and then added to the parallel gradient components to obtain the final modulated gradient.

[0055] Therefore, when the uncertainty of the current training batch is low, the modulation coefficient is large, and the vertical gradient component is more fully preserved, enabling the model to effectively learn new classes. When the uncertainty of the current training batch is high, the modulation coefficient is small, and the vertical gradient component is suppressed, making the model update closer to the stable direction of the base class, thereby reducing the risk of semantic drift and base class forgetting caused by erroneous supervision. Subsequently, the modulated gradient is used as the basis for updating model parameters, and the optimizer performs parameter updates. The processes of pseudo-label generation, uncertainty cleanup, calibration supervision target construction, segmentation loss calculation, round-level uncertainty estimation, gradient modulation, and parameter update are repeated until few-sample adaptation training on the current registration set is completed.

[0056] Figure 4 This is a flowchart of the multi-registration set training and summary evaluation process in an embodiment of the present invention.

[0057] Finally, as Figure 4 As shown, the above-mentioned few-shot adaptation training is performed on multiple independent few-shot registration sets to obtain multiple models or multiple evaluation results. Before training begins on each registration set, the optimizer state, learning rate scheduling state, running statistics state, and gradient modulation state are reset to ensure that the training states of different registration sets are independent of each other.

[0058] Step S9: Repeat steps S6 to S8 until all registration sets have been trained. Then, summarize the evaluation metrics of multiple registration sets to obtain the semantic segmentation results. The evaluation metrics include base class segmentation metrics, new class segmentation metrics, overall segmentation metrics, harmonic mean of base and new classes, class-level intersection-union ratio, class-level precision, class-level recall, and overall accuracy.

[0059] Through the above process, this embodiment forms a complete semantic segmentation method for few-shot 3D indoor scene point clouds, consisting of "pre-processing pseudo-label uncertainty purification + post-processing gradient uncertainty modulation". Specifically, the uncertainty pseudo-label filtering mechanism is used to improve the purity of the supervision signal entering the training loss, and the uncertainty-guided gradient modulation mechanism is used to control the model update direction and intensity based on the reliability of the training batch. Both mechanisms suppress noise propagation and base class forgetting during the few-shot adaptation process from the supervision input end and the optimization update end, respectively.

[0060] In addition, this embodiment also provides two methods that can be implemented independently, which can be used as a pre-cleaning step or a post-gradient modulation step in the above complete method flow, or can be embedded as independent modules in other point cloud semantic segmentation systems or neural network training tasks.

[0061] This embodiment also provides an independently implementable method for filtering uncertain pseudo-labels. This method can be used as a pre-processing pseudo-label cleanup step in the overall scheme described above, or it can be embedded as an independent module in any point cloud semantic segmentation system that generates pseudo-labels based on a 3D visual language model or a 3D basic model.

[0062] The uncertainty pseudo-label filtering method includes the following steps: obtaining the point-by-point category logical scores and initial pseudo-labels output by the 3D visual language model; normalizing the point-by-point category logical scores to obtain the point-by-point category probability distribution; calculating the point-by-point uncertainty based on the point-by-point category probability distribution; statistically analyzing the uncertainty distribution according to a preset filtering granularity; calculating the quantile threshold according to the retention ratio or filtering ratio; setting the pseudo-labels of points with uncertainty higher than the quantile threshold as ignore tags; retaining the pseudo-labels of points with uncertainty lower than or equal to the quantile threshold; and outputting the filtered pseudo-labels. Point-by-point uncertainty can be calculated using one or more of the following: information entropy, category interval, maximum probability, energy score, prediction variance, perturbation consistency, model ensemble difference, and category response stability. The filtering granularity can be category-level filtering, global-level filtering, scene-level filtering, region-level filtering, point cloud block-level filtering, or multi-level combined filtering. Preferably, category-level filtering is used when the distribution of new classes is uneven with few samples, to ensure that a certain number of high-confidence pseudo-label points are retained for each new class.

[0063] This independent method can transform traditional region-level or block-level hard screening of pseudo-labels into point-level uncertainty soft screening, thereby improving the purity of pseudo-label supervision signals in few-sample 3D point cloud segmentation.

[0064] This embodiment also provides an independently implementable uncertainty-guided gradient modulation method. This method can be used as a post-gradient modulation step in the overall scheme described above, or as an independent optimizer proxy mechanism applied to few-shot learning, continuous learning, incremental learning, long-tailed category learning, or other neural network training tasks that require balancing the preservation of old knowledge with the learning of new knowledge.

[0065] Uncertainty-guided gradient modulation methods include the following steps: After training on the original task or base class task, parameter gradients corresponding to multiple training batches are collected; the collected parameter gradients are organized into a gradient data matrix; orthogonalization or low-rank subspace estimation is performed on the gradient data matrix to obtain the parameter-level gradient basis; during training on a new task or with few samples, batch-level uncertainty is calculated based on the feature dispersion, supervision consistency, or prediction stability of the current training batch; the batch-level uncertainty is mapped to gradient modulation coefficients; the original gradients obtained by backpropagation of the current training batch are orthogonally decomposed to obtain parallel gradient components parallel to the gradient basis and vertical gradient components perpendicular to the gradient basis; the vertical gradient components are scaled using the gradient modulation coefficients; the parallel gradient components and the scaled vertical gradient components are synthesized into the modulated gradient; and the modulated gradient is used to update the parameters. Gradient basis construction methods include, but are not limited to, QR decomposition, singular value decomposition, principal component analysis, Gram-Schmidt orthogonalization, eigenvalue decomposition, or stochastic low-rank approximation. Modulation functions include, but are not limited to, the sigmoid function, exponential decay function, piecewise linear function, inverse proportional function, soft-gated function, or other monotonically non-increasing functions.

[0066] The essence of this method is to transform the reliability of the current training batch into a modulation factor in the gradient update geometry, so that the model maintains learning plasticity in reliable batches and enhances stability in unreliable batches.

[0067] Example 2 This embodiment provides a semantic segmentation system for point clouds of 3D indoor scenes based on uncertainty guidance.

[0068] Figure 5 This is a block diagram of a few-sample 3D indoor scene point cloud semantic segmentation system based on uncertainty guidance in an embodiment of the present invention.

[0069] like Figure 5 As shown, the uncertainty-guided few-sample 3D indoor scene point cloud semantic segmentation system in this embodiment includes a data preparation and preprocessing module 11, a 3D visual language model inference module 12, an uncertainty pseudo-label filtering module 13, an adaptive prototype completion module 14, a backbone network segmentation module 15, an uncertainty-guided gradient modulation module 16, a multi-registration set training and summary evaluation module 17, and a visualization and log output module 18.

[0070] The data preparation and preprocessing module 11 is used to perform the process in step S1 of embodiment 1, receive indoor 3D scene point cloud data and a small number of new class labeled samples, and construct training batches.

[0071] The 3D visual language model reasoning module 12 is used to execute the process in step S2 of embodiment 1, and outputs point-by-point category logical score, point-by-point visual feature, category-related response and initial pseudo label.

[0072] The uncertainty pseudo-label filtering module 13 is used to perform the process in step S3 of embodiment 1, calculate point-level uncertainty based on point-by-point category logical scores, and generate quantile thresholds according to the filtering granularity to filter high uncertainty pseudo-label points.

[0073] The adaptive prototype completion module 14 is used to perform the process in step S4 of embodiment 1, and to complete some ignored points based on the similarity between the ignored point features and the new visual prototype.

[0074] The backbone network segmentation module 15 is used to execute the process in steps S5 to S6 of Example 1, output point-by-point category prediction results and calculate segmentation loss according to the calibration supervision target.

[0075] The uncertainty-guided gradient modulation module 16 is used to execute the processes in steps S7 to S8 of Example 1, load the offline-constructed parameter-level orthogonal gradient basis, estimate the round-level uncertainty, calculate the modulation coefficients, and perform orthogonal decomposition and modulation on the original gradient.

[0076] The multi-registration set training and summary evaluation module 17 is used to perform the process in step S9 of Example 1, and to perform training and summarize evaluation metrics on multiple independent few-sample registration sets respectively.

[0077] The visualization and log output module 18 is used to record training status, pseudo-label filtering statistics, gradient modulation statistics, and evaluation results.

[0078] This embodiment also provides a specific application scenario for a few-sample 3D indoor scene point cloud semantic segmentation system based on uncertainty guidance, taking the task of 3D scene understanding of an indoor service robot in an office building environment as an example.

[0079] In this embodiment, the indoor service robot is deployed in indoor environments such as office buildings, laboratories, conference rooms, and corridors. It acquires indoor 3D point clouds using depth cameras, laser scanning equipment, or other 3D acquisition devices. The robot's existing model can already recognize common base categories such as walls, floors, doors, windows, tables, and chairs. During subsequent deployment, the user expects the robot to further recognize several new categories, such as charging piles, fire extinguishers, trash cans, mobile shelves, or other long-tailed indoor objects. Due to the high cost of labeling new categories, only a small number of labeled samples are provided for each new category.

[0080] Figure 6 This is a schematic diagram illustrating the working process of the uncertainty-guided few-sample 3D indoor scene point cloud semantic segmentation system in an embodiment of the present invention.

[0081] In this application scenario, such as Figure 6 As shown, the system first prepares and preprocesses the indoor 3D point cloud input data collected by the robot using module 11. This module performs parsing, coordinate normalization, sampling, voxelization, feature processing, and training batch construction on the point cloud to obtain point cloud batches that can be used for few-sample adaptation training. Point cloud batches can contain 3D coordinates, color, normal vectors, or other point attributes.

[0082] Subsequently, the system inputs the preprocessed point cloud into the 3D visual language model inference module 12. This module outputs point-by-point category logical scores, point-by-point visual features, category-related responses, and initial pseudo-labels based on the text descriptions of the newly added categories and the visual features of the point cloud. Due to occlusion, scan gaps, object stacking, and similar category appearances in the office building environment, the initial pseudo-labels may contain false positives and category confusion points. Therefore, the system constructs a visual prototype of the new category based on a small number of newly labeled samples and first screens the initial pseudo-labels using prototype similarity, removing pseudo-label points or pseudo-label regions that differ significantly from the corresponding new category prototypes.

[0083] After initial screening, the system further calls the uncertainty pseudo-label filtering module 13. This module calculates the point-by-point category probability distribution based on the point-by-point category logical scores output by the 3D visual language model, and obtains the point-level uncertainty based on information entropy, category interval, maximum probability, or other metrics. Figure 2 As shown, the system can employ intra-class filtering to calculate the uncertainty quantile threshold for each new category; or it can use global filtering to calculate the threshold uniformly for all pseudo-label points of new categories. For points with uncertainty higher than the threshold, the system sets their pseudo-labels to ignore them, preventing them from participating in subsequent loss calculations; for points with low uncertainty, the system retains their pseudo-labels as training supervision. Thus, the system reduces the interference of unreliable pseudo-labels on the learning of new categories at the training front end.

[0084] To avoid excessive filtering leading to insufficient supervision signals for the new categories, the system further processes ignored points through the adaptive prototype completion module 14. For some points marked as ignored but whose visual features are highly similar to the prototypes of the new categories, the system completes them with the corresponding new category labels; for points with insufficient similarity, the system continues to keep them as ignored. Subsequently, the system merges the completed pseudo-labels with a small number of real new category labels to form the calibration supervision targets for the current training batch.

[0085] During few-shot adaptation training, the backbone network segmentation module 15 outputs point-by-point semantic prediction results based on the current point cloud batch and calculates the segmentation loss based on the calibration supervision target. For points marked as ignored, the system does not calculate their loss to avoid the influence of highly uncertain supervision signals on model training. After backpropagation of the loss, the system obtains the original gradient corresponding to the current training batch.

[0086] like Figure 3 As shown, before the few-shot adaptation training begins, the system has already used the model trained on the base class to collect parameter gradients from multiple batches on the base class data, and constructed a parameter-level orthogonal gradient basis through orthogonalization or low-rank subspace estimation methods. This gradient basis is used to represent the stable optimization direction corresponding to the robot's original base class segmentation capability.

[0087] In each adaptation training batch, the uncertainty-guided gradient modulation module 16 first extracts features of newly added category points using the backbone network, and calculates round-level uncertainty based on the dispersion of features within the same newly added category. When the features of newly added category points are densely distributed, it indicates that the batch of samples is relatively complete and the pseudo-labels are relatively reliable; when the features of newly added category points are scattered, it indicates that the batch may have local occlusion, scan gaps, mislabeling, or pseudo-label noise. The system performs runtime statistical normalization on the round-level uncertainty and maps it to gradient modulation coefficients.

[0088] Subsequently, the system decomposes the original gradient into parallel gradient components and vertical gradient components using a parametric orthogonal gradient basis. The parallel gradient components represent update directions that are consistent with or similar to the original base class training directions, while the vertical gradient components represent update directions introduced by the adaptation of the new class. The system retains the parallel gradient components, scales the vertical gradient components according to the gradient modulation coefficients, and then combines the two into a modulated gradient.

[0089] When the newly added category samples collected by the robot in the current batch are relatively clear and the supervision signal is relatively reliable, the modulation coefficient is larger, allowing the system to fully learn the new category. When there is occlusion, local scanning, or false label noise in the current batch, the modulation coefficient is smaller, and the system suppresses the vertical gradient component, making the model update closer to the stable direction of the original base class, thereby reducing the damage to the segmentation ability of the few-sample adaptation for the original base class such as walls, floors, tables, and chairs. The optimizer updates the model parameters based on the modulated gradient.

[0090] like Figure 4As shown, to reduce the randomness caused by the selection of few samples, this embodiment can construct multiple independent registration sets and perform the above-mentioned adaptation training on each set. Before the training of each registration set begins, the system resets the optimizer state, learning rate scheduling state, running statistics state, and gradient modulation state. After all registration sets have been trained, the system summarizes the base class segmentation index, new class segmentation index, overall segmentation index, harmonic mean index, and category-level evaluation results to obtain the final semantic segmentation performance.

[0091] Through the above implementation methods, the indoor service robot can expand the supervision signal with pseudo-labels generated by the 3D visual language model when only a small number of newly labeled samples are provided. It can also reduce pseudo-label noise through uncertainty pseudo-label filtering, alleviate insufficient supervision through adaptive prototype completion, and suppress the damage of unreliable batches to model parameters through uncertainty-guided gradient modulation. Thus, it can maintain the segmentation ability of the original base class while learning new categories.

[0092] The role and effect of the embodiments According to the uncertainty-guided few-shot 3D indoor scene point cloud semantic segmentation method and system involved in this embodiment, in the indoor service robot navigation and scene understanding task, the system can quickly adapt to the new object categories in the robot's operating environment under the condition of providing only a small number of newly added category labeled samples. Compared with the method of directly using pseudo-labels of 3D visual language models for few-shot training, this embodiment removes high uncertainty pseudo-label points at the training front end through an uncertainty pseudo-label filtering mechanism, reducing the probability of erroneous pseudo-supervision entering the loss function; restores some reliable points that are highly consistent with the prototype of the new class through an adaptive prototype completion mechanism, alleviating the lack of supervision of the new category caused by over-filtering; and dynamically adjusts the gradient update direction at the training back end according to the reliability of the current batch through an uncertainty-guided gradient modulation mechanism, so that the model can fully learn the new category in reliable batches and reduce the destruction of the original base class knowledge in unreliable batches.

[0093] Therefore, the above embodiments can improve the robot's ability to segment new indoor object categories under the condition of limited indoor point cloud annotation costs, and reduce the impact of few-sample adaptation training on the segmentation performance of original categories such as walls, floors, doors, windows, tables and chairs. They are applicable to application scenarios such as indoor service robot navigation, intelligent building digital twins, augmented reality and 3D scene understanding.

[0094] Those skilled in the art should understand that this invention is not limited to the contents described in the above embodiments and specification. Any equivalent substitutions, modifications, or improvements made to the 3D visual language model, backbone network, gradient basis construction method, uncertainty measurement method, modulation function, number of registration sets, or application scenario without departing from the technical concept of this invention should fall within the protection scope of this invention.

Claims

1. A semantic segmentation method for point clouds in a few-sample 3D indoor scene based on uncertainty guidance, characterized in that, Includes the following steps: Step 1: Acquire indoor 3D scene point cloud data and a small number of new class labeled samples and preprocess them to obtain training data; Step 2: Input the preprocessed training data into the pre-trained 3D visual language model to obtain initial pseudo-labels. Construct a new class visual prototype based on the small number of new class labeled samples. Perform similarity matching between the initial pseudo-labels and the corresponding new class visual prototypes. Remove pseudo-label points or pseudo-label regions with large differences to obtain preliminary pseudo-labels. Step 3: Perform uncertainty point-level filtering on the initial screening pseudo-labels, and set pseudo-labels with high uncertainty points as ignore markers to obtain purified pseudo-labels; Step 4: Merge the purified pseudo-labels with a small number of new class real labels to form the calibration supervision target for the current training batch; Step 5: Input the 3D point cloud of the current training batch into the backbone network segmentation model, calculate the point-by-point segmentation loss according to the calibration supervision target, and perform few-sample adaptation training on the backbone network. Step 6: During the few-sample adaptation training process, the round-level uncertainty is estimated based on the new class supervision signal of the current training batch. The vertical component of the current gradient is modulated according to the round-level uncertainty to obtain the adjusted gradient. The modulated gradient is used as the basis for updating the model parameters, and the optimizer performs the parameter update. Step 7: Repeat steps 5 to 6 until all registration sets have been trained. Then, summarize the evaluation metrics of multiple registration sets to obtain the semantic segmentation results.

2. The semantic segmentation method for point clouds of few-shot 3D indoor scenes based on uncertainty guidance according to claim 1, characterized in that: In step 1, the indoor 3D scene point cloud data includes one or more of the following: 3D coordinates, color, normal vector, reflection intensity, and timestamp; the small number of newly labeled samples are used for new class adaptation training based on the existing segmentation capabilities of the base class model. The preprocessing includes point cloud parsing, coordinate normalization, sampling, voxelization, feature processing, data augmentation, and training batch construction of the indoor 3D scene point cloud data.

3. The semantic segmentation method for point clouds of 3D indoor scenes based on uncertainty guidance according to claim 1, characterized in that: In step 2, the preprocessed training data is input into the pre-trained 3D visual language model, and point-by-point category-related responses, point-by-point category logical scores, and point-by-point visual features are also obtained. A 3D visual language model is any model that can output point-level, region-level, or scene-level category-related responses based on 3D point clouds and category text information.

4. The semantic segmentation method for point clouds of few-shot 3D indoor scenes based on uncertainty guidance according to claim 3, characterized in that: Step 2 further includes: constructing a new class visual prototype based on a small number of new class labeled samples, the point-by-point visual features output by the 3D visual language model, or the point-by-point category-related response.

5. The semantic segmentation method for point clouds of few-shot 3D indoor scenes based on uncertainty guidance according to claim 1, characterized in that: In step 3, the uncertainty-level filtering includes: normalizing the point-by-point category logical scores output by the 3D visual language model to obtain a point-by-point category probability distribution; calculating the uncertainty of each point based on the point-by-point category probability distribution; and performing point-level filtering on the initially screened pseudo-labels according to a preset filtering granularity and quantile threshold strategy, setting pseudo-labels with high uncertainty points as ignore markers to obtain purified pseudo-labels. The uncertainty is one or more of the following: information entropy, class margin, maximum probability, class response energy, prediction variance, consistency of multiple perturbations, or model ensemble discrepancy. The filtering granularity includes intra-class filtering granularity and global filtering granularity.

6. The semantic segmentation method for point clouds of 3D indoor scenes based on uncertainty guidance according to claim 1, characterized in that: It also includes an adaptive prototype completion step: for points that are set as ignored, a completion judgment is made based on the similarity between their point features and the visual prototype of the new class. When the similarity is higher than the preset completion threshold, the point is completed as the corresponding new class label; otherwise, the ignored mark is maintained.

7. The semantic segmentation method for point clouds of 3D indoor scenes based on uncertainty guidance according to claim 1, characterized in that: It also includes an offline gradient basis construction step: using the model after the base class training is completed, parameter gradients corresponding to multiple training batches are collected on the base class data or base class validation data. For a preset parameter set, the parameter gradients obtained by backpropagation of each batch are organized into gradient vectors according to the parameter tensor. The gradient vectors of multiple batches are combined into a gradient data matrix. Orthogonalization or low-rank subspace estimation is performed on the gradient data matrix to obtain a parameter-level orthogonal gradient basis. The parametric-level orthogonal gradient basis is used to characterize the stable optimization directions formed during the base class training phase, which is the gradient subspace of the base class task. The gradient basis construction method is any one or more of QR decomposition, singular value decomposition, principal component analysis, Gram-Schmidt orthogonalization, eigenvalue decomposition, and stochastic low-rank approximation.

8. The semantic segmentation method for point clouds of 3D indoor scenes based on uncertainty guidance according to claim 1, characterized in that: In step 6, the round-level uncertainty is estimated as follows: Point-by-point features of the current training batch are extracted using the backbone network; the average distance, variance, or dispersion between the feature points of each new class in the current training batch and its class center is calculated; then, the intra-class dispersion of multiple new classes is averaged to obtain the original round-level uncertainty; and finally, the original round-level uncertainty is subjected to runtime statistical normalization. The gradient modulation coefficient is calculated based on the normalized round-level uncertainty, and the gradient modulation coefficient is negatively correlated with the round-level uncertainty. The gradient modulation method is as follows: using the parameter-level orthogonal gradient basis constructed offline, the original gradient of the current gradient is orthogonally decomposed to obtain parallel gradient components and vertical gradient components. The vertical gradient component is multiplied by the gradient modulation coefficient and then synthesized with the parallel gradient component to form the modulated gradient.

9. The semantic segmentation method for point clouds of few-shot 3D indoor scenes based on uncertainty guidance according to claim 1, characterized in that: In step 7, the evaluation metrics include one or more of the following: base class segmentation metric, new class segmentation metric, overall segmentation metric, harmonic mean of base class and new class segmentation metric, category-level intersection-union ratio, category-level precision, category-level recall, and overall accuracy.

10. A semantic segmentation system for point clouds of a few-sample 3D indoor scene based on uncertainty guidance, characterized in that, include: The data preparation and preprocessing module is used to acquire indoor 3D scene point cloud data and a small number of new class labeled samples and preprocess them to obtain training data. The three-dimensional visual language model inference module is connected to the data preparation and preprocessing module. It is used to input the preprocessed training data into the pre-trained three-dimensional visual language model to obtain initial pseudo-labels, construct new class visual prototypes based on the small number of new class labeled samples, perform similarity matching between the initial pseudo-labels and the corresponding new class visual prototypes, remove pseudo-label points or pseudo-label regions with large differences, and obtain preliminary pseudo-labels. An uncertainty pseudo-label filtering module is connected to the three-dimensional visual language model inference module. It is used to perform uncertainty point-level filtering on the initial screening pseudo-labels, setting pseudo-labels with high uncertainty points as ignore markers to obtain purified pseudo-labels. The backbone network segmentation module is connected to the uncertainty pseudo-label filtering module. It is used to merge the purified pseudo-labels with a small number of new class real labels to form the calibration supervision target of the current training batch. The 3D point cloud of the current training batch is input into the backbone network segmentation model. The point-by-point segmentation loss is calculated according to the calibration supervision target, and the backbone network is trained with few samples. An uncertainty-guided gradient modulation module, connected to the backbone network segmentation module, is used to estimate round-level uncertainty based on the new class supervision signal of the current training batch during the few-shot adaptation training process, modulate the vertical component of the current gradient according to the round-level uncertainty to obtain the adjusted gradient, and use the modulated gradient as the basis for updating the model parameters, so that the optimizer can perform parameter updates. The multi-registration set training and summary evaluation module is connected to the uncertainty-guided gradient modulation module and is used to repeat the operations of the backbone network segmentation module and the uncertainty-guided gradient modulation module until all registration sets are trained and semantic segmentation results are obtained.

Images