A Meta-Learning-Based Approach to Optimize Semantic Segmentation of Few-Shot Building Point Clouds Based on General Background Knowledge

Abstract

This invention discloses a method for semantic segmentation of few-shot building point clouds based on meta-learning and general background knowledge optimization. The method includes: generating foreground prototypes for each category in the support set using an adaptive prototype generation algorithm; learning background knowledge from all samples in the support set and all samples in the query set using a modeling algorithm to obtain a general background prototype, and optimizing the general background prototype using an adaptive background loss algorithm; filtering background point clouds in the task samples of the query set using a background filtering algorithm; calculating the similarity between the foreground point cloud of the task sample and the average foreground prototype and general background prototype of all categories in the support set using a prototype contrastive learning algorithm, and extracting the foreground point clouds of task samples with a similarity higher than a preset threshold to the average foreground prototype; and performing feature matching and category prediction on the foreground point clouds of the task samples using a non-metric learning algorithm. This invention can effectively reduce false positive predictions and eliminate interference information, thereby optimizing the target segmentation effect.

Description

Technical Field

[0001] This invention relates to the field of point cloud data processing technology. More specifically, this invention relates to a general background knowledge-based method for optimizing semantic segmentation of few-shot building point clouds using meta-learning. Background Technology

[0002] Point cloud semantic segmentation plays a crucial role in the digital preservation of architectural heritage. Few-sample point cloud semantic segmentation can generate point-by-point masks for unseen categories using only a minimal number of labeled point clouds. Although some progress has been made in this field, it has mainly focused on mining target object information, while background knowledge has not been fully utilized. This makes it difficult to distinguish non-target regions in segmentation tasks, interfering with target object recognition. Summary of the Invention

[0003] One object of the present invention is to solve at least the above-mentioned problems and to provide at least the advantages that will be described later.

[0004] Another objective of this invention is to provide a general background knowledge-based method for optimizing semantic segmentation of few-sample building point clouds, which can effectively reduce false positive predictions, eliminate interference information, and thus optimize the target segmentation effect.

[0005] To achieve these objectives and other advantages according to the present invention, a general background knowledge-based method for optimizing semantic segmentation of few-shot building point clouds based on meta-learning is provided, comprising:

[0006] An adaptive prototype generation algorithm is used to generate foreground and background prototypes for each category in the support set;

[0007] A modeling algorithm is used to learn background knowledge from all samples in the support set and all samples in the query set to obtain a general background prototype, and an adaptive background loss algorithm is used to optimize the general background prototype.

[0008] Based on a general background prototype, a background filtering algorithm is used to filter the background point cloud in the task samples of the query set and retain the foreground point cloud.

[0009] The prototype contrast learning algorithm is used to calculate the similarity between the foreground cloud of the task sample and the average foreground prototype and the general background prototype of all categories in the support set, respectively. The foreground cloud of the task sample with the similarity to the average foreground prototype of all categories in the support set is extracted if it is higher than a preset threshold.

[0010] A non-metric learning algorithm is used to perform feature matching and category prediction on the foreground point cloud of the task sample;

[0011] The support set contains labeled building point cloud samples, and the query set contains unlabeled building point cloud samples.

[0012] Preferably, the process of generating foreground and background prototypes for a certain category in the support set using an adaptive prototype generation algorithm specifically includes:

[0013] The support features and their binary masks of the support set samples are extracted using the backbone network extractor.

[0014] Based on the supporting features and their corresponding binary masks, the binary masks of the supporting features are extended to the same dimension as the supporting features, and then mask feature extraction is performed to obtain the foreground and background features of the support set samples.

[0015] Traverse other samples of the same category in the support set to obtain the foreground and background features of other samples of the same category in the support set;

[0016] For all samples of the same category in the support set, a weighted average pooling algorithm is used to calculate the foreground features and background features respectively, so as to obtain the foreground prototype and background prototype of that category.

[0017] Preferably, the process of learning background knowledge from all samples in the support set and all samples in the query set using a modeling algorithm to obtain a general background prototype, and then optimizing the general background prototype using an adaptive background loss algorithm, specifically includes:

[0018] The backbone network extractor extracts the support features of a sample in the support set and / or the query features of a sample in the query set. The support features and query features have the same dimension.

[0019] Build and initialize the generic background prototype to obtain the original generic background prototype;

[0020] The original general background prototype is expanded to the same dimension as the supporting features and / or query features, and then the original general background prototype is concatenated with the supporting features and / or query features.

[0021] A background prediction convolutional network is used to predict the background using support features concatenated with the original general background prototype and / or query features concatenated with the original general background prototype.

[0022] By traversing all samples in the support set and all samples in the query set, an adaptive background loss function is used to optimize the background prediction process in the background prediction convolutional network, thereby obtaining a general background prototype.

[0023] Preferably, the adaptive background loss function is the sum of the adaptive binary cross-entropy loss function and the cross-entropy loss function, wherein the adaptive binary cross-entropy loss function incorporates an adaptive weight factor, and the cross-entropy loss function incorporates a mask distribution dynamic weight factor.

[0024] Preferably, based on a general background prototype, the specific process of using a background filtering algorithm to filter the background point clouds in the task samples of the query set and retain the foreground point clouds includes:

[0025] The general background prototype is concatenated with the query features of the task samples in the query set, and then a convolutional unit is used to filter out the background point cloud in the query features, thereby preserving the foreground point cloud in the task samples.

[0026] Preferably, the specific process of calculating the similarity between the foreground cloud of the task sample and the average foreground prototype and the general background prototype of all categories in the support set using the prototype contrastive learning algorithm includes:

[0027] Calculate the average foreground prototype for all categories in the support set to obtain the average foreground prototype;

[0028] Expand the dimensions of the average foreground prototype and the general background prototype to match the dimensions of the query features of the task samples.

[0029] The cosine similarity between the foreground cloud in the task sample and the average foreground prototype and general background prototype of all categories in the support set is calculated separately. The cosine similarity calculation is scaled by introducing a temperature parameter τ to adjust the similarity calculation.

[0030] Preferably, the prototype contrastive learning algorithm is used to calculate the similarity between the foreground cloud of the task sample and the average foreground prototype and the general background prototype of all categories in the support set. In the process of extracting the foreground cloud of the task sample whose similarity with the average foreground prototype of all categories in the support set is higher than a preset threshold, the prototype contrastive loss function is used to optimize the learning process. The prototype contrastive loss function introduces a marginal threshold M to control the difference in similarity between positive and negative samples. The positive sample is the average foreground prototype and the negative sample is the general background prototype.

[0031] Preferably, the specific process of using a non-metric learning algorithm to perform feature matching and category prediction on the foreground point cloud of the task sample includes:

[0032] Calculate the cosine similarity between the feature vector of each query point in the foreground cloud of the task sample and the foreground prototype of each category in the support set, convert it to a negative value and amplify it using an exponential function to obtain the mask logits;

[0033] The mask logits of each query point in the foreground cloud of the task sample is transformed into a probability distribution using a normalized exponential function, and the prototype category with the highest probability value is selected as the predicted label for that query point.

[0034] Preferably, the backbone network extractor is a dynamic graph convolutional network based on a self-attention mechanism.

[0035] Preferably, the total loss function includes a main loss function, the adaptive background loss function, and the prototype comparison loss function, wherein both the adaptive background loss function and the prototype comparison loss function are set with adjustable weight factors.

[0036] This invention offers at least the following advantages: it is the first attempt to model and filter background regions in a few-shot point cloud semantic segmentation task, and also the first to apply it to a downstream scene task of architectural heritage point cloud semantic segmentation, effectively reducing false positive predictions; it proposes three algorithms / modules—Browser Modeling (BMM), Background Filtering (BFM), and Prototype Contrastive Learning (PCL)—to learn general background knowledge and eliminate interference information, thereby optimizing target segmentation performance; extensive experiments on the self-built ancient building dataset, the ArCH architectural heritage dataset, demonstrate that the proposed method achieves state-of-the-art performance, providing a new perspective for few-shot point cloud semantic segmentation and the digital preservation of architectural heritage.

[0037] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the method for optimizing the semantic segmentation of building point clouds based on meta-learning and general background knowledge, as described in an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram of the background prediction convolutional network described in an embodiment of the present invention;

[0040] Figure 3 This is a visualization diagram of each category in the self-built ancient building dataset described in this embodiment of the invention;

[0041] Figure 4 This is a visualization diagram of each category in the ArCH dataset described in this embodiment of the invention;

[0042] Figure 5 Comparison of qualitative results of semantic segmentation of point clouds using different methods on a self-built ancient architecture dataset with few samples;

[0043] Figure 6 A comparison of qualitative results of different methods for semantic segmentation of point clouds in the ArCH dataset using few samples;

[0044] Figure 7a A visual illustration of the adaptation process between task samples and different prototypes using t-SNE;

[0045] Figure 7bThe impact of different hyperparameters τ (temperature parameter) on the qualitative results of semantic segmentation of point clouds with few samples;

[0046] Figure 7c The impact of different hyperparameters M (marginal threshold) on the qualitative results of semantic segmentation of point clouds with few samples;

[0047] Figure 7d The impact of different hyperparameters γ (prototype contrast loss weights) on the qualitative results of semantic segmentation of few-sample point clouds;

[0048] Figure 7e The impact of different hyperparameters β (adaptive background loss weights) on the qualitative results of semantic segmentation of point clouds with few samples. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description.

[0050] It should be noted that, unless otherwise specified, the experimental methods described in the following embodiments are all conventional methods, and the reagents and materials described are all commercially available unless otherwise specified. In the description of this invention, the terms "lateral", "longitudinal", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0051] This few-shot 3D point cloud semantic segmentation applies general 3D point cloud semantic segmentation to few-shot scenarios. The model can learn the similarity and feature sharing between categories with only a few labels, improving its adaptability to new categories. It is suitable for architectural heritage scenarios with diverse styles and complex structures and details. Unlike most studies on few-shot 3D point cloud semantic segmentation that focus on directly segmenting target objects, this application is the first to propose a few-shot point cloud semantic segmentation method that utilizes complementary background knowledge to eliminate interference regions and optimize target recognition, and it is the first to be applied to the field of architectural heritage.

[0052] Most previous research has focused on designing human-preferred network architectures to solve various few-shot learning tasks. The emergence of contrastive learning leads this application to focus on mining better deep feature representations by comparing positive and negative samples. It aims to learn effective representations of data by learning the similarities and differences between samples.

[0053] In traditional machine learning, models typically require a large amount of training data and computational resources to achieve good performance on a specific task. The goal of meta-learning is to enable models to "learn how to learn." That is, to improve learning efficiency on new tasks by learning from the experience of different tasks. Its core ideas are: 1. Cross-task learning: extracting general knowledge from multiple tasks; 2. Rapid adaptation: achieving rapid learning on new tasks; 3. Few-shot learning: completing learning with fewer samples.

[0054] One solution to the problem of few-shot 3D point cloud semantic segmentation is meta-learning. Below, we first introduce the definition of using meta-learning for few-shot point cloud semantic segmentation. Specifically, all categories in the dataset can be divided into a training category set C. train (Also called the visible class collection) and the test class collection Each few-shot task, or scenario, E = {S, Q}, where S and Q are the support set and query set, is instantiated as an N-way K-shot point cloud segmentation task. The support set is represented as... K groups (from 1, 2, ..., K) of each of N categories support point clouds and the corresponding mask The query set is represented as Includes N sets of query point clouds and their corresponding ground truth masks. Typically, lookup mask It can only be used during the few-sample training phase, while only the support set mask is used during the testing phase. Generally, each point cloud is given Includes T sets of information with coordinates and additional features For example, color information.

[0055] In summary, the training of the few-shot point cloud semantic segmentation model (FS-PCS) consists of two phases: a pre-training phase and a meta-learning phase. The former mainly trains the backbone network, learning meaningful semantic features in a fully supervised manner. The latter focuses on training the model's ability to transfer learned knowledge from the support set S to the query set Q. Given an N-way K-shot task: during the training phase, the model uses the provided support set S and query set Q to train training categories C with semantic masks. train ( as well as Learning feature representations; during the testing phase, when faced with unseen categories, using a support set S containing a small number of labels, the model needs to identify new categories C in a query set Q that has no labels. test Generate point-by-point predictions (N+l (number 1) represents the background category).

[0056] As mentioned earlier, few-shot point cloud semantic segmentation models often struggle to distinguish background regions that can cause interference, posing a significant challenge to the accurate identification of architectural heritage components. Inspired by this observation, this application focuses on developing a model / method that can better learn background knowledge and eliminate background interference to optimize target recognition capabilities, rather than simply designing a model / method that directly segments the target. The "architectural heritage" referred to in the following meta-learning-based general background knowledge-optimized few-shot architectural point cloud semantic segmentation method refers to architectural cultural heritage.

[0057] like Figure 1 As shown, this invention provides a general background knowledge-based method for optimizing semantic segmentation of few-shot building point clouds using meta-learning, comprising:

[0058] S1. An adaptive prototype generation algorithm is used to generate foreground and background prototypes for each category in the support set.

[0059] The process of generating foreground and background prototypes for a specific category using an adaptive prototype generation algorithm specifically includes:

[0060] S11. The support features and their binary masks of the support set samples are extracted by the backbone network extractor.

[0061] The backbone network extractor uses DGCNN, a dynamic graph convolutional network, as the backbone feature extractor. Furthermore, this application introduces a self-attention network to capture the semantic correlation between building point clouds in the global context.

[0062] The support set samples are labeled building point cloud samples, and the query set samples are unlabeled building point cloud samples. For the input building point cloud samples, the support set samples are support point clouds. (where n∈{1,...,N}, k∈{1,...,K}), the sample in the query set is the query point cloud P. q Extract supporting features f s and query feature f q The calculation formula is as follows:

[0063]

[0064] Where φ represents the backbone network feature extractor, and N and d represent the number of point clouds and the number of channels for point cloud features, respectively.

[0065] S12. Based on the supporting features and their corresponding binary masks, the binary masks of the supporting features are extended to the same dimension as the supporting features, and then mask feature extraction is performed to obtain the foreground and background features of the support set samples.

[0066] Given supporting features f s With the corresponding binary mask and The mask is expanded to the same dimension as the supporting features, and then the mask features are extracted to extract the foreground features of the support set. and background features The calculation formula is as follows:

[0067] f bg =f s ☉Y s bg

[0068] f fg =f s ⊙Y s fg

[0069] Where ⊙ represents the Hardma factor.

[0070] S13. Traverse other samples of the same category in the support set to obtain the foreground and background features of other samples of the same category in the support set.

[0071] S14. For all samples of the same category in the support set, use a weighted average pooling algorithm to calculate the foreground features and background features respectively, and obtain the foreground prototype and background prototype of that category.

[0072] Based on foreground and background features, we generate a foreground prototype for the nth category using an adaptive weighted average pooling operation. and background prototype The calculation formula is as follows:

[0073]

[0074] Where x∈{1,...,N} represents the coordinate position information, w represents the adaptive weighting factor, and "|Δ|" represents the reduction factor that outputs 1 when Δ is true.

[0075] By using steps S11 to S14 above, traversing each category in the support set, a foreground prototype and a background prototype can be generated for each category.

[0076] The above describes the prototype generation representation in the N-way K-shot architectural heritage point cloud semantic segmentation task. Where n∈{1,...,N}. Through the adaptive prototype generation step, the model can initially understand the semantic information of the foreground and background in the support set from a small number of support samples, improving the model's discriminative knowledge and its ability to identify targets.

[0077] S2. A modeling algorithm is used to learn background knowledge from all samples in the support set and all samples in the query set to obtain a general background prototype, and an adaptive background loss algorithm is used to optimize the general background prototype.

[0078] The specific steps involved in this process are as follows:

[0079] S21. The support features of a certain sample in the support set and / or the query features of a certain sample in the query set are extracted by the backbone network extractor. The support features and query features have the same dimension.

[0080] The methods for extracting supporting features and query features have been explained in step S11 above, so they will not be repeated here.

[0081] S22. Construct and initialize the generic background prototype to obtain the original generic background prototype;

[0082] Background regions are typically present in most architectural point cloud scenes. Based on this commonality, it is recommended to use a generic background prototype to model background knowledge. It can be represented as... Here, the number 1 represents the dimension, and d is the channel dimension. First, the background prototype P needs to be randomly initialized. BG Then, during the few-shot semantic segmentation training phase, the general representation of the target category is learned from the support point cloud and the query point cloud, respectively.

[0083] S23. Extend the original general background prototype to the same dimension as the supporting features and / or query features, and then concatenate the original general background prototype with the supporting features and / or query features.

[0084] Specifically, given the supporting features f s and query feature f q , extended initialization of P BG to the same dimensional size as them Then, the supporting features and query features are concatenated with it respectively.

[0085] S24. Use a background prediction convolutional network to predict the background of support features that are concatenated with the original general background prototype and / or query features that are concatenated with the original general background prototype.

[0086] The structure of the background prediction convolutional network is as follows: Figure 2 As shown, the calculation formula is as follows:

[0087]

[0088] Where M(*) represents the background prediction convolutional network, This represents a splicing operation. It supports or queries probability results.

[0089] S25. Traverse each sample in the support set and each sample in the query set. During the background prediction process of the background prediction convolutional network, an adaptive background loss function is used for optimization to obtain a general background prototype.

[0090] In semantic segmentation tasks involving few-sample point clouds of architectural heritage, the data typically contains only the true label Y of the architectural category (target object). s and Y q This does not include the true labels of the background region. Based on this, in order to enable the learned general background prototype to more effectively predict the background region as we expect, we designed an adaptive background loss to optimize this learning metric during training. We first define an adaptive weight factor w, the specific calculation formula of which is shown below:

[0091]

[0092] Where ε is a small constant (1e) -6 This is used to avoid division by zero. This represents the proportion of foreground points within the target point. Using the above formula, the model can dynamically adjust the weights between the background region and the foreground target in the loss function, and effectively handle the difference in the proportion of background and foreground during training. This facilitates the model's continuous updating and learning of background knowledge during training.

[0093] Next, we defined an adaptive binary cross-entropy loss function. By adding this constraint, we can predict background noise. The specific calculation formula is shown below:

[0094]

[0095] Where N represents the total number of target points, and i represents the spatial location index.

[0096] The above formula aims to predict the reverse region belonging to the target region, i.e. The zero in the value, however, using only this constraint may lead to predictions. All values ​​are zero. Therefore, we add an auxiliary regularization to mitigate the above problem. Specifically, we first define a dynamic weight η based on the mask distribution of the support set and the query set. s / q The formula is as follows:

[0097]

[0098] Where N s / q These represent the number of support and query points, respectively. Through this formula, the model can dynamically adjust the ratio of foreground to background in the support and query areas, enhancing the model's adaptability to the background and preventing the model from over-focusing on the target area while ignoring the background area.

[0099] The cross-entropy loss function is then defined as follows:

[0100]

[0101] Where T is the total number of point clouds, and j represents the spatial location index. Therefore, our final adaptive background loss function is as follows:

[0102] L BG =L adBCE +L aux .

[0103] S3. Based on the general background prototype, a background filtering algorithm is used to filter the background point cloud in the task sample of the query set and retain the foreground point cloud.

[0104] After obtaining the modeled general background prototype representation in the previous step, we further use the extended background prototype. To filter out background noise in the query features after prototype learning. Specifically, we first... and f q The data is concatenated, and then a convolutional block (conv(*)) is used to filter out background noise points in the query points. The calculation formula is as follows:

[0105]

[0106] in, This represents the query features after filtering out background noise, and conv(*) represents a 1×1 convolutional block.

[0107] S4. The prototype contrast learning algorithm is used to calculate the similarity between the foreground cloud of the task sample and the average foreground prototype and the general background prototype of all categories in the support set, respectively. The foreground cloud of the task sample with a similarity higher than the average foreground prototype of all samples in the support set is extracted.

[0108] In the preceding steps, background regions were effectively eliminated during the few-shot model learning process through background modeling and background filtering. Although these operations significantly reduced background interference in the query features, some background information may still not be completely eliminated. To further improve the model's performance and ensure that it can not only accurately generate background masks but also achieve good distinction between foreground target point clouds and background noise point clouds, we propose a Prototype Contrastive Learning (PCL) method to guide the model to learn robust and discriminative category prototype representations. By calculating the similarity between query samples and foreground target prototypes and general background prototypes, we optimize the embedding of query samples in the feature space, helping the model overcome difficult samples and noise during training and reducing the risk of misclassification of building component targets.

[0109] Specifically, we first need to obtain the target prototype and the general background prototype of the query samples. For the target prototype, based on the previous steps, we obtain the foreground prototypes of each category in the support set, which serve as our positive sample library. Similarly, based on the previous steps, we obtain the expanded general background prototype, which serves as our negative sample library. Next, we calculate the average value of the foreground prototype features to obtain an average vector representing the foreground class features. The calculation formula is as follows:

[0110]

[0111] Where n represents the number of foreground categories. This represents the foreground prototype for each category. Then we will use the average foreground prototype. and general background prototype P BG The dimensions are expanded to match the query feature dimensions, resulting in and Calculate query features and as well as The cosine similarity is calculated, and a temperature parameter τ is introduced to scale the similarity calculation. The calculation formula is as follows:

[0112]

[0113] Where ||*|| represents the L2 norm of the vector, and "·" represents the dot product, thus we obtain the positive and negative sample libraries. Finally, we use a marginal threshold M to control the difference in similarity between positive and negative samples and calculate the prototype contrastive loss function, as shown in the following formula:

[0114]

[0115] Where i is the index of the query point, and N is the total number of query points. To query the sample label, ReLU(*) is the modified linear unit activation function.

[0116] S5. Use a non-metric learning algorithm to perform feature matching and category prediction on the foreground cloud of the task sample;

[0117] Previous operations optimized the feature embedding, making the foreground target and background noise more clearly separated and distinct in the feature space, providing an optimized and highly discriminative feature space for feature matching. Therefore, based on the optimized feature embedding, we employ a nonparametric metric learning algorithm to accurately match each query point to the most similar category prototype to achieve the final category prediction.

[0118] Specifically, we have query features after filtering the background. And a set of foreground prototypes and general background prototypes for specific classes. Finally, predict the labels for the query point belonging to the building target category and the background point. As shown in the formula:

[0119]

[0120] in is a scaling factor (set to 1.0), i is the current class index, and j is the index for summing all classes using softmax normalization.

[0121] Our total loss function includes the adaptive binary cross-entropy loss function L. adBCE and cross-entropy loss function L aux and prototype contrast loss function L contra And used to supervise the model's prediction of labels during the training phase. And the real label Y q The main loss function L in the processing procedure main The details are as follows:

[0122] L total =L main +β(L adBCE +L aux )+γL contra

[0123]

[0124] Where β and γ represent adjustable loss weighting factors.

[0125] Based on the same inventive concept, the present invention also provides a meta-learning-based general background knowledge optimization device for semantic segmentation of few-shot building point clouds. The meta-learning-based general background knowledge optimization device for semantic segmentation of few-shot building point clouds can be a personal computer, a server, or other device that implements the aforementioned meta-learning-based general background knowledge optimization method for semantic segmentation of few-shot building point clouds.

[0126] The semantic segmentation device for few-shot building point clouds based on meta-learning and general background knowledge optimization provided in this application includes:

[0127] The prototype generation module uses an adaptive prototype generation algorithm to generate foreground and background prototypes for each category in the support set.

[0128] The background modeling module uses a modeling algorithm to learn background knowledge from all samples in the support set and all samples in the query set to obtain a general background prototype, and then uses an adaptive background loss algorithm to optimize the general background prototype.

[0129] The background filtering module, based on a general background prototype, uses a background filtering algorithm to filter the background point clouds in the task samples of the query set, while retaining the foreground point clouds.

[0130] The prototype contrast learning module uses a prototype contrast learning algorithm to calculate the similarity between the foreground cloud of the task sample and the average foreground prototype and the general background prototype of all categories in the support set, and extracts the foreground cloud of the task sample with a similarity higher than a preset threshold with the average foreground prototype of all categories in the support set.

[0131] The prediction module uses a non-metric learning algorithm to perform feature matching and category prediction on the foreground cloud of the task samples.

[0132] All relevant content of each step involved in the aforementioned embodiments of the method for optimizing few-shot building point cloud semantic segmentation based on meta-learning general background knowledge can be referenced to the functional description of the corresponding functional module of the device for optimizing few-shot building point cloud semantic segmentation based on meta-learning general background knowledge in the embodiments of this application, and will not be repeated here.

[0133] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of this invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0134] Below, we validate the general background knowledge optimization method for few-shot building point cloud semantic segmentation based on meta-learning on different datasets:

[0135] (1) Self-built ancient architecture dataset: The research object is the Beiding Niangniang Temple complex built during the Xuande period of the Ming Dynasty (1426-1435). As an important node on the central axis of Beijing, the complex was built during the Xuande period of the Ming Dynasty and has a history of more than 580 years. The main buildings of the complex include the Hall of Heavenly Kings, the East Annex Hall, the Niangniang Hall, the Dongyue Hall and the Mountain Gate Hall. It is a typical traditional Chinese wooden architectural complex with distinctive architectural features: the roof adopts a combination of hip roof and gable roof with clear ridge, covered with green glazed tiles with yellow trim; the doors and windows are all carved wooden structures, and the platform is well preserved.

[0136] The experiment used the FARO Focus3D X130 3D laser scanner as the data acquisition device, which has an acquisition capacity of 976,000 points per second and a maximum scanning distance of 130 meters. Its integrated coaxial high-resolution camera system effectively ensured the accurate registration of color images and point cloud data.

[0137] The point cloud dataset constructed in this study covers three areas: the North Peak Niangniang Hall (Area 1), the Heavenly King Hall (Area 2), and the East Annex Hall (Area 3). The Niangniang Hall (Area 1) has five bays and a hipped roof, while the side halls have gable roofs and simple tiled roofs, respectively. The Heavenly King Hall (Area 2) has three bays and a gable roof, with four five-panel painted wooden doors and lattice windows on both the front and rear eaves. The East Annex Hall (Area 3) is a gable roof building. The dataset includes 10 target categories (e.g., Tiebeam, Window, Door, Column, Roof, Floor, Stylobate, Step, Walls, and Clutter). Figure 3 (As shown in Table 1). To address the needs of few-shot semantic segmentation tasks, this study employs the N-way K-shot format to partition the dataset, dividing the target category into two disjoint subsets: a training set and a test set. The specific partitioning scheme is detailed in Table 1. Furthermore, the point cloud size of the dataset was systematically statistically analyzed. Table 2 lists the number of point clouds in each region and the total data volume, while Table 3 shows the distribution of point cloud quantity for each target category.

[0138] Table 1

[0139]

[0140] Table 2

[0141]

[0142] Table 3

[0143]

[0144] (2) ArCH Dataset: Constructed jointly by the University of Turin, Italy, and multiple international research institutions, this large-scale point cloud semantic segmentation dataset is specifically designed for the protection of historical architectural heritage. The dataset contains 17 labeled point cloud sets and 10 unlabeled sets, covering various types of historical buildings. This study selected five typical architectural regions for experiments. Specific region features and target category information are detailed in Table 4, covering nine core architectural components such as Vaults and Columns. To maintain consistency in experimental methods, this study adopted the previously proposed N-way K-shot partitioning strategy, dividing the ArCH dataset target categories into two independent subsets: a training set and a test set. The specific partitioning scheme is shown in Table 5. Visual examples of each target category in the dataset can be found in [reference needed]. Figure 4 .

[0145] Table 4

[0146]

[0147] Table 5

[0148]

[0149] The self-built ancient architecture dataset and the ArCH dataset were divided into 1790 and 2205 blocks, respectively. 2048 point samples were randomly selected from each block. For each dataset, all semantic categories were divided into two disjoint subsets. We then trained on one subset and tested on the other, a process of double cross-validation. We sampled multiple N-way K-shot sets for model training and 100 sets for model evaluation.

[0150] We chose the mean Intersection over Union (mIoU), a widely used evaluation metric in semantic segmentation tasks, to validate the effectiveness of our method. mIoU is the average IoU (Intersection over Union) of the test class set. Its calculation formula is as follows:

[0151]

[0152] Where TP represents correctly classified positive classes, FP represents negative classes incorrectly classified as positive, TN represents correctly classified negative classes, and FN represents positive classes incorrectly classified as negative. Considering the different manifestations of splitting each architectural heritage category, we also report split-0(s) separately. 0 ) and split-1(s 1 The results and averages are used to facilitate a more comprehensive comparison.

[0153] We implemented the experiment in three phases: a pre-training backbone network phase, a few-shot training phase, and a few-shot testing phase. We used various techniques to augment the training samples, such as Gaussian jitter, random shifting, random scaling, and random rotation around the z-axis. For the pre-training phase, we used DGCNN as the extractor for both our support and query features. Pre-training was implemented on visible architectural heritage categories. We set the batch size to 32, the learning rate to 0.001, the weight decay to 0.0001, and the decay rate to 0.5. We trained the model for 150 epochs on our self-built dataset and the ArCH dataset using the Adam optimizer (β1 = 0.9, β2 = 0.999). In the few-shot training phase, we initialized the feature extraction of the pre-trained weights and used the Adam optimizer to update all our parameter weights. The initial learning rate was set to 0.001, decaying by half every 5000 iterations. In the prototype contrastive learning, the temperature parameter τ was set to 0.1, the marginal threshold M was set to 0.5, and the prototype contrastive loss weight γ was set to 0.05. The adaptive background loss weight β was set to 0.5. We trained the model for 40,000 iterations, with each set constructed from randomly selected architectural heritage categories. In the few-shot meta-test phase, we randomly sampled 100 sets from new, unseen categories (categories not learned in the previous training phases) for model evaluation. All our experiments were implemented using the PyTorch framework, and our CPU configuration consisted of a 12vCPU Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz and an RTX 2080Ti GPU.

[0154] Analysis of the self-built ancient building dataset: Table 6 presents a comprehensive comparative experiment between our proposed method and the state-of-the-art (SOTA) method, showcasing our quantitative results on the self-built ancient building dataset. Our proposed method consistently outperforms the compared methods in four few-shot point cloud semantic segmentation tasks, including 1 / 2-way and 1 / 5-shot scenarios. Specifically, we conducted experimental comparisons with seven methods. In the 1-way-1-shot setting, compared to the state-of-the-art method DPA, our average mean IoU is improved by 6.26%; compared to the baseline model AttProtoNet, we improve the IoU by 16.72% with split=0 and by 9.03% with split=1, for an average improvement of 12.86%. In the 1-way-5-shot, 2-way-1-shot, and 2-way-5-shot settings, our overall mean accuracy is improved by 2.94%, 1.46%, and 6.54% respectively compared to the best model in the same settings.

[0155] Table 6

[0156]

[0157] Note: The performance of the self-built ancient building dataset is measured using the mIoU (mean intersection-union ratio) metric. i This represents the i-th fold used for model testing.

[0158] Analysis of the ArCH dataset: Compared to the self-built ancient building dataset, ArCH represents a different architectural style and pattern, making it more challenging and containing more complex scenes. Table 7 presents the quantitative results for the ArCH dataset. Specifically, we conducted comparative experiments on seven methods. In the 1-way-1-shot setting, compared to the state-of-the-art method DPA, our average mIou is improved by 4.16%; compared to the baseline model AttProtoNet, we improve mIou by 14.37% with split=0 and by 23.24% with split=1, for an average improvement of 18.81%. In the 1-way-5-shot, 2-way-1-shot, and 2-way-5-shot settings, our overall average accuracy is improved by 1.63%, 2.93%, and 1.47% respectively compared to the best model in the same settings. Overall, our proposed method achieves better performance on the ArCH dataset, demonstrating its effectiveness and superiority.

[0159] Table 7

[0160]

[0161] Note: The performance of the ArCH dataset is measured using the mIoU (mean intersection-union ratio) metric. i This represents the i-th fold used for model testing.

[0162] Experimental results: Figure 5 and Figure 6 This section illustrates the qualitative results of our proposed method in a 1-way 1-shot setting for semantic segmentation of few-sample point clouds on both a self-built ancient architecture dataset and the ArCH dataset. Specifically, in the ancient architecture dataset, we visualized two representative ancient architecture regions that serve as both support and query sets: Beiding Niang Niang Temple and Hall of Heavenly Gods. We also showcased some target categories. It can be seen that our method accurately identifies several architectural categories, including roofs, walls, beams, and columns, and achieves effective and complete identification of indistinguishable fine components such as beams and columns (see reference). Figure 5(Lines 5, 6, 7, and 8). In the ArCH dataset, we also visualized two representative regions that are mutually supporting and query sets: 5_SMV_chapel_2to4 and 6_SMV_chapel_24. It can be seen that our method achieves more accurate and refined segmentation results for the Roof, Moldings, Wall, and Column categories compared to the AttMPTI method. Especially for the column category, our method achieves complete and accurate segmentation compared to AttMPTI, while the AttMPTI method exhibits significant misclassification and omissions when segmenting columns (see reference). Figure 6 (Lines 4 and 8). The results above fully demonstrate that our method possesses superior ability to distinguish between target and non-target objects and to resist background noise interference when dealing with different architectural styles and diverse building categories. This advantage stems from our method's optimization of the target recognition process by learning general background knowledge, making it highly promising for segmenting challenging and complex point cloud scenes of architectural heritage.

[0163] Ablation experiment:

[0164] Ablation experiments were conducted on a self-built ancient building dataset in a 1-way 1-shot setting to verify the importance of each module (prototype generation module, background modeling module, background filtering module, and prototype contrast learning module).

[0165] Impact of the prototype generator: In our experiments, we first evaluated the prototype generator. Our method improved performance by 1.73% (61.28 vs. 59.55) compared to the AttProtoNet method. It can be seen that the simple averaging method may affect the quality of prototype generation when the quality of the support set samples is inconsistent or there is noise. In contrast, the adaptive weighted averaging method can more flexibly integrate information from the support set, improving the model's generalization ability and segmentation accuracy.

[0166] Impact of the Background Modeling Module: As shown in Table 8, our designed background modeling module further improved the model's performance by 4.83% (66.11% vs. 61.28%) compared to the previous module, demonstrating its significant performance enhancement. By introducing a general background prototype, this module effectively captures common background features in architectural point cloud scenes, enabling the model to accurately distinguish background information unrelated to the target building and reducing background noise interference. In contrast, the baseline model, lacking dedicated background modeling, is prone to misclassifying the background as foreground objects, leading to decreased segmentation accuracy. Furthermore, the adaptive background loss dynamically adjusts the loss weights for background and foreground, balancing their influence and avoiding model bias when the background-to-foreground ratio is unbalanced. Learning general background knowledge enhances the model's generalization ability, enabling it to adapt to diverse background environments.

[0167] Table 8

[0168]

[0169] Impact of the Background Filtering Module: Table 9 shows our qualitative evaluation results of the effectiveness of the background filtering module. It can be seen that, compared to the first two modules, the model accuracy improved by 3.67% (69.78% vs. 66.11%). The background filtering module successfully reduced the interference of background information unrelated to the segmentation target on the model. This mechanism allows the model to focus more on feature learning of the target building, thereby improving segmentation accuracy.

[0170] Table 9

[0171]

[0172] Impact of Prototype Contrastive Learning: As shown in Table 8, we evaluated the prototype contrastive learning module. Based on the first three modules, the model accuracy improved by 2.63% (72.41% vs. 69.78%). While the background modeling and background filtering modules effectively reduced the interference of background noise on the model, some background information could still affect segmentation accuracy. After introducing the prototype contrastive learning module, by calculating the cosine similarity between the query sample and the foreground target prototype and the general background prototype, the model achieved a more robust and discriminative class prototype representation in the feature space. We also used t-SNE to analyze the adaptation process between the query sample and the prototype (e.g., Figure 7a The visualization results highlight that prototype contrastive learning enhances the ability to identify class prototypes and improves the model's generalization ability in the context.

[0173] The impact of hyperparameters: Figure 7b , 7cIn sections 7d and 7e, we reported the effects of hyperparameters τ (temperature parameter), M (marginal threshold), and γ (prototype contrast loss weight) and β (adaptive background loss weight) on a 1-way-1-shot setting. For the temperature parameter τ, in prototype contrast learning, it affects the scaling of similarity, influencing the model's perception of distance between features. We evaluated values ​​of [0.05, 0.1, 0.15, 0.2]. Lower temperature values ​​increase the contrast between similar and dissimilar samples, while higher temperature values ​​smooth the similarity scores. For the marginal threshold M, we evaluated values ​​of [0.3, 0.5, 0.7]. The marginal threshold determines the minimum separation between positive and negative samples, affecting the model's ability to classify buildings. A higher marginal threshold requires greater inter-class separation, thus improving discriminative power. Lower values ​​may lead to insufficient inter-class separation, increasing the risk of class confusion and reducing segmentation accuracy. We evaluated the prototype contrastive learning loss weights using values ​​of [0.01, 0.05, 0.075, 0.1]. We found that excessively high values ​​may cause the contrastive loss to dominate, affecting the optimization of the main task and leading to a decrease in overall segmentation performance or unstable training. For the adaptive background loss weights, we evaluated values ​​of [0.3, 0.5, 0.7, 1.0]. Values ​​that are too low may lead to inaccurate prediction of the background region, affecting the overall segmentation effect. Values ​​that are too high may cause the background loss to dominate, weakening the segmentation accuracy of the foreground target and resulting in inaccurate foreground category recognition.

[0174] Computational Complexity: In Table 9, we categorized computational complexity into two groups based on different benchmarks. AttMPTI and QGE were placed in the first group, while AtttProtonet, QGPA, DPA, and our method were placed in the second group. It can be seen that although the number of parameters in the second group is increased, the computational memory is smaller, the FPS is significantly better, the inference time is greatly reduced, and the accuracy is significantly improved. Compared with state-of-the-art methods DPA and QGPA, our method achieves a good balance between model performance and computational complexity, providing superior segmentation results with reasonable computational efficiency.

[0175] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A general background knowledge-based method for optimizing semantic segmentation of few-shot building point clouds using meta-learning, characterized in that, include: An adaptive prototype generation algorithm is used to generate foreground and background prototypes for each category in the support set; A modeling algorithm is used to learn background knowledge from all samples in the support set and all samples in the query set to obtain a general background prototype, and an adaptive background loss algorithm is used to optimize the general background prototype. Based on a general background prototype, a background filtering algorithm is used to filter the background point cloud in the task samples of the query set and retain the foreground point cloud. The prototype contrast learning algorithm is used to calculate the similarity between the foreground cloud of the task sample and the average foreground prototype and the general background prototype of all categories in the support set, respectively. The foreground cloud of the task sample with the similarity to the average foreground prototype of all categories in the support set is extracted if it is higher than a preset threshold. A non-metric learning algorithm is used to perform feature matching and category prediction on the foreground point cloud of the task sample; The support set contains labeled building point cloud samples, and the query set contains unlabeled building point cloud samples.

2. The method for semantic segmentation of few-shot building point clouds based on meta-learning and general background knowledge optimization as described in claim 1, characterized in that, The process of generating foreground and background prototypes for a specific category using an adaptive prototype generation algorithm specifically includes: The support features and their binary masks of the support set samples are extracted using the backbone network extractor. Based on the supporting features and their corresponding binary masks, the binary masks of the supporting features are extended to the same dimension as the supporting features, and then mask feature extraction is performed to obtain the foreground and background features of the support set samples. Traverse other samples of the same category in the support set to obtain the foreground and background features of other samples of the same category in the support set; For all samples of the same category in the support set, a weighted average pooling algorithm is used to calculate the foreground features and background features respectively, so as to obtain the foreground prototype and background prototype of that category.

3. The method for semantic segmentation of few-shot building point clouds based on meta-learning and general background knowledge optimization as described in claim 1, characterized in that, The process of learning background knowledge from all samples in the support set and all samples in the query set using a modeling algorithm to obtain a general background prototype, and then optimizing the general background prototype using an adaptive background loss algorithm, specifically includes: The backbone network extractor extracts the support features of a sample in the support set and / or the query features of a sample in the query set. The support features and query features have the same dimension. Build and initialize the generic background prototype to obtain the original generic background prototype; The original general background prototype is expanded to the same dimension as the supporting features and / or query features, and then the original general background prototype is concatenated with the supporting features and / or query features. A background prediction convolutional network is used to predict the background using support features concatenated with the original general background prototype and / or query features concatenated with the original general background prototype. By traversing all samples in the support set and all samples in the query set, an adaptive background loss function is used to optimize the background prediction process in the background prediction convolutional network, thereby obtaining a general background prototype.

4. The method for semantic segmentation of few-shot building point clouds based on meta-learning and general background knowledge optimization as described in claim 3, characterized in that, The adaptive background loss function is the sum of the adaptive binary cross-entropy loss function and the cross-entropy loss function. The adaptive binary cross-entropy loss function incorporates an adaptive weight factor, and the cross-entropy loss function incorporates a mask distribution dynamic weight factor.

5. The method for semantic segmentation of few-shot building point clouds based on meta-learning and general background knowledge optimization as described in claim 3, characterized in that, Based on a general background prototype, a background filtering algorithm is used to filter the background point clouds in the task samples of the query set, and the specific process of retaining the foreground point clouds includes: The general background prototype is concatenated with the query features of the task samples in the query set, and then a convolutional unit is used to filter out the background point cloud in the query features, thereby preserving the foreground point cloud in the task samples.

6. The method for semantic segmentation of few-shot building point clouds based on meta-learning and general background knowledge optimization as described in claim 3, characterized in that, The specific process of calculating the similarity between the foreground cloud of the task sample and the average foreground prototype and general background prototype of all categories in the support set using the prototype contrastive learning algorithm includes: Calculate the average foreground prototype for all categories in the support set to obtain the average foreground prototype; Expand the dimensions of the average foreground prototype and the general background prototype to match the dimensions of the query features of the task samples. The cosine similarity between the foreground cloud in the task sample and the average foreground prototype and general background prototype of all categories in the support set is calculated separately. The cosine similarity calculation is scaled by introducing a temperature parameter τ to adjust the similarity calculation.

7. The method for semantic segmentation of few-shot building point clouds based on meta-learning and general background knowledge optimization as described in claim 6, characterized in that, The prototype contrastive learning algorithm is used to calculate the similarity between the foreground cloud of the task sample and the average foreground prototype and the general background prototype of all categories in the support set. In the process of extracting the foreground cloud of the task sample with a similarity higher than a preset threshold with the average foreground prototype of all categories in the support set, the prototype contrastive loss function is used to optimize the learning process. The prototype contrastive loss function introduces a marginal threshold M to control the difference in similarity between positive and negative samples. The positive sample is the average foreground prototype and the negative sample is the general background prototype.

8. The general background knowledge-based semantic segmentation method for building point clouds with few samples, as described in claim 6, is characterized in that... The specific process of using a non-metric learning algorithm to perform feature matching and category prediction on the foreground point cloud of task samples includes: Calculate the cosine similarity between the feature vector of each query point in the foreground cloud of the task sample and the foreground prototype of each category in the support set, convert it to a negative value and amplify it using an exponential function to obtain the mask logits; The mask logits of each query point in the foreground cloud of the task sample is transformed into a probability distribution using a normalized exponential function, and the prototype category with the highest probability value is selected as the predicted label for that query point.

9. The general background knowledge-based semantic segmentation method for building point clouds with few samples, as described in claim 3, is characterized in that... The backbone network extractor is a dynamic graph convolutional network based on a self-attention mechanism.

10. The method for semantic segmentation of few-shot building point clouds based on meta-learning and general background knowledge optimization as described in claim 7, characterized in that, The total loss function includes the main loss function, the adaptive background loss function, and the prototype comparison loss function. Both the adaptive background loss function and the prototype comparison loss function are set with adjustable weight factors.

Images