General background knowledge optimization few-sample building point cloud semantic segmentation method based on meta-learning

Abstract

The invention discloses a meta-learning-based general background knowledge optimization few-sample building point cloud semantic segmentation method. The method comprises the steps of generating a foreground prototype for each category in a support set by adopting an adaptive prototype generation algorithm; learning background knowledge from all samples of the support set and all samples of the query set by using a modeling algorithm to obtain a general background prototype, and optimizing the general background prototype by using an adaptive background loss algorithm; filtering background point clouds in the task samples of the query set by adopting a background filtering algorithm; calculating the similarity between the foreground point clouds of the task samples and the average foreground prototype and the general background prototype of all categories of the support set by adopting a prototype contrast learning algorithm, and extracting the foreground point clouds of the task samples of which the similarity with the average foreground prototype is higher than a preset threshold value; and performing feature matching and category prediction on the foreground point cloud of the task sample by adopting a non-metric learning algorithm. According to the method, false positive prediction can be effectively reduced, interference information is eliminated, and the target segmentation effect is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud data processing. More specifically, the present invention relates to a general background knowledge-based meta-learning-based semantic segmentation method for few-shot architectural point clouds. Background Art

[0002] Point cloud semantic segmentation plays a key role in the digital preservation of architectural heritage. Few-shot point cloud semantic segmentation uses only a minimal number of annotated point clouds to generate point-by-point masks for unseen categories for segmentation. While research in this area has made some progress, it has primarily focused on mining target object information, while background knowledge has not been fully utilized. This makes it difficult to distinguish non-target areas during segmentation, which can interfere with target object recognition. Summary of the Invention

[0003] An object of the present invention is to solve at least the above problems and to provide at least the advantages which will be described hereinafter.

[0004] Another object of the present invention is to provide a general background knowledge-based meta-learning-based semantic segmentation method for optimizing few-sample building point clouds, which can effectively reduce false positive predictions, eliminate interference information, and thus optimize the target segmentation effect.

[0005] In order to achieve these objectives and other advantages according to the present invention, a general background knowledge optimization method for few-shot building point cloud semantic segmentation based on meta-learning is provided, comprising:

[0006] Adopting adaptive prototype generation algorithm to generate foreground prototype and background prototype for each category in support set;

[0007] A modeling algorithm is used to learn background knowledge from all samples in the support set and all samples in the query set to obtain a universal background prototype, and an adaptive background loss algorithm is used to optimize the universal background prototype.

[0008] Based on the general background prototype, a background filtering algorithm is used to filter the background point cloud in the task samples of the query set and retain the foreground point cloud;

[0009] The prototype contrast learning algorithm is used to calculate the similarity between the foreground point cloud of the task sample and the average foreground prototype and the universal background prototype of all categories in the support set. The foreground point cloud of the task sample whose similarity with the average foreground prototype of all categories in the support set is higher than a preset threshold is extracted.

[0010] A non-metric learning algorithm is used to perform feature matching and category prediction on the foreground point clouds of task samples.

[0011] The support set contains labeled building point cloud samples, and the query set contains unlabeled building point cloud samples.

[0012] Preferably, the process of using the adaptive prototype generation algorithm to generate a foreground prototype and a background prototype for a certain category in the support set specifically includes:

[0013] The support features and binary masks of the support set samples are extracted by the backbone network extractor;

[0014] Based on the support features and their corresponding binary masks, the binary masks of the support features are expanded to the same dimension as the support features, and then mask feature extraction is performed to obtain the foreground features and background features of the support set samples;

[0015] Traverse other samples of the same category in the support set to obtain the foreground features and background features of other samples of the same category in the support set;

[0016] For all samples of the same category in the support set, the weighted average pooling algorithm is used to calculate the foreground features and background features respectively to obtain the foreground prototype and background prototype of the category.

[0017] Preferably, the process of using a modeling algorithm to learn background knowledge from all samples in the support set and all samples in the query set to obtain a universal background prototype, and optimizing the universal background prototype using an adaptive background loss algorithm specifically includes:

[0018] The backbone network extractor is used to extract the support features of a sample in the support set and / or the query features of a sample in the query set, where the dimensions of the support features and the query features are the same;

[0019] Constructing and initializing a universal background prototype to obtain an original universal background prototype;

[0020] Expanding the original general background prototype to the same dimension as the supporting features and / or query features, and then concatenating the original general background prototype with the supporting features and / or query features;

[0021] Using a background prediction convolutional network to perform background prediction on support features concatenated with the original universal background prototype and / or query features concatenated with the original universal background prototype;

[0022] Each sample in the support set and each sample in the query set are traversed. During the background prediction process of the background prediction convolutional network, an adaptive background loss function is used for optimization to obtain a universal background prototype.

[0023] Preferably, the adaptive background loss function is the sum of an adaptive binary cross entropy loss function and a cross entropy loss function, an adaptive weight factor is introduced into the adaptive binary cross entropy loss function, and a mask distribution dynamic weight factor is introduced into the cross entropy loss function.

[0024] Preferably, based on the general background prototype, a background filtering algorithm is used to filter the background point cloud in the task samples of the query set, and the specific process of retaining the foreground point cloud includes:

[0025] The general background prototype is concatenated with the query features of the task samples in the query set, and then a convolution unit is used to filter out the background point cloud in the query features, thereby retaining the foreground point cloud in the task samples.

[0026] Preferably, the specific process of using the prototype contrast learning algorithm to calculate the similarity between the foreground point cloud of the task sample and the average foreground prototype and the universal background prototype of all categories in the support set includes:

[0027] Calculate the average value of the foreground prototypes of all categories in the support set to obtain the average foreground prototype;

[0028] Expand the dimensions of the average foreground prototype and the general background prototype to be consistent with the dimensions of the query features of the task samples;

[0029] The cosine similarity between the foreground point cloud in the task sample and the average foreground prototype and the general background prototype of all categories in the support set is calculated respectively. The temperature parameter τ used to adjust the similarity calculation is introduced in the cosine similarity calculation for scaling.

[0030] Preferably, a prototype contrast learning algorithm is used to calculate the similarity between the foreground point cloud of the task sample and the average foreground prototype and the universal background prototype of all categories in the support set. In the process of extracting the foreground point cloud of the task sample whose similarity with the average foreground prototype of all categories in the support set is higher than a preset threshold, a prototype contrast loss function is used to optimize the learning process. A marginal threshold M is introduced into the prototype contrast loss function to control the gap in similarity between positive and negative samples, where the positive sample is the average foreground prototype and the negative sample is the universal background prototype.

[0031] Preferably, the specific process of using the non-metric learning algorithm to perform feature matching and category prediction on the foreground point cloud of the task sample includes:

[0032] Calculate the cosine similarity between the feature vector of each query point in the foreground point cloud of the task sample and the foreground prototype of each category in the support set, convert it to a negative value and amplify it using an exponential function to obtain the mask logits;

[0033] The mask logits of each query point in the foreground point cloud of the task sample are converted into a probability distribution using a normalized exponential function, and the prototype category with the largest probability value is selected as the predicted label of the query point.

[0034] Preferably, the backbone network extractor is a dynamic graph convolutional network based on a self-attention mechanism.

[0035] Preferably, the total loss function includes the main loss function, the adaptive background loss function and the prototype contrast loss function, and both the adaptive background loss function and the prototype contrast loss function are provided with weight factors with adjustable proportions.

[0036] The present invention includes at least the following beneficial effects: it is the first attempt to model and filter the background area in the few-sample point cloud semantic segmentation task, and it is also the first time to apply it to the downstream scene task of architectural heritage point cloud semantic segmentation, which can effectively reduce false positive predictions; three algorithms / modules, background modeling (BMM), background filtering (BFM) and prototype contrastive learning (PCL), are proposed to learn common background knowledge and eliminate interference information, thereby optimizing the target segmentation effect; a large number of experiments on the self-built ancient building dataset and the Arch architectural cultural heritage dataset show that the method proposed in this application produces advanced performance, providing a new perspective for the few-sample point cloud semantic segmentation work and the digital protection of architectural heritage.

[0037] Other advantages, objectives and features of the present invention will be reflected in part from the following description and will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of the method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to an embodiment of the present invention;

[0039] Figure 2 Schematic diagram of a background prediction convolutional network according to an embodiment of the present invention;

[0040] Figure 3 This is a visualization diagram of each category in the self-built ancient building dataset described in an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of visualization of various categories in the ArCH dataset according to an embodiment of the present invention;

[0042] Figure 5 A comparison of qualitative results of semantic segmentation of few-shot point clouds using different methods in a self-built ancient architecture dataset.

[0043] Figure 6 A comparison of qualitative results of different methods for few-shot point cloud semantic segmentation in the Arch dataset;

[0044] Figure 7a A schematic diagram showing the use of t-SNE to visualize the adaptation process between task samples and different prototypes;

[0045] Figure 7bThe effect of different hyperparameters τ (temperature parameter) on the qualitative results of semantic segmentation of few-shot point clouds;

[0046] Figure 7c The effect of different hyperparameters Μ (margin threshold) on the qualitative results of semantic segmentation of few-shot point clouds;

[0047] Figure 7d The effect of different hyperparameters γ (prototype contrast loss weight) on the qualitative results of few-shot point cloud semantic segmentation;

[0048] Figure 7e The effect of different hyperparameters β (adaptive background loss weight) on the qualitative results of few-shot point cloud semantic segmentation. DETAILED DESCRIPTION

[0049] The present invention will be described in further detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0050] It should be noted that the experimental methods described in the following embodiments are conventional methods unless otherwise specified, and the reagents and materials are commercially available unless otherwise specified; in the description of the present invention, the terms "horizontal", "longitudinal", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention.

[0051] Few-shot 3D point cloud semantic segmentation applies general 3D point cloud semantic segmentation to few-shot scenarios. This model can learn similarities and feature sharing between categories with only a small number of labels, improving the model's adaptability to new categories. It is suitable for architectural heritage scenarios with diverse styles and complex structures and details. Unlike most research on few-shot 3D point cloud semantic segmentation, which focuses on directly segmenting the target object, this application is the first to propose a few-shot point cloud semantic segmentation method that uses complementary background knowledge to eliminate interference areas to optimize target recognition, and it is the first to apply this method to the field of architectural cultural heritage.

[0052] Most previous research has focused on designing artificially preferred network architectures to solve various few-shot learning tasks. The emergence of contrastive learning has led this application to focus on mining better deep feature representations by comparing positive and negative samples. It aims to learn effective representations of data by learning the similarities and differences between samples.

[0053] In traditional machine learning, models typically require extensive training data and computing resources to achieve good performance on specific tasks. The goal of meta-learning is to enable models to "learn to learn." This involves learning from past experiences across different tasks to improve learning efficiency on new tasks. Its core concepts include: 1. Cross-task learning: extracting common knowledge from multiple tasks; 2. Rapid adaptation: enabling rapid learning on new tasks; and 3. Few-shot learning: enabling learning with fewer samples.

[0054] One of the solutions to the problem of semantic segmentation of few-shot 3D point clouds is meta-learning. The following first introduces the definition of using meta-learning for semantic segmentation of few-shot point clouds. Specifically, all categories in the dataset can be divided into a training category set C train (also called visible class collection) and test class collection Each few-shot task or scenario E = {S, Q}, where S and Q are the support set and query set, is instantiated as an N-way K-shot point cloud segmentation task. The support set is represented as Contains K (from 1, 2, ..., K) groups of support point clouds for each of the N categories And the corresponding mask The query set is represented as Contains N sets of query point clouds and corresponding ground truth masks Normal query mask It can only be used in the few-shot training phase, while only the support set mask is used in the testing phase Generally, each point cloud Contains T groups of information with coordinates and additional features For example, color information.

[0055] In summary, the training of the Few-shot Point Cloud Semantic Segmentation Model (FS-PCS) consists of two stages, namely the pre-training stage and the meta-learning stage. The pre-training stage mainly trains the backbone network and learns meaningful semantic features in a fully supervised manner, while the meta-training focuses on the ability of the training model to transfer the learned knowledge from the support set S to the query set Q. Given an N-way K-shot task: In the training stage, the training category C with semantic masks in the provided support set S and query set Q is used. train ( as well as ) Learn feature representation; in the test phase, facing unseen categories, using a support set S with a small number of annotations, the model needs to identify new categories C without any labeled query set Q test , generating point-by-point predictions (N+1 (number 1) represents the background category).

[0056] As previously mentioned, few-shot point cloud semantic segmentation models often struggle to distinguish easily distracting background areas, posing a significant challenge to accurately identifying architectural heritage components. Inspired by this observation, this application focuses on developing a model / method that can better learn background knowledge and eliminate background interference to optimize target recognition, rather than simply designing a model / method that directly segments the target. The following method for optimizing few-shot architectural point cloud semantic segmentation based on general background knowledge through meta-learning refers to architectural cultural heritage.

[0057] like Figure 1 As shown, the present invention provides a general background knowledge optimization method for few-shot building point cloud semantic segmentation based on meta-learning, comprising:

[0058] S1, using the adaptive prototype generation algorithm to generate foreground prototypes and background prototypes for each category in the support set;

[0059] The process of using the adaptive prototype generation algorithm to generate foreground prototypes and background prototypes for a certain category in the support set specifically includes:

[0060] S11, extracting support features and binary masks of support set samples through the backbone network extractor;

[0061] The backbone network extractor adopts DGCNN, a dynamic graph convolutional network as the backbone feature extractor, and this application further introduces a self-attention network to capture the semantic correlation between building point clouds in a global context.

[0062] The samples in the support set are labeled building point cloud samples, and the samples in the query set are unlabeled building point cloud samples. (where n∈{1,...,N}, k∈{1,...,K}), the sample in the query set is the query point cloud P q , extract support features f s And the query feature f q , the calculation formula is as follows:

[0063]

[0064] Where φ represents the backbone network feature extractor, N and d represent the number of point clouds and the number of channels of point cloud features, respectively.

[0065] S12. Based on the support feature and its corresponding binary mask, the binary mask of the support feature is expanded to the same dimension as the support feature, and then mask feature extraction is performed to obtain the foreground feature and background feature of the support set sample;

[0066] Given support feature f s The corresponding binary mask and Expand the mask to the same dimension as the support feature, and then extract the support set foreground features using mask features and background characteristics The calculation formula is as follows:

[0067] f bg =f s ☉Y s bg

[0068] f fg =f s ⊙Y s fg

[0069] Where ⊙ represents the Hardma factor.

[0070] S13. Traverse other samples of the same category in the support set to obtain foreground features and background features of other samples of the same category in the support set.

[0071] S14. For all samples of the same category in the support set, the weighted average pooling algorithm is used to calculate the foreground features and background features respectively, and the foreground prototype and background prototype of the category are obtained.

[0072] Based on the foreground features and background features, we generate the foreground prototype of the category for the nth category through adaptive weighted average pooling operation and background prototype The calculation formula is as follows:

[0073]

[0074] Where x∈{1,...,N} represents the coordinate position information, w represents the adaptive weight factor, and “|Δ|” represents the reduction factor that outputs 1 when Δ is true.

[0075] By adopting the above steps S11 to S14 and traversing each category in the support set, a foreground prototype and a background prototype can be generated for each category.

[0076] The above is the prototype generation representation in the N-way K-shot architectural heritage point cloud semantic segmentation task where n∈{1,...,N}. Through the adaptive prototype generation step, the model can initially understand the semantic information of the foreground and background of the support set from a small number of support samples, improving the model's ability to discriminate knowledge and identify targets.

[0077] S2, using a modeling algorithm to learn background knowledge from all samples in the support set and all samples in the query set to obtain a universal background prototype, and then using an adaptive background loss algorithm to optimize the universal background prototype;

[0078] The process of this step specifically includes:

[0079] S21. Extracting support features of a sample in the support set and / or query features of a sample in the query set through a backbone network extractor, where the dimensions of the support features and the query features are the same;

[0080] The method of extracting support features and query features has been described in the aforementioned step S11, so it will not be repeated here.

[0081] S22, constructing and initializing a universal background prototype to obtain an original universal background prototype;

[0082] Background regions usually exist in most building point cloud scenes. Based on this commonality, it is recommended to use a general background prototype to model background knowledge. It can be expressed as The number 1 represents the dimension, and d is the channel dimension. First, we need to randomly initialize the background prototype P BG , and then learns its universal representation from the support point cloud and query point cloud respectively on the training target category in the few-shot semantic segmentation training phase.

[0083] S23, expanding the original general background prototype to the same dimension as the supporting features and / or query features, and then concatenating the original general background prototype with the supporting features and / or query features;

[0084] Specifically, given the support feature f s and query feature f q , expand the initialized P BG to the same dimensions as them Afterwards, the support features and query features are concatenated with it respectively.

[0085] S24, using a background prediction convolutional network to perform background prediction on the support features concatenated with the original universal background prototype and / or the query features concatenated with the original universal background prototype;

[0086] The structure of the background prediction convolutional network is as follows Figure 2 As shown, the calculation formula is as follows:

[0087]

[0088] Where M(*) represents the background prediction convolutional network, Represents the splicing operation, It is the support or query probability result.

[0089] S25. Traverse each sample in the support set and each sample in the query set, and use an adaptive background loss function for optimization in the background prediction convolutional network to obtain a universal background prototype.

[0090] In the semantic segmentation task of architectural heritage few-shot point cloud, the actual label Y usually only contains the building category (target object). s and Y q , which does not include the true label of the background area. On this basis, in order to make the learned general background prototype more effective in predicting the background area as we expect, we designed an adaptive background loss to optimize this learning metric during the training process. We first define an adaptive weight factor w, the specific calculation formula is as follows:

[0091]

[0092] where ε is a small constant (1e -6 ), used to avoid division by zero, is the ratio of foreground points to target points. This formula allows the model to dynamically adjust the weights between background areas and foreground objects in the loss function and effectively handle differences in background and foreground ratios during training, facilitating the model's continuous updating of background knowledge during training.

[0093] Next, we define the adaptive binary cross entropy loss function. By adding this constraint, we can predict the background noise. The specific calculation formula is as follows:

[0094]

[0095] Where N is the total number of target points and i is the spatial position index.

[0096] The above formula aims to predict the reverse region belonging to the target region, that is However, using only this constraint may result in a predicted All zeros. Therefore, we add an auxiliary regularization to alleviate the above problem. Specifically, we first define a dynamic weight η of the mask distribution based on the support set and the query set s / q , the formula is as follows:

[0097]

[0098] where N s / q Represent the number of support and query points respectively. Through this formula, the model can dynamically adjust the ratio of foreground and background in support and query, enhance the model's adaptability to the background, and avoid the model over-focusing on the target area and ignoring the background area.

[0099] The cross entropy loss function is then defined as follows:

[0100]

[0101] Where T is the total number of point clouds and j represents the spatial position index. Therefore, our final adaptive background loss function is as follows:

[0102] L BG =L adBCE +L aux .

[0103] S3, based on the general background prototype, the background filtering algorithm is used to filter the background point cloud in the task samples of the query set and retain the foreground point cloud;

[0104] After obtaining the general background prototype representation after modeling in the previous step, we further use the expanded background prototype To filter out the background noise in the query features after prototype learning. Specifically, we first and f q After concatenation, a convolution block conv(*) is used to filter out background noise points in the query point. The calculation formula is as follows:

[0105]

[0106] in, represents the query feature after filtering the background noise, and conv(*) represents a 1×1 convolutional block.

[0107] S4. Use the prototype contrast learning algorithm to calculate the similarity between the foreground point cloud of the task sample and the average foreground prototype and the universal background prototype of all categories in the support set, and extract the foreground point cloud of the task sample whose similarity with the average foreground prototype of all samples in the support set is higher than a preset threshold;

[0108] In the above steps, background modeling and background filtering effectively eliminate the background area in the few-shot model learning process. Although these operations significantly reduce the background interference in the query features, some background information may still not be completely eliminated. To further improve the performance of the model, we ensure that not only the background mask can be accurately generated, but also a good distinction can be achieved between the foreground target point cloud and the background noise point cloud. We propose a prototype contrastive learning (PCL) method to guide the model to learn a robust and discriminative category prototype representation. By calculating the similarity between the query sample and the foreground target prototype and the general background prototype, the embedding of the query sample in the feature space is optimized, helping the model overcome difficult samples and noise during training, and reducing the risk of the model misclassifying building component targets.

[0109] Specifically, we first need to obtain the target prototype and general background prototype of the query sample. For the target prototype, we obtain the foreground prototypes of each category in the support set according to the previous steps, which will serve as our positive sample library. We also obtain the expanded general background prototype according to the previous steps as our negative sample library. We then calculate the average value of the foreground prototype features to obtain an average vector representing the foreground class features. The calculation formula is as follows:

[0110]

[0111] Where n represents the number of foreground classes, Denote the foreground prototype of each category. Then we average the foreground prototype and the general background prototype P BG The dimension is expanded to be consistent with the query feature dimension, and we get and Calculate query features and as well as The cosine similarity of is calculated and the temperature parameter τ is introduced to adjust the similarity calculation for scaling. The calculation formula is as follows:

[0112]

[0113] Here, ||*|| represents the L2 norm of the vector, and “·” represents the dot product, thus we obtain the positive sample library and the negative sample library. Finally, we use a marginal threshold M to control the gap in the similarity between positive and negative samples and calculate the prototype contrast loss function. The calculation formula is as follows:

[0114]

[0115] Where i is the index of the query point, N is the total number of query points, To query the sample label, ReLU(*) is the rectified linear unit activation function.

[0116] S5. Use non-metric learning algorithm to perform feature matching and category prediction on the foreground point cloud of the task sample;

[0117] The previous operation optimizes feature embeddings, making foreground objects and background noise more separated and clear in the feature space, providing an optimized and highly discriminative feature space for feature matching. Therefore, based on the optimized feature embeddings, we employ a non-parametric metric learning algorithm to accurately match each query point to the most similar category prototype to achieve the final category prediction.

[0118] Specifically, we have the query features after filtering the background and a set of class-specific foreground prototypes and a generic background prototype Finally, the predicted labels of the query point belonging to the building target category and the background point As shown in the formula:

[0119]

[0120] in is a magnification factor (set to 1.0), i is the current category index, and j is the index of the softmax normalization summed over all categories.

[0121] Our total loss function consists of an adaptive binary cross entropy loss function L adBCE and the cross entropy loss function L aux And the prototype contrast loss function L contra , and used to supervise the training phase model to predict labels and the true label Y q The main loss function L of the processing process main , as shown below:

[0122] L total =L main +β(L adBCE +L aux )+γL contra

[0123]

[0124] Where β and γ represent the loss weight factors that can be adjusted.

[0125] Based on the same inventive concept, the present invention also provides a general background knowledge based meta-learning optimized few-sample building point cloud semantic segmentation device. The general background knowledge based meta-learning optimized few-sample building point cloud semantic segmentation device can be a personal computer, a server, or other devices that implement the aforementioned general background knowledge based meta-learning optimized few-sample building point cloud semantic segmentation method.

[0126] The embodiment of the present application provides a general background knowledge-based meta-learning-based semantic segmentation device for optimizing few-sample building point clouds, including:

[0127] The prototype generation module uses an adaptive prototype generation algorithm to generate foreground prototypes and background prototypes for each category in the support set;

[0128] The background modeling module uses a modeling algorithm to learn background knowledge from all samples in the support set and all samples in the query set to obtain a universal background prototype, and then uses an adaptive background loss algorithm to optimize the universal background prototype;

[0129] The background filtering module, based on the general background prototype, uses the background filtering algorithm to filter the background point cloud in the task samples of the query set and retain the foreground point cloud;

[0130] The prototype contrast learning module uses the prototype contrast learning algorithm to calculate the similarity between the foreground point cloud of the task sample and the average foreground prototype and the universal background prototype of all categories in the support set, and extracts the foreground point cloud of the task sample whose similarity with the average foreground prototype of all categories in the support set is higher than a preset threshold;

[0131] The prediction module uses a non-metric learning algorithm to perform feature matching and category prediction on the foreground point clouds of task samples.

[0132] All relevant contents of the steps involved in the aforementioned embodiment of the method for optimizing the semantic segmentation of few-sample building point clouds based on general background knowledge of meta-learning can be referred to the functional description of the functional modules corresponding to the device for optimizing the semantic segmentation of few-sample building point clouds based on general background knowledge of meta-learning in the embodiment of the present application, and will not be repeated here.

[0133] The division of modules in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present invention may be integrated into a single processor, or may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0134] Next, we use the above-mentioned meta-learning-based general background knowledge optimization method for few-shot building point cloud semantic segmentation to verify it on different datasets:

[0135] (1) Self-built ancient building dataset: The research object is the Beiding Niangniang Temple complex built during the Xuande period of the Ming Dynasty (1426-1435). As an important node on the central axis of Beijing, the complex was built during the Xuande period of the Ming Dynasty and has a history of more than 580 years. The main buildings of the complex include the Tianwang Hall, the East Side Hall, the Niangniang Hall, the Dongyue Hall, and the Shanmen Hall. It is a typical traditional Chinese wooden structure with distinctive architectural features: the roof is a combination of a hip roof and a gable roof, covered with green glazed tiles with yellow trim; the doors and windows are all hollow wooden structures, and the base is partially preserved.

[0136] The experiment used a FARO Focus3D X130 3D laser scanner as the data acquisition device. This device has a capture capacity of 976,000 points per second and a maximum scanning distance of 130 meters. Its integrated coaxial high-resolution camera system effectively ensures accurate registration of color images and point cloud data.

[0137] The point cloud dataset constructed in this study covers three areas: Beiding Niangniang Hall (Area1), Tianwang Hall (Area2) and East Side Hall (Area3). Among them, Niangniang Hall (Area1) is five bays wide, with the main body adopting a hip roof structure, and the side halls on both sides are respectively a hard mountain ridge roof and a simple tile roof; Tianqi Hall (Area2) is three bays wide, with a hard mountain ridge roof, and the front and back eaves are equipped with four five-panel painted wooden doors and sill windows; East Side Hall (Area3) is a hard mountain ridge roof building. The dataset contains 10 categories of targets (such as Tiebeam, Window, Door, Column, Roof, Floor, Stylobate, Step, Walls, Clutter) Figure 3 To meet the requirements of the few-shot semantic segmentation task, this study partitioned the dataset using the N-way K-shot format, dividing the target categories into two mutually exclusive subsets: a training set and a test set. The specific partitioning scheme is detailed in Table 1. Furthermore, a systematic statistical analysis of the point cloud size of the dataset was performed. Table 2 lists the number of point clouds in each region and the total data volume, and Table 3 shows the distribution of point clouds by target category.

[0138] Table 1

[0139]

[0140] Table 2

[0141]

[0142] Table 3

[0143]

[0144] (2)ArCH dataset: It was jointly constructed by the University of Turin, Italy and multinational research institutions. It is a large-scale point cloud semantic segmentation dataset specifically for the protection of historical architectural heritage. The dataset contains 17 labeled point cloud sets and 10 unlabeled sets, covering a variety of historical building types. This study selected five typical building areas to conduct experiments. The specific regional characteristics and target category information are detailed in Table 4, covering 9 types of core building components such as Vault (vault) and Column (column). In order to maintain the consistency of the experimental method, this study continues to use the N-way K-shot partitioning strategy proposed earlier, and divides the target categories of the Arch dataset into two independent subsets: training set and test set. The specific partitioning scheme is shown in Table 5. The visualization examples of each target category in the dataset can be referred to Figure 4 .

[0145] Table 4

[0146]

[0147] Table 5

[0148]

[0149] The Self-Built Ancient Architecture Dataset and the Arch Dataset were divided into 1790 and 2205 blocks, respectively. 2048 point samples were randomly sampled from each block. For each dataset, all semantic categories were divided into two disjoint class subsets. We then used one subset for training and the other for testing, a process known as two-way cross-validation. We sampled multiple N-way K-shot sets for model training and 100 sets for model evaluation.

[0150] We use the widely used semantic segmentation evaluation metric, mean-IoU, to verify the effectiveness of our method. mIoU is the average of the IoU (intersection over union) of the test set. It is calculated as follows:

[0151]

[0152] Where TP represents the correctly classified positive class, FP represents the negative class that is misclassified as the positive class, TN represents the accurately classified negative class, and FN represents the positive class that is misclassified as the negative class. Considering the different performance of each architectural heritage class split, we also report the split-0(s 0 ) and split-1(s 1 ) and averages for a more comprehensive comparison.

[0153] We conducted experiments in three phases: pre-training the backbone network, few-shot training, and few-shot testing. We augmented the training samples using various techniques, such as Gaussian jittering, random shifting, random scaling, and random rotation around the z-axis. For pre-training, we used DGCNN as our support and query feature extractor. Pre-training was performed on the visible architectural heritage category. We set the batch size to 32, the learning rate to 0.001, and the weight decay to 0.0001 and 0.5, respectively. We trained the model for 150 epochs using the Adam optimizer (β1 = 0.9, β2 = 0.999) on both our own dataset and the Arch dataset. During the few-shot training phase, we initialized the feature extraction with the pre-trained weights and updated all our parameter weights using the Adam optimizer. The initial learning rate was set to 0.001 and decayed by half every 5000 iterations. The temperature parameter τ, margin threshold M, and prototype contrastive loss weight γ in prototype contrastive learning were set to 0.1, 0.5, and 0.05, respectively. The adaptive background loss weight β was set to 0.5. We trained the model for 40,000 iterations, with each episode constructed from a randomly selected architectural heritage category. During the few-shot meta-test phase, we evaluated the model on 100 randomly selected episodes from unseen categories (categories not learned during the training phase). All experiments were implemented using the PyTorch framework, using a 12vCPU Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz and an RTX 2080Ti GPU.

[0154] Analysis of a Self-Constructed Ancient Architecture Dataset: In Table 6, we conduct comprehensive comparative experiments between our proposed method and the state-of-the-art methods (SOTA), demonstrating our quantitative results on a self-constructed ancient architecture dataset. Our proposed method consistently outperforms the comparison methods in four few-shot point cloud semantic segmentation tasks, including 1 / 2-way and 1 / 5-shot scenarios. Specifically, we conduct experimental comparisons against seven methods. In the 1-way-1-shot setting, compared to the state-of-the-art method DPA, our average mIou improves by 6.26%. Compared to the baseline model AttProtoNet, our mIou improves by 16.72% under split=0 and by 9.03% under split=1, for an average improvement of 12.86%. In the 1-way-5-shot, 2-way-1-shot, and 2-way-5-shot settings, our overall average accuracy improves by 2.94%, 1.46%, and 6.54%, respectively, compared to the best model under the same settings.

[0155] Table 6

[0156]

[0157] Note: The performance of the self-built ancient building dataset is measured using the mIoU (mean intersection over union) indicator. i represents the i-th fold used for model testing.

[0158] Analysis of the Arch Dataset: Compared to the self-built ancient architecture dataset, Arch represents different architectural styles and formats, making it a more challenging dataset with more complex scenes. Table 7 presents quantitative results on the Arch dataset. Specifically, we experimentally compare seven methods. In the 1-way-1-shot setting, our average mIou improves by 4.16% compared to the state-of-the-art method, DPA. Compared to the baseline model, AttProtoNet, our mIou improves by 14.37% when split=0 and by 23.24% when split=1, for an average improvement of 18.81%. In the 1-way-5-shot, 2-way-1-shot, and 2-way-5-shot settings, our overall average accuracy improves by 1.63%, 2.93%, and 1.47%, respectively, compared to the best model under the same settings. Overall, our proposed method achieves superior performance on the Arch dataset, demonstrating its effectiveness and superiority.

[0159] Table 7

[0160]

[0161] Note: The performance of the Arch dataset using the mIoU (mean intersection over union) metric is s i represents the i-th fold used for model testing.

[0162] Experimental results: Figure 5 and Figure 6 The qualitative results of our proposed method for semantic segmentation of few-shot point clouds in a 1-way 1-shot setting in a self-built ancient building dataset and the Arch dataset are presented. Specifically, in the ancient building dataset, we visualized two representative ancient building areas that serve as mutual support and query sets: Beiding Niang Niang Temple and Hall of Heavenly Gods. At the same time, we also presented some target categories. It can be seen that our method accurately identifies the building categories of roofs, walls, beams, and columns, and achieves effective and complete recognition of fine components that are difficult to distinguish: beams and columns (refer to Figure 5In the Arch dataset, we also visualized two representative regions of mutual support and query sets: 5_SMV_chapel_2to4 and 6_SMV_chapel_24. It can be seen that our method achieves more accurate and detailed segmentation for the categories of Roof, Moldings, Wall, and Column than the AttMPTI method. Especially for the column category, our method achieves complete and accurate segmentation compared to AttMPTI, while the AttMPTI method has significant misclassification and omission phenomena when segmenting columns (refer to Figure 6 These results demonstrate our method's superior ability to distinguish between target and non-target objects and its robustness to background noise across diverse architectural styles and categories. This advantage stems from our method's ability to optimize object recognition by learning common background knowledge, making it a promising candidate for segmenting challenging and complex architectural cultural heritage point cloud scenarios.

[0163] Ablation experiment:

[0164] Ablation experiments are conducted on a self-built ancient building dataset under a 1-way 1-shot setting to verify the importance of each module (prototype generation module, background modeling module, background filtering module, and prototype contrast learning module).

[0165] Impact of prototype generator: In our experiments, we first evaluated the prototype generator. Our method improved the performance by 1.73% (61.28 vs. 59.55) compared with the AttProtoNet method. It can be seen that the simple averaging method may affect the quality of prototype generation when the quality of support set samples is inconsistent or there is noise. In contrast, the use of adaptive weighted averaging method to generate prototypes can more flexibly integrate the information in the support set, thereby improving the generalization ability and segmentation accuracy of the model.

[0166] Impact of the background modeling module: As shown in Table 8, the background modeling module we designed further improved the performance of the model by 4.83% (66.11% vs. 61.28%) on the basis of the previous one. It can be seen that the background modeling module significantly improved the performance of the model. By introducing the general background prototype, the module effectively captures the common background features in the building point cloud scene, enabling the model to accurately distinguish background information unrelated to the target building and reduce the interference of background noise. In contrast, due to the lack of specialized background modeling, the baseline model easily misclassifies the background as a foreground target, resulting in a decrease in segmentation accuracy. In addition, the adaptive background loss dynamically adjusts the loss weights of the background and foreground, balancing the influence of the two and avoiding the bias of the model when the background and foreground ratio is unbalanced. The learning of general background knowledge improves the generalization ability of the model, enabling it to adapt to diverse background environments.

[0167] Table 8

[0168]

[0169] Impact of the Background Filtering Module: Table 9 shows our qualitative evaluation of the effectiveness of the background filtering module. It shows that compared to the first two modules, the model accuracy improved by 3.67% (69.78% vs. 66.11%). The background filtering module successfully reduces the interference of background information irrelevant to the segmentation target. This mechanism enables the model to focus more on learning the characteristics of the target building, thereby improving segmentation accuracy.

[0170] Table 9

[0171]

[0172] Impact of prototype contrast learning: As shown in Table 8, we evaluated the prototype contrast learning module. Based on the first three modules, the model accuracy was improved by 2.63% (72.41% vs. 69.78%). Although the background modeling module and the background filtering module effectively reduce the interference of background noise on the model, some background information may still affect the segmentation accuracy. After the introduction of the prototype contrast learning module, by calculating the cosine similarity between the query sample and the foreground target prototype and the general background prototype, the model achieves a more robust and highly recognizable category prototype representation in the feature space. We also used t-SNE to analyze the adaptation process between the query sample and the prototype (e.g. Figure 7a ), the visualization results emphasize that prototype contrast learning enhances the recognition ability of category prototypes and improves the generalization ability of the model in background environments.

[0173] The impact of hyperparameters: Figure 7b 、 7cIn Figures 7d, 7e, we report the effects of the hyperparameters τ (temperature parameter), M (margin threshold), γ (prototype contrast loss weight), and β (adaptive background loss weight) in the 1way-1shot setting, respectively. The temperature parameter τ affects the scaling of similarity during prototype contrast learning and the model's perception of the distance between features. We evaluated it using values ​​[0.05, 0.1, 0.15, 0.2]. Lower temperature values ​​increase the contrast between similar and dissimilar samples, while higher temperature values ​​smooth the similarity scores. We evaluated the margin threshold M using values ​​[0.3, 0.5, 0.7]. The margin threshold determines the minimum separation between positive and negative samples, affecting the model's ability to distinguish between categories. A higher margin threshold requires greater separation between categories, thereby improving discrimination. Lower values ​​may lead to insufficient separation between categories, increase the risk of category confusion, and reduce segmentation accuracy. We evaluated the prototype contrastive learning loss weights using values ​​of [0.01, 0.05, 0.075, 0.1]. We found that values ​​that were too high could make the contrastive loss too dominant, affecting the optimization of the main task, leading to a decrease in overall segmentation performance or unstable training. We evaluated the adaptive background loss weights using values ​​of [0.3, 0.5, 0.7, 1.0]. Values ​​that were too low could lead to inaccurate predictions of background regions and affect the overall segmentation effect. Values ​​that were too high could make the background loss too dominant, weakening the segmentation accuracy of foreground objects and causing inaccurate foreground category recognition.

[0174] Computational Complexity: In Table 9, we categorize the methods into two groups for computational complexity comparison based on different benchmarks. AttMPTI and QGE are grouped into the first group, and AtttProtonet, QGPA, DPA, and our method are grouped into the second group. It can be seen that while the second group of methods has an increased number of parameters, they use less computational memory, have a significant advantage in FPS, significantly shorten inference time, and significantly improve accuracy. Compared to the state-of-the-art methods, DPA and QGPA, our method strikes a good balance between model performance and computational complexity, providing superior segmentation results with reasonable computational efficiency.

[0175] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A general background knowledge optimization method for few-shot building point cloud semantic segmentation based on meta-learning, characterized by: include: Adopting adaptive prototype generation algorithm to generate foreground prototype and background prototype for each category in support set; A modeling algorithm is used to learn background knowledge from all samples in the support set and all samples in the query set to obtain a universal background prototype, and an adaptive background loss algorithm is used to optimize the universal background prototype. Based on the general background prototype, a background filtering algorithm is used to filter the background point cloud in the task samples of the query set and retain the foreground point cloud; The prototype contrast learning algorithm is used to calculate the similarity between the foreground point cloud of the task sample and the average foreground prototype and the universal background prototype of all categories in the support set. The foreground point cloud of the task sample whose similarity with the average foreground prototype of all categories in the support set is higher than a preset threshold is extracted. A non-metric learning algorithm is used to perform feature matching and category prediction on the foreground point clouds of task samples. The support set contains labeled building point cloud samples, and the query set contains unlabeled building point cloud samples.

2. The method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to claim 1, characterized in that: The process of using the adaptive prototype generation algorithm to generate foreground prototypes and background prototypes for a certain category in the support set specifically includes: The support features and binary masks of the support set samples are extracted by the backbone network extractor; Based on the support features and their corresponding binary masks, the binary masks of the support features are expanded to the same dimension as the support features, and then mask feature extraction is performed to obtain the foreground features and background features of the support set samples; Traverse other samples of the same category in the support set to obtain the foreground features and background features of other samples of the same category in the support set; For all samples of the same category in the support set, the weighted average pooling algorithm is used to calculate the foreground features and background features respectively to obtain the foreground prototype and background prototype of the category.

3. The method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to claim 1, wherein: The process of using a modeling algorithm to learn background knowledge from all samples in the support set and all samples in the query set to obtain a universal background prototype and then using an adaptive background loss algorithm to optimize the universal background prototype specifically includes: The backbone network extractor is used to extract the support features of a sample in the support set and / or the query features of a sample in the query set, where the dimensions of the support features and the query features are the same; Constructing and initializing a universal background prototype to obtain an original universal background prototype; Expanding the original general background prototype to the same dimension as the supporting features and / or query features, and then concatenating the original general background prototype with the supporting features and / or query features; Using a background prediction convolutional network to perform background prediction on support features concatenated with the original universal background prototype and / or query features concatenated with the original universal background prototype; Each sample in the support set and each sample in the query set are traversed. During the background prediction process of the background prediction convolutional network, an adaptive background loss function is used for optimization to obtain a universal background prototype.

4. The method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to claim 3, wherein: The adaptive background loss function is the sum of an adaptive binary cross entropy loss function and a cross entropy loss function, an adaptive weight factor is introduced into the adaptive binary cross entropy loss function, and a mask distribution dynamic weight factor is introduced into the cross entropy loss function.

5. The method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to claim 3, wherein: Based on the general background prototype, the background filtering algorithm is used to filter the background point cloud in the task samples of the query set. The specific process of retaining the foreground point cloud includes: The general background prototype is concatenated with the query features of the task samples in the query set, and then a convolution unit is used to filter out the background point cloud in the query features, thereby retaining the foreground point cloud in the task samples.

6. The method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to claim 3, wherein: The specific process of using the prototype contrast learning algorithm to calculate the similarity between the foreground point cloud of the task sample and the average foreground prototype and the common background prototype of all categories in the support set includes: Calculate the average value of the foreground prototypes of all categories in the support set to obtain the average foreground prototype; Expand the dimensions of the average foreground prototype and the general background prototype to be consistent with the dimensions of the query features of the task samples; The cosine similarity between the foreground point cloud in the task sample and the average foreground prototype and the general background prototype of all categories in the support set is calculated respectively. The temperature parameter τ used to adjust the similarity calculation is introduced in the cosine similarity calculation for scaling.

7. The method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to claim 6, wherein: A prototype contrastive learning algorithm is used to calculate the similarity between the foreground point cloud of the task sample and the average foreground prototype and the universal background prototype of all categories in the support set. In the process of extracting the foreground point cloud of the task sample whose similarity with the average foreground prototype of all categories in the support set is higher than a preset threshold, a prototype contrast loss function is used to optimize the learning process. A marginal threshold M is introduced into the prototype contrast loss function to control the similarity gap between positive and negative samples. The positive sample is the average foreground prototype, and the negative sample is the universal background prototype.

8. The method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to claim 6, wherein: The specific process of using the non-metric learning algorithm to perform feature matching and category prediction on the foreground point cloud of the task sample includes: Calculate the cosine similarity between the feature vector of each query point in the foreground point cloud of the task sample and the foreground prototype of each category in the support set, convert it to a negative value and amplify it using an exponential function to obtain the mask logits; The mask logits of each query point in the foreground point cloud of the task sample are converted into a probability distribution using a normalized exponential function, and the prototype category with the largest probability value is selected as the predicted label of the query point.

9. The method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to claim 3, wherein: The backbone network extractor is a dynamic graph convolutional network based on the self-attention mechanism.

10. The method for optimizing few-shot building point cloud semantic segmentation based on general background knowledge based on meta-learning according to claim 7, wherein: The total loss function includes a main loss function, the adaptive background loss function and the prototype contrast loss function, and both the adaptive background loss function and the prototype contrast loss function are provided with weight factors with adjustable proportions.

Images