Feature distribution guidance method based on weak supervision point cloud semantic segmentation

Abstract

The invention provides a feature distribution guidance method based on weak supervision point cloud semantic segmentation. The method comprises the following steps: S1, constructing a description function of semantic feature distribution; and S2, guiding weak supervised learning according to the description function. According to the weak supervision point cloud semantic segmentation task-oriented feature distribution guidance method (DGNet for short), by constraining the feature distribution condition under weak supervision learning, additional supervision information is provided, and the interpretability of a neural network to a prediction result is kept and increased.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud scene understanding under weakly supervised learning, and in particular, to a feature distribution guidance method based on weakly supervised point cloud semantic segmentation. Background Art

[0002] As a basic task for 3D scene understanding, the point cloud semantic segmentation task is widely used in various application fields such as autonomous driving, embodied intelligence, and virtual reality. Despite significant achievements in solving the disorder and unstructured nature of point clouds, the point cloud semantic segmentation task is still annotation-intensive, which hinders its further expansion in big data and large-scale models. To this end, the academic community is actively exploring the implementation of the point cloud semantic segmentation task in the form of weakly supervised learning. However, due to the extreme lack of supervision signals, learning point cloud semantic segmentation under sparse annotations faces huge challenges.

[0003] In recent years, researchers have made a lot of efforts to introduce additional supervision under sparse annotations. Existing work can be roughly divided into several representative paradigms: 1) Contrastive learning / perturbation consistency: Contrastive loss or consistency constraints are imposed between the network features of the original point cloud and the perturbed point cloud, respectively, so as to provide additional supervision information. 2) Self-training: The given sparse annotations are used as initial labels to train the segmentation network, and then fine-grained semantic prediction results are obtained on the point cloud scene. The network is continued to be trained by screening reliable prediction results as pseudo-annotations, thereby gradually improving its segmentation quality. 3) Similarity measurement: The supervision signal is transferred from labeled points to unlabeled points by leveraging low-level prior knowledge or network feature similarity.

[0004] Overall, despite some breakthroughs, these constraints are based on heuristic assumptions that ignore the inherent distribution of network features, leading to ambiguous interpretations of predictions.

[0005] The difficulty of solving this problem lies in: how to describe the semantic segmentation feature space under weakly supervised learning and how to strengthen this intrinsic distribution under weakly supervised learning.

[0006] The significance of solving this problem is to provide a function to mathematically describe the distribution of semantic features under supervised learning, which is conducive to exploring the differences and connections between weakly supervised learning models and fully supervised learning models; and to provide additional supervision information for weakly supervised learning based on this function, and use the analyzability of the function to explain the prediction results of the neural network at each point. Summary of the invention

[0007] The present invention provides a feature distribution guidance method based on weakly supervised point cloud semantic segmentation, and provides a feature distribution guidance method for weakly supervised point cloud semantic segmentation tasks (abbreviated as DGNet). By constraining the feature distribution under weakly supervised learning, additional supervision information is provided and the interpretability of the prediction results of the neural network is increased.

[0008] The technical solution of the present invention is as follows:

[0009] The feature distribution guidance method based on weakly supervised point cloud semantic segmentation of the present invention comprises the following steps: S1. constructing a description function of semantic feature distribution; and S2. guiding weakly supervised learning according to the description function.

[0010] Optionally, in the above-mentioned feature distribution guidance method based on weakly supervised point cloud semantic segmentation, the feature distribution under fully supervised learning is fitted by constructing a series of description functions of semantic feature distributions; the appropriate feature distribution description function is judged according to the fitting effect, and in the process of fitting the feature distribution, both training data and test data need to be fitted.

[0011] Optionally, in the above-mentioned feature distribution guidance method based on weakly supervised point cloud semantic segmentation, in step S2, a distribution guidance network is constructed according to the description function of the feature distribution to provide additional supervision information for weakly supervised point cloud semantic segmentation, and the distribution guidance network consists of two branches, namely, a weakly supervised learning branch and a distribution alignment branch.

[0012] Optionally, in the above-mentioned feature distribution guidance method based on weakly supervised point cloud semantic segmentation, the truncated cross entropy loss function needs to replace the conventional partial cross entropy loss function in the weakly supervised learning branch of the distribution guidance network; in the distribution alignment branch of the distribution guidance network, the feature learning process of the segmentation backbone network and the optimization process of the feature distribution description function are carried out iteratively, and a consistency loss function is introduced between the two branches to ensure the stability of the network training process.

[0013] Optionally, in the above-mentioned feature distribution guidance method based on weakly supervised point cloud semantic segmentation, step S2 includes the following steps:

[0014] Step 1. Construct a weakly supervised learning branch: The weakly supervised learning branch optimizes the network parameters only with sparse point annotations, and uses the truncated cross entropy loss function instead of the conventional cross entropy loss function, which is defined as:

[0015]

[0016] Where β∈[0,1] represents the threshold used to truncate the cross entropy loss function. When the predicted category probability of a point exceeds β, the cross entropy loss will be truncated, thereby avoiding overfitting at this point;

[0017] Step 2. Construct the distribution alignment branch: First, the network features are normalized and projected onto the unit hypersphere of the feature space, i.e., v i =norm(F φ (x i )), then the maximum likelihood estimation method is used to define the optimization objective function of the network:

[0018]

[0019] in, and Θ={α c ,κ c ,u c} respectively represent the normalized network embedding, the corresponding clustering results and the parameters of the hybrid model. The concentration parameter κ is fixed as a constant. It has category labels; it uses the nested expectation-maximum algorithm to optimize the three variables in the objective function simultaneously;

[0020] Step 3. A consistency loss is introduced to strengthen the information exchange between the two branches: the consistency loss is applied to the class probability map p of the weakly supervised learning branch and the posterior probability q of the distribution alignment branch in the form of cross entropy:

[0021]

[0022] Without elaborate tuning of the weights, the overall loss function is defined as follows:

[0023]

[0024] Optionally, in the above-mentioned feature distribution guidance method based on weakly supervised point cloud semantic segmentation, the expectation-maximum algorithm is as follows:

[0025] The expected step optimizes Θ and Taking the network features as input conditions, alternately optimize Θ and In order to improve the stability and computational efficiency of the algorithm, the features of the labeled positions of the weakly supervised branch are used to initialize u, and the posterior probability set As a result of clustering The probability assignment of i The definition is as follows:

[0026]

[0027] Posterior probability Not only as a weighting factor, but also used in the loss function of joint optimization;

[0028] The maximum step optimizes φ: between Θ and When fixed, φ is optimized through the back propagation process and the posterior probability Incorporate the optimization objective function of the network and reformulate it into a loss function as follows:

[0029]

[0030] Adding the discriminative loss from metric learning, the definition of the discriminative loss is as follows:

[0031]

[0032] According to the technical solution of the present invention, the beneficial effects produced are:

[0033] This paper designs a framework that directly constrains the distribution of point cloud semantic features to provide richer and more accurate supervision information for weakly supervised learning. Specifically, this paper proposes a feature distribution guidance method (abbreviated as DGNet) for weakly supervised point cloud semantic segmentation tasks, which provides additional supervision information and maintains the interpretability of the neural network's prediction results by constraining the feature distribution under weakly supervised learning.

[0034] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0036] Figure 1 is a flow chart of the feature distribution guidance method based on weakly supervised point cloud semantic segmentation of the present invention;

[0037] Figure 2 is a schematic diagram of a network architecture of the feature distribution guidance method of the present invention;

[0038] Figure 3 It is a posterior probability heat map of each category calculated by the network involved in the method of the present invention according to the feature distribution. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0040] This paper proposes a feature distribution guidance method (DGNet for short) for weakly supervised point cloud semantic segmentation tasks, which provides additional supervision information and increases the interpretability of the neural network's prediction results by constraining the feature distribution under weakly supervised learning.

[0041] This paper proposes a feature distribution guidance method (abbreviated as DGNet) for weakly supervised point cloud semantic segmentation tasks. By constraining the feature distribution under weakly supervised learning, it provides additional supervision information and maintains the interpretability of the neural network to the prediction results, so as to provide additional and accurate supervision information for weakly supervised learning and provide a probabilistic explanation for the network's prediction results according to Bayes' theorem. Without loss of generality, the point cloud representation of weakly supervised learning is Where X l and X u are the points with and without labels respectively. Y is X l The annotation on is a set of category indices. n and m are the number of point clouds and annotated points, respectively. After the point cloud is input to the segmentation head, the network feature F φ (x i ) is projected into the class probability vector p i , where φ is the parameter of the feature extraction network F. Partial cross entropy loss is used in traditional weakly supervised semantic segmentation:

[0042]

[0043] in, Represents y i Category in p i The probability in .

[0044] Working principle 1 of the present invention: The present invention emphasizes the depiction of feature distribution of weak supervision, and thereby provides additional supervision information for weakly supervised learning. Because the theoretical performance upper limit of weakly supervised learning is the performance under full supervision learning, the present invention attempts to model the feature distribution under full supervision and use it as a guide for weakly supervised learning. Modeling the semantic feature space is an open problem, so the present invention proposes to explore it from two aspects: distance metric and distribution pattern. On the one hand, the distance metric reflects the similarity relationship between features, and on the other hand, the distribution pattern determines the clustering result of the features.

[0045] (1) Distance metrics: Here, we consider two common distance metrics in clustering methods, namely Euclidean distance and cosine similarity. For two given feature vectors u and v, the Euclidean distance is ‖uv‖2 and the cosine similarity is The cosine similarity can be regarded as the vector inner product of the two eigenvectors u and v after normalization.

[0046] (2) Distribution model: An intuitive modeling method is the category prototype model. In this model, the average feature vector under each category is called the category prototype, and each feature is assigned a cluster label according to the distance from the category prototype. In addition, the present invention also considers a mixed model. Depending on the difference in the mixed model, it can be divided into a Gaussian mixed model with Euclidean distance as the distance metric and a von Mises-Fisher distribution mixed model with cosine similarity as the distance metric.

[0047] (3) In order to comprehensively evaluate, the present invention performs fitting verification on three classic network architectures, namely the DeepGCN network based on graph structure, the PointNeXt network based on convolution, and the PointTransformer network based on Transformer. For the hybrid model, the present invention uses maximum likelihood estimation to determine the parameters of the hybrid model. Since it is aimed at the semantic segmentation task, the fitting quality is evaluated by the mean intersection over union (mIoU) between the clustering results and the semantic segmentation true value. The performance of the training set and the validation set respectively represent the fitting ability and generalization ability of the feature distribution function. By comparing various possible combinations of feature distribution functions, the present invention finds that the von Mises-Fisher distribution hybrid model with cosine similarity as the distance metric has significant advantages in describing the fully supervised feature space. Therefore, the subsequent steps use the hybrid model to provide supervision information for weakly supervised learning.

[0048] Working Principle 2: The von Mises-Fisher function describes the normalized network characteristics v i =norm(F φ (x i The probability density function of the distribution of )) on the characteristic hypersphere is:

[0049] f(v i |u,κ)=C d (κ)exp(κu T v i ),

[0050] where u represents the average eigenvector of the von Mises-Fisher distribution, κ ≥ 0 represents the parameter controlling the distribution concentration, and C d (κ) represents the normalization constant. For the entire data category set The mixture model of von Mises-Fisher function can be defined as:

[0051]

[0052] in, The set of parameters representing the mixture model of the von Mises-Fisher function. c represents the proportion of the c-th category in the mixed model, and

[0053] like Figure 1 and Figure 2 As shown, the feature distribution guidance method based on weakly supervised point cloud semantic segmentation of the present invention comprises the following steps:

[0054] S1. Construct a description function of the semantic feature distribution.

[0055] By constructing a series of descriptive functions of semantic feature distribution, the feature distribution under fully supervised learning is fitted, and the appropriate feature distribution description function is determined based on the fitting effect.

[0056] In the process of constructing the description function of semantic feature distribution, two aspects are emphasized, namely how to measure the similarity relationship between features and how to model the feature distribution.

[0057] In the process of fitting the feature distribution, it is necessary to fit the features under full supervision. This is because the ultimate goal of weakly supervised learning is to achieve the performance of full annotation under the supervision of sparse annotation, so the feature distribution of full supervised learning can be used to guide weakly supervised learning. In the process of fitting the feature distribution, in order to make the fitting comprehensive, it is necessary to fit both training data and test data. In addition, the quality of the fitting is evaluated by the accuracy of point segmentation.

[0058] S2. Guide weakly supervised learning based on the description function.

[0059] According to the description function of feature distribution, a distribution guidance network is constructed to provide additional supervision information for weakly supervised point cloud semantic segmentation. The distribution guidance network consists of two branches, namely the weakly supervised learning branch and the distribution alignment branch.

[0060] In the weakly supervised learning branch of the distribution guidance network, the truncated cross entropy loss function needs to replace the conventional partial cross entropy loss function to avoid overfitting of the network on sparse annotations. In the distribution alignment branch of the distribution guidance network, the feature learning process of the segmentation backbone network and the optimization process of the feature distribution description function are iterative. In addition, the present invention introduces a consistency loss function between the two branches to ensure the stability of the network training process.

[0061] Among them, S2. guiding weakly supervised learning according to the description function specifically includes the following steps:

[0062] Step 1: Construct a weakly supervised learning branch. This branch optimizes network parameters only with sparse point annotations. Since the annotation ratio is extremely low, it is easy to cause the model to overfit on these limited annotated points. Therefore, the present invention proposes to use a truncated cross entropy loss function instead of the conventional cross entropy loss function, which is defined as:

[0063]

[0064] Where β∈[0,1] represents the threshold used to truncate the cross entropy loss function. When the predicted category probability of a point exceeds β, the cross entropy loss will be truncated, thus avoiding overfitting at this point.

[0065] In addition, by using the average feature vector on the labeled points, a robust initialization can be provided for the subsequent distribution alignment branch, which helps the clustering algorithm to achieve fast and stable convergence. However, due to the sparsity of the annotations, some classes in a scene may not have any labeled points, which hinders the initialization process. Therefore, the present invention additionally maintains a memory library to store the category prototypes of the entire data set. When there is no annotation for a certain class in a scene, the category prototype in the memory library will be used as a supplement for initialization.

[0066] Step 2: Construct a distribution alignment branch. This branch uses the mixture model of the von Mises-Fisher function to guide the feature space under weak supervision learning. First, the network features are normalized and projected onto the unit hypersphere of the feature space, that is, v i =norm(F φ (x i )). Then, the maximum likelihood estimation method is used to define the optimization objective function of the network:

[0067]

[0068] in, and Θ={α c ,κ c ,u c} respectively represent the normalized network embedding, the corresponding clustering results and the parameters of the hybrid model. In order to avoid the long tail problem and simplify the calculation, the concentration parameter κ is fixed as a constant in the implementation process of the present invention. Since the clustering initialization is category-aware, the clustering results is a class label. Although the main optimization goal is to learn the network parameters φ, and Θ also need to be dynamically optimized and updated to provide a more accurate description of the feature space. Therefore, the present invention develops a nested expectation-maximum algorithm to cope with the challenge of optimizing three variables simultaneously in the optimization objective function. Among them, the expectation-maximum algorithm is as follows:

[0069] · Expected step (optimize Θ and ): Take the network features as input conditions and alternately optimize Θ and In order to improve the stability of the algorithm

[0070] Qualitatively and computationally efficient, the present invention uses the features of the labeled positions of the weakly supervised branch to initialize u. The posterior probability set

[0071] As a result of clustering The probability assignment of i The definition is as follows:

[0072]

[0073] Posterior probability Not only is it used as a weighting factor, but it is also used in the loss function of joint optimization. In addition, in the inference phase, It can provide a probabilistic explanation for the prediction.

[0074] Maximum step (optimizing φ): between Θ and When φ is fixed, the present invention optimizes φ through the back propagation process.

[0075] And the posterior probability Incorporate the optimization objective function of the network and reformulate it into a loss function as follows:

[0076]

[0077] In addition, considering the importance of different decision boundaries in the mixture model, the present invention also adds a discriminant loss from metric learning. The definition of the discriminant loss is as follows:

[0078]

[0079] Step 3: A consistency loss is introduced to strengthen the information exchange between the two branches (weakly supervised learning branch and distribution alignment branch). This consistency loss is applied to the class probability map p of the weakly supervised learning branch and the posterior probability q of the distribution alignment branch in the form of cross entropy:

[0080]

[0081] Without elaborate tuning of the weights, the overall loss function is defined as follows:

[0082]

[0083] Experimental results:

[0084] The present invention conducted a test experiment on the indoor point cloud dataset S3DIS to verify the effectiveness of the invention in the weakly supervised semantic segmentation task. DeepGCN, PointNeXt and PointTransformer network architectures were used as backbone networks and compared on the average intersection-over-union evaluation index. In order to comprehensively test the weakly supervised performance, the present invention trained the invented network design under the settings of 0.1% annotation ratio and 0.01% annotation ratio. The verification results are shown in Table 1.

[0085] Table 1 Verification results of the test experiment of the present invention on the indoor point cloud dataset S3DIS

[0086]

[0087]

[0088] From the above, it can be seen that an important advantage of the present invention is that it provides good interpretability for the prediction. Figure 3 The posterior probability heat map of each category calculated by the network of the present invention according to the feature distribution is shown. Figure 3 As shown, the prediction results can be explained probabilistically using Bayes' theorem by comparing the posterior probabilities of each category. Categories with higher posterior probabilities under the mixture model of the von Mises-Fisher function are more likely to be predicted by the network. Therefore, Figure 3 The darker areas enlarged in the figure are more likely to be predicted by the network as the current category, which provides a probabilistic explanation for the network prediction.

[0089] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A feature distribution guidance method based on weakly supervised point cloud semantic segmentation, characterized in that: The following steps are involved: S1. constructing a description function of the semantic feature distribution; and S2. Guide weakly supervised learning based on the description function.

2. The feature distribution guidance method based on weakly supervised point cloud semantic segmentation according to claim 1, characterized in that: By constructing a series of description functions of the semantic feature distribution, the feature distribution under fully supervised learning is fitted; the appropriate feature distribution description function is judged according to the fitting effect, and in the process of fitting the feature distribution, both training data and test data need to be fitted.

3. The feature distribution guidance method based on weakly supervised point cloud semantic segmentation according to claim 1, characterized in that: In step S2, a distribution guidance network is constructed according to the description function of the feature distribution to provide additional supervision information for weakly supervised point cloud semantic segmentation, and the distribution guidance network consists of two branches, namely a weakly supervised learning branch and a distribution alignment branch.

4. The feature distribution guidance method based on weakly supervised point cloud semantic segmentation according to claim 3, characterized in that: In the weakly supervised learning branch of the distribution guidance network, the truncated cross entropy loss function needs to replace the conventional partial cross entropy loss function; in the distribution alignment branch of the distribution guidance network, the feature learning process of the segmentation backbone network and the optimization process of the feature distribution description function are iteratively performed, and a consistency loss function is introduced between the two branches to ensure the stability of the network training process.

5. The feature distribution guidance method based on weakly supervised point cloud semantic segmentation according to claim 2, characterized in that: Step S2 includes the following steps: Step 1. Construct a weakly supervised learning branch: The weakly supervised learning branch optimizes the network parameters only by sparse point annotations, and uses a truncated cross entropy loss function instead of the conventional cross entropy loss function, which is defined as: Where β∈[0,1] represents the threshold used to truncate the cross entropy loss function. When the predicted category probability of a point exceeds β, the cross entropy loss will be truncated, thereby avoiding overfitting at this point. Step 2. Construct the distribution alignment branch: First, the network features are normalized and projected onto the unit hypersphere of the feature space, i.e., v i =norm(F φ (x i )), then the maximum likelihood estimation method is used to define the optimization objective function of the network: in, and Θ={α c ,κ c ,u c } respectively represent the normalized network embedding, the corresponding clustering results and the parameters of the hybrid model. The concentration parameter κ is fixed as a constant. It has category labels; it uses the nested expectation-maximum algorithm to optimize the three variables in the objective function simultaneously; Step 3. A consistency loss is introduced to strengthen the information exchange between the two branches: the consistency loss is applied to the class probability map p of the weakly supervised learning branch and the posterior probability q of the distribution alignment branch in the form of cross entropy: Without elaborate tuning of the weights, the overall loss function is defined as follows:

6. The feature distribution guidance method based on weakly supervised point cloud semantic segmentation according to claim 5, characterized in that: in, The expectation-maximum algorithm is as follows: The expected step optimizes Θ and Taking the network features as input conditions, alternately optimize Θ and In order to improve the stability and computational efficiency of the algorithm, the features of the labeled positions of the weakly supervised branch are used to initialize u, and the posterior probability set As a result of clustering The probability assignment of i The definition is as follows: Posterior probability Not only as a weighting factor, but also used in the loss function of joint optimization; The maximum step optimizes φ: between Θ and When fixed, φ is optimized through the back propagation process and the posterior probability Incorporate the optimization objective function of the network and reformulate it into a loss function as follows: Adding the discriminative loss from metric learning, the definition of the discriminative loss is as follows:

Images