Small sample point cloud classification method based on nystrom and determinant sampling

Abstract

The application discloses a small sample point cloud classification method based on Nystrom and determinant sampling, which models and filters a diversity subset from an unlabeled training set through a sampling method improved based on DPP and Nystrom, and simultaneously combines a Chinese restaurant model in a Dirichlet process as a prior distribution, breaks the original label classification, and makes the generated training data of each class more representative and reasonable. Then, the meta-training set is input into a PointNet++ model and trained in cooperation with a MAML algorithm. The trained model is fine-tuned by using a small amount of target sample data, and the adaptation to a new task is quickly completed. The method mines the intra-class similarity and diversity of data, makes the data set distribution more uniform and reasonable, reduces the dependence of the model on the data volume, improves the generalization ability of the model, has good transferability, and is suitable for completing small sample point cloud classification work.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computers, and relates to a three-dimensional point cloud data classification method, in particular to a small sample point cloud classification method based on Nystrom and determinant sampling. BACKGROUND

[0002] Three-dimensional point cloud data has the advantages of being able to provide rich geometric, shape and scale information of three-dimensional objects or scenes, etc., is an important form of three-dimensional spatial information, and is widely used in scene reconstruction, unmanned driving and smart city fields. Manual labeling of point cloud data will consume a lot of manpower and resources, so the point cloud classification method is one of the research hotspots in the fields of computer graphics and 3D computer vision. However, due to the characteristics of insufficient scale and uneven distribution of point cloud data sets, the direct use of deep learning to process point cloud data has the problem of weak network generalization ability.

[0003] Small sample learning under two-dimensional image classification has also received considerable attention in recent years, but the corresponding three-dimensional point cloud classification problem has not been meaningfully explored. Since the labeled three-dimensional point cloud data set is usually small and has only a limited number of categories, the commonly used point cloud classification model can only use a few thousand samples of a few categories for training, which will lead to poor quality three-dimensional features, and the separation degree of their clusters is not as good as their visual counterparts. The clustering effect of pre-trained three-dimensional features is poor and shows a large intra-class distance. The conversion method suitable for two-dimensional image data cannot reduce the bias problem of three-dimensional point cloud data, especially the classification of three-dimensional point cloud data under small sample conditions is a challenging task. SUMMARY

[0004] In view of the deficiencies of the prior art, the application provides a small sample point cloud classification method based on Nystrom and determinant sampling, which uses fewer sample features for network training to overcome the negative impact of uneven distribution of data sets on classification effect.

[0005] The small sample point cloud classification method based on Nystrom and determinant sampling specifically includes the following steps:

[0006] Step 1, constructing an approximate value of a similarity kernel matrix based on Nystrom;

[0007] Collecting n point cloud samples with known labels, constructing a similarity kernel matrix M n×n , L ij is the i-th row and j-th element in the matrix M, used to represent the similarity between sample i and sample j, extracting a column subset from the kernel matrix M for deriving an approximate estimate of the kernel matrix:

[0008]

[0009] Where W∈R m×m Let S ∈ R be the similarity matrix of the set of sample points extracted from the kernel matrix M. (n-m)×m Let B represent the matrix of similarities between all sample points and non-sample points, where B ∈ R. (n-m)×(n-m) Let B be the similarity matrix of the non-sample point set. By calculating matrices W and S, the estimated value of matrix B can be obtained, thus deriving the approximate estimation kernel matrix.

[0010] As a preferred approach, the two-dimensional Euclidean distance between sample i and sample j is defined as the similarity between them.

[0011] Step 2, Determinant Point Process Sampling

[0012] The approximate estimated kernel matrix obtained in step 1 Perform eigenvalue decomposition into v n and λ n , where v n λ represents the eigenvalue. n Represent the eigenvectors and set an approximate estimation kernel matrix. The probability of each element being sampled is The sampling yields a submatrix |V|, where the elements of submatrix |V| are the feature vectors of the sampled elements. Then, the samples are processed according to probability. The eigenvectors of the submatrix |V| are extracted from the submatrix, and the elements corresponding to the eigenvectors are placed into the subset Y, thus completing the determinant point sampling process.

[0013] Step 3: Divide the meta-training set

[0014] Using the Chinese restaurant model of the Dirichlet process as the prior distribution, the subset Y obtained in step 2 is clustered to obtain the meta-training set for training the classification model.

[0015] Step 4: Classification Model Training

[0016] Construct a PointNet++ model and input the meta-training set obtained in step 3 into it. Train the PointNet++ model using the MAML algorithm, including inner loop learning and outer loop learning. In the inner loop learning, parameter updates are performed using multiple tasks. Then, in the outer loop learning, these parameters are used to compute the loss function value of the query set under the same task, and the network parameters are updated using the losses from all tasks.

[0017] Step 5: Model fine-tuning and point cloud classification

[0018] A small amount of point cloud samples with label classification targets is input into the trained PointNet++ model of step 4, and the model is fine-tuned. Then the point cloud data to be classified is input into the fine-tuned PointNet++ model to complete the small sample point cloud classification.

[0019] The present application has the following beneficial effects:

[0020] The method models and filters a group of diversity subsets from the unlabeled training set based on the improved sampling method of DPP and Nystrom, and combines the Chinese restaurant model in the Dirichlet process as a prior distribution to form a meta-training set, excavates the intra-class similarity and diversity of data, makes the generated training data of each class more representative and reasonable, improves the classification accuracy, successfully extends the MAML algorithm from two-dimensional image classification to three-dimensional point cloud classification, trains the PointNet++ model by using the MAML algorithm, finds a group of initialization parameters sensitive to all tasks. Finally, a few samples are fine-tuned to quickly adapt to new tasks. The method has good generalization ability and transferability, reduces the dependence of the model on the amount of data, improves the generalization ability of the model, and has very wide application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 It is a flowchart of the small sample point cloud classification method based on Nystrom and determinant sampling;

[0022] Figure 2 It is a comparison chart of uniform sampling and DPP sampling in the embodiment;

[0023] Figure 3 It is a clustering flowchart of the Chinese restaurant model in the embodiment;

[0024] Figure 4 It is a MAML-PointNet++ algorithm framework chart in the embodiment;

[0025] Figure 5 It is a visualization result based on DPP and Nystrom sampling in the embodiment; DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described below in combination with embodiments and drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the described embodiments, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0027] As Figure 1As shown, a small sample point cloud classification method based on Nystrom and determinant sampling, specifically comprising the following steps:

[0028] Step 1, constructing an approximate value of a similarity kernel matrix based on Nystrom;

[0029] Nystrom refers to an approximate method for estimating the entire similarity kernel matrix with partial sample features, which generates a low-rank approximation of the original matrix by extracting a column subset from the original matrix. Collect n point cloud samples with known labels, construct a similarity kernel matrix M n×n :

[0030]

[0031] L ij is the i-th row and j-th element in the matrix M, which is the Euclidean distance between samples i and j in two-dimensional space, used to represent the similarity between samples i and j. By Nystrom method, a column subset is extracted from the kernel matrix M, which is used to derive an approximate estimate of the kernel matrix:

[0032]

[0033] where W m×m is the similarity matrix of the sample point set extracted from the kernel matrix M, S (n-m)×m is a matrix composed of the similarity between all sample points and non-sample points, and B (n-m)×(n-m) is the similarity matrix of the non-sample point set. By calculating the matrices W, S, the estimated value of the matrix B can be represented, and the approximate estimate of the kernel matrix is derived as

[0034] Step 2, determinant point process sampling

[0035] Determinantal Point Process (DPP) calculates the probability of each element being selected into the subset through the determinant of the kernel matrix, thereby simplifying the complex probability calculation. Since the subset distribution given by DPP is negatively correlated with the probability of selecting two similar elements, the more dissimilar elements are more likely to be sampled simultaneously, therefore, under normal circumstances, the points obtained by DPP sampling are far apart and have a wider sample coverage, while uniform sampling is uniform, random and independent, so it is easy to produce clustering, and the points within the cluster are close to each other, i.e. the obtained result will have multiple points with close distances, reducing the diversity of sampling, as shown in Figure 2 The standard DPP sampling algorithm is as follows:

[0036]

[0037] According to the standard DPP algorithm, the approximate estimation kernel matrix obtained in step 1 is sampled The eigenvalues of the approximate estimation kernel matrix are first decomposed into v n and λ n , where v n represents the eigenvalue, and λ n represents the eigenvector, and the approximate estimation kernel matrix is set as The probability of sampling each element is The sub-matrix |V| is obtained by sampling, and the elements in the sub-matrix |V| are the eigenvectors of the sampled elements, and then the elements in the sub-matrix |V| are extracted according to the probability The eigenvectors in the sub-matrix |V| are extracted from the sub-matrix, and the elements corresponding to the eigenvectors are placed in the subset Y, and the determinant point process sampling is completed.

[0038] Step 3, dividing the meta-training set

[0039] The subset Y obtained in step 2 is clustered using the Chinese restaurant model of the Dirichlet process as the prior distribution to obtain the meta-training set for training the classification model. The Dirichlet process is a stochastic process used in Bayesian non-parametric data modeling, and the Chinese restaurant model is an interpretation of the Dirichlet process. The specific description of the model is as follows: it is assumed that there are an infinite number of tables in a Chinese restaurant, and each table can seat an infinite number of customers. The first customer chooses the first table, and each subsequent new customer will choose an occupied table with a probability proportional to the number of customers on the occupied table, or choose an empty table with a certain probability. When all customers have been seated, the customers seated at the same table are considered to be in the same group, and all customers are divided into several groups. The clustering task of point cloud data using the Chinese restaurant model can be compared to the above dining process. Each point cloud sample is regarded as a customer, and each clustering category is regarded as a table. After the first point cloud sample is divided into the first category, each subsequent point cloud sample is divided into an existing clustering category with a probability proportional to the number of samples in the current clustering category, or is divided into a new category with a certain probability. As the point cloud data grows, it automatically learns whether to add new categories, and eventually forms several point cloud clusters.

[0040] Step 4, classification model training

[0041] A PointNet++ model is constructed, and the meta-training set obtained in step 3 is input into the model. The PointNet++ model is trained by the MAML algorithm, including inner loop learning and outer loop learning. In the inner loop learning, multi-task parameter updating is used, and then in the outer loop learning, the parameters are used to calculate the loss function value of the query set under the same task, and the network parameters are updated using the loss of all tasks, which can help the model quickly converge on new tasks, as shown in Figure 4 .

[0042] Step 5, model fine-tuning and point cloud classification

[0043] A small amount of point cloud samples with labels as classification targets are input into the PointNet++ model trained in step 4 to fine-tune the model. Then, the point cloud data to be classified is input into the fine-tuned PointNet++ model to complete the small sample point cloud classification.

[0044] To illustrate the advantages of the method, comparative experiments are performed using point cloud data from the ModelNet40 dataset. 30 classes of sample data are randomly selected for model training tasks in steps 1-4, and the remaining 10 classes of sample data are used for model fine-tuning in step 5 and classification testing.

[0045] To more intuitively highlight the comparison effect of DPP sampling and Nystrom simplified sampling, the time consumption, cluster quantity and subset error of the two sampling methods are counted to evaluate the difference in subset quality generated by the two methods, and the results are shown in Table 1, Figure 5

[0046]

[0047] Table 1

[0048] Where Q represents the proportion of extracting column subsets when constructing an approximate kernel matrix, T represents the time consumption of sampling, N represents the cluster quantity of the final dataset, and error represents the difference between the subsets obtained by the two methods. In this embodiment, four proportions of Q, i.e. 1%, 5%, 10% and 20%, are selected for comparative experiments to explore the influence of different values of Q on the operation time, the cluster quantity of the final dataset and the error. As can be seen from Table 1, directly sampling the original point cloud samples using DPP will result in excessive time consumption, while the method of generating an approximate estimation matrix by Nystrom and then sampling by the determinant point process can simplify the calculation, greatly reduce the operation amount and improve the experimental efficiency. The experiment shows that the time consumption of Nystrom sampling increases with the increase of the proportion of extracting column subsets when constructing an approximate kernel matrix, while the error of the subset obtained by DPP sampling gradually decreases. When Q = 10%, compared with Q = 20%, the time consumption of sampling is shortened by half, but the error is not much different. In the subsequent classification test experiment, Q = 10% is set.

[0049] ​Based on the PointNet++ model constructed in step 4, the classification accuracy of the model MAML-PointNet++*(DPP) trained by using DPP sampling data and the model MAML-PointNet++*(Nystrom) trained by using DPP and Nystrom improved sampling data, and the model obtained by using three training methods of MAML algorithm (MAML-PointNet++), data preprocessing MAML algorithm (MAML-PointNet++*) and zero training (PointNet++) after different fine-tuning is compared, as shown in Table 2:

[0050]

[0051] Table 2

[0052] Wherein 10-way, 5-shot represents that there are 10 unknown categories in each test batch, and there are 5 training samples in each category, and 10-way, 10-shot represents that there are 10 unknown categories in each test batch, and there are 10 training samples in each category. As can be seen from Table 2, the accuracy of the method under different conditions is stably improved. In 10-way, 5-shot, the Nystrom method is flat with MAML-PointNet++*, and the DPP method is improved by 1.14% than MAML-PointNet++*. In 10-way, 10-shot, the Nystrom method is improved by 0.14% than MAML-PointNet++*, and the DPP method is improved by 1.05% than MAML-PointNet++*. As can be seen, the method can make the training data in each class of the data set more representative and reasonable by optimizing the sampling, and the classification accuracy under small sample data is improved.

[0053] In order to further explore the influence of the number of sample categories N in the meta test set obtained by clustering in step 3 and the number of samples K in each category on the classification accuracy, the present application uses ModelNet40 data set and MAML-PointNet++*(DPP) model to carry out comparative experiment, and the experimental results are shown in Table 3 and Table 4:

[0054] K 2 4 6 8 10 Accuracy 68.05 84.67 87.26 90.37 93.58

[0055] Table 3

[0056] N 2 4 6 8 10 Accuracy 95.36 92.33 89.34 87.78 86.04

[0057] Table 4

[0058] The experimental results show that the accuracy of small sample classification decreases as N increases, because the samples in the test set need to be divided into more categories, making classification more difficult. Conversely, as K increases, the accuracy of small sample classification will improve.

Claims

1. A small sample point cloud classification method based on Nystrom and determinant sampling, characterized in that: The method comprises the following steps: Step 1, constructing an approximate value of a similarity kernel matrix based on Nystrom; n unlabeled point cloud samples are collected, and a similarity kernel matrix M e R n×n A column subset is extracted from the kernel matrix M for deriving an approximate estimation kernel matrix: where W ∈ R m×m , denotes the similarity matrix of the set of sample points extracted from the kernel matrix M, S ∈ R (n-m)×m , denotes the matrix of similarity between all sample points and non-sample points, B ∈ R (n-m)×(n-m) is the similarity matrix of the set of non-sample points; the matrices W and S are calculated and the approximate estimated kernel matrix Step 2, sampling a determinant point process approximate estimation kernel matrix derived from step 1 performing determinant point process sampling to obtain a subset Y; Step 3, dividing a meta-training set Using a Chinese restaurant model of Dirichlet process as a prior distribution, the subset Y obtained in step 2 is clustered to obtain a meta-training set for training of a classification model; Step 4, training of a classification model A PointNet++ model is constructed, and the meta-training set obtained in step 3 is input into the PointNet++ model; the PointNet++ model is trained through a MAML algorithm; Step 5, model fine-tuning and point cloud classification A small amount of point cloud samples with labels as classification targets are input into the PointNet++ model trained in step 4, and the model is fine-tuned; then, point cloud data to be classified is input into the fine-tuned PointNet++ model, and small-sample point cloud classification is completed.

2. The method of claim 1, wherein the method is based on Nystrom and determinant sampling of small sample point clouds. A two-dimensional space Euclidean distance between two samples is used as similarity, and a similarity kernel matrix is constructed.

3. The small sample point cloud classification method based on Nystrom and determinant sampling of claim 1 or 2, wherein: In step 1 the approximate estimation kernel matrix is derived The proportion of the column subset extracted from the kernel matrix M is 10%.

4. The method of claim 1, wherein the method is based on Nystrom and determinant sampling for small sample point cloud classification. Approximating kernel matrices The process of sampling a determinant point process is: s2.1, approximating the kernel matrix performing eigenvalue decomposition into v n and λ n where v n denotes the eigenvectors and λ n denotes the eigenvalues; s2.2, setting an approximate estimation kernel matrix The probability of each element being sampled is A sub-matrix |V| is sampled, and elements in the sub-matrix |V| are eigenvectors of the sampled elements; s2.3, according to the probability Extract the eigenvectors in the submatrix |V| from the submatrix, and put the elements corresponding to the eigenvectors into the subset Y, complete the determinant point process sampling.

5. The method of claim 1, wherein: The process of clustering point cloud data using the Chinese restaurant model is as follows: the first point cloud sample is divided into the first category, and each subsequent point cloud sample is divided into an existing category with a probability proportional to the number of samples in the current clustering category, or is divided into a new category with a certain probability; with the growth of point cloud data, whether a new category is needed is automatically learned, and the number of new categories is unlimited; finally, all point cloud samples form several point cloud clusters.

6. The method of claim 1, wherein: The MAML algorithm comprises inner loop learning and outer loop learning; in the inner loop learning, parameters are updated using multiple tasks, and then in the outer loop learning, the parameters are used to calculate a loss function value of a query set under the same task, and network parameters are updated using the loss of all tasks.

Images