Three-dimensional point cloud feature extraction method based on adaptive multi-class Gaussian mixture model

Abstract

The invention discloses a three-dimensional point cloud feature extraction method based on an adaptive multi-class Gaussian mixture model, and the method comprises the steps: firstly designing a double-dependence hierarchical GMM learning model McGMM which supports the dynamic distribution of Gaussian mixture, and generating a more compact and precise anisotropic GMM representation; secondly, providing a point cloud feature extraction strategy of a self-adaptive multi-class GMM prototype, training an McGMM model through source data, obtaining various centroid GMM features, decoding the centroid GMM features into class prototype expressions, storing the class prototype expressions in a GMM library, and for any target sample, rapidly determining a pseudo label by executing probability density estimation; further, multi-class data with pseudo labels are fed back to the network, and the deep GMM features and class prototypes of a GMM library are subjected to consistency association, so that purification and refinement of the point cloud features are realized, and the representation performance of the point cloud features is effectively improved; and finally, establishing a joint regularization mechanism to optimize the learning process of the network.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular to a three-dimensional point cloud feature extraction method based on an adaptive multi-class Gaussian mixture model. Background Art

[0002] With the development of artificial intelligence technology, deep learning has been successfully applied in the field of computer vision, showing a strong ability to learn features. At present, deep learning has gradually expanded to the field of 3D vision, which can make use of massive data resources to fully explore the deep features of three-dimensional shapes, and has achieved remarkable research results around tasks such as three-dimensional object recognition, semantic segmentation and three-dimensional retrieval. However, unlike 2D visual images, three-dimensional models have characteristics such as sparseness, disorder and complexity. Most three-dimensional point cloud deep learning methods rely on the evaluation of spatial relationships, resulting in practical problems such as sensitivity to outliers and sparse distribution. Summary of the invention

[0003] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provides a three-dimensional point cloud feature extraction method based on an adaptive multi-class Gaussian mixture model.

[0004] The technical solution of the present invention is: a three-dimensional point cloud feature extraction method based on an adaptive multi-class Gaussian mixture model, wherein the adaptive multi-class Gaussian mixture model is provided with an encoder, a specific type decoder and a multi-class general decoder, wherein the encoder is a Pointnet network, and the specific type decoder and the multi-class general decoder are double-dependent hgmm modules consisting of a tree structure and a cross attention module, wherein the tree structure consists of four fully connected layers, and the cross attention module consists of mlp_1, mlp_2, mlp_3 and softmax() with three layers of parameter sharing, and training is performed in the following steps in sequence:

[0005] Step 1. Input the multi-class training samples divided by category into the encoder Pointnet to extract the initial point cloud model features, and then extract the GMM node features based on the decoder of the specific category;

[0006] Step 2. Adaptively allocate the mixing number of the Gaussian mixture model using Gaussian distribution weights;

[0007] Step 3. Calculate the GMM prototype representation c k ;

[0008] Step 4. Calculate pseudo labels for the input multi-class training samples and feed them into the adaptive multi-class Gaussian mixture model. Optimize the GMM features through the multi-class universal decoder and impose consistency constraints on the GMM prototype to obtain the predicted labels.

[0009] Step 5. Establish a joint objective loss function to obtain an optimized adaptive multi-class Gaussian mixture model.

[0010] The step 1 is specifically as follows:

[0011] Given a multi-class training dataset D = {x i}, x i is the i-th input sample, each sample x i Through the Pointnet network, the corresponding model feature F is extracted; the model feature F is input into a specific type decoder, and the Gaussian nodes are split through the tree structure. In the lth layer, given node V j and its eigenvalue z j ∈R d , and the given node V j The eigenvalue z j Other node characteristics z of the parent except the direct father node p Splice(), input to mlp_1, mlp_2, mlp_3 of three-layer parameter sharing, output parent weight pass The parent weight Convert to probability weight; at the same time, the given node V j The eigenvalue z j With sibling node feature z in the same level s Splice(), input to mlp_1, mlp_2, mlp_3 of three-layer parameter sharing, output sibling weights pass Weight the brothers Converted to probability weight; GMM node features are z j =∑ s∈S(j) ∑ p∈P(j) ω β,j z s .ω α,j z p Further updates.

[0012] The step 2 takes the Gaussian mixture Θ and the three-dimensional point cloud {X_i, z_i} as input, and outputs the weight of each Gaussian mixture;

[0013] Assume that the model consists of J Gaussian mixture Θ j Composition, calculate each point x i and Gaussian mixture Θ j The binary association between ij , to find the Gaussian mixture Θ j The maximum likelihood of each Gaussian mixture Θ j The weight ω j It is calculated by point density estimation according to the following formula:

[0014] p ij =π j B(z i |Θ j )

[0015]

[0016] Where N is the number of 3D point clouds, Θ represents the Gaussian mixture, j is the index of the Gaussian mixture, and p ij Indicates that the i-th point belongs to the j-th Gaussian mixture Θ j The probability of j represents the weight of the j-th Gaussian mixture.

[0017] The step 3 is as follows:

[0018] Using the updated GMM node features, calculate the prototype GMM features for each class separately Stored in a GMM library, denoted as G b {C k}

[0019]

[0020] Among them, S k represents the set of labeled samples in the kth category, f φ Represents the GMM features transformed by the decoder from the model features F.

[0021] The step 4 is specifically as follows:

[0022] 4.1 Given multi-class data T = {(x1, y1), (x2, y2)…(x t ,y t )},y i ∈{1,2,…C} is the corresponding category label, with x i As input, based on the prototype C in the GMM library k Calculate the pseudo label y by point probability estimation c ;

[0023]

[0024] Where: x i represents the i-th point in a point cloud model, N represents the total number of point clouds, G k represents the prototype of the kth class, g j Corresponding to the jth Gaussian component of the GMM prototype, p(X|G k ) represents the probability that the point cloud model X belongs to the Kth category;

[0025] 4.2 will be with the pseudo label yc The associated unclassified data is fed back to the network, and then passes through the encoder and multi-class general decoder to output the refined GMM feature f φ (z t ), and the GMM library prototype C k Use consistency constraints:

[0026]

[0027] Among them, d(,) represents the similarity distance (Euclidean distance), z t is the x in the latent space t The characteristics of C t is the GMM prototype, f φ () represents a universal decoder, and C represents the total number of categories;

[0028] The step 5 is specifically as follows:

[0029] 5.1 established a joint regularization function to facilitate the entire learning process. The chamfer distance optimizes the network parameters. This distance adjusts the mean and covariance of each Gaussian distribution in the Gaussian mixture model to ensure that the output Gaussian mixture model represents the maximum probability of the original point cloud:

[0030]

[0031] Where X2 represents the new shape generated from the original point cloud X1, obtained by interpolating vectors from the learned GMM parameters at any desired resolution;

[0032] 5.2 Use the KL divergence of the standard normal distribution to optimize the point feature representation in the latent space:

[0033]

[0034] Among them, X represents a point cloud model, Θ represents the GMM mixed model, L bg (X, Θ) represents the log-likelihood of point cloud X generated by Θ, Represents the mean and variance of the latent space obtained after the encoder, that is, the mean variance of the point cloud feature z distribution, represents the KL divergence of the latent space distribution in the standard normal distribution, and γ is the regularity coefficient;

[0035] 5.3 Using the prototype loss function L p To refine the GMM features, by minimizing the latent space f φ The intra-class changes in are used to enhance consistency, so that the learned GMM features are closer to the class center:

[0036]

[0037] Where D() is the Euclidean distance, is the centroid GMM feature;

[0038] 5.4 The adaptive multi-class Gaussian mixture model is optimized by the joint objective function L to ensure that the entire network can perform backpropagation:

[0039] L=λL e +γL d +ωL p

[0040] where λ, γ and ω are the regularization coefficients.

[0041] The present invention changes the method that most existing hierarchical GMM methods only consider unidirectional feature aggregation and use a fixed number of distributions. First, a dual-dependency hierarchical GMM learning model McGMM is designed, which supports dynamic distribution of Gaussian mixtures and generates more compact and accurate anisotropic GMM representations; secondly, a point cloud feature extraction strategy of adaptive multi-class GMM prototypes is proposed. The McGMM model is trained by source data to obtain the centroid GMM features of each class, and decoded into class prototype representations and stored in the GMM library. For any target sample, its pseudo-label can be quickly determined by performing probability density estimation; then, the multi-class data with pseudo-labels is fed back to the network, and the deep GMM features are consistently associated with the class prototypes of the GMM library to achieve the purification and refinement of point cloud features, effectively improving the representation performance of point cloud features. The present invention establishes a joint regularization mechanism to optimize the learning process of the network, and performs multi-task learning such as point cloud segmentation and classification on public data sets for verification. Experiments show that compared with other existing methods, the point cloud deep features extracted by the present invention have significant discriminability and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the overall structure of an embodiment of the present invention.

[0043] Figure 2 It is a schematic diagram of shapes reconstructed by extracting class prototypes layer by layer from HGMM according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The overall structure of the three-dimensional point cloud feature extraction method based on the adaptive multi-class Gaussian mixture model of the present invention is as follows: Figure 1As shown. The adaptive multi-class Gaussian mixture model is provided with an encoder, a specific type decoder and a multi-class general decoder. The encoder is a Pointnet network. The specific type decoder and the multi-class general decoder are double-dependent hgmm modules consisting of a tree structure and a cross attention module. The tree structure consists of four fully connected layers. The cross attention module consists of mlp_1, mlp_2, mlp_3 and softmax() with three layers of parameter sharing. The training is carried out in the following steps:

[0045] Step 1. Input the multi-class training samples divided by class into the encoder Pointnet, extract the initial point cloud model features, and then extract the GMM node features based on the decoder of a specific category, as follows:

[0046] Given a multi-class training dataset D = {x i}, x i is the i-th input sample, each sample x i Through the Pointnet network, the corresponding model feature F is extracted; the model feature F is input into a specific type decoder, and the Gaussian nodes are split through the tree structure. In the lth layer, given node V j and its eigenvalue z j ∈R d , and the given node V j The eigenvalue z j Other node characteristics z of the parent except the direct father node p Splice(), input to mlp_1, mlp_2, mlp_3 of three-layer parameter sharing, output parent weight pass The parent weight Convert to probability weight; at the same time, the given node V j The eigenvalue z j With sibling node feature z in the same level s Splice(), input to mlp_1, mlp_2, mlp_3 of three-layer parameter sharing, output sibling weights pass Weight the brothers Converted to probability weight; GMM node features are z j =∑ s∈S(j) ∑ p∈P(j) ω β,j z s .ω α,j z p Further updates.

[0047] Step 2. Use Gaussian distribution weights to adaptively allocate the mixing number of the Gaussian mixture model, which takes the Gaussian mixture Θ and the three-dimensional point cloud {X_i, z_i} as input, and outputs the weight of each Gaussian mixture;

[0048] Assume that the model consists of J Gaussian mixture Θ j Composition, calculate each point x i and Gaussian mixture Θ j The binary association between ij , to find the Gaussian mixture Θ j The maximum likelihood of each Gaussian mixture Θ j The weight ω j It is calculated by point density estimation according to the following formula:

[0049] p ij =π j B(z i |Θ j )

[0050]

[0051] Where N is the number of 3D point clouds, Θ represents the Gaussian mixture, j is the index of the Gaussian mixture, and p ij Indicates that the i-th point belongs to the j-th Gaussian mixture Θ j The probability of j represents the weight of the j-th Gaussian mixture.

[0052] Step 3. Calculate the GMM prototype representation c k :

[0053] Using the updated GMM node features, calculate the prototype GMM features for each class separately Stored in a GMM library, denoted as G b {C k}

[0054]

[0055] Among them, S k represents the set of labeled samples in the kth category, f φ Represents the GMM features transformed by the decoder from the model features F.

[0056] Step 4. Calculate pseudo labels for the input multi-class training samples and feed them into the adaptive multi-class Gaussian mixture model. Optimize the GMM features through the multi-class universal decoder and impose consistency constraints on the GMM prototype to obtain the predicted labels, as follows:

[0057] 4.1 Given multi-class data T = {(x1, y1), (x2, y2)…(x t ,yt )},y i ∈{1,2,…C} is the corresponding category label, with x i As input, based on the prototype C in the GMM library k Calculate the pseudo label y by point probability estimation c ;

[0058]

[0059] Where: x i represents the i-th point in a point cloud model, N represents the total number of point clouds, G k represents the prototype of the kth class, g j Corresponding to the jth Gaussian component of the GMM prototype, p(X|G k ) represents the probability that the point cloud model X belongs to the Kth category;

[0060] 4.2 will be with the pseudo label y c The associated unclassified data is fed back to the network, and then passes through the encoder and multi-class general decoder to output the refined GMM feature f φ (z t ), and the GMM library prototype C k Use consistency constraints:

[0061]

[0062] Among them, d(,) represents the similarity distance (Euclidean distance), z t is the x in the latent space t The characteristics of C t is the GMM prototype, f φ () represents a universal decoder, and C represents the total number of categories;

[0063] Step 5. Establish a joint objective loss function to obtain an optimized adaptive multi-class Gaussian mixture model:

[0064] 5.1 established a joint regularization function to facilitate the entire learning process. The chamfer distance optimizes the network parameters. This distance adjusts the mean and covariance of each Gaussian distribution in the Gaussian mixture model to ensure that the output Gaussian mixture model represents the maximum probability of the original point cloud:

[0065]

[0066] Where X2 represents the new shape generated from the original point cloud X1, obtained by interpolating vectors from the learned GMM parameters at any desired resolution;

[0067] 5.2 Use the KL divergence of the standard normal distribution to optimize the point feature representation in the latent space:

[0068]

[0069] Among them, X represents a point cloud model, Θ represents the GMM mixed model, L bg (X, Θ) represents the log-likelihood of point cloud X generated by Θ, Represents the mean and variance of the latent space obtained after the encoder, that is, the mean variance of the point cloud feature z distribution, represents the KL divergence of the latent space distribution in the standard normal distribution, and γ is the regularity coefficient;

[0070] 5.3 Using the prototype loss function L p To refine the GMM features, by minimizing the latent space f φ The intra-class changes in are used to enhance consistency, so that the learned GMM features are closer to the class center:

[0071]

[0072] Where D() is the Euclidean distance, is the centroid GMM feature;

[0073] 5.4 The adaptive multi-class Gaussian mixture model is optimized by the joint objective function L to ensure that the entire network can perform backpropagation:

[0074] L=λL e +γL d +ωL p

[0075] where λ, γ and ω are the regularization coefficients.

[0076] The model constructed by the embodiment of the present invention was verified using a public data set and compared with the existing technology. The results are shown in Table 1.

[0077] Table 1 Classification performance comparison on ModelNet40

[0078]

[0079] In the experiment, the present invention conducted two classification tests; one was to directly use the GMM prototype to determine the category of the input sample through probability density estimation, and the other was to input the GMM features obtained from McGMM into a multi-category SVM classifier for classification. The GMM prototype represents the distribution properties and obtains similar results to PointNet. The deep GMM features obtained from McGMM combine local geometric properties and global distribution information, which is nearly 2.0% higher than PointNet and PointNet++, and achieves comparable performance to KpConv and PAConv. Compared with the GMM deep learning method, McGMM has achieved state-of-the-art results in segmentation and generation tasks, and also has significant advantages in shape classification.

[0080] In addition, the present invention uses three types of prototypes, namely airplane, chair and guitar, to reconstruct a 3D point cloud model with N=2048 points on the shapenet dataset, and extracts the corresponding class prototypes from coarse to fine layers in the hierarchical GMM structure, corresponding to Gaussian mixtures of J=8, 16, 32 and 64, respectively. Figure 2 .from Figure 2 It can be seen that the class prototype is divided more and more finely from coarse to fine, and the semantic structure of each class is well revealed.

Claims

1. A three-dimensional point cloud feature extraction method based on an adaptive multi-class Gaussian mixture model, characterized in that: The adaptive multi-class Gaussian mixture model is provided with an encoder, a specific type decoder and a multi-class general decoder, wherein the encoder is a Pointnet network, the specific type decoder and the multi-class general decoder are double-dependent hgmm modules consisting of a tree structure and a cross attention module, wherein the tree structure consists of four fully connected layers, and the cross attention module consists of mlp_1, mlp_2, mlp_3 and softmax() with three layers of parameter sharing, and is trained in the following steps: Step 1. Input the multi-class training samples divided by category into the encoder Pointnet to extract the initial point cloud model features, and then extract the GMM node features based on the decoder of the specific category; Step 2. Adaptively allocate the mixing number of the Gaussian mixture model using Gaussian distribution weights; Step 3. Calculate the GMM prototype representation c k ; Step 4. Calculate pseudo labels for the input multi-class training samples and feed them into the adaptive multi-class Gaussian mixture model. Optimize the GMM features through the multi-class universal decoder and impose consistency constraints on the GMM prototype to obtain the predicted labels. Step 5. Establish a joint objective loss function to obtain an optimized adaptive multi-class Gaussian mixture model.

2. The three-dimensional point cloud feature extraction method based on adaptive multi-class Gaussian mixture model learning according to claim 1 is characterized in that The step 1 is specifically as follows: Given a multi-class training dataset D = {x i }, x i is the i-th input sample, each sample x i Through the Pointnet network, the corresponding model feature F is extracted; the model feature F is input into a specific type decoder, and the Gaussian nodes are split through the tree structure. In the lth layer, given node V j and its eigenvalue z j ∈R d , and the given node V j The eigenvalue z j Other node characteristics z of the parent except the direct father node p Splice(), input to mlp_1, mlp_2, mlp_3 of three-layer parameter sharing, output parent weight pass The parent weight Convert to probability weight; at the same time, the given node V j The eigenvalue z j With the sibling node feature z in the same level s Splice(), input to mlp_1, mlp_2, mlp_3 of three-layer parameter sharing, output sibling weights pass Weight the brothers Converted to probability weight; GMM node features are z j =∑ s∈S(j) ∑ p∈P(j) ω β， j z s ·ω α，j z p Further updates.

3. The three-dimensional point cloud feature extraction method based on adaptive multi-class Gaussian mixture model learning according to claim 2 is characterized in that The step 2 takes the Gaussian mixture Θ and the three-dimensional point cloud {X_i, z_i} as input, and outputs the weight of each Gaussian mixture; Assume the model consists of a Gaussian mixture Θ j Composition, calculate each point x i and Gaussian mixture Θ j The binary association between ij , to find the Gaussian mixture Θ j The maximum likelihood of each Gaussian mixture Θ j The weight ω j It is calculated by point density estimation according to the following formula: p ij =π j B(z i |I j ) Where N is the number of 3D point clouds, Θ represents the Gaussian mixture, j is the index of the Gaussian mixture, and p ij Indicates that the i-th point belongs to the j-th Gaussian mixture Θ j The probability of j represents the weight of the j-th Gaussian mixture.

4. The three-dimensional point cloud feature extraction method based on adaptive multi-class Gaussian mixture model learning according to claim 3 is characterized in that The step 3 is as follows: Using the updated GMM node features, calculate the prototype GMM features for each class separately Stored in a GMM library, denoted as G b (C k } Among them, S k represents the set of labeled samples in the kth category, f φ Represents the GMM features transformed by the decoder from the model features F.

5. The three-dimensional point cloud feature extraction method based on adaptive multi-class Gaussian mixture model learning according to claim 4 is characterized in that The step 4 is specifically as follows: 4.1 Given multi-class data T = {(x1, y1), (x2, y2)…(x t ,y t )},y i ∈{1, 2, ...C} is the corresponding category label, with x i As input, based on the prototype C in the GMM library k Calculate the pseudo label y by point probability estimation c ; Where: x i represents the i-th point in a point cloud model, N represents the total number of point clouds, G k represents the prototype of the kth class, g j Corresponding to the jth Gaussian component of the GMM prototype, p(X|G k ) represents the probability that the point cloud model X belongs to the Kth category; 4.2 will be with the pseudo label y c The associated unclassified data is fed back to the network, and then passes through the encoder and multi-class general decoder to output the refined GMM feature f φ (z t ), and the GMM library prototype C k Use consistency constraints: Among them, d(,) represents the similarity distance (Euclidean distance), z t is the x in the latent space t The characteristics of C t is the GMM prototype, f φ () represents a universal decoder, and C represents the total number of categories.

6. The three-dimensional point cloud feature extraction method based on adaptive multi-class Gaussian mixture model learning according to claim 5 is characterized in that The step 5 is specifically as follows: 5.1 established a joint regularization function to facilitate the entire learning process. The chamfer distance optimizes the network parameters. This distance adjusts the mean and covariance of each Gaussian distribution in the Gaussian mixture model to ensure that the output Gaussian mixture model represents the maximum probability of the original point cloud: in, X2 represents the new shape generated from the original point cloud X1, obtained by interpolating vectors from the learned GMM parameters at any desired resolution; 5.2 Use the KL divergence of the standard normal distribution to optimize the point feature representation in the latent space: Among them, X represents a point cloud model, Θ represents the GMM mixed model, L bg (X, Θ) represents the log-likelihood of point cloud X generated by Θ, Represents the mean and variance of the latent space obtained after the encoder, that is, the mean variance of the point cloud feature z distribution, represents the KL divergence of the latent space distribution in the standard normal distribution, and γ is the regularity coefficient; 5.3 Using the prototype loss function L p To refine the GMM features, by minimizing the latent space f φ The intra-class changes in are used to enhance consistency, so that the learned GMM features are closer to the class center: Where D() is the Euclidean distance, is the centroid GMM feature; 5.4 The adaptive multi-class Gaussian mixture model is optimized by the joint objective function L to ensure that the entire network can perform backpropagation: L=λL e +γL d +ωL p where λ, γ and ω are the regularization coefficients.

Images