Generative Devices and Methods Based on Point Cloud Probability Distribution Learning

Abstract

This invention discloses a point cloud probability distribution learning-based generation method, comprising: adding an improved mapping network to the generator to learn the probability distribution of the point cloud; combining style transfer to change the probability distribution of the point cloud, thereby changing the probability distribution of the generated point cloud positions; and constructing a complete classifier and discriminator for 3D point cloud generation based on a local neighborhood Transformer layer. This invention changes the probability distribution of point cloud positions through style transfer, making the generated point cloud more uniform and regular, thus improving the efficiency of model training. In the classifier and discriminator, a novel local neighborhood point cloud Transformer for point cloud learning is used. After obtaining the neighborhood through farthest point sampling and K-nearest neighbor classification algorithms, this invention calculates local self-attention and designs a pyramid-shaped network structure to better capture the contextual information in the point cloud.

Description

Technical Field

[0001] This invention relates to the field of data generation technology, and more specifically, to a generative model and method based on point cloud probability distribution learning. Background Technology

[0002] Research on 3D point cloud generation models is one of the hot topics in the field of computer vision. Two-dimensional images are not very effective at representing depth information between multiple objects in the real world, and are unsuitable for scenarios requiring both depth and localization information. From robot navigation to autonomous vehicles, augmented reality to healthcare, the challenges of using 3D datasets are gradually being addressed. Among various data models, raw point clouds are becoming popular as a compact, isomorphic representation, capable of capturing complex details of the environment. A 3D point cloud can be considered as an unordered, irregular set of points collected from the surface of an object, each point consisting of a Cartesian coordinate, along with other additional information such as surface normal estimates and RGB color values.

[0003] In recent years, point cloud generation models based on Generative Adversarial Networks (GANs) have emerged. GANs, proposed by Goodfellow et al. in 2014, are a generative model that has become a hot research area in artificial intelligence. Compared to two-dimensional images, where pixels are arranged in a regular grid, three-dimensional shapes are represented by points in a continuous three-dimensional space without a common structure. Therefore, 3D GANs often produce point clouds with significant inhomogeneity, where points are often unevenly distributed on the surface of the shape. Given that the number of points in each point cloud is fixed (current GANs typically have 2048), this inhomogeneity causes points to concentrate in one region, resulting in sparse or even hollow areas in other regions.

[0004] Without proper regularization, more points will cluster at the geometric center of the object or at the junctions of different semantic parts, resulting in a highly uneven shape in the point cloud. Furthermore, point cloud data is an unordered collection of points containing 3D coordinate information, which is insensitive to data order. The same point cloud data model can have multiple different storage orders, making traditional 3D GANs inefficient in training and generating point clouds. Summary of the Invention

[0005] To address the problems of existing technologies, embodiments of the present invention provide a generative model and method based on point cloud probability distribution learning. The technical solution is as follows:

[0006] On the one hand, a generative method based on point cloud probability distribution learning is provided, including:

[0007] Step 1): Add an improved mapping network to the generator to learn the probability distribution of the point cloud;

[0008] Step 2): Combine style transfer to change the probability distribution of the point cloud, thereby changing the probability distribution of the generated point cloud locations;

[0009] Step 3): Based on the local neighborhood Transformer layer, construct a complete classifier and discriminator for 3D point cloud generation.

[0010] Furthermore, the generator includes a mapping network and a tree-structured generation network; the mapping network is used to train and learn the probability distribution of the point cloud; the tree-structured generation network is used to train and learn the basic features of the point cloud.

[0011] Furthermore, the tree structure generation network consists of 7 feature tree modules, each of which includes an adaptive instance normalization module, a branching module, and a graph convolution module.

[0012] Further, step 1) specifically includes:

[0013] The mapping network is improved by outputting the results of the first four fully connected layers to the first layer of the tree structure generator network. Each fully connected layer has a dimension of 96, which is used to train and learn the probability distribution of the point cloud.

[0014] The feature vectors are trained through a fully connected layer to make their probability distribution more uniform. Then, they are mapped to each feature tree module through an affine transformation module. An adaptive instance normalization module is used to ensure that the point cloud distribution generated in each layer of the tree structure generation network is affected by the probability distribution of the feature vectors. The affine transformation module is a non-linear upsampling module that maps the dimension of the feature vectors to the corresponding dimension of each feature tree module.

[0015] Furthermore, the tree structure generation network performs branching and graph convolution operations at each layer to generate points for the next layer; all points generated in previous layers are stored and appended to the tree of the current layer. The tree starts from the root node, splits into child nodes through branching operations, and modifies the node values ​​through graph convolution operations; the branching module is used to increase the total number of points during the point cloud generation process, similar to upsampling in two-dimensional convolution.

[0016] Furthermore, in the generator, a different degree of branching is used for each layer, and 2048 points are obtained after the last layer of branching operation.

[0017] Further, step 2) specifically includes:

[0018] The AdaIN module combines the feature vector y learned through the mapping network with the basis vector x in the tree-structured generative network, aligning the mean and variance of the 3D point cloud position features to the mean and variance of the feature vector, thereby changing the distribution of the generated point cloud. The formula is shown below:

[0019]

[0020] Where σ(x) and σ(y) are the variances of the basis vectors and feature vectors, respectively, and μ(x) and μ(y) are the means of the basis vectors and feature vectors, respectively. StyleGAN generates two-dimensional images. The adaptive instance normalization module aligns the mean and variance of the generated image features to the features of the latent code passed through the mapping network to change the local features of the image. The grayscale and color information of the image controls the features of the two-dimensional image, while the features of the three-dimensional point cloud are controlled by the positional distribution of the point cloud.

[0021] Furthermore, step 3) specifically includes:

[0022] Local Neighbor Transformer Layer: After obtaining the local neighborhood using farthest point sampling and K-nearest neighbor classification algorithms, self-attention is calculated for points within the neighborhood; a reasonable K value is set so that different neighborhoods can overlap, further strengthening the relationship between neighborhoods; in the branch calculating weights and calculating the transformed features W... v Add position code δ above:

[0023]

[0024] The center point x obtained by sampling from the farthest point i A set of points within a local neighborhood (K nearest neighbors). Therefore, self-attention is applied within the local neighborhood of the center point of each farthest point.

[0025] Encoder: The encoder feeds the input coordinates into three stacked local neighborhood Transformer layers, where rich semantic feature representations of each point are learned within the local neighborhood. Then, a global Transformer layer is used, where no neighborhood partitioning is performed, so the value of K is set to 1. Finally, the output features are generated. In the local neighborhood Transformer network framework, taking an input of 2048 points as an example, the downsampling target number of points in each layer during the encoding stage is [512, 128, 64, 1], and the value of K in each layer is set to [16, 16, 32, 1]. The computation in the local neighborhood significantly reduces the number of model parameters, achieving O(n).

[0026] Classification: Classify point cloud data into N c There are several object categories; the feature vectors output by the encoder are input into two feedforward neural networks (LBRDs), and finally, a linear layer predicts the final classification score, with the class with the highest score being determined as the class label.

[0027] On the other hand, a generative model based on point cloud probability distribution learning is provided, including a generator; the generator includes a mapping network and a tree structure generative network; the mapping network is used to train and learn the probability distribution of the point cloud; the tree structure generative network is used to train and learn the basic features of the point cloud.

[0028] Furthermore, the tree structure generation network consists of 7 feature tree modules, each of which includes an adaptive instance normalization module, a branching module, and a graph convolution module.

[0029] The beneficial effects of the technical solution provided by the embodiments of the present invention are as follows:

[0030] (1) The probability distribution of point cloud locations is learned by training the latent code sampled from the Gaussian distribution through the mapping network. The distribution of the point cloud is aligned with the probability distribution of the latent code through style transfer, and the generated point cloud is more structured, which can improve the training efficiency of the model.

[0031] (2) The probability distribution learned by training the fully connected layers in the mapping network is more uniform, which can solve the problem of uneven point cloud generation.

[0032] (3) This invention designs a new local neighborhood self-attention Transformer layer for 3D point cloud processing. This layer is invariant to the arrangement of the point cloud and solves the problem of unevenness of the point cloud. Therefore, it is essentially suitable for point cloud processing. At the same time, the idea of ​​local attention in this layer makes our model have lower space complexity.

[0033] (4) Based on the local neighborhood self-attention layer, we constructed a high-performance local neighborhood Transformer classifier and discriminator, and applied it to the point cloud generation network. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of a generation method based on point cloud probability distribution learning in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the F-TreeGAN network structure in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the local neighborhood Transformer layer in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the classifier model in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0040] This invention provides a generation method based on point cloud probability distribution learning, see [link to relevant documentation]. Figure 1 ,include:

[0041] S100: Add an improved mapping network to the generator to learn the probability distribution of the point cloud;

[0042] S200: Combines style transfer to change the probability distribution of point clouds, thereby changing the probability distribution of the generated point cloud locations.

[0043] S300: Based on the local neighborhood Transformer layer, construct a complete classifier and discriminator for 3D point cloud generation.

[0044] Specifically, this invention trains a latent code in the mapping network to learn the probability distribution of the point cloud. The network outputs a feature vector with a uniform probability distribution. In the tree-structured generative network, each layer aligns the distribution of the point cloud with the distribution of the feature vector through style transfer, thereby changing the positional distribution of the generated point cloud and making the generated point cloud more structured. This not only solves the problem of uneven distribution but also improves the efficiency of generating realistic point clouds compared to previous 3D GAN point cloud generation methods. Furthermore, by changing the initial latent code, the local features of the generated point cloud can be altered.

[0045] In this embodiment, further explanation is provided as follows:

[0046] Tree structure generative network based on point cloud probability distribution learning

[0047] This embodiment proposes a tree-structured deep network based on TreeGAN, Feature-TreeGAN (abbreviated as F-TreeGAN). Inspired by StyleGAN, a mapping network is added to the generator to learn the probability distribution of point cloud locations. Style transfer is used to change the location distribution of the point cloud, and by changing the initial latent code in the mapping network, the effect of changing the local features of the point cloud can be achieved.

[0048] The generator mainly consists of two parts: a mapping network and a tree structure generation network. This embodiment improves the mapping network by outputting the results of the first four fully connected layers to the first layer of the tree structure generation network. Each fully connected layer has a dimension of 96, used to train and learn the probability distribution of the point cloud. The tree structure generation network consists of seven feature tree modules (FeatureTree-Block, or FT-Block for short). Each module has three parts: adaptive instance normalization (AdaIN), branching, and graph convolution (GraphConv), used to train and learn the basic features of the point cloud. The feature vectors are trained through fully connected layers to make their probability distribution more uniform, and then mapped to each feature tree module (FT-Block) through an affine transformation. The AdaIN module ensures that the point cloud distribution generated in each layer of the tree structure generation network is influenced by the probability distribution of the feature vectors. The Affine module performs non-linear upsampling, mapping the feature vector dimension to the corresponding dimension of each feature tree module. The network structure is as follows: Figure 2 As shown.

[0049] In each layer of the tree-structured generative network, branching and graph convolution operations are performed to generate points for the next layer. All points generated in previous layers are stored and appended to the tree of the current layer. The tree starts from the root node, splits into child nodes through branching operations, and modifies the node values ​​through graph convolution operations. The branching module is used to increase the total number of points during point cloud generation, similar to upsampling in 2D convolution. In the generator, a different degree of branching is used for each layer, resulting in 2048 points after the last layer's branching operation. Unlike traditional graph convolution, which updates its value based on the values ​​of its neighboring nodes, TreeGAN's Tree Graph Convolution (TreeGCN) updates its value based on the values ​​of each vertex's ancestors, introducing a tree structure to GCN. Since TreeGCN updates the current node's value based on the values ​​of its ancestor nodes, it can use ancestor information to improve the representational power of features.

[0050] Incorporating style transfer to alter the probability distribution of point clouds

[0051] To address the non-uniformity issue arising from tree-based networks, we employ methods from style transfer. The primary method in style transfer is Adaptive Instance Normalization (AdaIN), which aligns the mean and variance of the content image features to the mean and variance of the style image, thus achieving style transfer. In this embodiment, the AdaIN module combines the feature vector y learned through the mapping network with the basis vector x in the generator network. This aligns the mean and variance of the 3D point cloud location features to the mean and variance of the feature vector, effectively changing the distribution of the generated point cloud. The formula is shown below:

[0052]

[0053] Where σ(x) and σ(y) are the variances of the basis vectors and feature vectors, respectively, and μ(x) and μ(y) are the means of the basis vectors and feature vectors, respectively. StyleGAN generates two-dimensional images. The AdaIN module aligns the means and variances of the generated image features to the features of the latent code passed through the mapping network, thereby changing the local features of the image. The features of the two-dimensional image are controlled by the grayscale and color information of the image, while the features of the three-dimensional point cloud are controlled by the positional distribution of the point cloud. The latent code in StyleGAN learns the grayscale and color information of the image, while the latent code in the mapping network of this embodiment learns the probability distribution of the point cloud. Furthermore, the feature vector probability distribution obtained through training the fully connected layers of the mapping network is more uniform, making the point cloud distribution more uniform after aligning with the probability distribution of the feature vectors. This solves the problem of non-uniformity caused by the ancestor points of the tree structure. In addition, during the generation process, the AdaIN module is added to each layer of the tree-type generation network, making the generated point cloud more structured and improving the training efficiency of the network model.

[0054] Classifier based on local neighborhood Transformer

[0055] Self-attention is suitable for feature extraction from point clouds because point clouds are essentially collections of discrete positional information in three-dimensional space. Inspired by the concept of convolution in images, we use farthest point sampling and K-nearest neighbor classification algorithms to obtain local neighborhoods, and then calculate self-attention for points within those neighborhoods. We need to set a reasonable value for K so that different neighborhoods can overlap, further strengthening the relationship between them. This is discussed in the branch calculation of weights and the calculation of the transformed feature W. v Add position code δ above:

[0056]

[0057] The center point x obtained by sampling from the farthest point iA set of points within a local neighborhood (K nearest neighbors). Therefore, self-attention is applied within the local neighborhood of the center point of each farthest point. The local neighborhood Transformer layer structure is as follows: Figure 3 As shown.

[0058] The input is a set of feature vectors x and their associated 3D coordinates p. The local neighborhood Transformer layer facilitates information exchange between feature vectors within their local neighborhoods. Simultaneously, overlapping regions between neighborhoods also promote information exchange. Ultimately, all data points generate new feature vectors as the output. We employ a multi-head attention mechanism. [2] This allows the model to simultaneously focus on information from different representation subspaces. In summary, to make the Transformer suitable for point cloud processing, we improved it by constructing a local neighborhood self-attention layer with the Transformer at its core. This layer integrates a self-attention layer, linear projection, and a residual connection. This layer reduces the number of points and increases the feature dimension, laying the groundwork for building a pyramid-shaped network structure. A Maxpooling operator was added to the MLP before the output, which to some extent addresses the issue of point cloud permutation invariance.

[0059] Based on the local neighborhood Transformer layer, we constructed a complete classifier for 3D point cloud generation. For example... Figure 4 As shown.

[0060] The encoder's purpose, specifically the local neighborhood Transformer layer, is twofold: firstly, to encode the input points into a new high-dimensional feature vector; and secondly, to reduce the cardinality of the point set. We use this as the foundation of the encoder, constructing it with a pyramidal structure. The encoder feeds the input coordinates into three stacked local neighborhood Transformer layers, learning rich semantic feature representations for each point within the local neighborhood. This is followed by a global Transformer layer, where no neighborhood partitioning is performed, so the value of K is set to 1, ultimately generating the output features. In the local neighborhood Transformer network framework, taking an input of 2048 points as an example, the downsampling target number of points per layer in the encoding stage is [512, 128, 64, 1], and the K value in each layer is set to [16, 16, 32, 1]. Calculating within the local neighborhood significantly reduces the number of model parameters, achieving O(n).

[0061] Classification: Details of the classification network, such as Figure 4 As shown. The point cloud data is classified into N... cWe categorize objects into several classes (e.g., airplane, table, chair). The feature vectors output from the encoder are fed into two feedforward neural networks (LBRDs), which combine linear layers, batch normalization layers, and dropout layers, with the dropout rate set to 0.5. The linear layers then predict the final classification score, and the class with the highest score is determined as the class label.

[0062] In this embodiment, the probability distribution of point cloud locations is learned by training a mapping network from the latent code sampled from the Gaussian distribution. Style transfer is used to align the distribution of the point cloud with the probability distribution of the latent code, resulting in a more structured point cloud and improving the training efficiency of the model.

[0063] By mapping the probability distribution learned through fully connected layers in the network, a more uniform probability distribution can be achieved, which can solve the problem of uneven point cloud generation.

[0064] This invention designs a novel local neighborhood self-attention Transformer layer for 3D point cloud processing. This layer is invariant to the arrangement of the point cloud and solves the problem of point cloud non-uniformity, thus it is essentially suitable for point cloud processing. At the same time, the idea of ​​local attention in this layer makes our model have lower space complexity.

[0065] Based on the local neighborhood self-attention layer, we constructed a high-performance local neighborhood Transformer classifier and applied it to the point cloud generation network.

[0066] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A generative method based on point cloud probability distribution learning, characterized in that, include: Step 1): Add an improved mapping network to the generator to learn the probability distribution of the point cloud; Step 2): Combine style transfer to change the probability distribution of the point cloud, thereby changing the probability distribution of the generated point cloud locations; Step 3): Based on the local neighborhood Transformer layer, construct a complete classifier and discriminator for 3D point cloud generation; The generator includes a mapping network and a tree-structured generation network; the mapping network is used to train and learn the probability distribution of the point cloud; the tree-structured generation network is used to train and learn the basic features of the point cloud. The tree structure generation network consists of 7 feature tree modules, each of which includes an adaptive instance normalization module, a branching module, and a graph convolution module. Step 1) specifically refers to: The mapping network is improved by outputting the results of the first four fully connected layers to the first layer of the tree structure generator network. Each fully connected layer has a dimension of 96, which is used to train and learn the probability distribution of the point cloud. The feature vectors are trained through a fully connected layer to make their probability distribution more uniform. Then, they are mapped to each feature tree module through an affine transformation module. An adaptive instance normalization module is used to ensure that the point cloud distribution generated in each layer of the tree structure generation network is affected by the probability distribution of the feature vectors. The affine transformation module is a non-linear upsampling module that maps the dimension of the feature vectors to the corresponding dimension of each feature tree module. The tree structure generation network performs branching and graph convolution operations at each layer to generate points for the next layer; all points generated in previous layers are stored and appended to the tree in the current layer. The tree starts from the root node, splits into child nodes through branching operations, and modifies the values ​​of the nodes through graph convolution operations. The branching module is used to increase the total number of points during the point cloud generation process, similar to upsampling in 2D convolution; In the generator, a different degree of branching is used for each layer, and 2048 points are obtained after the last layer of branching operation; Step 2) specifically refers to: The AdaIN module combines the feature vector y learned through the mapping network with the basis vector x in the tree-structured generative network, aligning the mean and variance of the 3D point cloud position features to the mean and variance of the feature vector, thereby changing the distribution of the generated point cloud. The formula is shown below: Where σ(x) and σ(y) are the variances of the basis vectors and feature vectors, respectively, and μ(x) and μ(y) are the means of the basis vectors and feature vectors, respectively. StyleGAN generates two-dimensional images. The adaptive instance normalization module aligns the mean and variance of the generated image features to the features of the latent code passed through the mapping network to change the local features of the image. The grayscale and color information of the image controls the features of the two-dimensional image, while the features of the three-dimensional point cloud are controlled by the positional distribution of the point cloud.

2. The method as described in claim 1, characterized in that, Step 3) specifically refers to: Local Neighborhood Transformer Layer: After obtaining the local neighborhood using farthest point sampling and K-nearest neighbor classification algorithms, self-attention is calculated for the points within the neighborhood; a reasonable K value is set so that different neighborhoods can overlap, further strengthening the relationship between neighborhoods; in the branch calculating weights and calculating the transformed features W... v Add position code δ above: The center point x obtained by sampling from the farthest point i A set of points in a local neighborhood; therefore, self-attention is applied within the local neighborhood of the center point of each farthest point sampled; Encoder: The encoder feeds the input coordinates into three stacked local neighborhood Transformer layers, learning rich semantic feature representations for each point within the local neighborhood. Then, a global Transformer layer is used, where no neighborhood partitioning is performed; therefore, the value of K is set to 1. Finally, the output features are generated. In the local neighborhood Transformer network framework, the input consists of 2048 points. During the encoding stage, the target number of downsampled points per layer is [512, 128, 64, 1], and the value of K in each layer is set to [16, 16, 32, 1]. The computation in the local neighborhood significantly reduces the number of model parameters, achieving O(n). Classification: Classify point cloud data into N c There are several object categories; the feature vectors output by the encoder are input into two feedforward neural networks (LBRDs), and finally, a linear layer predicts the final classification score, with the class with the highest score being determined as the class label.

3. A device based on point cloud probability distribution learning, characterized in that it comprises: A generator, comprising a mapping network and a tree-structured generation network; The mapping network includes multiple fully connected layers for processing latent vectors to learn the probability distribution of point clouds, and maps feature vectors to various feature tree modules in the tree structure generation network through an affine transformation module. The tree structure generation network includes multiple feature tree modules, each feature tree module including an adaptive instance normalization module, a branching module and a graph convolution module, used to generate structural features of point clouds layer by layer; The apparatus based on point cloud probability distribution learning is used to execute the generation method based on point cloud probability distribution learning as described in any one of claims 1-2.

Images