A three-dimensional point cloud shape recognition method based on multi-view features

Abstract

The application belongs to the technical field of three-dimensional point cloud processing, and relates to a three-dimensional point cloud shape recognition method based on multi-view features, comprising the following steps: 1) obtaining point cloud data and corresponding classification labels; 2) pre-processing the original point cloud to obtain data-enhanced point cloud and multi-view depth maps; 3) constructing a classification model comprising a point cloud branch and a projection branch; in the point cloud branch, point clouds with similar semantic content as the sampling points are clustered into the same class, and multi-head self-attention is used to calculate the local attention within each cluster to capture long-distance dependence relationships of the point cloud; in the projection branch, multiple depth images of different views are subjected to a shallow convolutional network to generate discriminative features; 4) training the classification model; and 5) completing three-dimensional point cloud shape recognition classification; the application combines point cloud local features and global depth map features, captures long-distance dependence relationships, and can efficiently realize point cloud shape recognition classification accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional point cloud processing, and particularly relates to a three-dimensional point cloud shape recognition method based on multi-view features. BACKGROUND

[0002] Three-dimensional point cloud shape recognition technology aims to automatically detect, analyze and recognize the shape features of objects from three-dimensional point clouds, and has a wide application space and development prospect. In the fields of industrial automation and robot technology, three-dimensional point cloud shape recognition can help robots and automation systems better understand and manipulate objects, and improve production efficiency and quality. Unlike traditional two-dimensional images, point clouds are composed of the original coordinates of points in three-dimensional space, which are usually obtained by scanning. Due to the irregularity of point clouds, traditional convolutional neural networks (CNN) face challenges in processing such data, because CNN is usually designed to process two-dimensional images with fixed grid structures. In order to overcome this challenge, researchers have developed various methods to process point clouds, which can be divided into two categories according to the form of processing point clouds.

[0003] One method is to convert point clouds into formats that can be used by convolutional neural networks, such as voxels or projection images. Voxel-based methods convert point clouds into three-dimensional grids (voxels) and calculate features within each voxel. However, due to high memory requirements, this method is limited in its application to high-resolution point clouds. Projection-based methods use convolutional neural networks to extract image features and retain as many local features as possible by rendering views of a given three-dimensional shape. Projection-based methods often lose a large number of point cloud features during data processing due to excessive compression in one direction. In addition, these projection methods usually utilize mature two-dimensional deep neural network models such as ResNet and ViT. Another method is to directly process point clouds without any conversion. This type of method focuses on extracting local features by exploring new designs of local aggregation operators, but it is difficult to capture interactions between distant but similar points in three-dimensional space.

[0004] Most current methods focus on a single modality, such as relying on 3D point clouds or 2D images, ignoring the complementarity between different modalities. Considering that a single modality may not be sufficient to cover enough information for 3D shape recognition, it is necessary to use the information captured by 2D projection images to enhance 3D point cloud shape analysis, and make full use of the advantages of each modality. SUMMARY

[0005] To solve the above technical problems, the present application provides the following technical scheme: a three-dimensional point cloud shape recognition method based on multi-view features, comprising the following steps:

[0006] Step S1, obtaining a public classification dataset of three-dimensional point clouds and corresponding classification labels;

[0007] Step S2, data preprocessing; the point cloud in the data set obtained in step S1 is enhanced, then the point cloud obtained by the enhancement processing is projected, and a plurality of multi-view depth projection maps are generated by selecting v view angles;

[0008] Step S3, constructing a classification network structure; sequentially constructing a point cloud branch module and a projection branch module, then the point cloud local features obtained by the point cloud branch module are processed by maximum pooling, and the multi-view global features obtained by the projection branch module are spliced, the spliced features are aggregated by a classifier, and finally the prediction result of three-dimensional shape classification is obtained;

[0009] Step S4, training a classification model; using the sample data set obtained in step S2 to train the classification model constructed in step S3, using a cross-entropy loss function to obtain a loss value, and using a stochastic gradient descent method to update the parameters in the model until the loss value tends to be stable, and obtaining a trained classification model;

[0010] Step S5, completing the test: using the trained classification model in step S4 to complete the inference of the test point cloud data set, and obtaining the inference result of the point cloud data.

[0011] Preferably, in step S2, the specific operation of the enhancement processing includes: first, randomly scaling the point cloud; second, randomly shaking the point cloud.

[0012] Preferably, in step S3, the point cloud branch module is constructed in the following manner: the point cloud branch module is composed of four different stages, each stage includes a local embedding module and a specific number d of attention layers; the structure of each attention layer is consistent; at the beginning of each stage, the input point cloud is first processed by the local embedding module; the output of the local embedding module is then sent to multiple attention layers, in which the points cloud with similar semantic content are dynamically clustered into the same class using a balanced hierarchical clustering, and the local attention of each cluster is calculated using a multi-head self-attention mechanism to capture the long-distance dependence relationship of the point cloud; after four stages of feature extraction, the global context information of the point cloud local features is extracted.

[0013] More preferably, in step S3, the local embedding module is implemented by enhancing the local neighborhood of a specific point; receiving an input point coordinate and a point cloud feature wherein represents the embedding dimension of the i-1 layer, and N represents the number of input point clouds;

[0014] First, the farthest point sampling operation is performed to obtain the centroid point feature Then, the original point is queried using K-nearest neighbor and the adjacent features are extracted

[0015] To further integrate the local neighborhood information, the center point feature f i is merged with each neighboring feature f j ; the calculation of the local neighborhood feature is as follows:

[0016]

[0017] In formula (1), RP(x, k) is a matrix formed by repeating the vector x k times, k is the neighborhood size, f j -f i is used to evaluate the feature difference between the neighboring points relative to the centroid, f j represents the neighboring point feature, f i represents the center point feature, and ||·|| represents the concatenation operation.

[0018] To enhance the model's ability to perceive point cloud features within the local region, the features processed by RP(x, k) are concatenated with the original features, and the result is input into a convolutional layer to generate the output feature f' of the local embedding module; this process is defined as:

[0019]

[0020] In formula (2) and formula (3), mlp1 and mlp2 are composed of a convolutional layer, batch normalization, and an activation function, respectively, and max represents the max-pooling operation, which is used to aggregate the features of local blocks.

[0021] More preferably, in the step S3, the attention layer is composed of a multi-head self-attention module stacked by κ layers and a standard feedforward network; in the κth layer, given a feature vector X κ-1 , a new feature X κ is obtained by the following operation:

[0022]

[0023]

[0024] In formula (4) and formula (5), LN represents layer normalization; the output of the MSA module is sent to the FFN module after being processed by a layer normalization, and the FFN module contains two fully connected layers, the first layer expands the dimension from C to rC to effectively capture richer feature representations; the second layer then compresses the dimension back to C to achieve efficient computation.

[0025] Preferably, in the step S3, the projection branch module is constructed by a shallow convolutional network, which is used to extract multi-view global features from the depth projection map obtained in the step S2; the convolutional network includes four basic block layers, each of which is composed of two convolutional blocks and a pooling layer, and each convolutional block is composed of a convolutional layer, a batch normalization layer and an activation layer; the channel number and kernel size of the convolutional operation are dynamically adjusted according to the configuration parameters to adapt to the feature extraction requirements of different levels.

[0026] More preferably, in the step S3, a balanced hierarchical clustering method is used to realize dynamic clustering of the point cloud; specifically including:

[0027] For the input point cloud feature vector where N represents the number of point cloud features, and d represents the dimension of the point cloud feature. First, a linear projection operation is performed to map the original feature vector to a new feature space and further reconstruct three feature vectors Q, K and V, which are used for subsequent clustering operations; then are divided into two clusters of equal size (m), and the two cluster centroids c1 and c2 are calculated; for each point q in the feature vector Q i , the distance ratio to the two centroids is calculated to determine its cluster membership; through a sorting operation, the points in the point cloud feature vector Q are sorted according to their distance ratio to the centroid and are assigned to two clusters and The above balanced binary clustering operation can be summarized as:

[0028]

[0029] In formula (6), dist(x, y) represents the Euclidean distance between x and the centroid y in the feature space, and is the cluster to which these points are assigned, and argsort is the sorting operation that returns the sorted index value;

[0030] The average value of each cluster is used to update the cluster centroid; the balanced binary clustering iteratively updates the cluster centroid and the membership of each sample. In the l-th layer, Q is grouped into 2 l subsets Each subset has equal size |Q i | = N / 2 l ; then, K and V are divided into and Each subset shares the same index as Q i , ensuring that in the self-attention mechanism, Q i , K i and V icorrectly corresponding; finally, in each subset (Q i , K i , V i ) adopts a standard multi-head self-attention, and outputs Y i = MSA (Q i , K i , V i ) ; finally, the union set of all subsets is merged while maintaining its original order, ensuring that the extracted features can be combined in the order of the original point cloud, preserving the spatial information of the point cloud.

[0031] Preferably, in step S2, the projection module takes a depth map as input, where H and W represent the height and width of the depth image respectively, and v represents the number of depth maps; the depth image D is processed step by step by the network in step S3 and generates a feature map where represents the feature dimension of the i-th layer in the projection module; in order to capture distinctive features in different receptive fields, the feature map D i is divided into n bin feature sets where n ∈ {1, 2, 4, 8, 16, 32} represents the number of bins into which each scale feature map is divided, and each bin represents a part of the feature map at a certain scale; by dividing the feature map into multiple bins, the model can focus on features of different sizes; maximum pooling is performed on the bin features of each scale to obtain the most discriminative features of each scale feature map Finally, all the features from B1 to B4 are spliced as the output of the entire projection module.

[0032] Preferably, in step S3, the classifier is composed of a multi-layer perceptron, a batch normalization layer, an activation layer, a Dropout layer and a multi-layer perceptron.

[0033] The beneficial effects of the present application are:

[0034] 1. The present application improves the understanding ability and generalization ability of the model by combining point cloud and corresponding multi-view depth map, and can provide accurate three-dimensional object recognition and analysis in practical application.

[0035] 2. The present application clusters similar semantic content in the point cloud by balancing hierarchical clustering and calculates the local attention within each cluster, capturing long-range dependencies of the point cloud, which helps the model to understand and distinguish the shape and structure of complex objects, and has strong robustness. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of a three-dimensional point cloud shape recognition method based on multi-view features according to the present application.

[0037] Figure 2 is a multi-view depth map generated by an embodiment of the present application;

[0038] Figure 3 is a classification model network structure of an embodiment of the present application;

[0039] Figure 4 is a balanced hierarchical clustering schematic diagram of an embodiment of the present application;

[0040] Figure 5 is a point cloud branch different stage clustering visualization effect diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0041] The related technologies in the present application will be described in detail below with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0042] As shown in Figures 1 to 5 , the three-dimensional point cloud shape recognition method based on multi-view features of the present embodiment includes the following steps:

[0043] S1, obtaining a public classification data set of three-dimensional point clouds and corresponding classification labels;

[0044] S2, data preprocessing, specifically including:

[0045] S2.1, performing enhancement processing on the point clouds in the data set obtained in step S1, specifically including: performing random scaling on the point clouds. Secondly, random jitter is performed on the point clouds;

[0046] S2.2, projecting the point clouds obtained in step S2.1, and selecting v view angles to generate multi-view depth projection maps;

[0047] S3, constructing a classification network structure, specifically including the following steps:

[0048] S3.1, a point cloud branch module is constructed, which is composed of four different stages, each stage including a local embedding module and a certain number of attention layers d. The structure of each attention layer remains consistent. At the beginning of each stage, the input point cloud is first processed by the local embedding module. The output of the local embedding module is then fed into multiple attention layers, in which the point cloud with similar semantic content is dynamically clustered into the same class using balanced hierarchical clustering, and the local attention of each cluster is calculated using a multi-head self-attention mechanism to capture the long-range dependencies of the point cloud. After four stages of feature extraction, the global context information of the point cloud local features is extracted;

[0049] S3.2, a projection branch module is constructed, which is composed of a shallow convolutional network for extracting multi-view global features from the depth projection map obtained in step S2.2. The convolutional network includes four basic block layers, each composed of two convolutional blocks and a pooling layer, and each convolutional block is composed of a convolutional layer, a batch normalization layer and an activation layer. The channel number and kernel size of the convolution operation are dynamically adjusted according to the configuration parameters to adapt to the feature extraction requirements at different levels;

[0050] S3.3, the point cloud local features obtained in step S3.1 are processed by maximum pooling, and the multi-view global features obtained in step S3.2 are spliced. The spliced features are aggregated by a classifier, which is composed of a multi-layer perceptron, a batch normalization layer, an activation layer, a Dropout layer and a multi-layer perceptron, and finally the prediction result of three-dimensional shape classification is obtained;

[0051] S4, training the classification model: using the sample data set obtained in step S2 to train the classification model constructed in step S3, using the cross-entropy loss function to obtain the loss value, and using the stochastic gradient descent method to update the parameters in the model until the loss value tends to be stable, obtaining the trained classification model;

[0052] S5, complete the test: using the trained classification model in step S4 to complete the inference of the test point cloud data set, and obtaining the inference result of the point cloud data;

[0053] Further, the local embedding module in step S3.1 is implemented by enhancing the local neighborhood of a specific point. Taking the i-th stage (i>1) as an example, the input point coordinates and the point cloud features (the input point coordinates of the first stage are and the point cloud features are ), where represents the embedding dimension of the i-1 layer, and N represents the number of input point clouds. First, the farthest point sampling (FPS) operation is performed to obtain the centroid point features Subsequently, the original point is queried using a k-Nearest Neighbor (k-NN) and the neighboring features are extracted To further integrate the local neighborhood information, the center point feature f i is merged with each neighboring feature f j The local neighborhood feature is calculated as follows:

[0054]

[0055] where RP(x, k) forms a matrix by repeating the vector x for k times, k is the neighborhood size, f j -f i is used to evaluate the feature difference between neighboring points relative to the centroid, f j represents the neighboring point feature, f i represents the center point feature, and ||·|| represents the concatenation operation.

[0056] To enhance the model's ability to perceive point cloud features within a local region, the processed feature RP(x, k) is concatenated with the original feature, and the result is input into a convolutional layer to generate the output feature f' of the local embedding module. This process can be defined as:

[0057]

[0058] where mlp1 and mlp2 are composed of a convolutional layer, batch normalization, and an activation function, respectively, and max represents the max-pooling operation, which is used to aggregate the features of local blocks.

[0059] Further, the attention module in step S3.1 is composed of a multi-head self-attention (MSA) module stacked with κ layers and a standard feed-forward network (FFN). At the κth layer, given a feature vector X κ-1 , a new feature X κ is obtained by the following operation:

[0060]

[0061] where LN represents layer normalization. The output of the MSA module is sent to the FFN module after being processed by a layer normalization. The FFN module contains two fully connected layers. The first layer expands the dimension from C to rC to effectively capture richer feature representations. The second layer then compresses the dimension back to C to achieve efficient computation.

[0062] Further, in step S3.2, a balanced hierarchical clustering method is used to achieve dynamic clustering of point clouds. For the input point cloud feature vector where N denotes the number of point cloud features and d denotes the dimension of point cloud features. First, a linear projection operation is performed to map the original feature vector to a new feature space and further reconstruct into three feature vectors Q, K and V, which are used for subsequent clustering operations. Then the distances of each point q in Q to the two cluster centroids c1 and c2 are calculated to determine its cluster membership. The point cloud features are divided into two equal-sized (m) clusters, and the two cluster centroids c1 and c2 are calculated. For each point q in Q i , the distance ratio of q to the two centroids is calculated to determine its cluster membership. By a sorting operation, the points in Q are sorted according to their distance ratio to the centroids and assigned to two clusters and The above balanced binary clustering operation can be summarized as:

[0063]

[0064] where dist(x, y) denotes the Euclidean distance between x and centroid y in the feature space, and denotes the cluster assigned to these points, argsort is the sorting operation that returns the sorted index values.

[0065] The mean value of each cluster is used to update the cluster centroid. The balanced binary clustering iteratively updates the cluster centroids and the membership of each sample. At the l-th layer, Q is grouped into 2 l subsets Each subset has equal size |Q i | = N / 2 l . Then, K and V are divided into and Each subset shares the same index with Q i , ensuring that Q i , K i and V i can be correctly corresponded in the self-attention mechanism. Finally, the standard multi-head self-attention is adopted within each subset (Q i , K i , V i ) to output Y i = MSA(Q i , K i , V i ). The union set of all subsets is finally merged while maintaining its original order, ensuring that the extracted features can be combined in the order of the original point cloud, preserving the spatial information of the point cloud.

[0066] Further, the projection module in the step S2.2 is a depth map As input, where H and W represent the height and width of the depth image respectively, and v represents the number of depth maps. The depth image D is processed step by step by the network in step S3.2 and generates feature maps where represents the feature dimension of the i-th layer in the projection module. In order to capture the distinctive features in different receptive fields, the feature map D i is divided into n bin feature sets where n e {1, 2, 4, 8, 16, 32} represents the number of bins into which each scale feature map is divided, and each bin represents a part of the feature map at a certain scale. By dividing the feature map into multiple bins, it can be ensured that the model can pay attention to features of different sizes. The bin features of each scale are max-pooled to obtain the most discriminative features of each scale feature map Finally, all the features from B1 to B4 are spliced as the output of the entire projection module.

[0067] Embodiment

[0068] According to Figure 1 The flowchart of the embodiment of the present application is shown, and the specific steps are as follows:

[0069] S1, obtain a public classification dataset of three-dimensional point cloud and corresponding classification labels. The present embodiment carries out classification experiment on ModelNet40 dataset. The dataset covers 40 different categories of 12,311 computer aided design (CAD) models, of which 9,843 samples are used for training set and 2,468 samples are used for test set. All CAD models are correctly placed along the Z axis direction and are pre-normalized to the unit sphere [-1, 1] interval.

[0070] S2, data preprocessing, specifically including:

[0071] S2.1, the point cloud in the dataset obtained in step S1 is subjected to enhancement processing, and the specific operation includes: the point cloud is subjected to random scaling, and the scaling ratio is randomly selected between 0.75 and 1.5, i.e. the scaling factor e [0.75, 1.5]. Secondly, the point cloud is subjected to random jitter, and the jitter amplitude is limited between -0.2 and 0.2, i.e. the jitter amount e (-0.2, 0.2).

[0072] S2.2, the point cloud obtained by processing in step S2.1 is projected, and v views are selected to generate multi-view depth projection maps. By default, 6 views are selected, and the resolution is set to 112x112. Figure 2 As shown in the partial depth map diagram generated after projection of the present embodiment, it can be seen that the generated depth map is different from the natural scene with background, texture, color and other visual information, and there is domain offset with the natural scene image.

[0073] S3, constructing a classification network structure, Figure 3 The network structure flowchart of the embodiment is shown, and specifically includes the following steps:

[0074] S3.1, constructing a point cloud branch module, which is composed of four different stages, each stage containing a local embedding module and an attention layer with a specific number of layers d. In order to reduce the computational complexity and facilitate model convergence, the embodiment defaults to selecting the layer number configuration of [1, 1, 4, 2], and the embedding dimension is [24, 48, 96, 192]. The structure of each attention layer remains the same. The attention module is composed of a multi-head self-attention (MSA) module stacked with κ layers and a standard feed-forward network (FFN). At the κth layer, given a feature vector X κ-1 , a new feature X κ is obtained by the following operation:

[0075]

[0076]

[0077] where LN represents layer normalization. The output of the MSA module is sent to the FFN module after being processed by a layer normalization. The FFN module contains two fully connected layers, the first layer expands the dimension from C to rC to effectively capture richer feature representations, where the default value of r is set to 2. The second layer then compresses the dimension back to C to achieve efficient computation.

[0078] At the beginning of each stage, the input point cloud is first processed by the local embedding module. The local embedding module is implemented by enhancing the local neighborhood of a specific point. Taking the i-th stage (i > 1) as an example, the input point coordinates and point cloud features are received (the input point coordinates of the first stage are and the point cloud features are ), where C embi-1 represents the embedding dimension of the i-1th layer, and N represents the number of input point clouds. First, the farthest point sampling (FPS) operation is performed to obtain the centroid point feature Then, K-neighbor (k-NN) queries are used to query the original points and extract neighboring features In order to further integrate the local neighborhood information, the center point feature f i is combined with each neighboring feature f j . The calculation method of the local neighborhood feature is as follows:

[0079]

[0080] where RP(x, k) forms a matrix by repeating the vector x k times, k is the neighborhood size, and f j-f i for evaluating feature difference between neighboring points relative to the centroid, f j denotes neighboring point features, f i denotes center point features, ||·|| denotes concatenation operation.

[0081] To enhance the model's ability to perceive point cloud features within local regions, the processed features after the replication operation RP(x, k) are concatenated with the original features, and the result is input into a convolutional layer to generate the output features f' of the local embedding module. This process can be defined as:

[0082]

[0083] where mlp1 and mlp2 are composed of a convolutional layer, batch normalization, and an activation function, respectively, and max represents the max-pooling operation, which is used to aggregate the features of local blocks.

[0084] The output of the local embedding module is then fed into multiple self-attention layers, in which the point cloud with similar semantic content is dynamically clustered into the same class using balanced hierarchical clustering, and the attention of each cluster is calculated using the multi-head self-attention mechanism to capture the long-range dependencies of the point cloud. Figure 3 A schematic diagram of balanced hierarchical clustering is shown. For the input point cloud feature vector where N represents the number of point cloud features, and d represents the dimension of the point cloud features. First, a linear projection operation is performed to map the original feature vector to a new feature space and further reconstruct three feature vectors Q, K, and V, which are used for subsequent clustering operations. Then are divided into two equally sized clusters, and the centroids c1 and c2 of the two clusters are calculated. For each point q i in the feature vector Q, the distance ratio to the two centroids is calculated to determine its cluster membership. Through a sorting operation, the points in the point cloud feature vector Q are sorted according to their distance ratio to the centroids and assigned to two clusters and The above balanced binary clustering operation can be summarized as:

[0085]

[0086] where dist(x, y) represents the Euclidean distance between x and the centroid y in the feature space, and denote the clusters assigned to these points, and argsort is the sorting operation that returns the sorted index value.

[0087] The average of each cluster is used to update the cluster centroid. Balanced binary clustering iteratively updates the cluster centroid and the membership of each sample, at the l-th layer, Q is grouped into 2 l subsets Each subset has equal size |Q i | = N / 2 l . Subsequently, K and V are divided into and Each subset has the same index as Q i , ensuring that Q i , K i and V i can be correctly matched in the self-attention mechanism. Finally, within each subset (Q i , K i , V i ), the standard multi-head self-attention is adopted, and the output Y i = MSA(Q i , K i , V i ) is obtained. The union of all subsets is finally merged while maintaining its original order, ensuring that the extracted features can be combined in the order of the original point cloud, preserving the spatial information of the point cloud.

[0088] After four stages of feature extraction, the global context information of the local features of the point cloud is extracted. Figure 5 The effect of clustering at each stage is shown, and the point clouds belonging to the same cluster are assigned the same color, and different clusters are drawn using different colors. It can be seen that the embodiment can capture long-distance context, enabling the model to learn complex geometric features when facing complex scenes.

[0089] S3.2, a projection branch module is constructed, which is composed of a shallow convolutional network, used to extract multi-view global features from the depth projection map obtained in step S2.2 to alleviate domain shift. The convolutional network includes four basic block layers, each of which is composed of two convolutional blocks and a max pooling layer, and each convolutional block is composed of a convolutional layer, a batch normalization layer and an activation layer. The channel number and kernel size of the convolution operation are dynamically adjusted according to the configuration parameters to adapt to the feature extraction requirements at different levels, and in the embodiment, the default values are [24, 48, 96, 192].

[0090] The projection module takes the depth map as input, where H and W represent the height and width of the depth image respectively, and v represents the number of depth maps. The depth image D is processed step by step by the network in step S3.2 and generates a feature map where D i represents the feature dimension of the i-th layer in the projection module. In order to capture the distinctive features in different receptive fields, the feature map D i is divided into n bin feature sets where n e {1, 2, 4, 8, 16, 32} represents the number of bins into which each scale feature map is divided, and each bin represents a part of the feature map at a particular scale. By dividing the feature map into multiple bins, it can be ensured that the model can focus on features of different sizes. The bin features of each scale are max-pooled to obtain the most discriminative features of each scale feature map Finally, all the features from B1 to B4 are spliced as the output of the entire projection module.

[0091] S3.3, the point cloud features obtained in step S3.1 are respectively subjected to global pooling and max-pooling processing, and then spliced with the multi-view global features obtained in step S3.2. The spliced features are aggregated by a classifier composed of a multi-layer perception, a batch normalization layer, an activation layer, a Dropout layer and a multi-layer perception, to finally obtain a prediction result of three-dimensional shape classification.

[0092] S4, training the classification model: using the sample data set obtained in step S2 to train the classification model constructed in step S3, using the cross-entropy loss function to obtain the loss value, and using the stochastic gradient descent method to update the parameters in the model until the loss value tends to be stable, to obtain the trained classification model.

[0093] S5, complete the test: using the trained classification model in step S4 to complete the inference of the test point cloud data set, to obtain the inference result of the point cloud data.

[0094] The overall classification accuracy (Overall Accuracy, OA) and the average classification accuracy (meanClass Accuracy, mAcc) are used to measure the classification effect of the model, and the model parameter amount is used to measure the complexity of the model. The overall classification accuracy measures the classification accuracy of the model on the entire data set, and the higher the value, the better the classification performance of the model. The average classification accuracy measures the average value of the classification accuracy of the model on each class. The lower the model parameter amount, the lower the complexity of the model, and the lower the hardware requirement. The calculation formulas of OA and mAcc are as follows:

[0095]

[0096] where TP, TN, FP, and FN represent the number of samples correctly predicted as positive by the model, the number of samples correctly predicted as negative by the model, the number of samples incorrectly predicted as positive by the model, and the number of samples incorrectly predicted as negative by the model, respectively, C is the total number of classes, and TP i , TNi , FP i and FN i represent the number of true positives, true negatives, false positives and false negatives in the i-th class, respectively.

[0097]

[0098] Table 1 Classification results of different network models on ModelNet40 dataset

[0099] The comparative experimental results are shown in Table 1. Not all methods report the parameter amount and the average classification accuracy, and the parameter that is not reported is represented by a slash “ / ”. PointNet, PointNet++, PointConv, PCT and Point-BERT are all prior art methods. It can be found that the three-dimensional point cloud shape recognition method based on point cloud clustering and integrated multi-view features of the embodiment has certain improvement in the parameter amount, the overall classification accuracy and the average classification accuracy compared with the existing PointNet, PointNet++, PointConv, PCT and Point-BERT network. For example, compared with Point-BERT (the parameter amount is 39.1M) which requires a large amount of memory resources, the embodiment occupies less memory (3.7M) while achieving an excellent level of 93.8%, which highlights its efficient resource utilization capability. It proves the effectiveness of the method proposed in the embodiment on the classification task.

[0100] In summary, the present application improves the understanding ability and generalization ability of the model by combining point clouds and corresponding multi-view depth maps, and can provide accurate three-dimensional object recognition and analysis in practical applications. Moreover, the present application balances hierarchical clustering to cluster similar semantic content in the point cloud and calculates the local attention within each cluster to capture long-distance dependencies of the point cloud, which helps the model to understand and distinguish the shape and structure of complex objects, and has strong robustness. Therefore, the present application has a wide application prospect.

[0101] It should be emphasized that: the above is only the preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments still belongs to the scope of the technical solutions of the present application.

Claims

1. A method for recognizing the shape of a 3D point cloud based on multi-view features, characterized in that, Includes the following steps: Step S1: Obtain the public classification dataset of 3D point clouds and the corresponding classification labels; Step S2: Data preprocessing; The point cloud in the dataset obtained in step S1 is augmented, and then the augmented point cloud is projected to generate a multi-view depth projection map. Step S3: Construct the classification network structure; The point cloud branch module and the projection branch module are constructed sequentially. Then, the local point cloud features obtained from the point cloud branch module are processed by max pooling and then concatenated with the multi-view global features obtained from the projection branch module. The concatenated features are aggregated by a classifier to finally obtain the prediction result of the 3D shape classification. Step S4: Train the classification model; Use the sample dataset obtained in step S2 to train the classification model constructed in step S3, use the cross-entropy loss function to obtain the loss value, and use the stochastic gradient descent method to update the parameters in the model until the loss value tends to stabilize, and obtain the trained classification model. Step S5: Complete the test: Use the classification model trained in step S4 to perform inference on the test point cloud dataset and obtain the inference results of the point cloud data; In step S3, the point cloud branch module is constructed in the following way: the point cloud branch module consists of four different stages, each stage containing a local embedding module and an attention layer of number d; the structure of each attention layer remains consistent; at the beginning of each stage, the input point cloud is first processed by the local embedding module; the output of the local embedding module is then fed into multiple attention layers, in which balanced hierarchical clustering is used to dynamically cluster point clouds with similar semantic content into the same class, and a multi-head self-attention mechanism is used to calculate the local attention of each cluster to capture the long-distance dependencies of the point cloud; after feature extraction in four stages, global contextual information of the local features of the point cloud is extracted; In step S3, the local embedding module achieves this by enhancing the local neighborhood of a specific point; it receives the coordinates of the input point. and point cloud features in Let N represent the embedding dimension of the (i-1)th layer, and N represent the number of input point clouds; First, perform the farthest point sampling operation to obtain the centroid features. Then, K-nearest neighbor query was used to find the original point and extract the neighbor features. The center point feature f i With each neighboring feature f j Merging; the calculation method for local neighborhood features is as follows: In equation (1), RP(x,k) is the matrix formed by repeating vector x k times, k is the neighborhood size, and f j -f i To evaluate the characteristic differences between neighboring points relative to the centroid, f j f represents the feature of neighboring points. i The center point feature is represented by ||·||, and the concatenation operation is represented by ||·||. The features processed by RP(x,k) are concatenated with the original features, and the result is input into a convolutional layer to generate the output feature f′ of the local embedding module; this process is defined as: In equations (2) and (3), mlp1 and mlp2 are composed of convolutional layers, batch normalization and activation functions, respectively, and max represents max pooling operation, which is used to aggregate features of local blocks; In step S3, the attention layer consists of a multi-head self-attention module stacked with κ layers and a standard feedforward network; in the κ layer, given the feature vector X... κ-1 The new feature X is obtained through the following operations. κ : In equations (4) and (5), LN represents layer normalization. The output of the MSA module is fed into the FFN module after being processed by layer normalization. The FFN module contains two fully connected layers. The first layer expands the dimension from C to rC to effectively capture richer feature representations. The second layer then compresses the dimension back to C to achieve efficient computation.

2. The method for recognizing 3D point cloud shapes based on multi-view features according to claim 1, characterized in that, In step S2, the specific operations of the enhancement process include: first, randomly scaling the point cloud; second, randomly jittering the point cloud.

3. The method for recognizing 3D point cloud shapes based on multi-view features according to claim 1, characterized in that, In step S3, the projection branch module is constructed in the following ways: the projection branch module is composed of a shallow convolutional network, which is used to extract multi-view global features from the depth projection map obtained in step S2; the convolutional network includes multiple basic block layers, each basic block layer is composed of a convolutional block and a pooling layer, and the convolutional block is composed of a convolutional layer, a batch normalization layer and an activation layer; the number of channels and the kernel size of the convolution operation are dynamically adjusted according to the configuration parameters to adapt to the feature extraction requirements of different levels.

4. The method for recognizing 3D point cloud shapes based on multi-view features according to claim 3, characterized in that, In step S3, a balanced hierarchical clustering method is used to dynamically cluster the point cloud; specifically, it includes: For the input point cloud feature vector Where N represents the number of point cloud features and d represents the feature dimension, a linear projection operation is first performed to map the original feature vectors to a new feature space and further reconstruct them into three feature vectors Q, K, and V. These feature vectors are used for subsequent clustering operations. Then... Divide into two equal-sized clusters and calculate the centroids c1 and c2 of the two clusters; for each point q in each eigenvector Q. i The points in the point cloud feature vector Q are calculated based on their distance ratios to the two centroids to determine their cluster membership. Through a sorting operation, the points are sorted according to their distance ratios to the centroids and assigned to two clusters, C1 and C2. The above balanced binary clustering operation can be summarized as follows: In equation (6), dist(x,y) represents the Euclidean distance between x and centroid y in the feature space, C1 and C2 represent clusters assigned to points, and argsort is a sorting operation that returns the sorted index values. The mean of each cluster is used to update the cluster centroid; balanced binary clustering iteratively updates the cluster centroid and the membership of each sample, where Q is grouped into 2 at level l. l Subset Each subset has an equal size |Q i |=N / 2 l Subsequently, K and V were divided into... and Each subset and Q i Sharing the same index ensures that Q in the self-attention mechanism i K i and V i It can be correctly matched; finally, within each subset (Q) i K i V i Using standard multi-head self-attention, the output Y i =MSA(Q i ,K i V i The final union of all subsets. At the same time, the original order is maintained to ensure that the extracted features can be combined in the order of the original point cloud, thus preserving the spatial information of the point cloud.

5. The method for recognizing 3D point cloud shapes based on multi-view features according to claim 1, characterized in that, In step S2, the projection module uses a depth map As input, H and W represent the height and width of the depth image, respectively, and v represents the number of depth maps; the depth image D is processed stepwise by the network in step S3 to generate feature maps. in Represent the feature dimension of the i-th layer in the projection module; transform the feature map D i Divided into n bin feature sets Where n∈{1,2,4,8,16,32}, it represents the number of bins into which the feature map at each scale is divided, and each bin represents a part of the feature map at a specific scale; by dividing the feature map into multiple bins, it can be ensured that the model can pay attention to features of different sizes; Max pooling is performed on the bin features at each scale to obtain the most discriminative features of the feature map at each scale. Finally, all features from B1 to B4 are stitched together as the output of the entire projection module.

Images