Three-dimensional point cloud shape recognition method based on multi-view features

By combining point clouds and multi-view depth maps, using balanced hierarchical clustering and classifiers, the problem of difficult interactions between distant points in three-dimensional point cloud shape recognition is solved, and efficient three-dimensional object recognition and analysis is achieved, which improves the The robustness and generalization ability of the model.

CN118968493AActive Publication Date: 2024-11-15XI AN JIAOTONG UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410923082.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-15
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

Existing three-dimensional point cloud shape recognition technology is difficult to effectively capture the interaction between distant but similar points in three-dimensional space, and a single modal processing is not enough to cover enough information, ignoring the complementarity between different modalities.

Method used

Using a three-dimensional point cloud shape recognition method based on multi-view features, the three-dimensional point cloud dataset is obtained for enhanced processing and projection, and the point cloud branch module and projection branch module are constructed, combining maximum pooling and balanced hierarchical clustering to capture local and global Features are aggregated through classifiers to finally realize three-dimensional shape classification.

Benefits of technology

It improves the model's understanding and generalization ability, can accurately identify the shapes and structures of complex objects, is highly robust, and provides accurate three-dimensional object recognition and analysis in practical applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968493A_ABST
    Figure CN118968493A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of three-dimensional point cloud processing, and relates to a multi-view feature-based three-dimensional point cloud shape recognition method, which comprises the following steps of: 1) acquiring point cloud data and corresponding classification labels; 2) preprocessing the original point cloud and obtaining a point cloud after data enhancement and a multi-view depth map; 3) constructing a classification model containing point clouds and projection branches; in the point cloud branches, clustering the point clouds with similar semantic contents with the sampling points into the same class, calculating local attention in each cluster by adopting multi-head self-attention, and capturing a long-distance dependency relationship of the point clouds; in the projection branch, a plurality of depth images of different visual angles pass through a shallow convolutional network to generate discrimination features; 4) training a classification model; 5) completing three-dimensional point cloud shape recognition and classification; according to the method, the local features of the point cloud and the global depth map features are combined, the long-distance dependency relationship is captured, and the accuracy and robustness of point cloud shape recognition classification can be efficiently realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud processing, and in particular relates to a three-dimensional point cloud shape recognition method based on multi-view features. Background Art

[0002] 3D point cloud shape recognition technology aims to automatically detect, analyze and identify the shape features of objects from 3D point clouds, and has a wide range of applications and development prospects. In the fields of industrial automation, robotics, etc., 3D point cloud shape recognition can help robots and automation systems understand and operate objects more accurately, and improve production efficiency and quality. Unlike traditional two-dimensional images, point clouds consist of the original coordinates of points in three-dimensional space, which are usually obtained by scanning. Due to the irregularity of point clouds, traditional convolutional neural networks (CNNs) face challenges in processing such data, because CNNs are usually designed to process two-dimensional images with fixed grid structures. To overcome this challenge, researchers have developed a variety of methods for processing point clouds, and the existing methods can be divided into two types according to the form of processing point clouds.

[0003] One approach is to convert the point cloud into a format that can be used by convolutional neural networks, such as voxels or projection maps. Voxel-based methods convert point clouds into three-dimensional grids (voxels) and calculate features within each voxel. However, due to high memory requirements, their application on high-resolution point clouds is limited. Projection-based methods use convolutional neural networks to extract image features and retain as many local features as possible by rendering views of a given three-dimensional shape. Projection-based methods usually lose a large number of point cloud features during data processing due to excessive compression in one direction. In addition, these projection methods usually utilize mature two-dimensional deep neural network models, such as ResNet and ViT. Another approach is to process the point cloud directly without any conversion. This type of method focuses on extracting local features by exploring new designs of local aggregation operators, but it is difficult to capture the interactions between distant but similar points in three-dimensional space.

[0004] Most current methods focus on a single modality, such as relying on 3D point clouds or 2D images, ignoring the complementarity between different modalities. Considering that a single modality may not be enough to cover enough information for 3D shape recognition, a method is needed to enhance 3D point cloud shape analysis using the information captured by 2D projection images, making full use of the advantages of each modality. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: A three-dimensional point cloud shape recognition method based on multi-view features, comprising the following steps:

[0006] Step S1, obtaining a public classification data set of a three-dimensional point cloud and corresponding classification labels;

[0007] Step S2, data preprocessing: enhancing the point cloud in the data set obtained in step S1, then projecting the point cloud obtained by the enhanced processing, and selecting v perspectives to generate a multi-perspective depth projection map;

[0008] Step S3, constructing a classification network structure; sequentially constructing a point cloud branch module and a projection branch module, then splicing the point cloud local features obtained by constructing the point cloud branch module with the multi-view global features obtained by constructing the projection branch module after maximum pooling processing, and aggregating the spliced ​​features through a classifier to finally obtain a prediction result of the three-dimensional shape classification;

[0009] Step S4, training the classification model; using the sample data set obtained in step S2 to train the classification model constructed in step S3, using the cross entropy loss function to obtain the loss value, and using the stochastic gradient descent method to update the parameters in the model until the loss value tends to be stable, thereby obtaining a trained classification model;

[0010] Step S5, complete the test: use the classification model trained in step S4 to complete the reasoning of the test point cloud data set and obtain the reasoning result of the point cloud data.

[0011] Preferably, in step S2, the specific operations of the enhancement processing include: first, randomly scaling the point cloud; and second, randomly jittering the point cloud.

[0012] Preferably, in step S3, the method of constructing the point cloud branch module includes: the point cloud branch module is composed of four different stages, each stage includes a local embedding module and a specific number of attention layers d; the structure of each attention layer remains consistent; at the beginning of each stage, the input point cloud is first processed by the local embedding module; the output of the local embedding module is then sent to multiple attention layers, in which balanced hierarchical clustering is used to dynamically cluster point clouds with similar semantic content into the same class, and a multi-head self-attention mechanism is used to calculate the local attention of each cluster to capture the long-distance dependency of the point cloud; after four stages of feature extraction, global context information extraction of local features of the point cloud is achieved.

[0013] Preferably, in step S3, the local embedding module is implemented by enhancing the local neighborhood of a specific point; receiving the input point coordinates and point cloud features in represents the embedding dimension of the i-1th layer, and N represents the number of input point clouds;

[0014] First, perform the farthest point sampling operation to obtain the centroid point feature Then use K-nearest neighbors to query the original point and extract neighboring features

[0015] In order to further integrate the local neighborhood information, the center point feature f i With each neighboring feature f j Merge; the local neighborhood features are calculated as follows:

[0016]

[0017] In formula (1), RP(x,k) is a matrix formed by repeating vector x k times, k is the neighborhood size, and f j -f i is used to evaluate the feature differences between neighboring points relative to the centroid, f j represents the neighboring point features, f i represents the center point feature, ||·|| represents the splicing operation;

[0018] In order to enhance the model's ability to perceive point cloud features in local areas, the features processed by RP(x,k) are concatenated with the original features, and the results are input into a convolutional layer to generate the output features f′ of the local embedding module; this process is defined as:

[0019]

[0020] In formula (2) and formula (3), mlp1 and mlp2 are composed of a convolutional layer, batch normalization and an activation function respectively, and max represents the maximum pooling operation, which is used to aggregate the features of local blocks.

[0021] Preferably, in step S3, the attention layer is composed of a multi-head self-attention module stacked in layers of κ and a standard feed-forward network; in the κ layer, given the feature vector X κ-1 , the new feature X is obtained by the following operation κ :

[0022]

[0023]

[0024] In formula (4) and formula (5), LN represents layer normalization. The output of the MSA module is sent to the FFN module after a layer normalization process. The FFN module contains two fully connected layers. The first layer expands the dimension from C to rC to effectively capture richer feature representations. The second layer then compresses the dimension back to C for efficient calculation.

[0025] Preferably, in step S3, the method of constructing the projection branch module includes: the projection branch module is composed of a shallow convolutional network, which is used to extract multi-view global features from the depth projection image obtained in step S2; the convolutional network includes four basic block layers, each basic block layer is composed of two convolution blocks and a pooling layer, and the convolution block is composed of a convolution layer, a batch normalization layer and an activation layer; the number of channels and kernel size of the convolution operation are dynamically adjusted according to the configuration parameters to adapt to the feature extraction requirements at different levels.

[0026] Preferably, in step S3, a balanced hierarchical clustering method is used to achieve dynamic clustering of the point cloud; specifically, the method includes:

[0027] For the input point cloud feature vector Where N represents the number of point cloud features, d represents the feature dimension of the point cloud. First, a linear projection operation is performed to map the original feature vector to a new feature space and further reconstruct it into three feature vectors Q, K, and V. These feature vectors are used for subsequent clustering operations. Divide into two clusters of equal size (m) and calculate the two cluster centroids c1 and c2; for each point q in each feature vector Q i , calculate its distance ratio to the two centroids to determine its cluster membership; through the sorting operation, the points in the point cloud feature vector Q are sorted according to their distance ratio to the centroid and assigned to the two clusters and The above balanced binary clustering operation can be summarized as:

[0028]

[0029] In formula (6), dist(x,y) represents the Euclidean distance between x and the centroid y in the feature space. and Indicates the clusters assigned to these points, argsort is a sorting operation, and returns the sorted index value;

[0030] The average of each cluster is used to update the cluster centroid; balanced binary clustering iteratively updates the cluster centroid and the membership of each sample. At level l, Q is grouped into 2 l Subset Each subset has equal size |Q i |=N / 2 l ; Subsequently, K and V were divided into and Each subset is related to Q i Share the same index, ensuring that in the self-attention mechanism, Q i , K i and V ican correspond correctly; finally, in each subset (Q i , K i 、V i ) Using standard multi-head self-attention, output Y i =MSA(Q i ,K i ,V i ); finally merge the union of all subsets At the same time, its original order is maintained to ensure that the extracted features can be combined in the order of the original point cloud and the spatial information of the point cloud is retained.

[0031] Preferably, in step S2, the projection module uses the depth map As input, where H and W represent the height and width of the depth image respectively, and v represents the number of depth maps; the depth image D is processed step by step through the network in step S3 and generates a feature map in represents the feature dimension of the i-th layer in the projection module; in order to capture the distinctive features in different receptive fields, the feature map D i Divided into n bin feature sets Where n∈{1,2,4,8,16,32} represents the number of bins that each scale feature map is divided into, and each bin represents a part of the feature map at a specific scale. By dividing the feature map into multiple bins, it can ensure that the model can pay attention to features of different sizes. The bin features of each scale are pooled to obtain the most discriminative features of each scale feature map. Finally, all features from B1 to B4 are concatenated as the output of the entire projection module.

[0032] Preferably, in step S3, the classifier consists of a multi-layer perceptron, a batch normalization layer, an activation layer, a Dropout layer and a multi-layer perceptron.

[0033] The beneficial effects of the present invention are:

[0034] 1. The present invention improves the understanding and generalization capabilities of the model by combining point clouds and corresponding multi-view depth maps, and can provide accurate three-dimensional object recognition and analysis in practical applications.

[0035] 2. The present invention clusters similar semantic contents in point clouds through balanced hierarchical clustering, calculates local attention within each cluster, captures long-distance dependencies of point clouds, helps the model understand and distinguish the shapes and structures of complex objects, and has strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of a three-dimensional point cloud shape recognition method based on multi-view features of the present invention;

[0037] Figure 2 is a multi-view depth map generated by an embodiment of the present invention;

[0038] Figure 3 is a network structure diagram of a classification model according to an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of balanced hierarchical clustering according to an embodiment of the present invention;

[0040] Figure 5 It is a visualization effect diagram of clustering of point cloud branches at different stages according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the relevant technologies in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] like Figures 1 to 5 As shown, the three-dimensional point cloud shape recognition method based on multi-view features of this embodiment includes the following steps:

[0043] S1. Obtain a public classification dataset of 3D point clouds and corresponding classification labels;

[0044] S2. Data preprocessing, including:

[0045] S2.1, enhance the point cloud in the data set obtained in step S1, and the specific operations include: randomly scaling the point cloud. Secondly, randomly jittering the point cloud;

[0046] S2.2, projecting the point cloud obtained in step S2.1, and selecting v perspectives to generate a multi-perspective depth projection map;

[0047] S3. Constructing a classification network structure, specifically including the following steps:

[0048] S3.1. Construct a point cloud branch module, which consists of four different stages, each of which contains a local embedding module and a specific number of attention layers d. The structure of each attention layer remains consistent. At the beginning of each stage, the input point cloud is first processed by the local embedding module. The output of the local embedding module is then fed into multiple attention layers, where balanced hierarchical clustering is used to dynamically cluster point clouds with similar semantic content into the same class, and a multi-head self-attention mechanism is used to calculate the local attention of each cluster to capture the long-distance dependencies of the point cloud. After four stages of feature extraction, the global context information extraction of the local features of the point cloud is realized;

[0049] S3.2. Construct a projection branch module, which consists of a shallow convolutional network and is used to extract multi-view global features from the deep projection map obtained in step S2.2. The convolutional network includes four basic block layers, each of which consists of two convolutional blocks and a pooling layer. The convolutional block consists of a convolutional layer, a batch normalization layer, and an activation layer. The number of channels and kernel size of the convolution operation are dynamically adjusted according to the configuration parameters to meet the requirements of feature extraction at different levels;

[0050] S3.3, after the maximum pooling process, the local features of the point cloud obtained in step S3.1 are concatenated with the multi-view global features obtained in step S3.2. The concatenated features are aggregated through a classifier, wherein the classifier is composed of a multi-layer perceptron, a batch normalization layer, an activation layer, a dropout layer and a multi-layer perceptron, and finally a prediction result of the three-dimensional shape classification is obtained;

[0051] S4, training the classification model: using the sample data set obtained in step S2 to train the classification model constructed in step S3, using the cross entropy loss function to obtain the loss value, and using the stochastic gradient descent method to update the parameters in the model until the loss value tends to be stable, thereby obtaining a trained classification model;

[0052] S5. Complete the test: Use the classification model trained in step S4 to complete the reasoning of the test point cloud data set to obtain the reasoning result of the point cloud data;

[0053] Furthermore, the local embedding module in step S3.1 is implemented by enhancing the local neighborhood of a specific point. Taking the i-th stage (i>1) as an example, the input point coordinates are received and point cloud features (The input point coordinates in the first stage are Point cloud features are ),in represents the embedding dimension of the i-1th layer, and N represents the number of input point clouds. First, the farthest point sampling (FPS) operation is performed to obtain the centroid point features. Then use K-nearest neighbor (k-NN) to query the original point and extract neighboring features In order to further integrate the local neighborhood information, the center point feature f i With each neighboring feature f j Merge. The local neighborhood features are calculated as follows:

[0054]

[0055] Among them, RP(x,k) forms a matrix by repeating vector x k times, k is the neighborhood size, and f j -f i Used to evaluate the feature differences between neighboring points relative to the centroid, f j represents the neighboring point features, f i represents the center point feature, and ||·|| represents the splicing operation.

[0056] In order to enhance the model's ability to perceive point cloud features in local areas, the features processed by RP(x,k) are concatenated with the original features, and the results are input into a convolutional layer to generate the output features f′ of the local embedding module. This process can be defined as:

[0057]

[0058] Among them, mlp1 and mlp2 are composed of a convolutional layer, batch normalization and activation function respectively, and max represents the maximum pooling operation, which is used to aggregate the features of local blocks.

[0059] Furthermore, the attention module in step S3.1 is composed of a multi-head self-attention (MSA) module stacked with κ layers and a standard feed-forward network (FFN). In the κth layer, given the feature vector X κ-1 , the new feature X is obtained by the following operation κ :

[0060]

[0061] Where LN stands for layer normalization. The output of the MSA module is fed into the FFN module after a layer normalization process. The FFN module contains two fully connected layers. The first layer expands the dimension from C to rC to effectively capture richer feature representations. The second layer then compresses the dimension back to C for efficient computation.

[0062] Furthermore, in step S3.2, a balanced hierarchical clustering method is used to achieve dynamic clustering of point clouds. Where N represents the number of point cloud features and d represents the feature dimension of the point cloud. First, a linear projection operation is performed to map the original feature vector to a new feature space and further reconstruct it into three feature vectors Q, K and V. These feature vectors are used for subsequent clustering operations. Divide into two clusters of equal size (m) and calculate the two cluster centroids c1 and c2. For each point q in each feature vector Q i , calculate its distance ratio to the two centroids to determine its cluster membership. Through the sorting operation, the points in the point cloud feature vector Q are sorted according to their distance ratio to the centroid and assigned to the two clusters and The above balanced binary clustering operation can be summarized as:

[0063]

[0064] Where dist(x,y) represents the Euclidean distance between x and the centroid y in the feature space, and It represents the clusters assigned to these points, argsort is a sorting operation, and returns the sorted index value.

[0065] The average value of each cluster is used to update the cluster centroid. Balanced binary clustering iteratively updates the cluster centroid and the membership of each sample. At level l, Q is grouped into 2 l Subset Each subset has equal size |Q i |=N / 2 l Subsequently, K and V were divided into and Each subset is related to Q i Share the same index, ensuring that in the self-attention mechanism, Q i , K i and V i Finally, in each subset (Q i , K i 、V i ) Using standard multi-head self-attention, output Y i =MSA(Q i ,K i ,V i ). Finally, the union of all subsets is merged At the same time, its original order is maintained to ensure that the extracted features can be combined in the order of the original point cloud and the spatial information of the point cloud is retained.

[0066] Furthermore, the projection module in step S2.2 uses the depth map As input, H and W represent the height and width of the depth image respectively, and v represents the number of depth images. The depth image D is processed step by step through the network in step S3.2 and generates a feature map in Denotes the feature dimension of the i-th layer in the projection module. In order to capture the distinctive features in different receptive fields, the feature map D i Divided into n bin feature sets Where n∈{1,2,4,8,16,32} represents the number of bins that each scale feature map is divided into, and each bin represents a part of the feature map at a specific scale. By dividing the feature map into multiple bins, it ensures that the model can focus on features of different sizes. The bin features of each scale are max-pooled to obtain the most discriminative features of each scale feature map. Finally, all features from B1 to B4 are concatenated as the output of the entire projection module.

[0067] Example

[0068] according to Figure 1 The flowchart of the embodiment of the present invention is shown, and the specific steps are as follows:

[0069] S1. Obtain a public classification dataset of three-dimensional point clouds and corresponding classification labels. This embodiment performs classification experiments on the ModelNet40 dataset. The dataset covers 12,311 computer-aided design (CAD) models of 40 different categories, of which 9,843 samples are used for training sets and 2,468 samples are used for test sets. All CAD models are correctly placed along the Z-axis and are pre-normalized to the unit sphere [-1, 1] interval.

[0070] S2. Data preprocessing, including:

[0071] S2.1. Enhance the point cloud in the data set obtained in step S1. The specific operations include: randomly scaling the point cloud, with the scaling ratio randomly selected between 0.75 and 1.5, that is, the scaling factor ∈ [0.75, 1.5]. Secondly, randomly jitter the point cloud, with the jitter amplitude limited to between -0.2 and 0.2, that is, the jitter amount ∈ (-0.2, 0.2).

[0072] S2.2. Project the point cloud obtained in step S2.1, and select v perspectives to generate a multi-perspective depth projection map. By default, 6 perspectives are selected and the resolution is set to 112×112. Figure 2 The figure shows a schematic diagram of part of the depth map generated after projection in this embodiment. It can be seen that the generated depth map is different from the natural scene in that it has background, texture, color and other visual information, and has a domain offset from the natural scene image.

[0073] S3, build a classification network structure, Figure 3 The network structure flow chart of this embodiment is shown, which specifically includes the following steps:

[0074] S3.1. Construct a point cloud branch module, which consists of four different stages, each of which contains a local embedding module and a specific number of attention layers d. In order to reduce the computational complexity and facilitate model convergence, this embodiment selects the layer configuration of [1,1,4,2] by default, and the embedding dimension is [24,48,96,192]. The structure of each attention layer is consistent. The attention module consists of a multi-head self-attention (MSA) module stacked with κ layers and a standard feedforward network (FFN). In the κth layer, given the feature vector X κ-1 , the new feature X is obtained by the following operation κ :

[0075]

[0076]

[0077] Where LN stands for layer normalization. The output of the MSA module is fed into the FFN module after a layer normalization process. The FFN module contains two fully connected layers. The first layer expands the dimension from C to rC to effectively capture richer feature representations, where the default value of r is set to 2. The second layer then compresses the dimension back to C for efficient computation.

[0078] At the beginning of each stage, the input point cloud is first processed by the local embedding module. The local embedding module works by enhancing the local neighborhood of a specific point. For example, in stage i (i>1), the input point coordinates are received and point cloud features (The input point coordinates in the first stage are Point cloud features are ), where C embi-1 represents the embedding dimension of the i-1th layer, and N represents the number of input point clouds. First, the farthest point sampling (FPS) operation is performed to obtain the centroid point features. Then use K-nearest neighbor (k-NN) to query the original point and extract neighboring features In order to further integrate the local neighborhood information, the center point feature f i With each neighboring feature f j Merge. The local neighborhood features are calculated as follows:

[0079]

[0080] Among them, RP(x,k) forms a matrix by repeating vector x k times, k is the neighborhood size, and f j-f i Used to evaluate the feature differences between neighboring points relative to the centroid, f j represents the neighboring point features, f i represents the center point feature, and ||·|| represents the splicing operation.

[0081] In order to enhance the model's ability to perceive point cloud features in local areas, the features processed by the copy operation RP(x,k) are concatenated with the original features, and the result is input into a convolutional layer to generate the output features f′ of the local embedding module. This process can be defined as:

[0082]

[0083] Among them, mlp1 and mlp2 are composed of a convolutional layer, batch normalization and activation function respectively, and max represents the maximum pooling operation, which is used to aggregate the features of local blocks.

[0084] The output of the local embedding module is then fed into multiple self-attention layers, where balanced hierarchical clustering is used to dynamically cluster point clouds with similar semantic content into the same class, and a multi-head self-attention mechanism is used to calculate the attention of each cluster to capture the long-range dependencies of point clouds. Figure 3 The balanced hierarchical clustering diagram is shown. For the input point cloud feature vector Where N represents the number of point cloud features and d represents the feature dimension of the point cloud. First, a linear projection operation is performed to map the original feature vector to a new feature space and further reconstruct it into three feature vectors Q, K and V. These feature vectors are used for subsequent clustering operations. Divide into two clusters of equal size and calculate the two cluster centroids c1 and c2. For each point q in each feature vector Q i , calculate its distance ratio to the two centroids to determine its cluster membership. Through the sorting operation, the points in the point cloud feature vector Q are sorted according to their distance ratio to the centroid and assigned to the two clusters and The above balanced binary clustering operation can be summarized as:

[0085]

[0086] Where dist(x,y) represents the Euclidean distance between x and the centroid y in the feature space, and It represents the clusters assigned to these points, argsort is a sorting operation, and returns the sorted index value.

[0087] The average value of each cluster is used to update the cluster centroid. Balanced binary clustering iteratively updates the cluster centroid and the membership of each sample. At level l, Q is grouped into 2 l Subset Each subset has equal size |Q i |=N / 2 l Subsequently, K and V were divided into and Each subset is related to Q i The same index ensures that in the self-attention mechanism, Q i , K i and V i Finally, in each subset (Q i , K i 、V i ) Using standard multi-head self-attention, output Y i =MSA(Q i ,K i ,V i ). Finally, the union of all subsets is merged At the same time, its original order is maintained to ensure that the extracted features can be combined in the order of the original point cloud and the spatial information of the point cloud is retained.

[0088] After four stages of feature extraction, the global context information extraction of local features of point cloud is realized. Figure 5 The effect of clustering at each stage is shown. Point clouds belonging to the same cluster are given the same color, and different clusters are drawn with different colors. It can be seen that this embodiment can capture long-range context, enabling the model to learn complex geometric features when facing complex scenes.

[0089] S3.2. Construct a projection branch module, which consists of a shallow convolutional network and is used to extract multi-view global features from the depth projection map obtained in step S2.2 to mitigate domain offset. The convolutional network includes four basic block layers, each of which consists of two convolutional blocks and a maximum pooling layer. The convolutional block consists of a convolutional layer, a batch normalization layer, and an activation layer. The number of channels and kernel size of the convolution operation are dynamically adjusted according to the configuration parameters to adapt to the feature extraction requirements at different levels. In this embodiment, the default value is [24, 48, 96, 192].

[0090] The projection module uses the depth map As input, H and W represent the height and width of the depth image respectively, and v represents the number of depth images. The depth image D is processed step by step through the network in step S3.2 and generates a feature map in Denotes the feature dimension of the i-th layer in the projection module. In order to capture the distinctive features in different receptive fields, the feature map D i Divided into n bin feature sets Where n∈{1,2,4,8,16,32} represents the number of bins that each scale feature map is divided into, and each bin represents a part of the feature map at a specific scale. By dividing the feature map into multiple bins, it ensures that the model can focus on features of different sizes. The bin features of each scale are max-pooled to obtain the most discriminative features of each scale feature map. Finally, all features from B1 to B4 are concatenated as the output of the entire projection module.

[0091] S3.3, the point cloud features obtained in step S3.1 are processed by global pooling and maximum pooling respectively, and then spliced ​​with the multi-view global features obtained in step S3.2. The spliced ​​features are aggregated through a classifier, where the classifier consists of a multi-layer perceptron, a batch normalization layer, an activation layer, a dropout layer and a multi-layer perceptron, and finally the prediction result of the three-dimensional shape classification is obtained.

[0092] S4. Training the classification model: Use the sample data set obtained in step S2 to train the classification model constructed in step S3, use the cross entropy loss function to get the loss value, and use the stochastic gradient descent method to update the parameters in the model until the loss value tends to be stable, thereby obtaining a trained classification model.

[0093] S5. Complete the test: Use the classification model trained in step S4 to complete the reasoning of the test point cloud data set and obtain the reasoning result of the point cloud data.

[0094] This embodiment uses overall classification accuracy (OA) and mean Class Accuracy (mAcc) to measure the model classification effect, and uses the model parameter amount to measure the model complexity. Overall classification accuracy measures the classification accuracy of the model in the entire data set. The higher the value, the better the model classification performance. Mean Class Accuracy measures the average value of the classification accuracy of the model in each category. The lower the model parameter amount, the lower the model complexity and the lower the hardware requirements. The calculation formulas for OA and mAcc are as follows:

[0095]

[0096] Among them, TP, TN, FP, and FN represent the number of samples correctly predicted as positive by the model, the number of samples correctly predicted as negative by the model, the number of samples incorrectly predicted as positive by the model, and the number of samples incorrectly predicted as negative by the model, respectively. C is the total number of categories, and TP is the total number of samples correctly predicted as positive by the model. i TNi , FP i and FN i Represent the number of true positives, true negatives, false positives, and false negatives in the i-th class, respectively.

[0097]

[0098] Table 1 Classification results of different network models on the ModelNet40 dataset

[0099] The results of the comparative experiments are shown in Table 1. Not all methods report the number of parameters and the average classification accuracy. Those that do not report this parameter are indicated by a slash " / ". PointNet, PointNet++, PointConv, PCT and Point-BERT are all methods of the prior art. It can be found that the three-dimensional point cloud shape recognition method based on point cloud clustering and integrated multi-view features in this embodiment has certain improvements in parameter quantity, overall classification accuracy, and average classification accuracy compared to existing networks such as PointNet, PointNet++, PointConv, PCT and Point-BERT. For example, compared with Point-BERT (parameter quantity is 39.1M) which requires huge memory resources, this embodiment occupies less memory (3.7M) while achieving an excellent level of 93.8%, demonstrating its efficient resource utilization capabilities. It proves the effectiveness of the method proposed in this embodiment in classification tasks.

[0100] In summary, the present invention improves the understanding and generalization capabilities of the model by combining point clouds and corresponding multi-view depth maps, and can provide accurate three-dimensional object recognition and analysis in practical applications. In addition, the present invention clusters similar semantic content in the point cloud through balanced hierarchical clustering, calculates local attention within each cluster, captures long-distance dependencies of the point cloud, helps the model understand and distinguish the shape and structure of complex objects, and has strong robustness; therefore, the present invention has broad application prospects.

[0101] It should be emphasized that the above are only preferred embodiments of the present invention and do not limit the present invention in any form. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A three-dimensional point cloud shape recognition method based on multi-view features, characterized in that: The following steps are involved: Step S1, obtaining a public classification data set of a three-dimensional point cloud and corresponding classification labels; Step S2, data preprocessing; Performing enhancement processing on the point cloud in the data set obtained in step S1, then projecting the point cloud obtained by the enhancement processing, and generating a multi-view depth projection map; Step S3, constructing a classification network structure; The point cloud branch module and the projection branch module are constructed in sequence. Then, the local features of the point cloud obtained by the point cloud branch module are processed by maximum pooling and then spliced ​​with the multi-view global features obtained by the projection branch module. The spliced ​​features are aggregated through the classifier to finally obtain the prediction result of the 3D shape classification. Step S4, training the classification model; using the sample data set obtained in step S2 to train the classification model constructed in step S3, using the cross entropy loss function to obtain the loss value, and using the stochastic gradient descent method to update the parameters in the model until the loss value tends to be stable, thereby obtaining a trained classification model; Step S5, complete the test: use the classification model trained in step S4 to complete the reasoning of the test point cloud data set and obtain the reasoning result of the point cloud data.

2. The three-dimensional point cloud shape recognition method based on multi-view features according to claim 1, characterized in that: In step S2, the specific operations of the enhancement processing include: first, randomly scaling the point cloud; and second, randomly jittering the point cloud.

3. The three-dimensional point cloud shape recognition method based on multi-view features according to claim 1, characterized in that: In the step S3, the method of constructing the point cloud branch module includes: the point cloud branch module is composed of four different stages, each stage includes a local embedding module and an attention layer with a layer number d; the structure of each attention layer is consistent; at the beginning of each stage, the input point cloud is first processed by the local embedding module; the output of the local embedding module is then sent to multiple attention layers, in which balanced hierarchical clustering is used to dynamically cluster point clouds with similar semantic content into the same class, and a multi-head self-attention mechanism is used to calculate the local attention of each cluster to capture the long-distance dependency of the point cloud; after four stages of feature extraction, global context information extraction of local features of the point cloud is realized.

4. The three-dimensional point cloud shape recognition method based on multi-view features according to claim 3 is characterized in that: In step S3, the local embedding module is implemented by enhancing the local neighborhood of a specific point; receiving the input point coordinates and point cloud features in represents the embedding dimension of the i-1th layer, and N represents the number of input point clouds; First, perform the farthest point sampling operation to obtain the centroid point feature Then use K-nearest neighbors to query the original point and extract neighboring features The center point feature f i With each neighboring feature f j Merge; the local neighborhood features are calculated as follows: In formula (1), RP(x,k) is a matrix formed by repeating vector x k times, k is the neighborhood size, and f j -f i is used to evaluate the feature differences between neighboring points relative to the centroid, f j represents the neighboring point features, f i represents the center point feature, ||·|| represents the splicing operation; The features processed by RP(x,k) are concatenated with the original features and the result is input into a convolutional layer to generate the output features f′ of the local embedding module; The process is defined as: In formula (2) and formula (3), mlp1 and mlp2 are composed of convolutional layers, batch normalization and activation functions, respectively, and max represents the maximum pooling operation, which is used to aggregate the features of local blocks.

5. The three-dimensional point cloud shape recognition method based on multi-view features according to claim 3, characterized in that: In step S3, the attention layer is composed of a multi-head self-attention module stacked in κ layers and a standard feedforward network; in the κ layer, given the feature vector X κ-1 , the new feature X is obtained by the following operation κ : In formula (4) and formula (5), LN represents layer normalization. After layer normalization, the output of the MSA module is sent to the FFN module. The FFN module contains two fully connected layers. The first layer expands the dimension from C to rC to effectively capture richer feature representations. The second layer then compresses the dimension back to C for efficient computation.

6. The three-dimensional point cloud shape recognition method based on multi-view features according to claim 1, characterized in that: In the step S3, the method of constructing the projection branch module includes: the projection branch module is composed of a shallow convolutional network, which is used to extract multi-view global features from the deep projection image obtained in step S2; the convolutional network includes multiple basic block layers, each basic block layer is composed of a convolution block and a pooling layer, and the convolution block is composed of a convolution layer, a batch normalization layer and an activation layer; the number of channels and the kernel size of the convolution operation are dynamically adjusted according to the configuration parameters to adapt to the feature extraction requirements at different levels.

7. The three-dimensional point cloud shape recognition method based on multi-view features according to claim 6, characterized in that: In the step S3, a balanced hierarchical clustering method is used to achieve dynamic clustering of point clouds; specifically, the method includes: For the input point cloud feature vector Where N represents the number of point cloud features, d represents the feature dimension of the point cloud. First, a linear projection operation is performed to map the original feature vector to a new feature space and further reconstruct it into three feature vectors Q, K, and V. These feature vectors are used for subsequent clustering operations. Divide into two clusters of equal size and calculate the two cluster centroids c1 and c2; for each point q in each feature vector Q i , calculate its distance ratio to the two centroids to determine its cluster membership; through the sorting operation, the points in the point cloud feature vector Q are sorted according to their distance ratio to the centroid and assigned to the two clusters and The above balanced binary clustering operation can be summarized as: In formula (6), dist(x,y) represents the Euclidean distance between x and the centroid y in the feature space. and It represents the clusters assigned to the points, argsort is the sorting operation, and returns the sorted index value; The average of each cluster is used to update the cluster centroid; balanced binary clustering iteratively updates the cluster centroid and the membership of each sample. At level l, Q is grouped into 2 l Subset Each subset has equal size |Q i |=N / 2 l ; Subsequently, K and V were divided into and Each subset is related to Q i Share the same index, ensuring that in the self-attention mechanism, Q i , K i and V i can correspond correctly; finally, in each subset (Q i , K i 、V i ) Using standard multi-head self-attention, output Y i =MSA(Q i ,K i ,V i ); finally merge the union of all subsets At the same time, its original order is maintained to ensure that the extracted features can be combined in the order of the original point cloud and the spatial information of the point cloud is retained.

8. The three-dimensional point cloud shape recognition method based on multi-view features according to claim 1, characterized in that: In step S2, the projection module uses the depth map As input, where H and W represent the height and width of the depth image respectively, and v represents the number of depth maps; the depth image D is processed step by step through the network in step S3 and generates a feature map in Represents the feature dimension of the i-th layer in the projection module; the feature map D i Divided into n bin feature sets Where n∈{1,2,4,8,16,32} represents the number of bins that each scale feature map is divided into, and each bin represents a part of the feature map at a specific scale. By dividing the feature map into multiple bins, it can ensure that the model can pay attention to features of different sizes. Perform maximum pooling on the bin features of each scale to obtain the most discriminative features of the feature map of each scale Finally, all features from B1 to B4 are concatenated as the output of the entire projection module.

Citation Information

Patent Citations

  • Three-dimensional point cloud model classification method and system based on regional correlation modeling

    CN114299339A

  • Image super-resolution reconstruction method based on sparse adaptive clustering

    CN116777745A

  • Point cloud data classification and segmentation method based on multi-view adaptive graph convolutional network

    CN116912561A

  • Partial-to-partial point cloud registration method based on multiple attention fusion network

    CN117036431A

  • Deep hash media search method of double-branch attention mechanism

    CN118035483A