3D point cloud classification method based on attention mechanism, terminal device and storage medium

Abstract

The present disclosure provides a 3D point cloud classification method based on an attention mechanism, a terminal device and a computer readable storage medium, wherein the method comprises: creating an attention mechanism neural network model, the attention mechanism neural network model comprising M layers of attention mechanism convolution network layers; inputting 3D point clouds into the attention mechanism convolution network layers level by level to obtain point class features of all points in the 3D point clouds; and classifying the 3D point clouds based on the point class features of all points in the 3D point clouds. The present disclosure creates a neural network model based on an attention mechanism in view of the unordered characteristics of 3D point clouds, so that important points and their features in the 3D point clouds can be continuously focused on by the neural network during the learning process, and the attributes of edges in the point cloud can be considered, and the classification result accuracy of the point cloud is higher compared with ordinary neural networks.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, in particular to a 3D point cloud classification method based on an attention mechanism, a terminal device and a computer readable storage medium. BACKGROUND

[0002] A 3D point cloud is an unordered representation of the geometric structure of the surface of a 3D object, which can be acquired by 3D vision technology such as a 3D depth camera. The 3D point cloud can be applied in the fields of three-dimensional object reconstruction, reverse engineering design, unmanned driving, robots, etc. Therefore, in the field of machine vision, there is a trend of developing from 2D to 3D vision, and 3D point cloud data is also growing rapidly.

[0003] Unlike human vision, how to teach machines to accurately recognize 3D objects and efficiently classify 3D point clouds is a very challenging task. Although neural networks have been proposed to learn 3D point clouds to complete the classification of 3D point clouds and improve the accuracy of point cloud classification, the accuracy of point cloud classification using ordinary graph neural networks is low because the points in the point cloud have coordinates and the properties of the edges need to be considered in addition to the properties of the points. SUMMARY

[0004] The present disclosure provides a 3D point cloud classification method based on an attention mechanism, a terminal device and a computer readable storage medium to at least solve the problem of low accuracy of 3D point cloud classification using ordinary neural networks.

[0005] To achieve the above-mentioned purpose, the present disclosure provides a 3D point cloud classification method based on an attention mechanism, comprising:

[0006] creating an attention mechanism neural network model, wherein the attention mechanism neural network model comprises an attention mechanism convolution network layer connected by M layers;

[0007] inputting the 3D point cloud into the attention mechanism convolution network layer level by level to obtain the point class features of all points in the 3D point cloud; and

[0008] classifying the 3D point cloud based on the point class features of all points in the 3D point cloud.

[0009] In an embodiment, the method further comprises:

[0010] inserting a residual connection in the attention mechanism convolution network layer.

[0011] In an embodiment, before inputting the 3D point cloud into the attention mechanism convolution network layer level by level, the method further comprises:

[0012] Symmetric transformation is performed on the 3D point cloud to obtain a 3D point cloud subjected to symmetric transformation;

[0013] The 3D point cloud is input into the attention mechanism convolution network layer step by step to obtain point class features of all points in the 3D point cloud, including:

[0014] The 3D point cloud subjected to symmetric transformation is input into the attention mechanism convolution network layer step by step to obtain point class features of all points in the 3D point cloud.

[0015] In an embodiment, the symmetric transformation performed on the 3D point cloud includes:

[0016] The symmetric transformation is performed on the 3D point cloud by using a PointNet symmetric function.

[0017] In an embodiment, the 3D point cloud is input into the attention mechanism convolution network layer step by step to obtain point class features of all points in the 3D point cloud, including:

[0018] The 3D point cloud is input into the attention mechanism convolution network layer of the first layer to the attention mechanism convolution network layer of the Mth layer step by step to obtain final global point class features and local edge point class features of all points in the 3D point cloud;

[0019] The final global point class features and the local edge point class features of all points in the 3D point cloud are subjected to a pooling aggregation operation to obtain point class features of all points in the 3D point cloud.

[0020] In an embodiment, the method further includes:

[0021] The EdgeConv point cloud local feature extraction algorithm is inserted into the attention mechanism convolution network layer;

[0022] The 3D point cloud is input into the attention mechanism convolution network layer of the first layer to the attention mechanism convolution network layer of the Mth layer step by step to obtain final global point class features and local edge point class features of all points in the 3D point cloud, including:

[0023] The 3D point cloud is input into the attention mechanism convolution network layer of the first layer to the attention mechanism convolution network layer of the Mth layer step by step, and the local edge point class features of all points in the 3D point cloud in the attention mechanism convolution network layer of the first layer to the attention mechanism convolution network layer of the Mth layer are extracted based on the EdgeConv point cloud local feature extraction algorithm.

[0024] In an embodiment, the 3D point cloud is input into the attention mechanism convolution network layer of the first layer to the attention mechanism convolution network layer of the Mth layer step by step to obtain final global point class features and local edge point class features of all points in the 3D point cloud, including:

[0025] inputting the 3D point cloud into the first attention mechanism convolutional network layer, and extracting initial global point class features and initial local edge point class features of all points of the 3D point cloud;

[0026] pooling and aggregating the initial global point class features and the initial local edge point class features to obtain one-dimensional feature vectors of the initial global point class features and the initial local edge point class features;

[0027] sending the one-dimensional feature vectors into an encoder-decoder structure to learn shared attention weights of the one-dimensional feature vectors;

[0028] re-calibrating the shared attention weights by using a Sigmoid gating function to obtain calibrated attention weights of the initial global point class features and calibrated attention weights of the initial local edge point class features;

[0029] obtaining global point class features and local edge point class features of all points of the 3D point cloud in the first attention mechanism convolutional network layer based on the calibrated attention weights of the initial global point class features and the calibrated attention weights of the initial local edge point class features;

[0030] inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the first attention mechanism convolutional network layer into a next attention mechanism convolutional network layer to continue to obtain global point class features and local edge point class features of all points of the 3D point cloud in the next attention mechanism convolutional network layer;

[0031] inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the next attention mechanism convolutional network layer into a next attention mechanism convolutional network layer to continue to obtain global point class features and local edge point class features of all points of the 3D point cloud in the next attention mechanism convolutional network layer;

[0032] judging whether the next attention mechanism convolutional network layer is the Mth layer, if yes, inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the current attention mechanism convolutional network layer into the Mth attention mechanism convolutional network layer to obtain final global point class features and local edge point class features of all points of the 3D point cloud.

[0033] In an implementation, the 3D point cloud is input into the first attention mechanism convolutional network layer to the Mth attention mechanism convolutional network layer to obtain final global point class features and local edge point class features of all points of the 3D point cloud, including:

[0034] inputting the 3D point cloud into the first attention mechanism convolutional network layer, and extracting initial global point class features and initial local edge point class features of all points of the 3D point cloud;

[0035] pooling and aggregating the initial global point class feature and the initial local edge point class feature to obtain a one-dimensional feature vector of the initial global point class feature and the initial local edge point class feature;

[0036] sending the one-dimensional feature vector into an encoder-decoder structure to learn shared attention weights of the one-dimensional feature vector;

[0037] re-calibrating the shared attention weights by using a Sigmoid gating function to obtain calibrated attention weights of the initial global point class feature and calibrated attention weights of the initial local edge point class feature;

[0038] obtaining global point class features and local edge point class features of all points of the 3D point cloud in the first layer of the attention mechanism convolutional network layer based on the calibrated attention weights of the initial global point class feature and the calibrated attention weights of the initial local edge point class feature;

[0039] inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the first layer of the attention mechanism convolutional network layer into the next layer of the attention mechanism convolutional network layer to continue to obtain global point class features and local edge point class features of all points of the 3D point cloud in the next layer of the attention mechanism convolutional network layer;

[0040] inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the next layer of the attention mechanism convolutional network layer into the next layer of the attention mechanism convolutional network layer to continue to obtain global point class features and local edge point class features of all points of the 3D point cloud in the next layer of the attention mechanism convolutional network layer;

[0041] determining whether the next layer of the attention mechanism convolutional network layer is the Mth layer, if yes, inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the current layer of the attention mechanism convolutional network layer and the global point class features and the local edge point class features of all points of the 3D point cloud in the first layer to the (M-1)th layer of the attention mechanism convolutional network layer into the Mth layer of the attention mechanism convolutional network layer to obtain the final global point class features and the local edge point class features of all points of the 3D point cloud.

[0042] In an embodiment, the pooling and aggregation operation of the final global point class features and the local edge point class features of all points of the 3D point cloud comprises:

[0043] performing the pooling and aggregation operation on the final global point class features and the local edge point class features of all points of the 3D point cloud by using the average pooling and the maximum pooling respectively.

[0044] In an implementation, the 3D point cloud is classified based on the point class features of all points in the 3D point cloud, including:

[0045] The point class features of all points in the 3D point cloud are input into an MLP multi-layer perception system, and the 3D point cloud is classified based on the point class features of all points in the 3D point cloud in the MLP multi-layer perception system to obtain a classification result.

[0046] In an implementation, after the 3D point cloud is classified based on the point class features of all points in the 3D point cloud in the MLP multi-layer perception system, further including:

[0047] The classification result is analyzed by using a Triplet loss function to obtain an analysis result;

[0048] Based on the analysis result, it is determined whether the loss value of the classification result meets an expected value, and if not, the step of classifying the 3D point cloud based on the point class features of all points in the 3D point cloud in the MLP multi-layer perception system is re-executed to classify the 3D point cloud based on the point class features of all points in the 3D point cloud until the loss value of the classification result meets the expected value.

[0049] To achieve the above object, the disclosure further provides a terminal device comprising a memory and a processor, the memory storing a computer program, and when the processor executes the computer program stored in the memory, the processor executes the attention mechanism-based 3D point cloud classification method.

[0050] To achieve the above object, the disclosure further provides a computer readable storage medium storing a computer program, and when the processor executes the computer program, the processor executes the attention mechanism-based 3D point cloud classification method.

[0051] The technical solution provided by the disclosure can include the following beneficial effects:

[0052] The attention mechanism-based 3D point cloud classification method provided by the disclosure creates an attention mechanism neural network model, which includes M layers of attention mechanism convolution network layers, then inputs the 3D point cloud into the attention mechanism convolution network layers level by level to obtain the point class features of all points in the 3D point cloud, and classifies the 3D point cloud based on the point class features of all points in the 3D point cloud. The disclosure creates a neural network model based on the attention mechanism for the unordered characteristics of the 3D point cloud, and in the learning process, the important points and their features in the 3D point cloud can be continuously focused on by the neural network, and the attributes of the edges in the point cloud can be considered. Compared with ordinary neural networks, the classification result accuracy of the point cloud is higher.

[0053] Additional features and advantages of the present disclosure will be set forth in the description that follows, and in part will be apparent from the description, or can be learned by practice of the present disclosure. The objectives and other advantages of the present disclosure will be realized and attained by the structure particularly pointed out in the description and claims. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings are included to provide a further understanding of the technical solutions of the present disclosure, and constitute a part of the specification, and are used to explain the technical solutions of the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions of the present disclosure.

[0055] Figure 1 A flowchart of a 3D point cloud classification method based on an attention mechanism provided by an embodiment of the present disclosure;

[0056] Figure 2 A flowchart of another 3D point cloud classification method based on an attention mechanism provided by an embodiment of the present disclosure;

[0057] Figure 3 One of flowcharts of yet another 3D point cloud classification method based on an attention mechanism provided by an embodiment of the present disclosure;

[0058] Figure 4 A flowchart of extracting global point class features and local edge point class features of all points in a 3D point cloud of each AMN layer in an embodiment of the present disclosure;

[0059] Figure 5 The second flowchart of yet another 3D point cloud classification method based on an attention mechanism provided by an embodiment of the present disclosure;

[0060] Figure 6 A flowchart of performing pooling and aggregation operations on global point class features and local edge point class features of all points in a 3D point cloud in an embodiment of the present disclosure;

[0061] Figure 7 A structural schematic diagram of a terminal device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the specific embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure, and are not used to limit the present disclosure.

[0063] It should be noted that the terms "first", "second", etc. in the description of the present disclosure and the claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence; and, without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other at will.

[0064] In the following description, the suffixes such as "module", "component", or "unit" used to represent elements are only for the convenience of the description of the present disclosure, and have no specific meaning. Therefore, "module", "component", or "unit" can be used interchangeably.

[0065] To solve the above problems, the present embodiment provides a 3D point cloud classification method based on attention mechanism, as shown in Figure 1 The method comprises steps S101-S103.

[0066] In step S101, an attention mechanism neural network model is created, which comprises M layers of attention mechanism convolution network layers.

[0067] In this embodiment, the creation of the attention mechanism neural network (AMN) model is achieved by adding an attention mechanism to the convolutional neural network (CNN) model. The attention mechanism can be regarded as a combination function, which highlights the influence of a certain key input on the output by calculating the probability distribution of attention. For unordered point clouds, important points such as contour points and skeleton points and their features can be continuously focused on by the neural network. In addition to considering point attributes, edge attributes are also considered, which can more accurately perform classification tasks compared to ordinary neural networks.

[0068] In an embodiment, the parameters of the attention mechanism neural network model, such as weight, bias parameters, etc., can be continuously trained to make the created attention neural network model have higher accuracy.

[0069] In step S102, the 3D point cloud is input into the attention mechanism convolution network layer level by level, and the point class features of all points in the 3D point cloud are obtained.

[0070] Specifically, the 3D point cloud is input into the M-layer connected attention mechanism neural network layer AMN step by step to extract the class features of the point cloud (the first step is to map the three-dimensional vector to a 64-dimensional high-dimensional space), that is, the output of the upper AMN network is used as the input of the lower AMN network, and more levels and richer semantic feature representations of the point cloud can be obtained through the multi-level AMN network in the form of hierarchical feature representation.

[0071] It should be noted that the point class feature mentioned in the embodiment is a feature used for classifying the 3D point cloud.

[0072] In an implementation manner, in order to avoid gradient disappearance and accelerate feature convergence, the method further includes the following step: inserting a residual connection in the attention mechanism convolutional network layer. The residual connection allows the original information of the low layer to be directly transmitted to the subsequent high layer, so that the high layer focuses on the learning of the residual, and the degradation of the model is avoided.

[0073] Further, considering the unordered nature of the 3D point cloud, the input 3D point cloud is first subjected to a symmetric transformation, so that the order of the input point cloud is unchanged, and the point cloud permutation invariance is ensured, so as to facilitate subsequent work and improve the accuracy and efficiency of point cloud classification. The specific manner is described in detail in the following embodiment, which is not repeated here.

[0074] In step S103, the 3D point cloud is classified based on the point class features of all points in the 3D point cloud.

[0075] Compared with the ordinary convolutional neural network, the attention mechanism neural network model created based on the unordered characteristics of the point cloud and the attention mechanism can focus on the important points and their features in the point cloud which are continuously paid attention to by the neural network after receiving the input 3D point cloud, and can consider the edge attributes (distance between points), so that the point cloud classification task can be more accurately performed.

[0076] Please refer to Figure 2 , Figure 2 Another attention mechanism-based 3D point cloud classification method provided by the embodiment of the present disclosure considers that the permutation invariance transformation of the point cloud when performing deep learning is also an important problem of 3D point cloud classification. On the basis of the previous embodiment, the 3D point cloud is first subjected to a symmetric transformation, and then the 3D point cloud subjected to the symmetric transformation is input into the model for learning, and finally the classification of the point cloud is completed. Specifically, before the 3D point cloud is input into the attention mechanism convolutional network layer step by step (step S102), step S201 is further included, and step S102 is further divided into step S102a.

[0077] It should be noted that, with the continuous maturity of deep learning technology, how to use deep learning tools to realize the classification processing of 3D point cloud has become a major research direction, but 3D point cloud is unstructured data, and point cloud is scattered and sparse in three-dimensional space, represented as an xyz coordinate point, and does not have a structured grid to help ordinary neural networks perform convolution operations, such as convolutional neural network. At the same time, 3D point cloud is represented as an N x 3 matrix in CNN operation, where the row vector N represents the number of point clouds, and the column vector represents the xyz three-dimensional coordinates of each point. Geometrically, two completely different matrices can represent the same point cloud set, but the running results of two different matrices in CNN are completely different, so the permutation invariant transformation of the point cloud is also an important part of 3D point cloud processing.

[0078] In step S201, the 3D point cloud is subjected to symmetry transformation to obtain a 3D point cloud subjected to symmetry transformation.

[0079] Specifically, taking an N x 3 original 3D point cloud PC as input, the row vector N represents the number of points in the point cloud set, and the column vector represents the xyz three-dimensional coordinates of each point. A 3 x 3 symmetry transformation matrix is generated by a symmetry transformation network for point cloud geometric transformation. After the N x 3 original point cloud PC is multiplied by the 3 x 3 symmetry transformation matrix, an N x 3 3D point cloud set PC' subjected to symmetry transformation is obtained.

[0080] In this embodiment, the PointNet symmetry function is used to perform symmetry transformation on the 3D point cloud.

[0081] By using PointNet (vanilla) to perform symmetry transformation on the point cloud, the global features of the 3D point cloud can be extracted point by point.

[0082] In related technologies, when using deep learning methods to process point clouds, it is often necessary to convert them into a specific perspective depth image (2D mapping) or a more regular format such as three-dimensional voxel (Voxel) in order to define the weight-sharing convolution operation. Or directly use the PointNet algorithm, based on the PointNet classification network, to perform input transformation and feature transformation on the input point cloud, and then perform feature aggregation, but this architecture cannot capture the local structure caused by the metric space point, and the local structure is very important for 3D point cloud convolution architecture, because it will limit its ability to identify fine-grained patterns and generalize to complex scenes. In this embodiment, the 3D point cloud is subjected to symmetry transformation by the PointNet symmetry function before being input into the model, and then input into the attention mechanism convolutional neural network layer, which can realize the extraction of global and local features of the point cloud.

[0083] In step S102a, the 3D point cloud after the symmetry transformation is input into the attention mechanism convolution network layer by layer to obtain the point class features of all points in the 3D point cloud.

[0084] Because of the unordered nature of the point cloud, the input point cloud is transformed by a symmetry function method such as PointNet (vanilla), and under the condition that the function is continuous on any Hausdorff space and the number of network layers is sufficient, PC' can be considered to be infinitely close to PC, so as to align all points to a regular space, make the input order of the point cloud unchanged, guarantee the point cloud permutation invariance, and then input into the convolution network for subsequent work.

[0085] Please refer to Figure 3 , Figure 3 Another kind of 3D point cloud classification method based on attention mechanism is provided for the embodiments of the present disclosure. Considering that the point cloud classification in the prior art lacks effective local region feature analysis, the embodiments insert a point cloud local feature extraction algorithm in each attention mechanism convolution neural network layer, combine the global point class features and the local point class features of the point cloud, and complete the classification of the 3D point cloud. Specifically, the 3D point cloud after the symmetry transformation is input into the attention mechanism convolution network layer by layer to obtain the point class features of all points in the 3D point cloud (step S102a), which is further divided into step S102a' and step S102a''.

[0086] In step S102a', the 3D point cloud after the symmetry transformation is input into the first layer of the attention mechanism convolution network layer to the Mth layer of the attention mechanism convolution network layer to obtain the final global point class features and the local edge point class features of all points in the 3D point cloud.

[0087] As described above, the 3D point cloud can extract its global features after the transformation, but still lacks effective local region features. The embodiments insert a local feature extraction algorithm in each layer of the attention mechanism convolution network AMN layer to obtain the point cloud local features. In an implementation manner, the point cloud local features are extracted by inserting the EdgeConv point cloud local feature extraction algorithm. The method further includes the following steps:

[0088] The EdgeConv point cloud local feature extraction algorithm is inserted in the attention mechanism convolution network layer.

[0089] The 3D point cloud is input into the first layer of the attention mechanism convolution network layer to the Mth layer of the attention mechanism convolution network layer to obtain the final global point class features and the local edge point class features of all points in the 3D point cloud (step S102a), which includes the following steps:

[0090] The 3D point cloud is input into the attention mechanism convolution network layer of the 1st layer to the attention mechanism convolution network layer of the Mth layer, and the local edge point class features of all points in the 3D point cloud in the attention mechanism convolution network layer of the 1st layer to the attention mechanism convolution network layer of the Mth layer are extracted based on the EdgeConv point cloud local feature extraction algorithm.

[0091] In this embodiment, the point cloud local feature extraction method such as EdgeConv is introduced to obtain the point cloud local region feature. Specifically, the local region feature structure of each point and the surrounding adjacent points is represented, and the multi-scale local point cloud cluster is used to increase the robustness. Therefore, the AMN layer can extract the point cloud local shape feature and obtain the global feature at the same time, while maintaining the overall point cloud permutation invariance. This hierarchical feature representation fully combines the global point class feature and the local edge class feature to obtain better point cloud classification performance.

[0092] In step S102a", the final global point class feature and the local edge point class feature of all points in the 3D point cloud are subjected to a pooling aggregation operation to obtain the point class feature of all points in the 3D point cloud.

[0093] Further, the AMN layer of this embodiment will also calculate the attention weight value of the global point class feature and the local edge class feature respectively through the attention mechanism, recalibrate the class features generated by each network layer, and multiply the weight matrix to input the adjusted features into the next layer AMN network. The use of the attention mechanism can make the important points (such as contour points and skeleton points) and their features in the point cloud continue to be focused on by the neural network, so as to better perform the classification task. The following takes two implementation ways as examples:

[0094] In one implementation, the global point class feature and the local edge point class feature of the 3D point cloud are obtained by using the output feature of the Mth layer. Specifically, the 3D point cloud is input into the attention mechanism convolution network layer of the 1st layer to the attention mechanism convolution network layer of the Mth layer to obtain the final global point class feature and the local edge point class feature of all points in the 3D point cloud (step S102a), including the following steps a-h:

[0095] a. The 3D point cloud is input into the attention mechanism convolution network layer of the 1st layer to extract the initial global point class feature and the initial local edge point class feature of all points in the 3D point cloud.

[0096] Specifically, the point cloud after the symmetric transformation (PointNet or other methods can be adopted based on the above embodiments) is input into the AMN layer to obtain the global feature of each point in the 3D point cloud. Due to the lack of effective local region features, the local edge point class feature of the point cloud is extracted by inserting a point cloud local edge feature extraction algorithm such as EdgeConv into the AMN layer, and then input into the first layer of AMN to obtain the initial global point class feature and the initial local edge point class feature.

[0097] b. The initial global point class feature and the initial local edge point class feature are pooled and aggregated to obtain a one-dimensional feature vector of the initial global point class feature and the initial local edge point class feature.

[0098] c. The one-dimensional feature vector is sent to an encoder-decoder structure to learn the shared attention weight of the one-dimensional feature vector.

[0099] d. The shared attention weight is recalibrated by using a Sigmoid gating function to obtain the attention weight of the calibrated initial global point class feature and the attention weight of the calibrated initial local edge point class feature.

[0100] e. Based on the attention weight of the calibrated initial global point class feature and the attention weight of the calibrated initial local edge point class feature, the global point class feature and the local edge point class feature of all points in the 3D point cloud in the first layer of the attention mechanism convolutional network layer are obtained.

[0101] f. The global point class feature and the local edge point class feature of all points in the 3D point cloud in the first layer of the attention mechanism convolutional network layer are input into the next layer of the attention mechanism convolutional network layer to continue to obtain the global point class feature and the local edge point class feature of all points in the 3D point cloud in the next layer of the attention mechanism convolutional network layer.

[0102] g. The global point class feature and the local edge point class feature of all points in the 3D point cloud in the next layer of the attention mechanism convolutional network layer are input into the next layer of the attention mechanism convolutional network layer to continue to obtain the global point class feature and the local edge point class feature of all points in the 3D point cloud in the next layer of the attention mechanism convolutional network layer.

[0103] For steps b-f, the specific operation principle of the AMN layer is combined with Figure 4 It can be understood that the operation of the AMN layer of the second layer to the M-1 layer is also applicable. Specifically, the global point class feature (including the initial global point class feature of the first layer) and the local edge point class feature (including the initial local edge point class feature of the first layer) are first input into the maximum pooling aggregation to obtain a one-dimensional vector, and then the one-dimensional vector is sent to the encoder-decoder architecture with shared weights to learn the attention weight value of the global / edge feature to determine its influence on the output.

[0104] wherein the encoder is configured to find certain patterns in the original data, and generate the most meaningful context concise representation by compressing the original dimensional data, and the decoder is configured to utilize the context to up-sample to the original dimension to generate the high-dimensional output data such as descriptive knowledge, and the purpose of the encoder-decoder architecture is to improve the quality of the representation generated by the network by explicitly modeling the degree of interdependence between the channels of the convolutional features, and the network can allocate different attention weight values to the input according to the representation, so as to realize a recalibration mechanism for the feature channels, so that the network can selectively emphasize important information features and suppress unimportant features. Then, the attention weights of the global point class features and the edge local class features of each point are obtained through the Sigmoid gating function, the weight values are controlled between 0 and 1, and the sum of the weights of each feature channel (global feature and edge local feature) is 1. The global point class features and the edge local class features are rescaled by multiplying the original information with the weight matrix, and the influence of the global point class features and the edge local class features on the output is adjusted. The recalibrated global point class features and edge local class features of the current layer are connected and sent to the next AMN network.

[0105] h. determining whether the next attention mechanism convolutional network layer is the Mth layer, if yes, inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the attention mechanism convolutional network layer of the current layer into the attention mechanism convolutional network layer of the Mth layer to obtain the final global point class features and the local edge point class features of all points in the 3D point cloud.

[0106] In the embodiment, the aligned point cloud is sent into a series of connected attention mechanism convolutional network layers, and the global point class features and the local edge class features are represented by layered features, and the global point class features and the local edge point class features of all points output by the Mth (1024) layer will have better point cloud classification performance.

[0107] In another embodiment, in order to fully utilize the feature outputs of each AMN layer and obtain complete global context aggregation information, the outputs of each AMN layer are connected and sent into the last AMN (1024) layer to generate an N x 1024 output result. Specifically, the 3D point cloud is input into the first layer of the attention mechanism convolutional network layer to the Mth layer of the attention mechanism convolutional network layer to obtain the final global point class features and the local edge point class features of all points in the 3D point cloud (step S102a), including the following steps:

[0108] a. inputting the 3D point cloud into the first layer of the attention mechanism convolutional network layer to extract the initial global point class features and the initial local edge point class features of all points of the 3D point cloud;

[0109] pooling and aggregating the initial global point class feature and the initial local edge point class feature to obtain a one-dimensional feature vector of the initial global point class feature and the initial local edge point class feature;

[0110] b. inputting the one-dimensional feature vector into an encoder-decoder structure to learn shared attention weights of the one-dimensional feature vector;

[0111] c. re-calibrating the shared attention weights by using a Sigmoid gating function to obtain attention weights of the calibrated initial global point class feature and attention weights of the calibrated initial local edge point class feature;

[0112] d. obtaining global point class features and local edge point class features of all points in the 3D point cloud at a first layer of the attention mechanism convolutional network layer based on the attention weights of the calibrated initial global point class feature and the attention weights of the calibrated initial local edge point class feature;

[0113] e. inputting the global point class features and the local edge point class features of all points in the 3D point cloud at the first layer of the attention mechanism convolutional network layer into a next layer of the attention mechanism convolutional network layer to continue obtaining global point class features and local edge point class features of all points in the 3D point cloud at the next layer of the attention mechanism convolutional network layer;

[0114] f. inputting the global point class features and the local edge point class features of all points in the 3D point cloud at the next layer of the attention mechanism convolutional network layer into a next layer of the attention mechanism convolutional network layer to continue obtaining global point class features and local edge point class features of all points in the 3D point cloud at the next layer of the attention mechanism convolutional network layer;

[0115] g’. judging whether the next layer of the attention mechanism convolutional network layer is the Mth layer, if yes, inputting the global point class features and the local edge point class features of all points in the 3D point cloud at the current layer of the attention mechanism convolutional network layer and the global point class features and the local edge point class features of all points in the 3D point cloud at the first layer to the (M-1)th layer of the attention mechanism convolutional network layer into the Mth layer of the attention mechanism convolutional network layer to obtain the final global point class features and the local edge point class features of all points in the 3D point cloud.

[0116] It should be noted that the difference between the present example and the previous example is that the information input into the Mth layer of the AMN layer in the present example is the output result of each layer from the first layer to the (M-1)th layer, while the information input into the Mth layer of the AMN layer in the previous example is the output result of the (M-1)th layer. Compared with the previous example, the present example can make more full use of the feature output of each AMN layer, and accordingly, the degree of information redundancy is greater.

[0117] In an embodiment, a pooling aggregation operation is performed on the final global point class features and the local edge point class features of all points in the 3D point cloud (step S102b), including the following steps:

[0118] The final global point class features and the local edge point class features of all points in the 3D point cloud are respectively subjected to a pooling aggregation operation by using average pooling aggregation and maximum pooling aggregation.

[0119] It can be understood that average pooling and maximum pooling are two common ways of pooling operation, wherein average pooling can retain background information, and is operated in the form of sliding window on the feature map (similar to the window sliding of convolution), and the average value in the window is taken as the result, so that the feature map is down-sampled and the overfitting phenomenon is reduced; maximum pooling can extract feature texture and reduce the influence of useless information. After the global point class features and the local edge point class features of all points in the 3D point cloud are extracted, the two kinds of pooling aggregation operations are used for aggregation in this embodiment, so that the overfitting phenomenon can be reduced while the redundant information is reduced.

[0120] This embodiment takes another example (the output results of each AMN layer are connected to the input of the Mth layer) as an example, and combines the results shown in Figure 5 and Figure 6 The N x 1024 point cloud global point class features and the local edge class features of the final output are obviously redundant for representing the 3D point cloud, and by simultaneously using average pooling aggregation and maximum pooling aggregation, two vector representations are obtained, which are sent to the encoder-decoder architecture with shared weights, the influence weight values of the two aggregation methods on the final output results are obtained through the Sigmoid gating function, then the output of each pooling operation is readjusted using the attention weight values, and finally the outputs of the readjusted average pooling and maximum pooling are connected to obtain a 1 x 1024 dimensional vector, which is the global feature of the N point cloud (i.e. the point class features of all points in the 3D point cloud). This global feature aggregation operation based on attention mechanism has greater advantages compared to using only a single aggregation method, and can improve the point cloud classification efficiency by about 15%.

[0121] Further, the 3D point cloud is classified based on the point class features of all points in the 3D point cloud (step S103), including the following steps:

[0122] The point class features of all points in the 3D point cloud are input into the MLP multi-layer perception system, and the 3D point cloud is classified based on the point class features of all points in the 3D point cloud in the MLP multi-layer perception system to obtain a classification result.

[0123] It can be understood that, in addition to the input and output layers, the Multilayer Perceptron (MLP) can have multiple hidden layers in the middle. The simplest MLP contains only one hidden layer, i.e., a three-layer structure. In this embodiment, the full connection structure of the Multilayer Perceptron MLP is used to realize the classification of the point cloud. Specifically, a 1024-dimensional global aggregation feature vector (i.e., the point class feature of all points in the 3D point cloud) is input into a three-layer fully connected MLP multi-layer perception system (MLP(512, 256, c)) for completing the final classification of the point cloud. The last layer of the network is MLP(c), which outputs a 1xc-dimensional vector. c is the number of classification categories of the final output. The value of each column is the score value of the point cloud belonging to the category. Then, the classification result is obtained according to the score value of each point in each category.

[0124] Further, in order to further improve the accuracy of 3D point cloud classification, the Triplet loss function is used for optimization in this embodiment to make the obtained classification result optimal. Specifically, after classifying the 3D point cloud based on the point class feature of all points in the 3D point cloud in the MLP multi-layer perception system, the following steps are further included:

[0125] analyzing the classification result by using the Triplet loss function to obtain an analysis result;

[0126] judging whether the loss value of the classification result meets an expected value based on the analysis result. If not, the step of classifying the 3D point cloud based on the point class feature of all points in the 3D point cloud in the MLP multi-layer perception system is re-executed until the loss value of the classification result meets the expected value.

[0127] In an implementation, the step of classifying the 3D point cloud based on the point class feature of all points in the 3D point cloud in the MLP multi-layer perception system can be re-executed by continuously optimizing the parameters of the MLP multi-layer perception system.

[0128] Specifically, the Triplet loss function is obtained according to the following formula:

[0129]

[0130] In the above formula, L(F, F + , F - ) represents the loss of the point cloud classification relationship in the classification result. The smaller the loss is, the higher the classification accuracy is. F represents the class feature matching degree between any two point pairs in the classification result. F + represents the class feature matching degree of the positive matching point pair. F -a class feature matching degree representing a negative matching point pair, m is a threshold parameter, wherein m, F + and F - are given numerical values.

[0131] Based on the same technical concept, the embodiment of the present disclosure correspondingly further provides a terminal device, as shown in the figure, comprising a memory 71 and a processor 72, the memory 71 stores a computer program, when the processor 72 runs the computer program stored in the memory 71, the processor 72 executes the 3D point cloud classification method based on the attention mechanism. Figure 7

[0132] Based on the same technical concept, the embodiment of the present disclosure correspondingly further provides a computer readable storage medium, which stores a computer program, when the processor executes the computer program, the processor executes the 3D point cloud classification method based on the attention mechanism.

[0133] In summary, the 3D point cloud classification method based on deep learning proposed by the embodiment of the present disclosure, by symmetric transformation alignment of 3D point cloud, and sending the aligned point cloud into a series of connected attention mechanism convolution network layer, and through hierarchical feature representation of global point class feature and local edge class feature, to obtain better point cloud classification performance, in this process, in order to avoid gradient disappearance, residual connection is inserted in the convolution layer. Then, for the final output class feature, through attention mechanism, average pooling aggregation and maximum pooling aggregation are used at the same time, and through attention mechanism, the aggregation operation is recalibrated to improve the classification processing performance, finally, the aggregated feature vector is sent into the multi-layer perception system to complete the classification of 3D point cloud, and the Triplet loss function is introduced, by continuously optimizing the model parameters to make the loss function L value minimum, to achieve the purpose of self-supervised optimization, further improve the point cloud classification accuracy.

[0134] ​Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof. In the hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as known to those skilled in the art, communication media typically includes computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and can include any information delivery medium.

[0135] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present disclosure, and not to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A 3D point cloud classification method based on an attention mechanism, characterized in that, The method comprises the following steps: creating an attention mechanism neural network model, wherein the attention mechanism neural network model comprises M layers of attention mechanism convolution network layers connected in sequence; inserting an EdgeConv point cloud local feature extraction algorithm in each attention mechanism convolution network layer; performing a symmetric transformation on the 3D point cloud to obtain a symmetrically transformed 3D point cloud; gradually inputting the symmetrically transformed 3D point cloud into the first layer of attention mechanism convolution network layer to the Mth layer of attention mechanism convolution network layer, extracting the final global point class features and the final local edge point class features of all points in the 3D point cloud in the first layer of attention mechanism convolution network layer to the Mth layer of attention mechanism convolution network layer, and extracting the final global point class features and the final local edge point class features of all points in the 3D point cloud in the first layer of attention mechanism convolution network layer to the Mth layer of attention mechanism convolution network layer based on the EdgeConv point cloud local feature extraction algorithm; performing a pooling aggregation operation on the final global point class features and the final local edge point class features of all points in the 3D point cloud to obtain the point class features of all points in the 3D point cloud, wherein the average pooling aggregation and the maximum pooling aggregation are simultaneously used to obtain two vector representations, which are sent into a shared weight encoder-decoder architecture, the influence weight values of the two aggregation modes on the final output result are obtained through a Sigmoid gating function, the outputs of the respective pooling operations are then adjusted using the attention weight values, and finally the outputs of the recalibrated average pooling and maximum pooling are connected to obtain a 1×1024-dimensional vector, i.e., the point class features of all points in the 3D point cloud; and classifying the 3D point cloud based on the point class features of all points in the 3D point cloud.

2. The method of claim 1, wherein, The method further comprises the following steps: inserting a residual connection in the attention mechanism convolution network layer.

3. The method of claim 1, wherein, performing a symmetric transformation on the 3D point cloud, comprising: performing a symmetric transformation on the 3D point cloud using a PointNet symmetric function.

4. The method of claim 1, wherein, gradually inputting the 3D point cloud into the first layer of attention mechanism convolution network layer to the Mth layer of attention mechanism convolution network layer to obtain the final global point class features and the final local edge point class features of all points in the 3D point cloud, comprising: inputting the 3D point cloud into the first layer of attention mechanism convolution network layer to extract the initial global point class features and the initial local edge point class features of all points in the 3D point cloud; performing a pooling aggregation operation on the initial global point class features and the initial local edge point class features to obtain a one-dimensional feature vector of the initial global point class features and the initial local edge point class features; sending the one-dimensional feature vector into an encoder-decoder structure to learn the shared attention weight of the one-dimensional feature vector; recalibrating the shared attention weight using a Sigmoid gating function to obtain the attention weight of the recalibrated initial global point class features and the attention weight of the recalibrated initial local edge point class features; obtaining the global point class features and the local edge point class features of all points in the 3D point cloud in the first layer of attention mechanism convolution network layer based on the attention weight of the recalibrated initial global point class features and the attention weight of the recalibrated initial local edge point class features; inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the first attention mechanism convolution network layer into a next attention mechanism convolution network layer, and continuously obtaining the global point class features and the local edge point class features of all points of the 3D point cloud in the next attention mechanism convolution network layer; inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the first attention mechanism convolution network layer into a next attention mechanism convolution network layer, and continuously obtaining the global point class features and the local edge point class features of all points of the 3D point cloud in the next attention mechanism convolution network layer; determining whether the next attention mechanism convolution network layer is the Mth layer, if yes, inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the current attention mechanism convolution network layer into the Mth attention mechanism convolution network layer, and obtaining the final global point class features and the final local edge point class features of all points of the 3D point cloud.

5. The method of claim 1, wherein, inputting the 3D point cloud into the first attention mechanism convolution network layer, and extracting initial global point class features and initial local edge point class features of all points of the 3D point cloud; pooling and aggregating the initial global point class features and the initial local edge point class features to obtain a one-dimensional feature vector of the initial global point class features and the initial local edge point class features; sending the one-dimensional feature vector into an encoder-decoder structure to learn shared attention weights of the one-dimensional feature vector; re-calibrating the shared attention weights by using a Sigmoid gating function to obtain calibrated attention weights of the initial global point class features and calibrated attention weights of the initial local edge point class features; obtaining the global point class features and the local edge point class features of all points of the 3D point cloud in the first attention mechanism convolution network layer based on the calibrated attention weights of the initial global point class features and the calibrated attention weights of the initial local edge point class features; inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the first attention mechanism convolution network layer into a next attention mechanism convolution network layer, and continuously obtaining the global point class features and the local edge point class features of all points of the 3D point cloud in the next attention mechanism convolution network layer; inputting the global point class features and the local edge point class features of all points of the 3D point cloud in the first attention mechanism convolution network layer into a next attention mechanism convolution network layer, and continuously obtaining the global point class features and the local edge point class features of all points of the 3D point cloud in the next attention mechanism convolution network layer; ​ If the next attention mechanism convolutional network layer is the Mth layer, the global point class feature and the local edge point class feature of all points in the 3D point cloud in the attention mechanism convolutional network layer and the global point class feature and the local edge point class feature of all points in the 3D point cloud in the first to (M-1)th attention mechanism convolutional network layers are input into the Mth attention mechanism convolutional network layer, and the final global point class feature and the final local edge point class feature of all points in the 3D point cloud are obtained.

6. The method of claim 1, wherein, The final global point class feature and the final local edge point class feature of all points in the 3D point cloud are subjected to a pooling aggregation operation, including: The final global point class feature and the final local edge point class feature of all points in the 3D point cloud are subjected to a pooling aggregation operation, including:

7. The method of claim 1, wherein, The 3D point cloud is classified based on the point class feature of all points in the 3D point cloud, including: The point class feature of all points in the 3D point cloud is input into an MLP multi-layer perception system, and the 3D point cloud is classified based on the point class feature of all points in the 3D point cloud in the MLP multi-layer perception system to obtain a classification result.

8. The method of claim 7, wherein, After the 3D point cloud is classified based on the point class feature of all points in the 3D point cloud in the MLP multi-layer perception system, further including: The classification result is analyzed by using a Triplet loss function to obtain an analysis result; It is judged whether the loss value of the classification result meets an expected value based on the analysis result, and if not, the step of classifying the 3D point cloud based on the point class feature of all points in the 3D point cloud in the MLP multi-layer perception system is returned to classify the 3D point cloud based on the point class feature of all points in the 3D point cloud again until the loss value of the classification result meets the expected value.

9. A terminal device, comprising: The computer program is executed by the processor, and the processor executes the attention mechanism-based 3D point cloud classification method according to any one of claims 1 to 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor, and the processor executes the attention mechanism-based 3D point cloud classification method according to any one of claims 1 to 8.

Images