A point cloud classification and segmentation network based on point cloud multi-scale parallel feature extraction and attention mechanism

Abstract

The application discloses a kind of to solve prior art defects and propose a kind of based on point cloud parallel multi-scale feature extraction and attention mechanism's point cloud classification segmentation network Parallel-Net, specifically related to a kind of based on point cloud multi-scale parallel feature extraction and attention mechanism's point cloud classification segmentation network, to improve accuracy rate.The method uses parallel multi-scale feature extraction and cross attention mechanism, including the following steps:1) input point cloud data, carries out feature learning;2) by down-sampling algorithm carries out feature extraction and sampling;3) introduce self-attention mechanism calculation position correlation;4) use cross attention to process multiple down-sampled point cloud features;5) apply up-sampling algorithm to transfer feature information.The application can effectively extract 3D point cloud features, improve point cloud classification and entity segmentation accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D point cloud object classification and object segmentation feature extraction, in particular to a point cloud classification and point cloud entity segmentation network based on parallel multi-scale feature extraction, residual connection and up-sampling of point cloud feature fusion. BACKGROUND

[0002] Point cloud data refers to a set of vectors in a three-dimensional coordinate system. Scanning data is recorded in the form of points, and each point contains three-dimensional coordinates, and some may contain color information (RGB) or intensity information (Intensity).

[0003] The background of point cloud classification and segmentation technology can be traced back to the development of computer vision and machine learning. The background of point cloud classification and segmentation technology can be attributed to the research and development of three-dimensional data processing algorithms, including the progress of sensor technology, the application of deep learning methods, etc. These technologies have promoted the continuous development and application of point cloud classification and segmentation technology. Point cloud classification and segmentation technology refers to the algorithm and method of classifying and segmenting points in point cloud data according to their categories. It has important applications in computer vision, autonomous driving, robot perception, etc. By classifying and segmenting objects such as roads, vehicles, pedestrians, etc., it can help autonomous driving systems better perceive and understand the surrounding environment, and realize intelligent decision-making and path planning.

[0004] Point cloud classification and segmentation tasks face the following challenges and difficulties:

[0005] 1. Point cloud data may be incomplete and uneven due to occlusion, missing and sparsity in the acquisition process, which brings difficulties to classification and segmentation algorithms.

[0006] 2. Due to sensor noise, environmental interference and other factors, point cloud data often contains noise and outliers, reducing the accuracy and robustness of classification and segmentation algorithms.

[0007] 3. Point cloud data has invariance under different perspectives, scales and poses. Classification and segmentation algorithms need to be able to recognize and learn these invariance features in order to accurately classify and segment objects.

[0008] To address these challenges, researchers have proposed a series of point cloud classification and segmentation algorithms and methods. Among them, methods based on deep learning such as PointNet, PointNet++, PointCNN, Point transformer, etc. can perform end-to-end processing and learning on point cloud data by using fully connected or convolutional neural networks (CNN) to operate on point cloud data, extract local features and perform classification and segmentation. But the above methods are aimed at extracting local features from multi-layer feature fusion results, and due to the serial learning of point cloud features, the output accuracy of the network is not high. Therefore, the present application proposes a point cloud classification and segmentation network based on point cloud multi-scale parallel feature extraction and attention mechanism to solve the above technical defects of the prior art, which has better effect on point cloud data classification and entity segmentation. SUMMARY

[0009] The present application aims to provide a 3D point cloud information feature extraction method Parallel-Net network based on the residual connection in Resnet, matrix operation in Transformer and SA (Set Abstraction) and upsampling in PointNet++, to improve the accuracy of point cloud classification and entity segmentation.

[0010] To achieve the above purpose, the present application adopts the following technical scheme: a point cloud classification and segmentation network structure based on point cloud parallel multi-scale feature extraction and cross attention mechanism, comprising the following steps:

[0011] Step one, input the point cloud data set, and perform the first feature learning through the full connection layer. The input point cloud data is converted into initial feature representation through the full connection layer.

[0012] Step two, use the down-sampling algorithm for feature extraction and sampling. Feature extraction captures the local features of point cloud through local region message passing, while sampling is used to reduce the number of point clouds and preserve the global structure information. Through these two steps, higher level feature representation can be obtained.

[0013] Step three, use the self-attention mechanism to calculate the attention weight of each position in the point cloud input sequence according to the correlation of different positions. By calculating the similarity between each point and other points, the attention degree of each point is obtained, then these weights are multiplied with the input vector and summed to get the weighted representation, so as to realize the attention degree regulation of different positions.

[0014] Step four, use cross attention algorithm to process the down-sampled point cloud data features with different point cloud feature scales. The correlation between different sequence point cloud feature data is considered when calculating the attention weight. In this way, the expression ability and performance of the model to the relationship between different features are enhanced.

[0015] Step five, use the up-sampling algorithm to convert the point cloud data with known resolution into higher resolution point cloud grid. Then, through a convolution-like operation, the feature information of each point is transferred from the low-resolution local area to the high-resolution local area, and the feature fusion of the obtained point cloud features of different scales is completed.

[0016] Step six, design the network structure of Parallel-Net, design the corresponding loss function and optimizer according to different data sets of point cloud classification and segmentation tasks, and optimize and adjust the network parameters in the training process.

[0017] Further, for the input point cloud data set X ∈ R N×6 , a fully connected layer composed of an input layer, a normalization layer and an activation layer is designed for the first feature learning, and a neural network model containing multiple fully connected layers or convolutional layers and activation functions is used to learn the local feature representation of the point cloud, that is, to obtain the attention score of each point. Through the first feature learning, the global information of the point cloud data is effectively extracted.

[0018] Further, by down-sampling, smaller-scale down-sampled point cloud data is generated, which can extract important features from the original point cloud data while reducing the amount of computation.

[0019] Further, by giving the input point cloud data sequence X, we first obtain three different sequences through linear transformation: query sequence, key sequence and value sequence. Next, the attention weight between points is calculated, and finally the output vector C contains important information of each element in the point cloud data feature sequence, which is weighted according to the correlation between elements.

[0020] Further, through the cross attention mechanism learning of local point cloud features and global point cloud features, the local features are obtained as the correlation between the query sequence and the key sequence, and the global features are obtained as the important information of the value sequence.

[0021] Further, the input point cloud is divided into regions, and in each region, a specific interpolation method is used to up-sample the point cloud. The up-sampled point cloud is merged with the original point cloud, and the distance and weight between each newly generated point and its nearest neighbor point are calculated to obtain the coordinates of the new point.

[0022] Further, during the experiment, the cross-entropy loss function is used to measure the difference between the predicted results and the true labels, and the SGD (Stochastic Gradient Descent) optimizer, the Adam optimizer and the AdamW optimizer are used for experiments,

[0023] In the point cloud classification task, the ModelNet40 dataset is used, and during the training process, the number of points collected by each model is N=1024, and the initial point cloud data feature dimension d=6, and the input point cloud data is strengthened before training;

[0024] In the point cloud entity segmentation task, the ShapeNet dataset is used, and during the training process, the number of points collected by each model is N=1024, and the initial point cloud data feature dimension d=9, and the input point cloud data is strengthened before training, such as range scaling, region translation and random inactivation.

[0025] During the training, the batch_size of the data input is set to 4-16 for testing, the epoch is set to 200-1000 for testing, and the eps is set to 1e-8, and the target value is reached through different training methods.

[0026] The present application has the following beneficial effects:

[0027] The present application adopts a parallel computing method for multi-scale down-sampling local feature extraction. By down-sampling the point cloud data at different scales, we can capture different levels of details and global structures of the point cloud data. Next, for each down-sampled point cloud dataset, we introduce a self-attention mechanism to learn the correlation between each position, thereby enhancing the model's attention to important information within the point cloud data. To further improve the model's expression ability and performance, we introduce a cross-attention mechanism. This mechanism considers the relevance between multiple down-sampled point cloud data features, and captures their relationships by calculating attention weights. In this way, we can better model the interactions between different features in the point cloud data, improving the model's accuracy and robustness. Finally, an up-sampling algorithm is used to convert the point cloud data at a known resolution to a higher resolution point cloud grid. Through operations similar to convolution, the feature information of low-resolution local regions is transferred to high-resolution local regions, so that the model can better capture the details and accurate information of the point cloud data.

[0028] In summary, the present application provides an effective point cloud data feature extraction method through parallel multi-scale feature extraction, self-attention mechanism, cross-attention mechanism and up-sampling, thereby improving the accuracy and performance of point cloud classification and entity segmentation tasks. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 Network structure for point cloud classification and segmentation task of the present application; DETAILED DESCRIPTION

[0030] The embodiments of the present application will be described in detail below with reference to the accompanying drawings, so that the purposes, technical solutions, features and the like of the present application are easier to understand. Obviously, the described embodiments are only a part of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application. The embodiments are only used to explain the present application, but not to limit the present application.

[0031] Step one, input the point cloud dataset, for the input point cloud dataset X ∈ R N×6 , design a fully connected layer composed of an input layer, a normalization layer and an activation layer for the first feature learning, then the output of the multilayer perceptron in the first feature learning is:

[0032] α i =softmax(MLP(X i ))

[0033] Where X i represents the point cloud data in the i-th local region, and α i is the attention score of each point. Here MLP is a neural network model containing multiple fully connected layers (or convolutional layers) and activation functions, which is used to learn the local feature representation of point cloud. After mapping each point feature through MLP, the softmax function is applied to obtain the attention score of each point. Through the first feature learning, the global information of the point cloud data is effectively extracted.

[0034] Step two, feature extraction and sampling are performed through a downsampling algorithm. Feature extraction captures local features of point cloud through message passing in local regions, while sampling is used to reduce the number of point clouds and preserve global structural information.

[0035] The down-sampling formula can be expressed as:

[0036] Where:

[0037] α is the input point cloud data after multilayer perceptron learning, α = {α1, α2, … α n}, where each α i represents a feature vector of a feature point.

[0038] f MLP(.) is a multi-layer perceptron that converts the feature vector of each point into a higher-level feature representation.

[0039] g(.) is a transformation function that can transform the point cloud to increase the invariance of the model.

[0040] f downsample (α) is a down-sampling function that selects important points through a max-pooling operation and generates a smaller-scale point cloud data after down-sampling.

[0041] In this formula, N' represents the number of points in the down-sampled point cloud, which is usually much smaller than N. By applying this formula, down-sampling can extract important features from the original point cloud data.

[0042] Step three, self-attention mechanism, we use the self-attention mechanism to calculate the attention weight of each position by considering the correlation between different positions in the global and local features of the point cloud. Then, multiply these weights with the input vector and sum them up to get the weighted representation, so as to realize the attention degree control of different positions.

[0043] The formula of self-attention mechanism is as follows:

[0044] 1. Given the input point cloud data sequence X, we first get three different sequences through linear transformation: query sequence Q, key sequence K and value sequence V.

[0045] Q = X · WQ + bQ

[0046] K = X · WK + bK

[0047] V = X · WV + bV

[0048] Where W Q , W K and W V are learnable weight matrices, and b Q , b K and b V are learnable bias vectors.

[0049] Next, calculate the attention weight matrix A, where each element A ij represents the correlation between the query sequence Q i and the key sequence K j :

[0050]

[0051] Where the softmax function is used to normalize the attention score, is a scaling factor that controls the stability of the attention weights. During the calculation of the attention scores, the dot product of the query sequence Q and the key sequence K is used, and the dot product result is divided by scaled.

[0052] Finally, we use the attention weight matrix A to perform a weighted sum of the value sequence V to obtain the output vector C of the self-attention mechanism:

[0053] C = A · V

[0054] The output vector C contains important information about each element in the point cloud data feature sequence X, which is weighted according to the relevance between elements.

[0055] Step four, cross-attention. By using the cross-attention algorithm, the association between multiple down-sampled point cloud data features is established, and the relevance between different sequence point cloud feature data is considered when calculating the attention weight, and the point cloud feature information of different sequences is transmitted to other input point cloud feature sequences. Thus, the model's expression ability and performance are enhanced.

[0056] The point cloud global feature serves as the query sequence Q, and the point cloud local feature serves as the key sequence K and the value sequence V. We first calculate the similarity score matrix S between the query sequence and the key sequence:

[0057] S = Q · KT

[0058] where Q is the query sequence, K is the key sequence, ^T represents the transpose operation, and · represents matrix multiplication.

[0059] Next, we normalize the similarity score matrix (S) to obtain the attention weight matrix (A):

[0060] A = softmax(S)

[0061] In the attention weight matrix (A), each row of elements represents the similarity distribution of the corresponding query and key. By performing a weighted sum of the attention weight matrix (A) and the value sequence (V), we obtain the final output vector (C):

[0062] C = A · V

[0063] Finally, the output vector C contains important information about the value sequence V that is weighted according to the relevance between the query sequence Q and the key sequence K.

[0064] Step five, upsampling algorithm. The point cloud data at a known resolution is converted into a higher resolution point cloud grid through the upsampling algorithm. Then, through an operation similar to convolution, the feature information of each point is passed from the low-resolution local area to the high-resolution local area, and the specific steps are as follows:

[0065] 1. The input point cloud is divided into regions by hierarchical clustering or gridding.

[0066] 2. In each region, the point cloud is upsampled using a specific interpolation method.

[0067] 3. The upsampled point cloud is merged with the original point cloud.

[0068] 4. Finally, the distance and weight between each newly generated point and its nearest neighbor are calculated to obtain the coordinates of the new point.

[0069] Step six, design the network training method of Parallel-Net. According to the different data sets of point cloud classification and segmentation tasks, design the corresponding loss function and optimizer, and optimize and adjust the network parameters in the training process.

[0070] 1. Loss function and optimizer settings

[0071] During the experimental process of the present application, the cross-entropy loss function is used to measure the difference between the predicted results and the true labels. And respectively using SGD (Stochastic Gradient Descent) optimizer, Adam optimizer and AdamW optimizer for experiment.

[0072] 2. Training data set selection

[0073] In the point cloud classification task, the present application uses ModelNet40 data set. ModelNet40 data set contains about 12,311 object models of 40 different categories. Each object model is divided into two parts: training set and test set. The training set contains 9,843 objects, and the test set contains 2,468 objects. In addition, in order to enhance the diversity of the data set, each object model has 12 different transformations, including translation, rotation, scaling, etc. During the training process, the number of point clouds collected by each model is N = 1024, and the initial point cloud data feature dimension d = 6. Before training, the input point cloud data is strengthened, such as range scaling, regional translation, and random inactivation.

[0074] In the point cloud entity segmentation task, the present application uses the ShapeNet dataset, which contains more than 50 object categories, covering a wide range of everyday objects such as chairs, tables, cars, airplanes, animals, etc. Each object is represented in the form of a three-dimensional mesh (3D mesh) and has different poses and scales. The ShapeNet dataset provides rich annotation information, including object classification labels, geometric information, part segmentation, pose, and scene semantics, etc. The number of samples in the ShapeNet dataset is relatively large, and the specific number of samples is as follows: the total number of models is 51300, the number of training set samples is 40913, the number of validation set samples is 4888, and the number of test set samples is 5499. In the training process, the number of point clouds collected for each model is N = 1024, and the initial point cloud data feature dimension d = 9. The input point cloud data is strengthened before training, such as range scaling, region translation, and random inactivation.

[0075] 3. Determine the initialization parameters of network training

[0076] In the training, the batch_size of the data input is set to 4-16 for testing, the epoch is set to 200-1000 for testing, and the eps is set to 1e-8.

[0077] Step seven, network performance analysis of Parallel-Net

[0078] In the point cloud classification and segmentation task, the commonly used indicators include the following:

[0079] Accuracy: represents the ratio of the number of correctly classified or segmented point clouds to the total number of point clouds. Its formula is:

[0080]

[0081] Average Intersection over Union (mIoU): also known as the Jaccard index, is an indicator for evaluating the performance of the model by calculating the ratio of the intersection area to the union area between the predicted results and the true labels. Its formula is:

[0082]

[0083] where N represents the number of categories, and IOU_i represents the Intersection over Union of the i-th category. For each category i, its Intersection over Union (IOU_i) can be calculated using the following formula:

[0084] IOU_i = (intersection area) / (union area)

[0085] Where "intersection area" represents the area of intersection between the prediction result and the true label, and "union area" represents the area of union between the prediction result and the true label.

[0086] F1 score (F 1Score ): It comprehensively considers the precision and recall, and can measure the comprehensive performance of the model in classification or segmentation tasks. The higher the F1 score, the higher the recall rate while maintaining high accuracy. Its formula is:

[0087]

[0088] Where the calculation of precision and recall is as follows:

[0089]

[0090]

[0091] Mean Square Error (MSE): MSE calculates the average of the square of the difference between the prediction result and the true label, and the smaller the value, the closer the model prediction result to the true label. Its formula is

[0092]

[0093] Where N represents the number of samples, y i represents the true value of the i-th sample, represents the prediction value of the i-th sample by the model. For each sample i, calculate the difference between its prediction value and true value, and then square it. Sum the squared differences of all samples and divide by the number of samples N to get the mean square error (MSE).

[0094] Parallel-Net's classification performance on the ModelNet40 dataset, with an overall accuracy (OA) of 92.7% and a class average accuracy (mAcc) of 90.6%.

[0095]

[0096] Table 1: Parallel-Net's classification performance on the ModelNet40 dataset

[0097]

[0098] Table 2: Parallel-Net's classification performance on the ShapeNet dataset

[0099] Through comprehensive experimental analysis, compared with the Point Transformer network, the Parallel-Net network has similar classification and entity segmentation performance, but the parameter quantity and operation quantity are better than those of the Point Transformer network, and compared with other network models, the classification and segmentation accuracy is improved.

[0100] The above further detailed description of the application in combination with the specific embodiments cannot be regarded as the specific implementation of the application being limited to these descriptions, and equivalent substitutions or obvious deformations can be made without departing from the concept and premise of the application, and the performance or use is the same, which should be regarded as the protection scope of the application.

Claims

1. A point cloud classification and segmentation network based on point cloud multi-scale parallel feature extraction and attention mechanism, comprising the following steps: Step one, input the point cloud dataset, and perform the first feature learning through the full connection layer, and convert the input point cloud data into initial feature representation through the full connection layer; Step two, feature extraction and sampling are performed by using a down-sampling algorithm, feature extraction captures the local features of the point cloud through message passing in the local region, and sampling is used to reduce the number of point clouds and retain the global structure information, through these two steps, higher level feature representation can be obtained; Step three, the self-attention mechanism is used to calculate the attention weight of each position in the point cloud input sequence, the attention degree of each point is obtained by calculating the similarity between each point and other points, then the weights are multiplied with the input vector and summed to obtain the weighted representation, so as to realize the attention degree regulation of different positions; Step four, cross-attention algorithm is used to process the down-sampled point cloud data features with different point cloud feature scales, the correlation between different sequence point cloud feature data is considered when calculating the attention weight, in this way, the expression ability and performance of the model to the relationship between different features are enhanced; Step five, the up-sampling algorithm is used to convert the point cloud data with known resolution into higher resolution point cloud grid, then a convolution-like operation is used to pass the feature information of each point from the low resolution local region to the high resolution local region, and the feature fusion of the obtained point cloud features with different scales is completed; Step six, the network structure of Parallel-Net is designed, the corresponding loss function and optimizer are designed according to different datasets of point cloud classification and segmentation task, and the network parameters are optimized and adjusted during the training process.

2. The point cloud classification and segmentation network based on point cloud multi-scale parallel feature extraction and attention mechanism according to claim 1, characterized in that, For the input point cloud dataset X, a full connection layer composed of an input layer, a normalization layer and an activation layer is designed to perform the first feature learning, a neural network model containing multiple full connection layers or convolution layers and activation functions is used to learn the local feature representation of point cloud, i.e. to obtain the attention score of each point, through the first feature learning, the global information of point cloud data is effectively extracted.

3. The point cloud classification and segmentation network based on point cloud multi-scale parallel feature extraction and attention mechanism according to claim 1, characterized in that, Through down-sampling, smaller scale point cloud data after down-sampling is generated, in the case of reducing the amount of calculation, down-sampling can extract important features from the original point cloud data.

4. The point cloud classification and segmentation network based on point cloud multi-scale parallel feature extraction and attention mechanism according to claim 1, characterized in that, By giving the input point cloud data sequence X, we first get three different sequences through linear transformation: query sequence, key sequence and value sequence, then calculate the attention weight between points and points, and finally the output vector C contains the important information of each element in the point cloud data feature sequence, which is weighted according to the correlation between elements.

5. The point cloud classification and segmentation network based on point cloud multi-scale parallel feature extraction and attention mechanism according to claim 1, characterized in that, Through the cross-attention mechanism learning of local point cloud features and global point cloud features, the local features are obtained as the correlation between the query sequence and the key sequence, and the global features are obtained as the important information of the value sequence.

6. The point cloud classification and segmentation network based on point cloud multi-scale parallel feature extraction and attention mechanism according to claim 1, characterized in that, The input point cloud is regionally divided, in each region, a specific interpolation method is used to up-sample the point cloud, the up-sampled point cloud is merged with the original point cloud, and the distance and weight between each newly generated point and its nearest neighbor point are calculated to obtain the coordinates of the new point.

7. The point cloud classification and segmentation network based on point cloud multi-scale parallel feature extraction and attention mechanism according to claim 1, characterized in that: During the experiment, the cross-entropy loss function is used to measure the difference between the predicted results and the true labels, and the SGD optimizer, Adam optimizer and AdamW optimizer are used for experiments, In the point cloud classification task, the ModelNet40 dataset is used, and during the training process, the number of points collected by each model is N = 1024, and the initial point cloud data feature dimension d = 6. The input point cloud data is strengthened before training. In the point cloud entity segmentation task, the ShapeNet dataset is used, and during the training process, the number of points collected by each model is N = 1024, and the initial point cloud data feature dimension d = 9. The input point cloud data is strengthened, scaled, regionally translated, and randomly deactivated before training. During the training, the batch_size of the data input is set to 4~16 for testing, the epoch is set to 200~1000 for testing, and the eps is set to 1e-8. Through different ways of training, the target value is reached.

Images