Efficient point cloud analysis method based on Mamba-Transform hybrid architecture

Abstract

The invention belongs to the technical field of point cloud image analysis, and more specifically, relates to an efficient point cloud analysis method based on a Mamb-Transform hybrid architecture. The method comprises the following steps: firstly, grouping original point cloud data to obtain global features of each group of point cloud data, and performing position embedding on the global features; then, reordering the obtained global features of each group of point cloud data by using Morton ordering, so that a Mamb-Transform model can causally capture the structure of the point cloud data; then, the global features after reordering are input into a Mamba-Transform model to be subjected to feature processing, and high-dimensional features are obtained; and finally, carrying out classification prediction on the high-dimensional features processed by the Mamba-Transform model through a classification prediction head. The problems that in the prior art, the calculation requirement is large, the method is difficult to apply to long sequence modeling, and the capacity of capturing and utilizing the global context is insufficient are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud image analysis, and more specifically, relates to an efficient point cloud analysis method based on a Mamba-Transformer hybrid architecture. Background Art

[0002] Emerging applications based on 3D vision are growing rapidly, and 3D point clouds are gaining more and more attention. Point clouds are widely used in robotics, 3D graphics, autonomous driving, and augmented reality / virtual reality / mixed reality (AR / VR / MR). To meet the growing application needs, it is particularly important to study and develop algorithms for efficient storage and processing of point cloud data. Traditional methods usually focus on encoding the local geometric features of points. Given the significant achievements of deep learning in image processing, it is particularly urgent to study neural network architectures suitable for point clouds. The current research trend is focused on developing deep neural networks for various point cloud processing tasks.

[0003] 3D point cloud is a data set of points in three-dimensional space. Point cloud is used to represent the 3D surface of an object. Each point consists of three coordinates (X, Y, Z) that uniquely identify its position relative to the orthogonal axes. Point cloud in the usual sense contains a large number of points. Point cloud is unorganized and has no specific order. This disordered nature requires consideration of its particularity when processing point cloud and design methods.

[0004] At present, deep learning methods have become the mainstream for the classification and segmentation of point cloud data. The more commonly used ones are PointNet, PointNet++ and PointTransformer.

[0005] PointNet is the first deep learning model that directly processes point cloud data. It learns the corresponding spatial encoding for each point in the input point cloud, and then uses the features of all points to obtain a global point cloud feature. The global features extracted by PointNet can complete the classification task well, but the local feature extraction ability is poor, which makes it difficult to analyze complex scenes. To solve this problem, researchers proposed PointNet++. Based on PointNet, PointNet++ introduces a multi-level feature extraction structure, which effectively improves the local feature extraction ability. It recursively divides the point cloud into multiple local areas and applies PointNet to each area for feature extraction, thereby obtaining rich local features. At the same time, PointNet++ also fuses features at different levels through upsampling operations to form global features. This enables PointNet++ to achieve significant performance improvements in 3D point cloud classification and segmentation tasks. PointTransformer realizes the construction of network models based on self-attention networks. By designing a self-attention layer for point clouds, combining position encoding to build a Transformer block, and using the self-attention mechanism, it can achieve tasks including semantic segmentation, component segmentation, and recognition.

[0006] Among them, Transformer has become one of the basic architectures for point cloud analysis tasks with its excellent global modeling capabilities. However, although the powerful long-range modeling capabilities of Transformer intuitively improve the analysis performance, the comprehensive mechanism inevitably leads to high computational complexity and memory usage. Due to the quadratic complexity of the attention mechanism and limited computing resources, it is difficult to extend to long sequence modeling.

[0007] Chinese patent document CN118485998A discloses a state space model based on point cloud data serialization technology, which makes full use of the scalability advantages of structured unstructured point clouds and introduces a reordered variant of the standard space filling curve to enhance the global modeling capability of SSM. The reordered point tokens are then sent to a series of mamba blocks to capture the point cloud structure at will. However, the model lacks reasonable sequence modeling and can only process data step by step, limiting its ability to capture and utilize global context in a forward pass.

[0008] In view of this, the present invention designs an efficient point cloud analysis method based on the Mamba-Transformer hybrid architecture. Summary of the invention

[0009] The present invention aims to overcome at least one of the defects of the above-mentioned prior art and provide an efficient point cloud analysis method based on the Mamba-Transformer hybrid architecture to solve the problems of large computational requirements, difficulty in applying to long sequence modeling, and insufficient ability to capture and utilize global context in the prior art. SUMMARY OF THE INVENTION

[0011] In order to overcome the shortcomings of the prior art, the present invention designs an efficient point cloud analysis hybrid architecture based on Mamba. By introducing Morton sorting to sort the point cloud, efficient sequence modeling is performed. By constructing a hybrid architecture that alternates between Mamba modules and self-attention modules and processes them through MLP, the powerful analysis capabilities of Transformer and the efficiency of Mamba are effectively combined to improve the expression and generalization capabilities of the model.

[0012] The detailed technical scheme of the present invention is as follows:

[0013] An efficient point cloud analysis method based on a Mamba-Transformer hybrid architecture, the method comprising:

[0014] S1. Group the original point cloud data, obtain the global features of each group of point cloud data, and embed the global features;

[0015] S2. Using Morton sorting, the global features of each set of point cloud data are reordered so that the Mamba-Transformer model can causally capture the structure of the point cloud data;

[0016] S3, input the reordered global features into the Mamba-Transformer model for feature processing to obtain high-dimensional features;

[0017] S4, classify and predict the high-dimensional features processed by the Mamba-Transformer model through the classification prediction head;

[0018] The classification prediction head includes two layers of hidden classification heads. The classification prediction head first compresses high-dimensional features to 256 dimensions, then extracts effective features through two linear layers, batch normalization, ReLU and Dropout layers, and finally maps the extracted effective features to the dimensions of the target classification.

[0019] Preferably, according to the present invention, the specific steps of step S1 are as follows:

[0020] S11. Use the farthest point FPS algorithm to sample the original point cloud data and select num_group center points to represent the center of each group;

[0021] S12, using the KNN algorithm to find the neighborhood of each center point, and obtain the index of the nearest neighbor point of each center point;

[0022] S13, normalizing each center point and neighborhood, subtracting the coordinates of the center point from the coordinates of the nearest neighbor point in each neighborhood, to obtain the normalized neighborhood and center point;

[0023] S14, performing the first convolution coding on the normalized neighborhood and center point to extract initial features, taking the maximum value of the initial features of each group to obtain preliminary global features; then performing feature splicing on the preliminary global features and the initial features;

[0024] S15, performing a second convolution process on the concatenated features to obtain the features of each group, and then taking the maximum value of the features of each group to obtain the final global features;

[0025] S16. Use the position embedding module to convert the center point coordinates of the point cloud data into a high-dimensional embedding vector.

[0026] Preferably, according to the present invention, the specific steps of step S2 are as follows:

[0027] S21. First, the sorting process starts from the center point, and the first point among all the center points is selected as the first point to be sorted. The sorted points form a sorted point cluster.

[0028] S22. Next, a greedy algorithm is used to determine the next point according to the point closest to the sorted point cluster, and the closest point to the current sorted point cluster is gradually selected, and the points are sorted based on the spatial relationship between the center points of each group of point clouds;

[0029] S23, adjusting the feature tokens and the position information pos accordingly according to the order, to obtain re-ordered tokens and pos;

[0030] S24. Finally, the sorted tokens and pos are fused element by element to form the final point cloud tokens.

[0031] Preferably, according to the present invention, the Mamba-Transformer model adopts a layered strategy, in odd layers, Mamba modules are used for long sequence modeling, in even layers, self-attention modules are used, and at the same time, an MLP layer is introduced between the Mamba module and the self-attention module for nonlinear transformation and processing of the hybrid architecture.

[0032] Preferably, according to the present invention, in the Mamba module, the input data is first processed by layer normalization LN, and then linearly transformed through a linear layer, and then local features are extracted using deep convolution DW. The output of the deep convolution is activated by the SiLU activation function and sent to the state space model SSM to further model the sequence information. The output of the SSM is processed by layer normalization LN again, and then multiplied with the input data that has been normalized, linearly transformed by another linear layer, and activated by the SiLU activation function, and then passed through the linear layer and the MLP layer and added with the original input data to form a residual connection to obtain the output of the Mamba module.

[0033] Preferably, according to the present invention, the self-attention module of the self-attention mechanism generates attention weights by calculating the similarity between points, emphasizes the relationship between neighboring points, dynamically adjusts the weights of points, highlights important local features, and effectively captures the local structure of the point cloud; at the same time, the generated context vector contains global information, which enhances the understanding of the global structure of the point cloud;

[0034] The final formula of the self-attention module can be expressed as:

[0035] MultiHead(Q,K,V)=Concat(head 1 ,head 2 ,...,head i )W O (1)

[0036] Each head i The calculation formula is:

[0037]

[0038] In formula (1) and formula (2), Q represents the query vector set Query, K represents the key vector set Key, V represents the value vector set Value, head i represents the i-th head of the self-attention module, W O represents the final linear transformation matrix, Q i , K i 、V i Represents query, key and value respectively, d k represents the feature dimension of each head, and T represents the transposition operation.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The present invention rearranges point cloud tokens by introducing Morton sorting. This sorting technology can enhance the capture of local continuity and spatial characteristics of data, thereby effectively improving the model's ability to process point cloud data.

[0041] (2) The present invention constructs a hybrid architecture that alternates between Mamba modules and self-attention modules and processes them through MLP, which effectively combines the high efficiency of Mamba and the powerful analytical ability of Transformer. It can better acquire and utilize global context information and improve the expressiveness and generalization capabilities of the model. The hybrid architecture of the present invention can optimize the utilization of computing resources while ensuring performance, enhance the analysis capability of point cloud data, and realize comprehensive extraction and precise modeling of point cloud data features. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the efficient point cloud analysis method of the present invention.

[0043] Figure 2 This is a graph showing the results of experiments on the ScanObjectNN dataset using the method of the present invention and other methods.

[0044] Figure 3 This is a graph showing the experimental results of the method described in the present invention and other methods on the ModelNet40 dataset.

[0045] Figure 4 This is a graph showing the results of experiments on the ShapeNetPart dataset using the method described in the present invention and other methods. DETAILED DESCRIPTION

[0046] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0047] Embodiment 1,

[0048] Ginseng Figure 1 This embodiment provides an efficient point cloud analysis method based on a Mamba-Transformer hybrid architecture, the method comprising:

[0049] S1. Group the original point cloud data, obtain the global features of each group of point cloud data, and embed the global features into positions; the specific steps are as follows:

[0050] S11. Use the farthest point FPS algorithm to sample the original point cloud data and select num_group center points to represent the center of each group;

[0051] S12, using the KNN algorithm to find the neighborhood of each center point, and obtain the index of the nearest neighbor point of each center point;

[0052] S13, normalizing each center point and neighborhood, subtracting the coordinates of the center point from the coordinates of the nearest neighbor point in each neighborhood, so that the coordinates of the center point of each neighborhood point become (0, 0, 0), and obtaining the normalized neighborhood and center point;

[0053] S14, performing the first convolution coding on the normalized neighborhood and center point to extract initial features, taking the maximum value of the initial features of each group to obtain preliminary global features; then performing feature splicing on the preliminary global features and the initial features;

[0054] S15, performing a second convolution process on the concatenated features to obtain the features of each group, and then taking the maximum value of the features of each group to obtain the final global features;

[0055] S16. Use the position embedding module to convert the center point coordinates of the point cloud data into a high-dimensional embedding vector.

[0056] S2. Using Morton sorting, the global features of each set of point cloud data are reordered so that the Mamba-Transformer model can causally capture the structure of the point cloud data. The specific steps are as follows:

[0057] S21. First, the sorting process starts from the center point, that is, the local center of each point in the point cloud, and the first point among all the center points is selected as the first point to be sorted. The sorted points form a sorted point cluster.

[0058] S22. Next, a greedy algorithm is used to determine the next point according to the point closest to the sorted point cluster, and the closest point to the current sorted point cluster is gradually selected, and the points are sorted based on the spatial relationship between the center points of each group of point clouds;

[0059] S23, adjusting the feature tokens and the position information pos accordingly according to the order, to obtain re-ordered tokens and pos;

[0060] S24. Finally, the sorted tokens and pos are fused element by element to form the final point cloud tokens.

[0061] S3, input the reordered global features into the Mamba-Transformer model for feature processing to obtain high-dimensional features;

[0062] The Mamba-Transformer model adopts a layered strategy. In the odd layers, the Mamba module is used for long sequence modeling to efficiently capture global feature interactions. However, relying solely on the Mamba module for modeling may lead to the loss of local features. Therefore, in the even layers, the self-attention module is used. This method of alternating the use of the Mamba module and the self-attention mechanism can not only efficiently process long sequence data, but also ensure that the model achieves a balance in capturing global and local features, and achieves more comprehensive feature extraction. In order to further ensure the effective fusion of local and global features, a hybrid architecture of MLP layers for nonlinear transformation and processing is introduced between layers to ensure that local features and global information can be smoothly connected and fused, while enhancing the nonlinear representation ability of features, further improving the overall performance of the model.

[0063] Mamba incorporates time-varying parameters into SSM and proposes a hardware-aware algorithm to achieve efficient training and inference. In the Mamba module, the input data is first processed by layer normalization (LN), then linearly transformed through a linear layer, and then local features are extracted using depth-wise convolution (DW). The output of the depth-wise convolution is activated by the SiLU activation function and sent to the state space model (SSM) to further model the sequence information. The output of the SSM is then processed by layer normalization again, and then multiplied with the input data that has been normalized, linearly transformed by another linear layer, and activated by the SiLU activation function, and then passed through the linear layer and MLP layer and added with the original input data to form a residual connection to obtain the output of the Mamba module.

[0064] The self-attention module is used to perform weighted aggregation on the input features. The module calculates the self-attention weights of the input matrix through the query, key, and value vectors, and then adjusts the feature values ​​according to these weights.

[0065] The self-attention module calculates the similarity between points, generates attention weights, emphasizes the relationship between neighboring points, dynamically adjusts the weights of points, highlights important local features, and effectively captures the local structure of the point cloud. At the same time, the generated context vector contains global information, which enhances the understanding of the global structure of the point cloud.

[0066] The final formula of the self-attention module can be expressed as:

[0067] MultiHead(Q,K,V)=Concat(head 1 ,head 2 ,...,head i )W O (1)

[0068] Each head i The calculation formula is:

[0069]

[0070] In formula (1) and formula (2), Q represents the query vector set Query, which is used to find the most relevant key vector, K represents the key vector set Key, which is used to measure the relevance to the query vector, and V represents the value vector set Value, which is used for the final weighted summation. i represents the i-th head of the self-attention module, W O represents the final linear transformation matrix, Q i , K i 、V i Represents query, key and value respectively, d k represents the feature dimension of each head, and T represents the transposition operation.

[0071] Without loss of generality, the attention formula can be computed in a windowed manner similar to before.

[0072] Use gamma_1 and gamma_2 to control the scaling of LayerScale to help the model stabilize in the deep structure. Through this alternating inter-layer design, an efficient model with feature extraction capabilities can be created. And through normalization, average pooling and other operations, global features can be obtained;

[0073] S4, classify and predict the high-dimensional features processed by the Mamba-Transformer model through the classification prediction head Task Head;

[0074] The classification prediction head includes two layers of hidden classification heads. The classification prediction head first compresses high-dimensional features to 256 dimensions, then extracts effective features through two linear layers, batch normalization, ReLU and Dropout layers, and finally maps the extracted effective features to the dimension of the target classification. This layered design can effectively improve the expressiveness of features, and at the same time use normalization and Dropout to improve the stability and generalization ability of the model, which is suitable for preventing overfitting in large-scale training processes.

[0075] Experimental example

[0076] a. In order to evaluate the effectiveness of the proposed method in point cloud classification tasks, a detailed experimental comparison was conducted on the ScanObjectNN dataset. The experimental results are shown in Figure 2 As shown in the figure, the classification accuracy of the proposed method (labeled as "ours") and several other popular methods, including PointNet, PointNet++, PointCNN, DGCNN, Transformer-base and Point-MABE on the OBJ-BG, OBJ-ONLY and PB-T50-RS sub-datasets are listed. Figure 2 As can be seen in the figure, the proposed method achieved the highest classification accuracy on all three sub-datasets. Specifically, on the OBJ-BG sub-dataset, the proposed method achieved an accuracy of 89.50%, which is about 1.2% higher than the second place. On the OBJ-ONLY sub-dataset, the proposed method achieved an accuracy of 88.12%, leading the other methods. In the most challenging PB-T50-RS sub-dataset, the proposed method also showed strong performance, achieving an accuracy of 83.41%, which is better than all other competitors. In addition, it can be noted that when the complexity of the dataset increases, the performance of most methods will decline, especially on the PB-T50-RS sub-dataset. However, the proposed method still performs well when facing more difficult datasets, showing good robustness and generalization ability. Overall, these experimental results show that the proposed method has obvious advantages in handling point cloud classification tasks, especially when dealing with more challenging datasets.

[0077] b. In order to verify the effect of the proposed method on the 3D shape classification task, experiments were conducted on the ModelNet40 dataset. Figure 3 As shown in the figure, the classification accuracy of the proposed method (labeled as "ours") and some existing mainstream methods, including PointNet, PointNet++, PointCNN, PointMAE, PointMamba and PCM* on ModelNet40 are listed. Figure 3 As can be seen in Figure 1, the proposed method achieves the best classification accuracy of 92.9% on the ModelNet40 dataset, surpassing all other baseline methods. This result shows that the proposed method has significant advantages in dealing with 3D shape classification tasks and can more effectively capture and utilize geometric features in point cloud data.

[0078] In general, experiments on the ModelNet40 dataset demonstrate the effectiveness and superiority of the method proposed in this paper.

[0079] c. In order to verify the effect of the proposed method on the 3D shape component segmentation task, experiments were conducted on the ShapeNetPart dataset. Figure 4 As shown in the figure, the classification mIoU and instance mIoU scores of the proposed method (labeled as "Ours") and some existing mainstream methods, including PointNet, PointNet++, DGCNN, APES, PointMAE, PointMamba and PCM* on ShapeNetPart are listed. Figure 4 It can be seen that the method of the present invention has achieved the best classification mIoU of 84.07% on the ShapeNetPart dataset. This result shows that the method of the present invention has significant advantages in processing three-dimensional shape part segmentation tasks and can more accurately understand and segment complex three-dimensional models.

[0080] In general, experiments on the ShapeNetPart dataset demonstrate the effectiveness and superiority of the method proposed in this paper.

Claims

1. An efficient point cloud analysis method based on Mamba-Transformer hybrid architecture, characterized in that: The method comprises: S1. Group the original point cloud data, obtain the global features of each group of point cloud data, and embed the global features; S2, using Morton sorting, re-sorting the global features of each set of point cloud data; S3, input the reordered global features into the Mamba-Transformer model for feature processing to obtain high-dimensional features; S4, classify and predict the high-dimensional features processed by the Mamba-Transformer model through the classification prediction head; The classification prediction head includes two layers of hidden classification heads. The classification prediction head first compresses high-dimensional features to 256 dimensions, then extracts effective features through two linear layers, batch normalization, ReLU and Dropout layers, and finally maps the extracted effective features to the dimensions of the target classification.

2. According to claim 1, an efficient point cloud analysis method based on the Mamba-Transformer hybrid architecture is characterized in that: The specific steps of step S1 are as follows: S11. Use the farthest point FPS algorithm to sample the original point cloud data and select num_group center points to represent the center of each group; S12, using the KNN algorithm to find the neighborhood of each center point, and obtain the index of the nearest neighbor point of each center point; S13, normalizing each center point and neighborhood, subtracting the coordinates of the center point from the coordinates of the nearest neighbor point in each neighborhood, to obtain the normalized neighborhood and center point; S14, performing the first convolution coding on the normalized neighborhood and center point to extract initial features, taking the maximum value of the initial features of each group to obtain preliminary global features; then performing feature splicing on the preliminary global features and the initial features; S15, performing a second convolution process on the concatenated features to obtain the features of each group, and then taking the maximum value of the features of each group to obtain the final global features; S16. Use the position embedding module to convert the center point coordinates of the point cloud data into a high-dimensional embedding vector.

3. The efficient point cloud analysis method based on the Mamba-Transformer hybrid architecture according to claim 1 is characterized in that: The specific steps of step S2 are as follows: S21. First, the sorting process starts from the center point, and the first point among all the center points is selected as the first point to be sorted. The sorted points form a sorted point cluster. S22. Next, a greedy algorithm is used to determine the next point according to the point closest to the sorted point cluster, and the closest point to the current sorted point cluster is gradually selected, and the points are sorted based on the spatial relationship between the center points of each group of point clouds; S23, adjusting the feature tokens and the position information pos accordingly according to the order, to obtain re-ordered tokens and pos; S24. Finally, the sorted tokens and pos are fused element by element to form the final point cloud tokens.

4. The efficient point cloud analysis method based on the Mamba-Transformer hybrid architecture according to claim 1 is characterized in that: The Mamba-Transformer model adopts a layered strategy. In odd layers, Mamba modules are used for long sequence modeling, and in even layers, self-attention modules are used. At the same time, an MLP layer is introduced between the Mamba module and the self-attention module for a hybrid architecture of nonlinear transformation and processing.

5. The efficient point cloud analysis method based on the Mamba-Transformer hybrid architecture according to claim 3 is characterized in that: In the Mamba module, the input data is first processed by layer normalization LN, then linearly transformed through a linear layer, and then local features are extracted using deep convolution DW. The output of the deep convolution is activated by the SiLU activation function and sent to the state space model SSM to further model the sequence information. The output of the SSM is processed by layer normalization LN again, and then multiplied with the input data that has been normalized, linearly transformed by another linear layer, and activated by the SiLU activation function. The data is passed through the linear layer and the MLP layer and the original input data is added to form a residual connection to obtain the output of the Mamba module.

6. The efficient point cloud analysis method based on the Mamba-Transformer hybrid architecture according to claim 3 is characterized in that: The self-attention module calculates the similarity between points, generates attention weights, emphasizes the relationship between neighboring points, dynamically adjusts the weights of points, highlights important local features, and effectively captures the local structure of the point cloud. At the same time, the generated context vector contains global information, which enhances the understanding of the global structure of the point cloud. The final formula of the self-attention module can be expressed as: MultiHead(Q,K,V)=Concat(head1,head2,...,head i )W O (1) Each head i The calculation formula is: In formula (1) and formula (2), Q represents the query vector set Query, K represents the key vector set Key, V represents the value vector set Value, head i represents the i-th head of the self-attention module, W O represents the final linear transformation matrix, Q i , K i 、V ii Represents query, key and value respectively, d k represents the feature dimension of each head, and T represents the transposition operation.

Images