A three-dimensional point cloud classification and segmentation method based on Mamba and attention fusion mechanism
Patent Information
- Application Number
- CN202611519285.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-29
- Publication Date
- 2026-10-30
AI Technical Summary
点卷积网络通常依赖固定邻域进行局部特征聚合，难以充分刻画中心点与邻域点之间的细粒度关系；Transformer虽然能够建立远距离依赖，但其自注意力计算复杂度随点数增加而快速增长；状态空间模型能够降低序列建模复杂度，但由于点云本身具有无序性，直接进行序列化处理容易破坏局部空间邻域关系，影响局部几何特征表达
[0034]1.本发明通过构建中心点与邻域点之间的关联特征，并结合查询向量、键向量、值向量及相对位置编码对邻域特征进行自适应聚合，使网络能够根据邻域点的特征相关性和空间位置动态分配特征权重，从而增强局部几何结构、部件边界及细粒度形状信息的表达能力。
Smart Images
Figure CN123016100A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D point cloud data processing technology, specifically to a 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism. Background Technology
[0002] With the continuous development of perception technologies such as LiDAR, RGB-D cameras, and 3D scanning equipment, 3D point clouds have become an important data carrier for describing the geometric structure of real-world scenes. Compared to 2D images, point clouds can directly reflect the spatial position and geometric shape of objects, and are therefore widely used in fields such as robot perception, autonomous driving, intelligent manufacturing, digital twins, and 3D reconstruction. Point cloud classification aims to identify the overall category to which a point cloud belongs, while point cloud component segmentation requires further prediction of the semantic component category corresponding to each point. Both are fundamental tasks in the field of 3D vision.
[0003] In recent years, with the development of deep learning, point cloud analysis methods have gradually shifted from traditional geometric feature design to end-to-end deep learning models. Early methods, represented by PointNet and its improved networks, achieved point cloud classification and segmentation by directly processing unordered point sets. Subsequently, point convolutional networks further enhanced local geometric modeling capabilities. In recent years, attention mechanisms such as Transformer have been introduced into point cloud analysis to improve long-distance dependency modeling capabilities through global feature interactions. The latest research has begun to explore applying State Space Model (SSM) to point cloud sequence modeling, improving the efficiency of global feature modeling while maintaining linear computational complexity.
[0004] Despite the progress made by the aforementioned methods, certain limitations remain. Point convolutional networks typically rely on fixed neighborhoods for local feature aggregation, making it difficult to fully characterize the fine-grained relationships between the center point and its neighbors. While Transformers can establish long-distance dependencies, their self-attention computational complexity increases rapidly with the number of points. State-space models can reduce the complexity of sequence modeling, but due to the inherent disorder of point clouds, direct serialization can easily disrupt local spatial neighborhood relationships, affecting the representation of local geometric features. Furthermore, most existing state-space models employ unidirectional modeling, and their utilization of bidirectional contextual information in point clouds remains insufficient.
[0005] Therefore, how to enhance the ability to model local geometric relationships and global context while maintaining low computational complexity, and effectively integrate local features with long-distance dependency information, remains a key problem that urgently needs to be solved in the field of 3D point cloud classification and component segmentation. Summary of the Invention
[0006] To address the problems existing in the prior art, this application proposes a 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism. It constructs a 3D point cloud analysis framework consisting of local neighborhood feature construction, center-to-neighborhood attention modeling, bidirectional state space feature modeling, and hierarchical classification and segmentation network. The aim is to improve the ability to express local geometric features and global context modeling of point clouds, and improve the accuracy and robustness of 3D point cloud classification and segmentation while maintaining low computational complexity.
[0007] To achieve the above technical objectives, this application provides the following technical solution:
[0008] A 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism includes the following steps:
[0009] A 3D point cloud classification and segmentation model is constructed, which includes an input embedding layer, multiple cascaded feature encoding modules, multiple cascaded feature recovery modules, and a task output layer; each feature encoding module consists of a center-to-neighborhood attention module and a bidirectional Mamba module.
[0010] Point cloud data is acquired, including point cloud coordinates, labels, and attribute information; the original point cloud is input into the input embedding layer to obtain initial features, the initial features of each point being composed of its three-dimensional spatial coordinates and attribute information; the original point cloud is divided into different local neighborhoods, and the center point features and neighborhood point features of each local neighborhood are obtained based on the initial features, and then the center point features and neighborhood point features are encoded to obtain center point encoded features and neighborhood point encoded features;
[0011] In the center-to-neighborhood attention module of the first feature encoding module: the correlation between the center point encoding features and the neighboring point encoding features is modeled to obtain the associated features; based on the associated features, the neighboring point encoding features are weighted and aggregated by combining position encoding and attention mechanism to obtain the local geometric enhancement features of each center point;
[0012] The local geometric enhancement features are input into the bidirectional Mamba module of the first feature encoding module. First, the local geometric enhancement features are spatially serialized using a space-filling curve to obtain spatially serialized features. Then, the spatially serialized features are modeled in the state space along the forward and reverse directions of the sequence to obtain bidirectional state features. The spatially serialized features are then linearly mapped and normalized to obtain gated features. Finally, the bidirectional state features and gated features are fused and modulated to obtain local-global fused features.
[0013] The remaining feature encoding modules take the local-global fusion features output by the previous feature encoding module and the corresponding center point coordinates as input, and perform downsampling step by step to complete multi-scale feature encoding; then, multiple cascaded feature recovery modules perform upsampling step by step, and restore the spatial resolution and feature expression step by step through feature interpolation and cross-layer fusion, and the last feature recovery module outputs the multi-scale fusion features;
[0014] Multi-scale fused features are input into the task output layer to obtain prediction results. During the model training phase, a loss function is used to calculate the difference between the prediction results and the labels, and the model parameters are updated based on the loss function until the convergence condition is met. The trained model is then used to complete the 3D point cloud classification and point cloud component segmentation tasks.
[0015] Furthermore, the local neighborhood is sampled from the original point cloud using the farthest point sampling method, and neighborhood points are obtained by searching for each center point using the K-nearest neighbor algorithm; the input embedding layer includes a multilayer perceptron and a shared multilayer perceptron. The multilayer perceptron is used to extract initial features from the original point cloud, and the shared multilayer perceptron is used to encode the center point features and neighborhood point features to obtain center point encoded features and neighborhood point encoded features.
[0016] Furthermore, the correlation between the modeling center point encoding features and the neighboring point encoding features yields the following specific correlation features:
[0017] For any local neighborhood, the center point's encoded features are copied along the neighborhood dimension and then independently differiated element-wise with the encoded features of each neighboring point to obtain the association features between the center point and each neighboring point.
[0018] Furthermore, the weighted aggregation of neighborhood point encoding features based on association features, combined with position encoding and attention mechanisms, to obtain the local geometric enhancement features of each center point specifically involves:
[0019] The centroid encoding features are linearly mapped to the query vector, and the associated features are linearly mapped to the key vector and value vector, respectively. Attention scores are calculated based on the query vector and key vector, and normalized along the neighborhood dimension to obtain the attention weights between the centroid and each neighboring point.
[0020] A positional code is generated based on the relative spatial distance between the center point and neighboring points; the positional code is fused with the value vector, and then attention weights are used to weight and aggregate the positional coded value vector to obtain the neighborhood aggregation feature; the neighborhood aggregation feature is residually connected with the center point encoding feature, and then processed through a linear layer and a batch normalization layer in sequence to obtain the local geometric enhancement feature;
[0021] By traversing all center points and their corresponding local neighborhoods, the local geometric enhancement features of each center point are obtained.
[0022] Furthermore, the spatial serialization of the local geometric enhancement features using a space-filling curve to obtain the spatially serialized features is specifically as follows:
[0023] A Hilbert curve sorting index is generated based on the three-dimensional spatial coordinates of each center point. The local geometric enhancement features corresponding to each center point are rearranged according to the Hilbert curve sorting index, so that the local geometric enhancement features corresponding to the center points that are spatially adjacent or close maintain an adjacent or close arrangement relationship in the one-dimensional sequence, thus obtaining spatial serialized features. At the same time, the reverse sorting index, which is the opposite of the sorting index, is recorded.
[0024] Furthermore, the process of performing state-space modeling on the spatial serialization features along the forward and reverse directions of the sequence to obtain bidirectional state features, and then performing linear mapping and normalization on the spatial serialization features to obtain gating features, specifically involves:
[0025] The bidirectional Mamba module consists of parallel state-space feature branches and gated feature branches;
[0026] In the state space feature branch: the spatial serialized features are processed by the first linear layer and the forward depth-separable convolution to obtain the forward state space modeling input features; after the spatial serialized features are processed by the first linear layer, they are reversed along the sequence dimension to obtain the reverse sequence features; the reverse sequence features are processed by the reverse depth-separable convolution to obtain the reverse state space modeling input features.
[0027] The forward state space modeling input features are input into the forward state space model to obtain forward state features; the reverse state space modeling input features are input into the reverse state space model to obtain temporary reverse state features; the temporary reverse state features are reversed again along the sequence dimension to obtain reverse state features corresponding to the positions of the forward state features; the forward state features and reverse state features are added element by element to obtain bidirectional state features.
[0028] In the gated feature branch: the spatial serialized features are processed through a second linear layer and batch normalization to obtain the gated features.
[0029] Furthermore, the specific steps of fusing and modulating the bidirectional state features and gating features to obtain the local-global fused features are as follows:
[0030] The bidirectional state features and the gated features are multiplied element-wise, and the multiplication result is mapped through the third linear layer to obtain the fused output features under the Hilbert curve sorting index. The fused output features are then reverse sorted according to the inverse sorting index to restore the original spatial arrangement order corresponding to each center point, thus obtaining the local-global fused features of each center point.
[0031] Furthermore, the task output layer includes a classification head and a component segmentation head; the classification head performs global aggregation on the multi-scale fusion features to obtain a one-dimensional global feature vector, and the one-dimensional global feature vector is passed through a fully connected layer to output the prediction result of the overall category of the point cloud; the component segmentation head performs feature interpolation on the multi-scale fusion features to restore the original point cloud resolution, and then outputs the prediction result of the component category to which each point in the original point cloud belongs through a fully connected layer.
[0032] Furthermore, the loss function is a labeled smooth cross-entropy loss function, and the model parameters are updated iteratively through the AdamW optimizer.
[0033] By employing the above technical solution, the present invention has at least the following beneficial effects:
[0034] 1. This invention constructs the association features between the center point and the neighboring points, and combines query vectors, key vectors, value vectors and relative position encoding to adaptively aggregate the neighborhood features, enabling the network to dynamically allocate feature weights according to the feature correlation and spatial position of the neighboring points, thereby enhancing the ability to express local geometric structure, component boundaries and fine-grained shape information.
[0035] 2. This invention uses Hilbert curves to spatially sequence local geometric enhancement features based on the spatial coordinates of the center point, ensuring that adjacent or close center points remain adjacent or close in the one-dimensional sequence. It also performs bidirectional modeling of the Hilbert sequence using forward and backward state space models, and restores the order of the backward state features and fused output features, thereby improving the modeling ability of global context information and long-distance dependencies in point clouds, while ensuring the correspondence between the output features and the original spatial positions.
[0036] 3. This invention constructs a hierarchical network consisting of an input embedding layer, multiple cascaded feature encoding modules, and multiple cascaded feature recovery modules. By downsampling at each level, multi-scale features at different spatial resolutions are extracted, and feature interpolation and cross-layer fusion are used to recover local spatial details, thereby improving the overall point cloud category recognition and component boundary segmentation capabilities.
[0037] 4. This invention combines the center-to-neighborhood attention mechanism with the bidirectional Mamba state space modeling mechanism, which enhances the expression of local geometric relationships while achieving efficient bidirectional global context modeling. This can improve the accuracy and robustness of point cloud classification and component segmentation while maintaining low computational complexity. Attached Figure Description
[0038] Figure 1 This is a flowchart of a 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism proposed in this invention;
[0039] Figure 2 This is a schematic diagram of the center-to-neighborhood attention module in the embodiment;
[0040] Figure 3 This is a schematic diagram of the bidirectional Mamba module in the embodiment;
[0041] Figure 4 This is a schematic diagram of the structure of the 3D point cloud classification and segmentation model in the embodiment;
[0042] Figure 5 This is a schematic diagram of the component segmentation results of point cloud data in the embodiment. Detailed Implementation
[0043] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.
[0044] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0045] like Figure 1 As shown, this invention proposes a 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism, which specifically includes the following steps:
[0046] S1. Construct a 3D point cloud classification and segmentation model, which includes an input embedding layer, multiple cascaded feature encoding modules, multiple cascaded feature recovery modules, and a task output layer; each feature encoding module consists of a center-to-neighborhood attention module and a bidirectional Mamba module.
[0047] In this embodiment, as a preferred implementation, such as Figure 4 As shown, the 3D point cloud classification and segmentation model includes four feature encoding modules and three feature recovery modules.
[0048] S2. Obtain point cloud data, which includes point cloud coordinates, labels, and attribute information; the point cloud coordinate set is denoted as... ,in, This indicates the number of points contained in the point cloud. Indicates the first The three-dimensional spatial coordinates of each point are denoted as and the attribute information of each point is denoted as . It includes RGB color information, reflection intensity information, normal vector information, or other auxiliary features; at the same time, only during the training phase, the point cloud data is further associated with the overall category label or the component category label to which each point belongs.
[0049] The original point cloud is input into the input embedding layer for feature mapping, resulting in... The initial features of each point are composed of its three-dimensional spatial coordinates and attribute information. In this embodiment, the input embedding layer consists of a multilayer perceptron and a shared multilayer perceptron. First, the multilayer perceptron is used to perform point-by-point feature mapping on the three-dimensional coordinates and attribute information of each point in the original point cloud, and the output feature dimension is set to 64, with the convolution kernel size being [missing information]. The activation function used is ReLU;
[0050] The original point cloud is then divided into different local neighborhoods. In this preferred embodiment, the local neighborhoods are sampled from the original point cloud using the farthest point sampling method (FPS), with each center point corresponding to a local neighborhood. The value of G ranges from 1 / 4 to 3 / 4 of the number of original points N. In this embodiment, G = N / 4 is used to reduce the amount of subsequent calculations and preserve the overall spatial distribution characteristics of the point cloud.
[0051] Then, by using the K-nearest neighbor algorithm to search for neighboring points for each center point, the i-th... A local neighborhood is represented as , This represents the number of neighboring points corresponding to each center point. The value range is from 4 to 128. In this embodiment, K = 32 is chosen to balance the coverage of local geometric structures and computational efficiency.
[0052] Based on the initial features, the center point features and neighborhood point features of each local neighborhood are obtained; for the th A local neighborhood, with the initial features of its center point Let be the feature of the center point, and let the th be the feature corresponding to the center point. Initial features of neighboring points The features of the center point and the neighborhood points are denoted as the features of the neighboring points. Then, a shared multilayer perceptron is used to encode the features of the center point and the neighborhood points to obtain the center point encoded features. and neighborhood point encoding features The formula is expressed as:
[0053] , ;
[0054] in, This indicates a shared multilayer perceptron, where the center point and neighboring points use the same... And parameters, to perform unified feature encoding on the features of the center point and the features of the neighboring points;
[0055] By arranging the encoded features of the neighboring points corresponding to each center point according to the center point dimension and the neighborhood dimension, a local neighborhood point feature tensor can be constructed. :
[0056] ;
[0057] in, This represents the feature dimension used for local neighborhood feature extraction in the feature encoding module; in this embodiment, ;
[0058] Then, stack the center-encoded features corresponding to the G center points row by row to obtain the center feature matrix. :
[0059] ;
[0060] In this embodiment, the number of center points is taken. Feature Dimension That is, the size of the central feature matrix is ;
[0061] Will and As input to the subsequent feature encoding module, it provides the center point encoding features and neighborhood point encoding features of each local neighborhood.
[0062] S3. In the center-to-neighborhood attention module of the first feature encoding module: model the correlation between the center point encoded features and the neighboring point encoded features to obtain the correlated features;
[0063] In a preferred embodiment, for any local neighborhood, the center point's encoded features are copied along the neighborhood dimension, and then element-wise differencing is performed independently with the encoded features of each neighboring point to obtain the association features between the center point and each neighboring point. The formula is as follows:
[0064] ;
[0065] in, Indicates the first The central point and its first The association features between neighboring points.
[0066] Based on the associated features, the neighborhood point encoding features are weighted and aggregated by combining position encoding and attention mechanisms to obtain the local geometric enhancement features of each center point;
[0067] As a preferred embodiment, such as Figure 2As shown, in this embodiment, the specific process of obtaining local geometric enhancement features is as follows:
[0068] The centroid encoding features are linearly mapped to the query vector, and the associated features are linearly mapped to the key vector and value vector, respectively. The formula is as follows:
[0069] ;
[0070] ;
[0071] ;
[0072] in, Represents the query vector. Represents the key vector. Represents a value vector; , and These represent the corresponding learnable linear mapping matrices. Through the above mapping, the features of the center point are used to characterize the query information of the current local region, and the association features between the center point and its neighboring points are used to generate key vectors and value vectors, thereby providing a feature basis for subsequent attention weight calculation.
[0073] In this application, attention scores are calculated based on the query vector and key vector, and then normalized along the neighborhood dimension to obtain the attention weights between the center point and each neighboring point. The formula is as follows:
[0074] ;
[0075] in, Indicates the first The center point and the first Attention weights among neighborhood points, which are used to measure the contribution of different neighborhood points to the local geometric representation of the center point; This means normalization along the neighborhood dimension, so that the sum of the attention weights of the K neighboring points corresponding to the same center point is 1;
[0076] The location code is generated based on the relative spatial distance between the center point and its neighboring points. Let the first point be... The coordinates of the center points are , No. The first center point corresponds to the The coordinates of the neighboring points are Then the relative positions between the two Represented as
[0077] ;
[0078] Input the relative position into the position encoding mapping function to obtain the first... The central point and its first Position encoding between the coordinates of neighboring points The formula is expressed as:
[0079] ;
[0080] in, This represents a positional encoding mapping function that introduces the spatial relationship between the center point and its neighboring points through positional encoding. This enables subsequent neighborhood feature aggregation to not only rely on feature similarity but also to incorporate local geometric positional relationships.
[0081] The positional encoding and value vector are fused, and then attention weights are used to weight and aggregate the positional-encoded value vector to obtain the neighborhood aggregation feature, which is expressed by the formula:
[0082] ;
[0083] ;
[0084] in, This represents the position-encoded value vector, indicating the first position. Neighborhood aggregation features of each center point Indicates the first The set of neighborhood points of a central point;
[0085] The neighborhood aggregation features and the center point encoding features are residually concatenated and then processed sequentially through a linear layer and a batch normalization layer to obtain local geometric enhancement features, expressed by the formula:
[0086] ;
[0087] in, Indicates the first Local geometric enhancement features at each center point This represents a linear mapping operation. This indicates a batch normalization operation;
[0088] By traversing all center points and their corresponding local neighborhoods, the local geometric enhancement features of each center point are obtained. After processing by the center-to-neighborhood attention module, the key geometric information in each local neighborhood of the original point cloud is adaptively enhanced and aggregated to the corresponding center point. The local geometric enhancement features of each center point serve as the input to the subsequent bidirectional Mamba module.
[0089] S4. Input the local geometric enhancement features into the bidirectional Mamba module of the first feature encoding module. First, use the space filling curve to spatially serialize the local geometric enhancement features to obtain spatially serialized features.
[0090] As a preferred embodiment, in this example, the specific process dimension for obtaining spatial serialization features is as follows:
[0091] A Hilbert curve sorting index is generated based on the three-dimensional spatial coordinates of each center point, and the index is sorted according to the Hilbert curve. The local geometric enhancement features corresponding to each center point are rearranged so that the local geometric enhancement features corresponding to spatially adjacent or close center points maintain an adjacent or close arrangement relationship in the one-dimensional sequence, thus obtaining spatially serialized features. Simultaneously record the inverse sorted index, which is the opposite of the sorted index. This is used to subsequently restore the original spatial arrangement order of each center point.
[0092] Then, state-space modeling (SSM) is performed on the spatial serialization features along the forward and reverse directions of the sequence to obtain bidirectional state features. Finally, the spatial serialization features are linearly mapped and normalized to obtain gated features.
[0093] As a preferred embodiment, in this embodiment, such as Figure 3 As shown, the specific process for obtaining the bidirectional state features and gating features is as follows:
[0094] The bidirectional Mamba module consists of parallel state-space feature branches and gated feature branches; the forward depthwise separable convolution and the backward depthwise separable convolution use the same network structure to process forward and backward sequence features respectively, and employ mutually independent learnable parameters;
[0095] In the state-space feature branch: the spatially serialized features are processed by the first linear layer and the forward depthwise separable convolution to obtain the forward state-space modeling input features. After processing the spatially serialized features through the first linear layer, they are reversed along the sequence dimension to obtain the reverse sequence features. The inverse sequence features are processed by inverse depthwise separable convolution to obtain inverse state space modeling input features. The formula is expressed as:
[0096] ;
[0097] ;
[0098] ;
[0099] in, Indicates the first linear layer. This indicates forward depthwise separable convolution processing. This indicates inverse depthwise separable convolution processing; Indicates reversal along the sequence dimension;
[0100] Input the features into the forward state space model to obtain the forward state features. Input the input features into the inverse state space model to obtain temporary inverse state features. The temporary reversed state feature is then reversed again along the sequence dimension to obtain the reversed state feature corresponding to the position of the forward state feature. The formula is expressed as:
[0101] ;
[0102] ;
[0103] ;
[0104] in, This represents the forward state space model. The forward state space model is used to model the dependencies between point cloud features along the forward direction of the Hilbert sequence, while the reverse state space model is used to model the dependencies in the opposite direction along the reversed Hilbert sequence. By reversing the temporary reverse state features again, the reverse state features and the forward state features are made to correspond one-to-one in the sequence position.
[0105] The bidirectional state features are obtained by adding the forward and backward state features element by element, as expressed by the formula:
[0106] ;
[0107] in, Indicates a two-way state characteristic;
[0108] In the gated feature branch: the spatially serialized features are processed through a second linear layer and batch normalization to obtain the gated features, expressed by the formula:
[0109] ;
[0110] in, Indicates the second linear layer. This indicates a gating feature.
[0111] By fusing and modulating bidirectional state features and gating features, local-global fused features are obtained;
[0112] As a preferred embodiment, in this embodiment, such as Figure 3 As shown, the specific process for obtaining local-global fused features is as follows:
[0113] The bidirectional state features and the gated features are multiplied element-wise. The result of the multiplication is then mapped through a third linear layer to obtain the fused output features under the Hilbert curve sorting index, as expressed in the formula:
[0114] ;
[0115] in, Represents the third linear layer mapping. This indicates element-wise multiplication. This represents the fused output features; during the element-wise multiplication process, the gating features play a role in filtering and regulating the bidirectional state features.
[0116] The fused output features are inverted based on the inverted sort index to restore the original spatial arrangement order corresponding to each center point, thus obtaining the local-global fused features of each center point. The formula is expressed as follows:
[0117] ;
[0118] in, This indicates that the features are rearranged according to the specified index; This indicates the local-global fusion feature.
[0119] S5. After processing by the first feature encoding module, the center points in the original point cloud aggregate the feature information of their local neighborhoods, and the size of the output local-global fusion feature is N / 4×128.
[0120] Then as Figure 4 As shown, the remaining feature encoding modules take the local-global fusion features and corresponding center point coordinates output by the previous feature encoding module as input, resample the center points in the previous level center point set, and construct the corresponding local neighborhood. The process of local neighborhood construction, center-to-neighborhood attention processing, and bidirectional Mamba processing is executed sequentially to perform downsampling at each level to complete multi-scale feature encoding. In subsequent feature encoding modules, the center point features and neighborhood point features of the constructed local neighborhood are provided by the local-global fusion features output by the previous feature encoding module. The output feature sizes of the second, third, and fourth feature encoding modules are N / 16×256, N / 64×512, and N / 256×1024, respectively.
[0121] Then, multiple cascaded feature recovery modules perform step-by-step upsampling, and the spatial resolution is gradually restored to N / 4×128 through feature interpolation and cross-layer fusion. The last feature recovery module outputs multi-scale fused features.
[0122] S6. Input the multi-scale fused features into the task output layer to obtain the prediction results;
[0123] In a preferred embodiment, the task output layer includes a classification head and a component segmentation head. The classification head performs global aggregation on the multi-scale fusion features to obtain a one-dimensional global feature vector. The one-dimensional global feature vector is passed through a fully connected layer to output the prediction result of the overall category of the point cloud. The component segmentation head performs feature interpolation on the multi-scale fusion features to restore the original point cloud resolution, and then outputs the prediction result of the component category to which each point in the original point cloud belongs through a fully connected layer.
[0124] During the model training phase, a loss function is used to calculate the difference between the predicted result and the label, and the model parameters are updated based on the loss function until the convergence condition is met. In this embodiment, the loss function is the labeled smooth cross-entropy loss function, and the total number of categories is set to 1. The original tag is The label smoothing coefficient is The target label after label smoothing is represented as:
[0125] ;
[0126] Suppose the model predicts that it belongs to the th The probability of class is Then the labeled smooth cross-entropy loss is expressed as:
[0127] ;
[0128] in, Indicates the total number of categories. Indicates the label smoothing coefficient. Indicates the original label. This indicates the target label after label smoothing. This indicates that the model predicts the sample belongs to the first... The probability of a class This represents the label smooth cross-entropy loss.
[0129] The model parameters are updated iteratively using the AdamW optimizer, with the initial learning rate set to 0.01, the weight decay coefficient set to 0.05, and the batch size set to 32.
[0130] Use the trained model to complete the tasks of 3D point cloud classification and point cloud component segmentation.
[0131] This concludes the description of the entire process of the 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism proposed in this invention. In this embodiment, to verify the effectiveness of the proposed method in point cloud classification tasks, comparative tests were conducted using the ModelNet40 dataset and the ScanObjectNN dataset, respectively.
[0132] ModelNet40 is a synthetic point cloud dataset composed of 3D computer-aided design models, containing 40 object classes; ScanObjectNN is a point cloud dataset collected from real scenes, including three test settings: OBJ_BG, OBJ_ONLY, and PB_T50_RS. Among them, OBJ_BG includes background point disturbances, OBJ_ONLY only retains the point cloud of the target object, and PB_T50_RS introduces translation, rotation, and scaling perturbations on top of background noise. On the ScanObjectNN dataset, each input sample contains 2048 points; on the ModelNet40 dataset, each input sample contains 1k points.
[0133] In this embodiment, Overall Accuracy (OA) is used to evaluate the overall recognition performance of each model in the point cloud classification task. Overall Accuracy (OA) represents the proportion of correctly classified samples out of all test samples, and its calculation formula is as follows:
[0134] ;
[0135] in, This represents the total number of categories for point cloud classification tasks; Indicates the true category is the first Class and predicted category is the first The number of samples in each class; This indicates that both the true category and the predicted category are the first. The number of samples in each class; This represents the total number of samples that were correctly classified. This indicates the total number of all test samples.
[0136] The results of the comparative tests are shown in Table 1 below:
[0137] Table 1. Comparative Test Results of Classification Tasks
[0138]
[0139] In this table, DGCNN stands for Dynamic Graph Convolutional Neural Network, KPConv for Kernel Point Convolution, PCT for Point Cloud Transformer, and PointTrans for Point Transformer; Params(M) represents the number of model parameters, in millions. FLOPs(G) represents the number of floating-point operations, in billions. The proposed method achieves an overall accuracy of 94.4% on the ModelNet40 dataset, and overall accuracies of 89.7%, 90.3%, and 87.4% under the three test settings of ScanObjectNN: OBJ_BG, OBJ_ONLY, and PB_T50_RS, respectively. Notably, even under the PB_T50_RS test setting, which includes background interference and translation, rotation, and scaling perturbations, the proposed method still achieves an overall accuracy of 87.4%, indicating that it can adapt well to complex point cloud data. Furthermore, the proposed method has a parameter count of 7.59M and FLOPs of 3.74G, achieving high classification accuracy while maintaining low parameter and computational costs, demonstrating that the center-to-neighborhood attention module and the bidirectional Mamba module effectively enhance point cloud feature representation.
[0140] To examine the performance of the proposed method in point cloud component segmentation, this embodiment also uses the ShapeNetPart dataset for experiments. This dataset contains 16 object classes and 50 component annotations, with each point cloud sample inputting 2048 points. Some experimental results are shown below. Figure 5 As shown in the figure, the first row represents the actual values of the point cloud components, and the second row represents the predicted values output by the method proposed in this invention; different colors are used to distinguish different component categories. As can be seen from the figure, the predicted values output by the method proposed in this invention can better maintain the continuity of the object's local structure and accurately delineate the component connection areas and fine-grained boundaries.
[0141] Furthermore, to demonstrate the superiority of the two key modules in the proposed method—the center-to-neighborhood attention module and the bidirectional Mamba module—and their synergistic effect, this embodiment sets up ablation experiments with different module combinations and varies the number of neighborhood points to examine the impact of the local neighborhood range on the model's classification performance. The experimental results are shown in Tables 2 and 3 below:
[0142] Table 2 Ablation Experiment Results
[0143]
[0144] Table 3. Experimental results of changing the neighborhood range
[0145]
[0146] As shown in Table 2, even when only the center-to-neighborhood attention module or the bidirectional Mamba module is enabled, the model of this invention shows a significant overall accuracy improvement compared to the existing models in Table 1 on the ModelNet40 dataset, while the number of parameters and computational cost are within an acceptable range. When both modules are enabled simultaneously, the overall accuracy can be further increased by 0.9-1.2 percentage points. Considering that the performance of existing models on the ModelNet40 dataset is close to saturation, this improvement is statistically significant and has practical application value in the field. Moreover, the number of parameters and computational cost of the complete model has not increased dramatically, verifying the superiority of this application in achieving a comprehensive balance between accuracy and efficiency.
[0147] As shown in Table 3, the overall accuracy of the method proposed in this invention does not differ significantly across different neighborhood sizes. This demonstrates that the method proposed in this application maintains stable performance under different neighborhood sizes and exhibits good robustness. Those skilled in the art can reasonably select the neighborhood size based on the trade-off between computational resources and accuracy, reflecting the flexibility and applicability of the technical solution in this application.
[0148] In summary, this invention discloses a 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism, which can improve the ability to express local geometric features and global context modeling of point clouds, and improve the accuracy and robustness of 3D point cloud classification and segmentation while maintaining low computational complexity.
[0149] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.
[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).
[0151] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism, characterized in that, Specifically, the following steps are included: A 3D point cloud classification and segmentation model is constructed, including an input embedding layer, multiple cascaded feature encoding modules, multiple cascaded feature recovery modules, and a task output layer. Each feature encoding module consists of a center-to-neighborhood attention module and a bidirectional Mamba module; The original point cloud is input into the input embedding layer to obtain initial features, and the center point encoding features and neighborhood point encoding features are obtained based on the initial features. In the center-to-neighborhood attention module of the first feature encoding module: the correlation between the center point encoded features and the neighbor point encoded features is modeled to obtain the associated features; Based on the association features, the encoding features of the neighborhood points are weighted and aggregated to obtain the local geometric enhancement features of each center point; The local geometric enhancement features are input into the bidirectional Mamba module of the first feature encoding module and spatially serialized to obtain spatially serialized features. State space modeling is performed on the spatial serialization features along the forward and reverse directions of the sequence to obtain bidirectional state features. Linear mapping and normalization are then performed on the spatial serialization features to obtain gated features. The bidirectional state features and gated features are fused and modulated to obtain local-global fused features. The remaining feature encoding modules perform step-by-step downsampling to complete multi-scale feature encoding; the feature recovery module performs step-by-step upsampling, and outputs multi-scale fused features through feature interpolation and cross-layer fusion. Multi-scale fused features are input into the task output layer to obtain the prediction result.
2. The 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism according to claim 1, characterized in that, The process of inputting the original point cloud into the input embedding layer to obtain initial features, and obtaining center point encoding features and neighborhood point encoding features based on the initial features, specifically involves: The original point cloud is divided into different local neighborhoods. The center point of each local neighborhood is sampled from the original point cloud using the farthest point sampling method. Neighboring points are obtained by searching for each center point using the K-nearest neighbor algorithm. The input embedding layer includes a multilayer perceptron and a shared multilayer perceptron. The multilayer perceptron is used to extract initial features from the original point cloud and obtain the center point features and neighboring point features of each local neighborhood based on the initial features. The shared multilayer perceptron is used to encode the center point features and neighboring point features to obtain the center point encoded features and the neighboring point encoded features.
3. The 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism according to claim 1, characterized in that, The correlation between the modeling center point coding features and the neighboring point coding features yields the following specific correlation features: For any local neighborhood, the center point's encoded features are copied along the neighborhood dimension and then independently differiated element-wise with the encoded features of each neighboring point to obtain the association features between the center point and each neighboring point.
4. The 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism according to claim 1, characterized in that, The weighted aggregation of neighborhood point encoding features based on association features to obtain the local geometric enhancement features of each center point is specifically as follows: The centroid encoding features are linearly mapped to the query vector, and the associated features are linearly mapped to the key vector and value vector, respectively. Attention scores are calculated based on the query vector and key vector, and normalized along the neighborhood dimension to obtain the attention weights between the centroid and each neighboring point. A positional code is generated based on the relative spatial distance between the center point and neighboring points; the positional code is fused with the value vector, and then attention weights are used to weight and aggregate the positional coded value vector to obtain the neighborhood aggregation feature; the neighborhood aggregation feature is residually connected with the center point encoding feature, and then processed through a linear layer and a batch normalization layer in sequence to obtain the local geometric enhancement feature; By traversing all center points and their corresponding local neighborhoods, the local geometric enhancement features of each center point are obtained.
5. The 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism according to claim 1, characterized in that, The local geometric enhancement features are spatially serialized using space-filling curves to obtain spatially serialized features, specifically: A Hilbert curve sorting index is generated based on the three-dimensional spatial coordinates of each center point. The local geometric enhancement features corresponding to each center point are rearranged according to the Hilbert curve sorting index, so that the local geometric enhancement features corresponding to the center points that are spatially adjacent or close maintain an adjacent or close arrangement relationship in the one-dimensional sequence, thus obtaining spatial serialized features. At the same time, the reverse sorting index, which is the opposite of the sorting index, is recorded.
6. The 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism according to claim 1, characterized in that, The process of performing state-space modeling on the spatial serialization features along the forward and reverse directions of the sequence to obtain bidirectional state features, and then performing linear mapping and normalization on the spatial serialization features to obtain gated features, specifically: The bidirectional Mamba module consists of parallel state-space feature branches and gated feature branches; In the state space feature branch: the spatial serialized features are processed by the first linear layer and the forward depth-separable convolution to obtain the forward state space modeling input features; After the spatial serialization features are processed by the first linear layer, they are reversed along the sequence dimension to obtain the reverse sequence features. The reverse sequence features are then processed by the reverse depthwise separable convolution to obtain the reverse state space modeling input features. Input the forward state space modeling input features into the forward state space model to obtain the forward state features; Input the input features into the inverse state space model to obtain temporary inverse state features. Reverse the temporary inverse state features along the sequence dimension to obtain inverse state features corresponding to the positions of the forward state features. Add the forward state features and inverse state features element by element to obtain bidirectional state features. In the gated feature branch: the spatial serialized features are processed through a second linear layer and batch normalization to obtain the gated features.
7. A 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism according to claim 5, characterized in that, The specific steps for fusing and modulating the bidirectional state features and gating features to obtain the local-global fused features are as follows: The bidirectional state features and the gated features are multiplied element-wise, and the multiplication result is mapped through the third linear layer to obtain the fused output features under the Hilbert curve sorting index. The fused output features are then reverse sorted according to the inverse sorting index to restore the original spatial arrangement order corresponding to each center point, thus obtaining the local-global fused features of each center point.
8. The 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism according to claim 1, characterized in that, The task output layer includes a classification head and a component segmentation head. The classification head performs global aggregation on the multi-scale fusion features to obtain a one-dimensional global feature vector. The one-dimensional global feature vector is passed through a fully connected layer to output the prediction result of the overall category of the point cloud. The component segmentation head performs feature interpolation on the multi-scale fusion features to restore the original point cloud resolution. Then, it outputs the prediction result of the component category to which each point in the original point cloud belongs through a fully connected layer.
9. A 3D point cloud classification and segmentation method based on Mamba and attention fusion mechanism according to claim 1, characterized in that, During the model training phase, a labeled smooth cross-entropy loss function is used, and the model parameters of the 3D point cloud classification and segmentation model are iteratively updated through the AdamW optimizer.