Mama-based point cloud semantic segmentation method and device, equipment and medium

By adopting a Mamba-based method in large-scale scene cloud semantic segmentation, using serialized encoding and local and global Mamba feature extraction, combined with grid upsampling module, the problem of point cloud segmentation accuracy and computing efficiency in large-scale scenarios is solved. , realizes the combination of high-precision segmentation and low computational complexity.

CN119006814APending Publication Date: 2024-11-22HUAQIAO UNIVERSITY
View PDF 0 Cites 10 Cited by

Patent Information

Application Number
CN202411076605.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In large-scale scenarios, the accuracy of three-dimensional point cloud semantic segmentation is difficult to reach the ideal level, and the existing algorithms have considerable computing resource requirements and processing time when processing high-density point cloud data. How to reduce computational complexity while ensuring high-precision segmentation Degree and model parameters are an important challenge.

Method used

Using Mamba-based method, efficient serialization encoding and sorting is used, combined with local Mamba to extract the fine features of each layer and the global Mamba fusion of the features of the first few layers, enhance collaboration and communication between levels, and through grid upsampling and networking The grid mapping module restores point cloud data to the original number of points, reducing the amount of computing resources and model parameters.

Benefits of technology

It improves the accuracy of scene segmentation, shortens the network convergence time, avoids the quantization error and time overhead of data format conversion, and the designed network model has fewer parameters, which significantly improves processing speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006814A_ABST
    Figure CN119006814A_ABST
Patent Text Reader

Abstract

According to the Mama-based large-scale point cloud semantic segmentation method, the Mama-based large-scale point cloud semantic segmentation device, the Mama-based large-scale point cloud semantic segmentation equipment and the medium provided by the invention, the point cloud semantic segmentation model is constructed based on Mama; the constructed semantic segmentation model of the point cloud scene is composed of a trunk feature extraction network, a grid pooling and local grouping module, an interlayer auxiliary network, a global Mama module, a grid up-sampling and grid mapping decoding network and a point-by-point classification module. According to the method, the feature thought of the Mama model is fused, the local Mama is adopted to perform local feature extraction, the global Mama fuses the local features extracted by the local Mama and captures lost information, information fusion at different levels is realized, richer large-scale point cloud feature representation is obtained, and the precision of semantic segmentation of the point cloud scene is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of large-scale point cloud semantic segmentation based on deep learning, and in particular to a point cloud semantic segmentation method in large-scale complex scenes based on Mamba. Background Art

[0002] Semantic segmentation of large-scale scene point clouds is a prerequisite for robot scene perception and further processing. 3D point cloud semantic segmentation is widely used in many fields such as autonomous driving and remote sensing. With the rapid development of technologies such as computer vision and artificial intelligence, applications such as autonomous driving, robot navigation, augmented reality, automatic grasping, and smart city construction have received strong support. In these applications, the system needs to deal with complex and difficult-to-understand scenes. In order to accurately understand these complex scenes, the targets in them must be accurately segmented.

[0003] Compared with two-dimensional image data, three-dimensional point clouds can better obtain scene feature information because they are not affected by factors such as scale, angle, and light. This enables three-dimensional point clouds to more accurately describe the object information in the scene, thus providing favorable conditions for target segmentation in complex scenes. The data format of three-dimensional point clouds contains the spatial position information of each point, which helps to capture the geometric shape and spatial distribution of objects, and provides richer features for target segmentation. However, three-dimensional data itself has characteristics such as irregular arrangement and sparsity, which makes the direct segmentation of point cloud scene targets face huge challenges.

[0004] When processing large-scale scene point clouds, the segmentation accuracy often fails to meet the ideal level and requirements. Large-scale point cloud data usually contains millions of points, which significantly increases the computational complexity and memory requirements. How to improve the segmentation accuracy of the network while reducing computing resources and model parameters remains a difficult problem that needs to be solved urgently. Many existing algorithms perform well when processing small-scale point clouds, but often perform poorly when faced with large-scale scenes. In order to achieve high-precision point cloud semantic segmentation in large-scale scenes, researchers have proposed various methods, including but not limited to point cloud downsampling, feature extraction, and optimization algorithms.

[0005] In addition, with the development of deep learning technology, methods based on deep neural networks have shown strong performance in point cloud semantic segmentation tasks. Classic models such as PointNet, PointNet++, and DGCNN process point cloud data through different strategies, extract effective features, and perform accurate segmentation. However, the application of these models in large-scale point cloud scenarios still has limitations, especially when processing high-density point cloud data, the computing resource requirements and processing time are considerable.

[0006] In general, 3D point cloud semantic segmentation is of great significance in many practical applications, but it also faces many technical challenges. How to improve computational efficiency and reduce model complexity while ensuring high-precision segmentation is an important direction for future research. Summary of the invention

[0007] The technical problem to be solved by the present invention is how to improve computational efficiency and reduce the complexity of the model while ensuring high-precision segmentation. Therefore, the present invention proposes a large-scale scene point cloud semantic segmentation method based on Mamba, which aims to enhance the collaborative communication between layers and reduce the demand for computing resources and model parameters through efficient serialization encoding and sorting without relying on complex neighbor construction methods (such as KNN) and combining local Mamba to extract the fine features of each layer and global Mamba to fuse the features of the previous layers.

[0008] The present invention adopts the following technical solution: a point cloud semantic segmentation method based on Mamba, comprising the following steps:

[0009] S1 obtains a point cloud semantic segmentation dataset and preprocesses the point cloud semantic segmentation dataset;

[0010] S2 serializes the preprocessed data set into point cloud and extracts features through sparse convolution to obtain the initial features of the point cloud;

[0011] S3 divides the serialized point cloud into several non-overlapping patches and performs local Mamba in each patch to extract local features of the point cloud;

[0012] S4 performs hierarchical feature extraction and learns the association between aggregated information at each layer;

[0013] S5 inputs the features learned at different levels into the global Mamba to further fuse the information aggregated at each level;

[0014] S6 constructs a grid upsampling and grid mapping module to restore the point cloud data to the original number of points and generate the learned refined global semantic information for each point;

[0015] S7 builds a point-by-point classification module and uses a multi-classification cross entropy loss function to supervise network learning, maps the number of channels of each point in the learned point cloud to the number corresponding to the category, and predicts the label corresponding to each point.

[0016] Preferably, the point cloud semantic segmentation dataset includes ScanNetv2 and S3DIS datasets, and the preprocessing operation on the dataset in S1 includes dividing the dataset into a training set T 1 , test set T 2The point cloud data is enhanced using a validation set V, wherein the steps of data enhancement include discarding, rotating, scaling, flipping, jittering, distorting, and cropping.

[0017] Preferably, the step of obtaining the initial features of the point cloud in S2 includes serializing the point cloud data using four different space filling curves including Z-order curve, Trans Z-order curve, Hibert curve and Trans Hibert curve to map the 3D point cloud data to a one-dimensional sequence and then extracting features through 3D sparse convolution to obtain the initial features of the point cloud.

[0018] Preferably, local Mamba feature extraction and grid downsampling are performed alternately in layers in S4 to extract key point features. At each stage, the spatial size is compressed while the dimension of the feature is increased to learn the association between the aggregated information of each layer.

[0019] Preferably, in S5, the association between the information learned from different layers is further fused with the information of each layer of local Mamba using global Mamba and the information missing from local Mamba is captured.

[0020] Preferably, a grid upsampling and grid mapping module is constructed in S6. In the grid downsampling stage in the encoder, the position information of each point is recorded. In the upsampling stage, grid division and feature mapping are used to map the fused point set features to the features of each point in the original point set. Finally, the point cloud data is restored to the original number of points to generate the learned refined global semantic information for each point.

[0021] Preferably, in S7, a point-by-point classification module is constructed for the upsampled and restored points, and a multi-classification cross entropy loss function is used to supervise network learning, the number of channels of each point in the learned point cloud is mapped to the number corresponding to the category, and the label corresponding to each point is predicted.

[0022] A Mamba-based point cloud semantic segmentation device, characterized in that it includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program can be executed by the processor to implement a Mamba-based large-scale point cloud semantic segmentation method as described above.

[0023] A computer-readable storage medium, characterized in that computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a Mamba-based point cloud semantic segmentation method as described above is implemented.

[0024] The present invention has the following beneficial effects: In complex scenes, the present invention constructs a global Mamba module that is conducive to the interaction of inter-layer features, improves the accuracy of scene segmentation and accelerates the convergence speed of the network. The present invention directly processes 3D point cloud data, avoids the quantization error and time overhead of data format conversion, can significantly improve the processing speed, and the designed network model uses fewer parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 This is a flow chart of the point cloud semantic segmentation method for complex scenes based on local Mamba and global Mamba proposed in the present invention;

[0027] Figure 2 The overall structure of the point cloud scene semantic segmentation model of the present invention;

[0028] Figure 3 This is the visualization result of the present invention on the ScanNetV2 dataset;

[0029] Figure 4 This is the visualization result of the present invention on the S3DIS dataset. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0031] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0032] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0033] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0034] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0035] Example

[0036] The following are only preferred implementations of the present invention. The protection scope of the present invention is not limited to the following embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention.

[0037] Get the point cloud semantic segmentation dataset and divide it into training set T 1 , test set T2 and validation set V; using training set T 1 , complete E rounds of supervised parameter training for the point cloud segmentation network. To ensure the basic training effect, E≥50, in this experiment, ScanNetv2 sets the training round E=900. Due to the small scale of S3DIS data, the training round E=3000 is set to ensure training stability. Each round uses the validation set V to verify the partial segmentation performance, and saves the network parameters corresponding to the round with the best verification result; the test set T is used 2 (It is T 2 ? ) is T 2 , the performance of the trained point cloud segmentation network is tested to obtain the prediction results of each scene point; the input dimension of the original point cloud data is N p *C p , where N p Represents the number of points in the point cloud, C p Represents the original feature vector of each point, which includes xyz coordinates and RGB color. Before the point cloud data is input into the point cloud scene semantic segmentation model, it is preprocessed, including discarding, rotating, scaling, flipping, jittering, twisting and cropping, and then converted into npy, pth data format as the input data of the model.

[0038] The specific definitions of the linear layer (Linear), layer normalization (LayerNorm), depthwise separable convolution (DWConv), and selective state space model (SelectiveSSM) used in the model are as follows:

[0039] (1) Linear layer uses a linear layer to map point cloud coordinate information to feature information

[0040] Linear=f′ in =Wf p +b

[0041] Among them, W is the weight matrix and b is the bias vector.

[0042] (2) Layer Normalization is a normalization technique used to accelerate the training of neural networks and improve their performance. The basic idea of ​​LayerNorm is to normalize the features of each input sample instead of normalizing the features of small batches of samples. The following is the mathematical expression of LayerNorm:

[0043] Step 1: Calculate the mean and variance:

[0044]

[0045] Step 1: Standardize each feature hi :

[0046]

[0047] Here, ∈ is a small constant used to prevent division by zero errors.

[0048] Step 3: Zoom and Pan:

[0049]

[0050] Among them, γ and β are trainable parameters for scaling and translation, respectively.

[0051] Combining the above steps, the complete formula of LayerNorm is:

[0052]

[0053] (1) Depthwise Separable Convolution (DWConv) can be divided into two steps: Depthwise Convolution and Pointwise Convolution

[0054] Step 1: Depthwise Convolution

[0055] For each input point i and its feature vector F i , deep convolution applies convolution kernels to each channel of the point cloud separately. Assume that the input feature is F∈R N * C , where N is the number of points and C is the number of feature channels. The output of the depthwise convolution can be expressed as:

[0056] Y i (d) =K (d) ⊙F i

[0057] Where: Y i (d) is the output feature after deep convolution, K (d) is a deep convolution kernel, applied to each feature channel, and ⊙ represents a point-by-point multiplication operation (convolution is performed independently on each channel).

[0058] Step 2: Pointwise Convolution

[0059] Point-by-point convolution linearly combines the feature channels of each point and outputs a new feature representation. Assume that the output feature of the deep convolution is Y i(d) ∈R N*C , the formula for point-by-point convolution is:

[0060]

[0061] Y i (p) It is the output feature after point-by-point convolution, and W is the weight matrix of point-by-point convolution, with a size of C×C′.

[0062] Represents a matrix multiplication operation

[0063] Combining the above two steps, the overall formula for applying depthwise separable convolution to each point is:

[0064]

[0065] (2) In the Mamba model, the formula of Selective State Space Models (SSMs) is as follows:

[0066] The selective state space model achieves selective processing by making the parameters of the state space model dependent on the input data. The specific formula includes the following parts:

[0067] First, the basic structure of the state-space model can be expressed as:

[0068] h(t)=Ah(t-1)+BI(t) (Is this x(t) the I(t) in the claims? If so, let’s unify it.) It is the I(t) in the claims.

[0069] Where h(t) is the hidden state at time step t, I(t) is the input at time step t, and A and B are the parameter matrices of the model.

[0070] In Selective SSM, the parameter matrices A and B are functions of the input data, which allows the model to dynamically adjust its behavior based on the input data. The formula is expressed as:

[0071] h(t)=A(I(t))h(t-1)+B(I(t))I(t)

[0072] Specifically: A(I(t)) and B(I(t)) indicate that these matrices are functions of the input data I(t).

[0073] The output y(t) is calculated by the output matrix C and the input signal passed directly:

[0074] y(t)=Ch(t)+DI(t)

[0075] where C and D can also be functions of the input data, but are usually kept fixed.

[0076] Combining the above parts, the comprehensive formula of Selective SSM is:

[0077] h(t)=A(I(t))h(t-1)+B(I(t))I(t)

[0078] y(t)=Ch(t)+DI(t)

[0079] Among them, h(t) is the hidden state at time step t, I(t) is the input at time step t, A and B are the parameter matrices of the model, the observation matrix C describes how the state vector is mapped to the output signal, and the direct transfer matrix D describes how the input signal directly affects the output signal;

[0080] The process of building a point cloud scene semantic segmentation model includes:

[0081] Firstly, a 3D point cloud backbone feature extraction network is constructed, including an initialization feature encoding module and a sequentially cascaded L-layer feature extraction network;

[0082] The first-layer feature encoding module consists of a point cloud serialization layer and a first-layer feature encoding layer.

[0083] In the point cloud serialization layer, the input of the Mamba network requires the use of four different space filling curves including Z-order curve, Trans Z-order curve, Hibert curve, and Trans Hibert curve to serialize the point cloud data and map the 3D point cloud data to a one-dimensional sequence. The use of space filling curves can maintain the spatial proximity of the point cloud and reduce the computational complexity. The following is the specific implementation and use:

[0084]

[0085] Among them, p represents the position coordinates (x, y, z) of the point, b represents the batch index, g represents the grid size, << represents the left shift operator, k represents the number of bits of left shift, φ -1 represents the inverse mapping function of the space-filling curve, represents the integer operator, and |: represents the bitwise OR operator. The overall formula means: First, The point position p is divided by the grid size g and rounded down to get the discrete grid coordinates. Then, the inverse mapping function φ of the space filling curve is used -1 Map the discrete grid coordinates into a one-dimensional sequence. (b<<k) Shift the binary representation of the batch index b to the left by k bits to reserve enough bits for the position encoding of the points. Finally, the batch index after the left shift is bitwise ORed with the point position code to generate the final serialized code, which contains both the batch information and the position information of the input point cloud.

[0086] The first feature encoding layer consists of a cascade of 3D sparse convolutional layers, normalization layers, and activation layers. The input point cloud is serialized and sent to the first feature encoding layer. The features of the point cloud data are converted from the input space to the feature space. The output of the first feature encoding module is used as the input of the cascade backbone feature extraction network.

[0087] The backbone feature extraction network includes L layers of local Mamba feature extraction modules with different depths cascaded in sequence to extract point cloud features. Each local Mamba layer includes a conditional position encoding (xCPE) layer, a layer normalization layer (LayerNorm), a Mamba block layer (Mamba block) and a multi-layer perceptron layer (MLP). First, the output obtained from the first-layer feature encoding module is passed through the xCPE layer to perform position encoding on the point cloud data to enhance the expression ability of the position feature. Then, it is input into the Mamba block layer through the LayerNorm layer. After the output of the Mamba block layer, it is sent to the MLP layer through the LayerNorm layer and finally output through the MLP layer. In this example, L=5 is set, and the stacking depth of each layer is set as [2,2,2,6,2]. The point cloud needs to be sampled in the last L-1 layers. By setting a cascade of five layers of feature extraction layers, the selective state space model (SSM) is used to enhance the feature extraction capability and extract local features of the point cloud.

[0088] Specifically, in the xCPE layer, the initial features are obtained for the first-layer feature encoding module, and a sparse convolution layer with skip connection is added before MambaBlock. The formula is as follows:

[0089] X=X+batch_norm(linear(depth_wise_conv(X))

[0090] In the MambaBlock layer, there are LayererNorm layer, SelectiveState Space Model, Depthwise Separable Convolutional Layer (DWConv) and Residual Connection. A standard MambaBlock includes several steps: First, the input Z l-1 Perform layer normalization to get Z′ l-1 , and then perform depth convolution to get Z′ l And linear transformation to get Z″ l Finally, the feature Z in each patch is extracted through the selective state space model (Selective SSM) and residual connectionl The formula is as follows:

[0091] Z′ l-1 =LayerNorm(Z l-1 )

[0092] Z′ l =σ(DWConv(Linear(Z′ l-1 )))

[0093] Z″ l =σ(Linear(Z′ l-1 ))

[0094] Z l =Linear(SelectiveSSM(Z′ l ☉Z″ l )+Z′ l-1

[0095] Second, we construct a grid pooling module and a patch grouping module. The traditional sampling-based pooling method uses the farthest point sampling (FPS) and aggregates the information of the previous layer through the nearest neighbor query (KNN). This pooling method is not well aligned in space and is computationally complex. To solve this problem, we use a more efficient and effective grid pooling method. Use a unified grid to divide the point set S = (P, F) into multiple non-overlapping subsets [S 1 , S 2 , ..., S n ]Then the features of the points in each subset are max-pooled to obtain the features of the subset f i ′, average pooling is performed on the positions of the points in each subset to obtain the center position of the subset Finally, the linear projection U is applied to the features to generate a new point set The formula is as follows:

[0096] f′ i =MaxPool({f j U}|f j ∈F i )

[0097]

[0098] In this example, the initial grid size is 0.02 meters, which is the same as the voxelized grid size in the data preprocessing stage. The grid size multiplier is set to [x4.0, x2.0, x2.0, x2.0], which means that the grid size in each stage is [0.08, 0.16, 0.32, 0.64] meters respectively. The points after grid pooling are regrouped using patch groups and sent to the next layer of backbone feature extraction module.

[0099] Third, after the Local Mamba local feature extraction module of the M layer and the Grid DownPooling layer of the M-1 layer, in order to capture the information lost by the Local Mamba, this example proposes a Global Mamba module to extract global features from the point cloud. After the local feature learning and grid pooling operations of the M layer, each point passed to the Global Mamba module often represents a larger receptive field than the points of the first M layers. Here, the use of the Global Mamba module can better integrate the information of the previous layers and take into account the computational efficiency.

[0100] Specifically, Global Mamba includes the following steps: First, the features learned by the M-layer Local Mamba are firstly Hilbert input (Hilbert serialization) to learn the relative position encoding of the input coordinates and feature encoding of the input features. Then the learned relative position information and feature information are added element by element. Finally, the global information of the first M layers is input into MambaBlock to learn, and then the output features are layer-normalized using LayerNorm to stabilize the training process. The formula is as follows:

[0101] RPE=Linear(δ(Linear(f p )))

[0102] f out =LayerNorm(SelectiveSSM(RPE(f p )+Linear(f in )))

[0103] Among them, f p represents the input coordinates, δ represents the GELU activation function, RPE represents the relative position encoding, and f in represents the input feature, f out Represents the output of the global Mamba, where LayerNorm is the normalization of the input of each layer, Linear is the linear transformation, and SelectiveSSM is the selective state space model.

[0104] Fourth, grid upsampling (Grid UpPooling) is used to build an upsampling decoding network and a point-by-point classification module. The upsampling interpolation network includes an upsampling module and a grid mapping module. The upsampling module uses a cascade feature propagation layer U that forms a symmetrical structure with the sampling layer. i (i=1,2,3,4) is upsampled, and the fine features are propagated back from the key points to each original point, so that the number of points is restored. The traditional upsampling method obtains the original point features through linear interpolation calculation. This method is based on the principle of nearest neighbor. It finds the nearest neighbor points in the existing point cloud data and generates new points based on their position and attribute information. This paper proposes a mapping-based upsampling module. Specifically, first, the position of each point is recorded in the GridPooling process of the encoding (encoder) stage, and these positions will be used to remap the features to the original points during the upsampling process. Then, in the upsampling stage, the features in the fused point set can be mapped to the features of each point in the original point set using the grid division information in the pooling stage. The formula is as follows:

[0105]

[0106] in, represents the upsampled feature, f′ j Represents the features after pooling, (p i ,f i ) belongs to the subset M j . In each subset, the features of each point are obtained from the pooled point set. This method ensures that the features can be correctly restored during the upsampling process and that local and global spatial information can be retained. Finally, LocalMamba blocks are used to optimize the features restored during the upsampling process. In this example, the depth of each stack of LocalMamba blocks is set to [1,1,1,1]. Compared with the traditional linear interpolation method, since the point set obtained by the grid mapping in our upsampling stage is the point set of the original position, the spatial information is better preserved.

[0107] Finally, the cross entropy loss function is used as the loss function for training the point cloud semantic segmentation network to supervise network learning; the cross entropy loss function is a common loss function for point cloud semantic segmentation tasks. Specifically, the cross entropy loss function measures the difference between the predicted category distribution and the true category distribution. Its function is defined as follows:

[0108]

[0109] Where N is the number of samples, C is the number of categories, and y i,c is the true label (one-hot encoding) that sample i belongs to category c. is the probability that sample i is predicted to be category c. In specific experiments, the cross entropy loss function is usually used in conjunction with the Softmax activation function. The Softmax function converts the logits output by the network into a probability distribution, and then uses the cross entropy loss to calculate the difference between the predicted distribution and the true distribution.

[0110] To verify the effectiveness of this solution, the following is a further explanation based on experimental data: For the evaluation of segmentation effect, in addition to visual intuition, quantitative evaluation is more important. Start from the two aspects of training efficiency and segmentation effect. In terms of training efficiency: the training efficiency is analyzed by the changes in training loss (loss) and training accuracy (Acc) with the number of training times; in terms of segmentation effect: the two most classic evaluation indicators, intersection over union (IoU) and accuracy (Acc), are used, and IoU and Acc are used to evaluate the segmentation effect of each semantic category. The average intersection over union (mIoU), average accuracy (mAcc), and overall accuracy (OA) are used to evaluate the overall segmentation effect.

[0111] Intersection over union (IoU) represents the ratio of the intersection and union of the prediction set and the label set. The formula is as follows:

[0112]

[0113] Accuracy (Acc) represents the ratio of the number of correct samples in a certain classification to the total number of samples, and mean Accuracy (mAcc) is the mean of the accuracies of all categories.

[0114]

[0115] The confusion matrix is ​​also called the error matrix, which is represented by an n*n matrix. It is shown as follows:

[0116] Predicted as positive label Predicted as negative label Positive label data True Positive(TP) False Negative(FN) Negatively labeled data False Positive (FP) True Negative(TN)

[0117] Precision (P) refers to the number of samples correctly classified into a certain category and the number of samples classified into that category.

[0118]

[0119] The recall rate indicates the ratio of the number of correctly classified positive samples to the number of positive samples.

[0120]

[0121] TP represents samples with positive labels and positive predictions, FN represents samples with positive labels and negative predictions, FP represents samples with negative labels and positive predictions, and TN represents samples with negative labels and negative predictions. In addition, mean intersection over union (mIoU), category average accuracy (mAcc), and overall accuracy (OA) are also commonly used metrics. mIoU is the average value of IoU, mAcc represents the average value of each category, and OA represents the proportion of correctly predicted points in the total number of point clouds.

[0122] The experiment was tested and run on Ubuntu 22.04.4LTS, based on the Pytorch=1.12 deep learning framework, the development environment was PyCharm 2021.3.3, and the calculation was completed on Nvidia RTX 4090. The experimental environment configuration is shown in Table 1

[0123] Table 1: Experimental environment configuration of the present invention

[0124]

[0125] To implement the present invention, two mainstream public point cloud indoor scene datasets ScanNetV2 and S3DIS are used as validation datasets, and the experimental results on these two datasets illustrate the specific implementation process. The S3DIS dataset consists of 271 rooms in six areas from three different buildings, including 13 categories. According to the standard protocol, area 5 is retained during the training process and used for S3DIS testing. The ScanNetv2 dataset contains 1201 training scenes and 312 validation scans reconstructed from RGB-D frames. The model uses the reconstructed grid sampled point cloud as input, and each point is assigned a semantic label from 20 categories.

[0126] The processed training set data in the dataset is input into the constructed network model, saved in the training process, and verified by the validation set to obtain a set of weights with the best performance.

[0127] Table 2 shows the comparative experimental results of the proposed method with other advanced algorithms on the S3DIS dataset. The proposed method brings significant performance improvement, achieving segmentation performance of 91.7%, 79.6% and 72.5% on OA, mAc and mIoU respectively. It is significantly better than the existing methods under the evaluation index of mAc, and the other two indicators also maintain advantages.

[0128] Table 2: Comparison of the proposed method with other methods on S3DIS Area5

[0129]

[0130] Comparison of generalization experimental results: Table 3 shows the comparison results of the proposed method with other advanced algorithms on the ScanNetv2 dataset. The model method on this dataset is more inclined to the comparison of the two mIoU indicators on the validation set and the test set. The proposed model reaches 77.4% on the Test mIoU, and its overall performance is better than other methods.

[0131] Table 3: Comparison of the proposed method with other methods on the ScanNet V2 dataset

[0132]

[0133]

[0134] This method uses an average of 20GB of multi-card video memory during training. It is worth noting that the method of the present invention achieves excellent performance on S3DIS Area5, with mIoU 0.5% higher than the Stratified Transformer method, 2.1% higher than the PointTransformer method, and 5.4% higher than the KPConv method, while mAc is also 1.5%, 3.1%, and 6.8% higher, respectively. A good average intersection-over-union ratio is achieved on ScanNetV2, with Test mIoU improving by 1.8% over OACNNs, 0.8% higher than OctFormer, and 3.2% higher than Point Transformer V2. The network does not need to be pre-trained on the data set first, and the network converges faster than the same series of Transformer models.

[0135] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A point cloud semantic segmentation method based on Mamba, characterized in that: The following steps are included in order: Obtain a point cloud semantic segmentation dataset and divide it into a training set T1, a test set T2, and a validation set V; use the training set T1 to complete E rounds of supervised parameter training for the point cloud scene semantic segmentation model, where E is the number of training rounds and E≥50; Construct a 3D point cloud backbone feature extraction network to map the point cloud from the input space to the feature space, perform hierarchical feature extraction, and learn the association between the aggregated information at each layer; Construct an initial feature encoding module to serialize the preprocessed data set into point cloud and extract features through sparse convolution to obtain the initial features of the point cloud; Construct local Mamba modules and grid downsampling modules to perform hierarchical feature extraction, learn the association between aggregated information at each layer, and promote local feature fusion; Construct a global Mamba module, use the global Mamba to further fuse the information of each local Mamba layer and capture the missing information of the local Mamba; Constructing grid upsampling and grid mapping modules to restore point cloud data to the original number of points and generate learned refined global semantic information for each point; Construct a point-by-point classification module, use the multi-classification cross entropy loss function to supervise network learning, map the number of channels of each point in the learned point cloud to the number corresponding to the category, and predict the label corresponding to each point.

2. The Mamba-based point cloud semantic segmentation method according to claim 1, characterized in that: The step of obtaining the initial features of the point cloud includes serializing the preprocessed data set using four different space filling curves including Z-order curve, Trans Z-order curve, Hibert curve, and Trans Hibert curve to map the 3D point cloud data to a one-dimensional sequence and extract features through 3D sparse convolution to obtain the initial features of the point cloud; the expression is: Among them, p represents the position coordinates (x, y, z) of the point, b represents the batch index, g represents the grid size, << represents the left shift operator, k represents the number of bits of left shift, φ -1 represents the inverse mapping function of the space-filling curve, represents the rounding operator, and | represents the bitwise OR operator.

3. The Mamba-based point cloud semantic segmentation method according to claim 1, characterized in that: The local Mamba module and grid downsampling module use Patch grouping to divide a large point cloud into several overlapping Patch neighborhoods, and then use the local Mamba module to learn local features for the point cloud in each Patch, including the following steps: first, the output obtained from the first-layer feature encoding module is passed through the xCPE layer to perform position encoding on the point cloud data to enhance the expression ability of the position feature, and then input into the Mamba block layer through the LayerNorm layer, and after the output of the Mamaba block layer, it is sent to the MLP layer through the LayerNorm layer, and finally, it is output through the MLP layer and sent to the next layer of Encoder for grid downsampling.

4. The Mamba-based point cloud semantic segmentation method according to claim 3, characterized in that: The position encoding comprises the following steps: using a uniform grid to divide the point set S = (P, F) into a plurality of non-overlapping subsets [S1, S2, ..., S n ]Then the features of the points in each subset are max-pooled to obtain the features of the subset f i ′ , average pooling is performed on the positions of the points in each subset to obtain the center position of the subset Finally, the linear projection U is applied to the features to generate a new point set 5. The Mamba-based point cloud semantic segmentation method according to claim 1, characterized in that: The global Mamba module consists of the following steps: First, a linear layer is used to map the point cloud coordinate information to feature information Linear=f′ in =Wf p +b Where W is the weight matrix, b is the bias vector, and f p ∈R N*3 represents the point cloud coordinates, f′ in ∈R N*C Represents the feature information after coordinate mapping; Among them, h i is the input feature, σ 2 is the variance of the input feature, μ is the mean of the input feature, ∈ is a small constant to prevent division by zero, γ is the scaling parameter, β is the translation parameter, and y i is the normalized output feature; Among them, h(t) is the hidden state at time step t, I(t) is the input at time step t, A and B are the parameter matrices of the model, the observation matrix C describes how the state vector is mapped to the output signal, and the direct transfer matrix D describes how the input signal directly affects the output signal; RPE=Linrar(δ(Linrar(f p ))) f out =LayerNorm(SelectiveSSM(RPE(f p )+Linear(f in ))) Among them, f p represents the input coordinates, δ represents the GELU activation function, RPE represents the relative position encoding, and f in represents the input feature, f out Represents the output of the global Mamba; LayerNorm is the normalization of the input of each layer, Linear is the linear transformation, and SelectiveSSM is the selective state space model.

6. The Mamba-based point cloud semantic segmentation method according to claim 1, characterized in that: Construct a grid upsampling and grid mapping module. In the grid downsampling stage in the encoder, record the position information of each point. In the upsampling stage, use grid division and feature mapping to map the fused point set features to the features of each point in the original point set. Finally, restore the point cloud data to the original number of points to generate the learned refined global semantic information for each point. The expression is as follows: in, represents the upsampled features, f ′j Represents the features after pooling, (p i ,f i ) belongs to the subset M j .

7. The Mamba-based point cloud semantic segmentation method according to claim 1, characterized in that: A point-by-point classification module is constructed for the upsampled and restored points, and the multi-classification cross entropy loss function is used to supervise the network learning. The number of channels of each point in the learned point cloud is mapped to the number corresponding to the category, and the label corresponding to each point is predicted.

8. A point cloud semantic segmentation device based on Mamba, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program can be executed by the processor to implement a large-scale point cloud semantic segmentation method based on Mamba as claimed in claim 1.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a Mamba-based point cloud semantic segmentation method as claimed in claim 1 is implemented.

Citation Information

Cited By

  • Long-term time sequence prediction method of double-path Mama for data processing

    CN119557605A

  • 3D point cloud target detection method based on state space model

    CN119741697A

  • 3D printing concrete interface pore structure visual segmentation method based on point cloud

    CN119991701A

  • Robot action prediction method and device, equipment and storage medium

    CN120023835A

  • Robot action prediction method, device, equipment and storage medium

    CN120023835B