Point cloud segmentation method and system

Abstract

The invention relates to a point cloud segmentation method and system, and relates to the technical field of point cloud segmentation, and the method comprises the steps: obtaining point cloud data, constructing an octree for the point cloud data, carrying out the information enhancement of nodes in the octree, and obtaining a point cloud octree after the information enhancement; generating a point cloud sequence X based on the point cloud octree after information enhancement, and performing Patch coding on the point cloud sequence X; and carrying out point cloud segmentation on the point cloud sequence X after Patch coding. According to the method, the problem of over-segmentation or under-segmentation of point cloud segmentation in the prior art can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud segmentation, and in particular to a point cloud segmentation method and system. Background Art

[0002] Point clouds, as carriers of rich information in three-dimensional space, have been widely used in fields such as robotic perception, autonomous driving, and virtual reality. Point cloud segmentation, as a core step in classifying data, has long been a research focus. However, due to the sparsity and disorder of point clouds, their development has relied heavily on deep learning methods to extract essential features and process point cloud scenes. Existing deep learning methods can be generally divided into two categories: voxel-based and point-based. Voxel-based methods utilize uniform voxel sampling to extract point cloud features, reducing the 3D data of the point cloud to accommodate 2D convolutions. However, the dense mapping computations incur significant overhead. Even with methods that organize sparse point clouds to avoid redundant voxel cube construction, point cloud processing is limited to local regions and is difficult to adapt to larger point cloud scenes. Point-based methods, exemplified by PointNet and PointNet++, directly extract feature information from the raw point cloud, aggregate point cloud neighborhood features by constructing a k-nearest neighbor search tree, and downsample the point cloud using methods such as farthest point sampling (FPS) and grid sampling. However, both when aggregating neighborhood features using the K-Nearest Neighbors (KNN) algorithm and when sampling point clouds using FPS, point-based methods incur significant computational overhead, severely limiting their generalization capabilities for point cloud scenarios. Furthermore, deep learning methods that utilize voxels and points are all based on convolutional neural networks (CNNs). The receptive field size of this CNN model limits its ability to process point clouds, making it difficult to capture the contextual dependencies of point cloud models, thereby increasing the probability of these methods making errors in point cloud segmentation tasks.

[0003] To expand the network's receptive field and improve its ability to process point clouds, researchers introduced the Transformer into deep learning methods, designed an architecture based on the sensitivity of the attention mechanism to data, and achieved some progress. However, the use of the attention mechanism will exponentially increase the time and space complexity of the algorithm, and consume a huge amount of resources. Even if the use of window division or hash voxel methods can significantly reduce the time and space complexity, this will cause the network to focus too much on local information, sacrificing the global perception of the point cloud, causing such methods to easily cause over-segmentation of the entire point cloud. Of course, there are also some methods that can expand the perception of global information, but these methods often have difficulty paying attention to individual point cloud information, ignore the processing of detailed features of the point cloud, and are unable to organize the global information of the point cloud, which seriously affects the accuracy of the point cloud segmentation task and often leads to under-segmentation.

[0004] In summary, the current methods for point cloud segmentation have the problem of over-segmentation or under-segmentation, and it is urgent to find a method to balance this problem. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem of over-segmentation or under-segmentation in point cloud segmentation in the prior art.

[0006] To solve the above technical problems, the present invention provides a point cloud segmentation method, comprising:

[0007] Obtain point cloud data, construct an octree for the point cloud data, and perform information enhancement on the nodes in the octree to obtain an information-enhanced point cloud octree;

[0008] Generate a point cloud sequence X based on the information-enhanced point cloud octree, and perform Patch encoding on the point cloud sequence X;

[0009] Perform point cloud segmentation on the point cloud sequence X after Patch encoding.

[0010] In one embodiment of the present invention, the method for enhancing information of nodes in an octree includes:

[0011] During the construction process, the information of the corresponding parent node to the root node is added to each node of the octree as a global association. At the same time, the information of the remaining leaf nodes at the same depth as the current node is included as a local association to obtain the information-enhanced point cloud octree.

[0012] In one embodiment of the present invention, the formula for performing information enhancement on any node in the octree is:

[0013] φ′ feat (p i )=Concat(φ feat (p i),φ partfeat (p i(2-8) ),φ overallfeat (p iI ,p iI-1 ,...,p0))

[0014] Among them, φ feat (p i ) is the node p i Self-information, φ partfeat (p i(2-8) ) for its p i The local information provided by the seven adjacent nodes, φ overallfeat (p iI ,p iI-1 ,...,p0) is the node p i The global information provided by all associated parent nodes, p iI For p i The corresponding parent node has iI<d; d is the depth of the octree, p0 is the root node, and concat is the data concatenation function.

[0015] In one embodiment of the present invention, the method for generating a point cloud sequence X based on the information-enhanced point cloud octree includes:

[0016] Morton code is used to construct binary key values ​​for point cloud nodes of the same depth in the octree. The scanning method of the binary key value is expressed as:

[0017] key(x,y,z)=x0y0z0..x. i y i z i ..x. d-1 y d-1 z d-1

[0018] Among them, key(x,y,z) represents the key value of the octree node (x,y,z), d is the depth of the octree, (x i ,y i ,z i ) is the position code of the i-th node;

[0019] The point cloud sequence X is obtained by scanning the octree nodes using binary key values, which is expressed as:

[0020]

[0021] Among them, L is the sequence length; M is the point cloud feature mapping dimension; x t is an element in the sequence X and x t Scanned by key(x,y,z); x tIt is a spatial-semantic joint representation derived from the mapping of octree nodes and is expressed as:

[0022] x t =Concat(φ coard (p t ),φ feature (f t ),φ oct (c t ))

[0023] Among them, φ coord (p t ) is the node p t Spatial information, φ feature (f t ) is the node p t Attribute information, φ oct (c t ) is p t The contextual relevance information in the octree.

[0024] In one embodiment of the present invention, the method for performing point cloud segmentation on the patch-encoded point cloud sequence X includes:

[0025] The point cloud sequence X after Patch encoding is feature aggregated through the Point-TransMamba architecture, and then the aggregated features are segmented by the decoder, wherein the Point-TransMamba architecture includes a first mamba layer and multiple Transformer-Mamba hybrid layers, the first mamba layer is used to perform point cloud feature expansion on the point cloud sequence X after Patch encoding, and the Transformer-Mamba hybrid layer is used to perform point cloud feature aggregation on the data after point cloud feature expansion.

[0026] In one embodiment of the present invention, the Transformer-Mamba hybrid layer includes a Transformer layer, a second mamba layer and a multi-layer perceptron MLP connected in sequence, the Transformer layer is used to extract local feature information of point cloud data, the second mamba layer is used to extract global feature information of point cloud data, and the multi-layer perceptron MLP is used to aggregate local and global feature information of point cloud data.

[0027] In one embodiment of the present invention, the first mamba layer and the second mamba layer both include a first RMSNorm layer, the first RMSNorm layer is connected to the first branch and the second branch respectively, the first branch includes a first linear layer, the first linear layer is connected to the first sub-branch and the second sub-branch respectively, the first sub-branch includes a first DW convolution, a first activation function σ, a feedforward SSM, and a second RMSNorm layer connected in sequence; the second sub-branch includes a second DW convolution, a second activation function σ, a feedback SSM, and a third RMSNorm layer connected in sequence; the second branch includes a second linear layer;

[0028] The output of the second RMSNorm layer is element-wise added to the output of the second linear layer to obtain a first summation result; the output of the third RMSNorm layer is element-wise added to the output of the second linear layer to obtain a second summation result; the first summation result is element-wise added to the second summation result to obtain a third summation result; the third summation result is summed with the data input into the first RMSNorm layer after passing through the third linear layer.

[0029] In one embodiment of the present invention, the first mamba layer satisfies:

[0030]

[0031] Among them, X0 is used as the point cloud sequence X and input into the first mamba layer to obtain the output sequence K is the overall transformation matrix of the state space model, and is obtained by continuously updating the state of the first mamba layer. The formula is:

[0032]

[0033] in, Δ is the sampling step of the learnable point cloud; I is the identity matrix; A, B, and C are projected from the input sequence X0 to the output sequence The matrix that realizes state recording, input control, and state transition in the process; k is used to record the state change of each step in the sequence.

[0034] In one embodiment of the present invention, the Transformer layer satisfies:

[0035]

[0036] Among them, X0 is used as the point cloud sequence X and input into the first mamba layer to obtain the output sequence is the output of the Transformer layer; Atten represents multi-head attention; OctDil is the octree expansion window constructed when constructing the point cloud sequence; LN is the normalization layer.

[0037] To solve the above technical problems, the present invention provides a point cloud segmentation system, comprising:

[0038] Acquisition module: used to acquire point cloud data, build an octree for the point cloud data, enhance the information of the nodes in the octree, and obtain the point cloud octree after information enhancement;

[0039] Sequence generation module: used to generate point cloud sequence X based on the information enhanced point cloud octree and perform patch encoding on the point cloud sequence X;

[0040] Segmentation module: used to perform point cloud segmentation on the point cloud sequence X after Patch encoding.

[0041] The above technical solution of the present invention has the following advantages over the prior art:

[0042] The point cloud segmentation method described in the present invention, based on the spatial recursion of an octree to sort unordered point clouds, supplements the local and global correlation information between octree nodes with adjacent nodes at the same depth and the recursive root node to meet the input requirements of the first Mamba layer and the Transformer layer. On this basis, to ensure that the network can both model the global features of the point cloud and fully obtain the local information of a single point cloud, the present invention designs a Point-TransMamba architecture for application on three-dimensional point clouds to solve the problem of window attention losing the ability to process information outside the window and eliminate the problem of the Mamba layer ignoring the local information of a single point cloud during the spatial state reasoning process. In other words, the present invention uses Mamba to recursively transfer the global state along the octree level to model global information; and uses the Transformer layer to perform window attention on non-empty nodes at the same level to fully process the information of a single point cloud to aggregate the features of the neighborhood point cloud. This Point-TransMamba architecture applied to the octree not only inherits Mamba's linear and efficient modeling capabilities for disordered point clouds to understand the overall point cloud model, but also retains the feature information of individual point clouds that is ignored by SSM through the local sensitivity of the Transformer layer. It ensures the maintenance of point cloud topological consistency and algorithm performance in a hybrid drive manner, and solves the problems of over-segmentation and under-segmentation in the point cloud segmentation process to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0044] Figure 1 is a flow chart of the method of the present invention;

[0045] Figure 2It is a visualization architecture diagram of the method of the present invention;

[0046] Figure 3 Schematic diagram of global information association and local information association of octree nodes of the present invention;

[0047] Figure 4 This is a schematic diagram of a two-dimensional scan of a z-order curve of the present invention;

[0048] Figure 5 This is a schematic diagram of the octree sorting point cloud sampling points and performing z-order sorting according to the present invention;

[0049] Figure 6 This is a schematic diagram of the structure of the first mamba layer and the second mamba layer of the present invention;

[0050] Figure 7 This is a schematic diagram of the Transformer-Mamba hybrid layer structure of the present invention;

[0051] Figure 8 Schematic diagram of the calculation process of the ClS decoder of the present invention;

[0052] Figure 9 Schematic diagram of the calculation process of the SeG decoder of the present invention;

[0053] Figure 10 This is a diagram showing the point cloud segmentation effects of each method;

[0054] Figure 11 This is a comparison chart of the scale experiment of the Point-TransMamba architecture segmentation model;

[0055] Figure 12 This is a comparison of the segmentation effects of Point-TransMamba architectures of different scales. DETAILED DESCRIPTION

[0056] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0057] Example 1

[0058] Reference Figure 1 As shown, the present invention relates to a point cloud segmentation method, comprising:

[0059] Obtain point cloud data, construct an octree for the point cloud data, and perform information enhancement on the nodes in the octree to obtain an information-enhanced point cloud octree;

[0060] Generate a point cloud sequence X based on the information-enhanced point cloud octree, and perform Patch encoding on the point cloud sequence X;

[0061] Perform point cloud segmentation on the point cloud sequence X after Patch encoding.

[0062] The following is a detailed introduction to this embodiment:

[0063] In the point cloud preprocessing stage, this embodiment improves the information processing method of the octree to complete the processing of the point cloud data. First, an octree is constructed for the normalized point cloud data. During the construction process, the corresponding parent node to the root node information is added to each octree node (except the root node) as a global association. At the same time, the information of the remaining leaf nodes at the same depth is included as a local association. Connections are added from both the overall and detailed perspectives to improve the richness of point node information, ensure frequent and smooth communication of information, and thus ensure the geometric correlation within the point cloud. After that, the point cloud is serialized and sorted, and windows are constructed and the internal data is shuffled to divide new windows (this process is called expansion) to meet the needs of information transmission between windows, and provide data support for the subsequent processing of the Mamba layer and the Transformer layer, while ensuring the normal operation of the hybrid framework in the point cloud segmentation stage.

[0064] In the point cloud segmentation stage, this embodiment designs a Point-TransMamba architecture applied to three-dimensional point clouds to process point cloud sequences that have completed spatial mapping through preprocessing and Patch encoding. Figure 2 As shown, Num represents the number of calculations of the Point-TransMamba architecture. Different numbers determine the operation process of the algorithm. When Num = 1 or 3, it is calculated normally in the architecture. When Num = 2 or 4, it is input into the corresponding decoder to complete the corresponding segmentation task. Different numbers represent different adapted data sets; N represents the number of TransMamba layers, which determines the number of aggregation times of point cloud features in the hybrid architecture. In this stage, the preprocessed point cloud is first passed through the first mamba layer to complete the feature expansion, and then the coarse and fine granularity information of the point cloud is processed through the TransMamba layer (composed of three modules: Transformer layer, second mamba layer and MLP layer) to realize feature aggregation. After that, it is input into the subsequent network through downsampling processing to ensure the full expansion of point cloud information to meet the in-depth processing requirements of subsequent steps. Finally, the extracted and aggregated point cloud information is input into the decoder (i.e., according to the task requirements). Figure 2 This approach effectively solves the problem of window attention being unable to process point cloud information outside the window, and the spatial state model ignoring the local information of a single point cloud during state reasoning, which leads to over-segmentation or under-segmentation in the point cloud model segmentation process.

[0065] (1) Point cloud preprocessing

[0066] See also Figure 2 In the point cloud preprocessing stage, this embodiment will perform certain processing on the original point cloud to meet the input requirements of different modules of the Point-TransMamba architecture in the subsequent point cloud segmentation stage. The implementation process will be described in detail below.

[0067] First, a GPU-capable point cloud octree is constructed from the original point cloud to generate a sorted point cloud. Prior to serialization, to improve the semantic information density and spatial hierarchy of the sequence, this embodiment expands the information contained in the internal nodes of the sorted octree point cloud to maintain the geometric connectivity within the point cloud.

[0068] Therefore, this embodiment performs the following processing on the information contained in these nodes. For ease of explanation, this embodiment uses the node p1 in the octree constructed with p0 as the root node and octree depth d as an example, where the parent node of p1 is denoted as p I (I<d), parent node p I The 7 leaf nodes adjacent to node p1 are denoted as p 2-8 First, this embodiment sets the leaf node p 2-8 All the information of p1 is aggregated and the aggregated information is provided to p1 to supplement the local information of p1 at the same depth and build local associations. I The information contained in is provided to p1, providing high-level information and providing part of the global association. Finally, iterate the parent node p I The parent node of the node is aggregated to the root node p0, and this part of information is added to the node p1 to provide another part of the global association, thus realizing cross-level transmission. Thus, this embodiment can obtain enhanced node data information by adding the local information association of adjacent nodes and the global information association of iterative parent nodes on the basis of the original information of node p1. Its visual representation is as follows Figure 3 shown.

[0069] Therefore, the enhanced information after adding association to the original node information is formulated as shown in formula (1):

[0070] φ′ feat (p1)=Concat(φ feat (p1),φ partfeat (p 2-8 ),φ overallfeat (p I ,p I-1 ,...,p0)) (1)

[0071] Among them, concat is a data splicing function used to splice information; φ feat (p1) is the node p1’s own information, φ partfeat (p 2-8 ) is its adjacent node p2-8 The local correlation information provided, φ overallfeat (p I ,p I-1 ,...,p0) is the global information provided by all associated parent nodes of node p1, and I<d.

[0072] This embodiment further extends this to all sorted point clouds after octree processing. For the enhanced information of any node of the octree, its form is shown in formula (2):

[0073] φ′ feat (p i )=Concat(φ feat (p i ),φ partfeat (p i(2-8) ),φ overallfeat (p iI ,p iI-1 ,...,p0)) (2)

[0074] Among them, concat is a data splicing function used to splice information; φ feat (p i ) is the node p i Self-information, φ partfeat (p i(2-8) ) for its p i The local correlation information provided by the seven adjacent nodes, φ overallfeat (p iI ,p iI-1 ,...,p0) is the node p i The global information provided by all associated parent nodes, p iI For p i The corresponding parent node has iI<d.

[0075] After this processing, this embodiment constructs a point cloud octree sequence after expanding the node information. The Morton code is used to construct binary key values ​​for point cloud nodes of the same depth. The form is shown in formula (3). The scanning method of these key values ​​is as follows: Figure 4 As shown:

[0076] key(x,y,z)=x0y0z0..x. i y i z i ..x. d-1 y d-1 z d-1 (3)

[0077] Among them, key(x,y,z) represents the key value of the octree node (x,y,z), and d is the depth of the octree, so (x i ,yi ,z i ) is the position code of the i-th node.

[0078] To facilitate the subsequent description of the sorting strategy, this embodiment takes part of the point cloud at the same depth in the octree as an example, such as Figure 5 (a) is a schematic diagram of the octree sorting point cloud sampling points. Figure 5 The gray box shown in (a) is the actual point cloud location. Figure 4 The 2nd order z-order curve shown is scanned to obtain the following Figure 5 The schematic diagram of z-order sorting of the octree-sorted point cloud sampling points shown in (b) is used to set the key value with the help of formula (3) to obtain the ordered point cloud sequence X, which is expressed as: Among them, x in X t Obtained by scanning key (x, y, z), L is the sequence length, M is the point cloud feature mapping dimension, x t It is a spatial-semantic joint representation derived from the mapping of octree nodes, and the information it contains is shown in formula (4):

[0079] x t =Concat(φ coard (p t ),φ feature (f t ),φ oct (c t )) (4)

[0080] Among them, φ coord (p t ) is the node p t Spatial information, φ feature (f t ) is p t Attribute information (such as color, label, etc.), φ oct (c t ) is p t The contextual relevance information in the octree.

[0081] In order to meet the needs of the local attention mechanism, this embodiment performs window processing on the point cloud sequence. Specifically, the point cloud data in the window can be expanded by dilation, that is, the data of different window partitions are mixed to form a new window to improve the sensitivity of the attention mechanism to the data. For example, the octree expansion window in the following formula (15) is used. Since the octree expansion window belongs to the existing technology, it will not be described in detail.

[0082] At this point, this embodiment has completed the point cloud preprocessing work, and can obtain the point cloud data of the associated local and overall information of the serialized and divided windows, and pass it to the point cloud segmentation stage to complete the segmentation task.

[0083] (2) Point cloud segmentation

[0084] See also Figure 2 Through point cloud preprocessing, this embodiment can obtain an ordered point cloud sequence X carrying correlation information. In this stage, the task of point cloud segmentation is achieved by identifying these data. The network core relies on the Point-TransMamba architecture. Therefore, the implementation process of the Point-TransMamba architecture will be mainly introduced below.

[0085] This embodiment designs a hybrid architecture for 3D applications, namely the Point-TransMamba architecture, to solve the problem of ignoring fine-grained information perception when capturing information. The Point-TransMamba architecture combines the advantages of window attention's sensitivity to fine-grained features and Mamba's contextual association to better analyze and process point cloud models. The structure of this hybrid architecture is as follows: Figure 2 As shown in Figure 1, it consists of a first Mamba layer (also called the Pre-Mamba layer) and multiple Transformer-Mamba hybrid layers (TransMamba layers for short). The first Mamba layer is used to expand the internal features of the encoded point cloud sequence X, while the TransMamba layer aggregates the expanded point cloud feature information using a windowed attention mechanism (i.e., the Transformer layer) and a spatial state model. The alternating effect of these layers significantly enhances the network's ability to aggregate point cloud features. The specific implementation of these structures is described below.

[0086] See also Figure 7 The Transformer-Mamba hybrid layer (i.e., TransMamba layer) includes a Transformer layer, a second mamba layer, and a multi-layer perceptron MLP connected in sequence. The Transformer layer is used to extract local feature information of the point cloud data, the second mamba layer is used to extract global feature information of the point cloud data, and the multi-layer perceptron MLP is used to aggregate the local and global feature information of the point cloud data as the output of the Transformer-Mamba hybrid layer.

[0087] See also Figure 6 and Figure 7, the first mamba layer and the second mamba layer both include a first RMSNorm layer, the first RMSNorm layer is connected to the first branch and the second branch respectively, the first branch includes a first linear layer, the first linear layer is connected to the first sub-branch and the second sub-branch respectively, the first sub-branch includes a first DW convolution, a first activation function σ, a feedforward SSM (i.e., Forward SSM), and a second RMSNorm layer connected in sequence; the second sub-branch includes a second DW convolution, a second activation function σ, a feedback SSM (i.e., Backward SSM), and a third RMSNorm layer connected in sequence; the second branch includes a second linear layer.

[0088] The output of the second RMSNorm layer is element-wise added to the output of the second linear layer to obtain a first summation result; the output of the third RMSNorm layer is element-wise added to the output of the second linear layer to obtain a second summation result; the first summation result is element-wise added to the second summation result to obtain a third summation result; the third summation result is summed with the data input into the first RMSNorm layer after passing through the third linear layer.

[0089] The structure of the first mamba layer is as follows Figure 6 and Figure 7 As shown in the figure, this layer linearly processes the RMS normalized (RMSNorm) data and inputs them into the forward spatial state model (Forward SSM) and the backward spatial state model (Backward SSM) to infer the state. Then, it normalizes and integrates these two parts of the state with the initial state. Finally, the output result completes the expansion of the point cloud and is passed to the next stage of calculation. The following will mainly introduce the operation mechanism of SSM. The point cloud sequence X sorted by the octree strategy is used as the input sequence of this layer. For the convenience of subsequent explanation, the point cloud sequence X is rewritten as X0 as the initial input sequence of the network, so there is also X0 = [x t ] t＝1 L , and in the first mamba layer the input sequence X0 will be processed by the linear system to obtain the output sequence The following is the calculation process of the first mamba layer.

[0090] First, this embodiment ignores the discrete characteristics of the point cloud and converts x t Rewritten as a continuous function And define the output as So from input x0(t) to output The implementation process is shown in Equations (5) and (6). Equation (5) is the state equation that records local memory; Equation (6) is the observation equation that realizes global projection.

[0091] h′(t)=Ah(t)+Bx0(t) (5)

[0092]

[0093] Among them, h(t) represents the potential state, and the four matrices A, B, C, and D respectively realize the four functions of state recording, input control, state transition, and jump connection.

[0094] Next, this embodiment introduces the sparse discrete characteristics of the point cloud into equations (5) and (6). Before that, the zero-order hold (ZOH) method is used to appropriately process A and B to meet the needs of discrete data. The processing method is shown in equations (7) and (8):

[0095]

[0096] Where Δ is the sampling step of the learnable point cloud and I is the identity matrix.

[0097] Therefore, in this embodiment, equations (5) and (6) can be rewritten to meet the needs of point cloud data, as shown in equations (9) and (10):

[0098]

[0099] Among them, h k represents the state at time t, That is the input and output at that moment, and there is It should be noted that equations (3) and (4) conform to continuous characteristics. In this embodiment, equations (3) and (4) are derived from continuous characteristics to equations (9) and (10) of discrete characteristics. Therefore, there is no matrix D corresponding to the jump connection in equation (3).

[0100] From this embodiment, the actual iterative process of the point cloud sequence at the first mamba layer can be deduced as shown in formula (11):

[0101]

[0102] Among them, K is obtained by continuously updating the state of the feedforward and feedback SSM layers in the first mamba layer of the network layer. Its value is calculated as shown in formula (12):

[0103]

[0104] After the first mamba layer expands the features, the output sequence It will enter multiple TransMamba layers to complete the aggregation of point cloud features. The structure of TransMamba layer is as follows Figure 7As shown, this layer mainly consists of the Transformer layer, the second Mamba layer and the multi-layer perceptron MLP. The calculation description of the TransMamba layer is shown in formula (13). After one layer of calculation Will be converted into a new sequence Y0:

[0105]

[0106] Among them, TransMamba is the general term for the alternating framework of Transormer layer, second Mamba layer and multi-layer perceptron MLP. is the initial input of this layer, and Y0 is the initial output.

[0107] From this, we can deduce the data processing method of the subsequent TransMamba layer, as shown in formula (14):

[0108] Y i+1 =TransMamba(Y i ),i=0,...,N-2 (14)

[0109] Among them, Y i Y is the input of the subsequent TransMamba layer. i+1 For its output, N represents the number of TransMamba layers, and the actual number of layers is determined according to different task requirements.

[0110] For the calculation process of the TransMamba layer, this embodiment will take the input sequence The first layer calculation of the output sequence Y0 is used as an example.

[0111] first The local information of the data in the sequence is aggregated by the Transformer layer using a window method, and then input into the multi-head attention operation. The calculation process is shown in formula (15):

[0112]

[0113] Among them, Atten (Multi-Attention) represents multi-head attention, OctDil (Octree Dilation) is the octree expansion window constructed when constructing the point cloud sequence, and LN (Layer Norm) is the normalization layer.

[0114] It should be noted that, in this embodiment, if the octree expansion window is not used when constructing the point cloud sequence, equation (15) can be adjusted to

[0115] Calculated by the Transformer layer, It will continue to enter the second Mamba layer to realize the aggregation of global information. Since the second Mamba layer is the same as the first Mamba layer, the calculation method is consistent with the above. The calculation method of this part can be derived from formula (11), as shown in (16):

[0116]

[0117] Where MLP(·) is the perception processing process, and K0 is calculated in the same way as K in formula (11), as shown in formula (12).

[0118] Therefore, this embodiment can obtain the calculation process of the remaining N-1 layers of TransMamba, and the process is shown in formula (17):

[0119] Y i+1 =MLP(K i+1 ×(Atten(OctDil(LN(Y i )))+Y i )),i=0,...,N-2 (17)

[0120] Among them, K i+1 Calculated by formula (12), Y i is the input, Y i+1 is the actual output, i is the iteration of the TransMamba calculation process, and N is the number of TransMamba layers.

[0121] At this point, this embodiment completes the entire processing of point cloud features within the Point-TransMamba architecture. This data will then be fed into a downsampling layer (which expands the resolution of the point cloud data, providing a foundation for deeper processing of the point cloud data) to complete data expansion and prepare for further processing. Through multiple layers of alternating Point-TransMamba architecture and downsampling, this embodiment can obtain a mapping result of aggregated point cloud information at all coarse and fine granularity, denoted as Y. This point cloud information Y will then be fed into different decoders to complete the segmentation task.

[0122] For the ModelNet40 dataset, the Point-TransMamba architecture is executed 2 times, and the point cloud information is processed by combining global pooling and MLP layers. Y , its architecture is as follows Figure 8 The calculation form is shown in formula (18):

[0123] ClS=MLP(globalpool(Y)) (18)

[0124] Among them, globalpool is the global pooling function, and ClS is the processing result of the ModelNet40 dataset.

[0125] For the segmentation task related to the ScanNet dataset, the Point-TransMamba architecture is executed 4 times and the data is parsed in the form of Feature Pyramid Networks (FPN), which is as follows Figure 9 As shown in the figure, the point cloud information Y is processed by upsampling and downsampling to obtain multi-scale information, and then the data is segmented by the prediction head. Its composition is as follows Figure 9 The calculation method is shown in formula (19):

[0126] SeG=pred(FPN(Y)) (19)

[0127] Among them, FPN represents the feature pyramid network structure processing process, pred is the prediction head, and SeG is the processing result of the ScanNet dataset.

[0128] So far, this embodiment has obtained the running process of all tasks in the point cloud segmentation stage, and trained the overall hybrid framework in a supervised manner to meet the point cloud segmentation task requirements required by actual scenarios.

[0129] Experimental analysis

[0130] The following experiments will be conducted on the Point-TransMamba hybrid architecture designed in this embodiment to verify the effectiveness of the method.

[0131] (1) Dataset

[0132] This embodiment uses two different data sets to analyze the actual effect of the hybrid architecture. For the classification task, this embodiment uses ModelNet40, which contains 12,311 CAD models of 40 categories. During the experiment, 9,843 models are used as training sets and 2,468 models are used as test sets. In order to improve the training effect, this embodiment adopts the same data preprocessing and enhancement method as PointNet, that is, random anisotropic scaling, random rotation in the range of [0,π], and random translation in the range of [-0.2,0.2]. For the segmentation task, this embodiment uses the ScanNet dataset, which is a large-scale dataset containing three-dimensional scans of indoor scenes, including 1,513 indoor scenes of 20 semantic categories in total, with an average number of scan points of 148,000. During the experiment, 1,213 scenes were used as training sets and 300 scenes were used as test sets. This embodiment also performs data augmentation on the ScanNet dataset, that is, randomly rotates this part of the training data in the range of [-π, π], randomly scales it in the range of [0.75, 1.25], and randomly translates it in the range of [-0.1, 0.1].

[0133] The experimental environment of this embodiment is under Ubuntu20.04, the training graphics card is NIVIDIA RTX4090D, and the operating environment is built on Anaconda, configured as python3.10 and applying torch2.1.0 and cuda118. For the classification task, this embodiment adopts the framework of Octformer (O) as the training benchmark, which uses Adam optimizer to train 200 cycles in a multi-step decay mode (MultiStepLR), and the learning rate lr is set to 0.001. The remaining parameters of these two training methods are consistent, while the training batch size batchsize is 32, the weight decay weight_decay is 0.05, the octree recursion depth is 6, and the number of graphics cards used for training is 1. For the segmentation task, this embodiment also uses Adam optimizer to train 600 cycles in a MultiStepLR mode, the learning rate lr is set to 0.0015, the batch size batchsize is 16, the weight decay weight_decay is 0.05, the octree recursion depth is 11, and the number of graphics cards used for training is 3.

[0134] (2) Evaluation criteria

[0135] The evaluation standard used in this embodiment is the accuracy (accu) evaluation of the ModelNet40 dataset, that is, the proportion of correctly classified labels in the total is calculated. The calculation formula is shown in formula (20):

[0136]

[0137] Among them, pred represents the number of labels correctly classified by the network, and logits is the total number of labels.

[0138] This embodiment uses Mean Intersection over Union (MIoU) as the evaluation criterion to process the ScanNet dataset, which is shown in formula (21):

[0139]

[0140] Among them, i is the correct label, j is the wrong label; p ij Indicates that i The prediction is j, recorded as the number of false negatives (FN); p ji Indicates that j is predicted as i, which is recorded as the number of false positives (FP); p ii Indicates that i The prediction is i, k is the number of true positives (TP), and k is the number of categories in the dataset.

[0141] (3) Experimental results

[0142] Figure 10 The results show a comparison of the hybrid architecture approach of this embodiment with the Transformer-based OctFormer and Mamba-based Point-Mamba segmentation methods. Experiments demonstrate that this embodiment's hybrid architecture approach, by integrating both global and local point cloud information and alternating between Transformer and Mamba processing, can better analyze point cloud data, resolve mis-segmentation issues, and achieve superior segmentation results.

[0143] (4) Ablation experiment

[0144] (4.1) Octree sorting ablation

[0145] Unlike normal construction, this embodiment adds global and local associations to each node of the octree to enhance the richness of node information, thereby improving the performance of classification and segmentation tasks. To demonstrate the effectiveness of this approach, this embodiment compared the two sorting methods on the ModelNet40 dataset, with the results shown in Table 1. Experiments demonstrate that the additional node association method maintains the internal set associations of the point cloud, better processing the coarse and fine-grained features of the point cloud, and better completing related tasks. Compared to the method without adding associations, the accuracy rate increased by 1.5%, further demonstrating the advantages of this strategy.

[0146] Table 1 Comparison of results of octree sorting information processing methods

[0147]

[0148] (4.2) Network architecture combination experiment

[0149] To validate the advantages of hybrid architectures for point cloud segmentation tasks, this example compared the performance of multiple different architectures using the ModelNet40 dataset. Ten experiments were conducted, using MLP, Transformer, Mamba, and the hybrid architecture of this example as benchmarks. The results are shown in Table 2. The experiments demonstrate that the hybrid architecture takes the lead due to its unique advantage in aggregating global and local point cloud features. Transformer and Mamba standalone combinations ranked second and third, respectively, with the remaining combinations performing less well. This is because networks using the same architecture can better bridge information transfer between previous and next modules, ensuring that data is not lost or discarded during processing. Because Transformer alone lacks Mamba to handle the transfer and association of out-of-window information, it lacks some consideration of global information, resulting in inferior classification performance compared to the method of this example. The experimental results further demonstrate that the hybrid architecture of Transformer and Mamba of this example can better aggregate coarse-grained and fine-grained point cloud features, achieving higher experimental accuracy.

[0150] Table 2 Comparison of network architecture combination classification experimental results

[0151]

[0152] (4.3) ModelNet40 dataset architecture scale experiment

[0153] In order to compare the impact of hybrid architecture scale on the experiment, this embodiment focuses on the T rans M am b a The number of layers is set to design an ablation experiment. This embodiment sets 4 layers and 8 layers respectively to compare with the original 6 layers. The results are shown in Table 3. The experiment proves that when the number of layers is 6, the network performance and running time of the hybrid architecture are balanced and the best effect is achieved. Too many or too few layers are better. TransMamba The number of layers will affect the capacity of the network, resulting in reduced accuracy.

[0154] Table 3 ModelNet40 dataset architecture scale (i.e. TransMam b a Comparison of experimental results (number of layers)

[0155]

[0156] (4.4) ScanNet dataset hybrid architecture segmentation model scale experiment

[0157] In order to fully demonstrate the performance of the Point-TransMamba architecture on the ScanNet dataset, this embodiment designs an ablation experiment based on the impact of models of different scales on the segmentation task. However, due to the limitations of the experimental environment and the fact that the ScanNet dataset itself is too large, this embodiment mainly compares two scale models, namely the Point-TransMamba_seg_small model with the number of TransMamba layers being [2,2,6,2] and the Point-TransMamba_seg_base model with the number of layers being [2,2,18,2] (which is the main model of the method of this embodiment). The remaining parameters of the two models remain the same, and the experimental results are shown in Figure 2. Figure 11 shown. Figure 11 Voting refers to the voting strategy applied in PCT, which can improve performance.

[0158] Experiments show that Point-TransMamba_seg_base sets more T in the third downsampling stage rans M am b a The layer helps to more fully aggregate and analyze the features of point cloud data to achieve better segmentation. At the same time, the actual segmentation effects of the two models are as follows Figure 12 As shown, it can be seen that the two models have obvious differences in the processing of details of the point cloud scene, and the larger-scale model of this embodiment can achieve better results.

[0159] (5) Method comparison

[0160] To verify the actual performance of the method of this embodiment on the ModelNet40 dataset, this embodiment collected data and actual performance from papers such as PointNet, PCNN, and PointMamba, and compared them with the method of this embodiment. The results are shown in Table 4. As can be seen from Table 3, the method of this embodiment achieves better results on the classification task by using a hybrid architecture of Transformer layers and Mamba on the octree, surpassing the single baseline method. This shows that the hybrid architecture used by the method of this embodiment can solve the problem that a single Mamba or Transformer layer cannot process the entire domain information of the point cloud by processing point cloud data through alternating Transformer layers and Mamba. It more accurately identifies point cloud information and classifies the segmented labels. At the same time, this framework achieves better results than the method of crudely connecting two models, demonstrating the advantages of the alternating processing of the method of this embodiment. These experiments further demonstrate the operational advantages of the hybrid architecture on the ModelNet40 dataset.

[0161] Table 4 Comparison of experimental accuracy of ModelNet40 dataset

[0162]

[0163] Among them, PointNet++(1k) and PointNet++(5k) represent the different results when the PointNet++ point cloud sampling value is 1k and 5k, respectively.

[0164] For segmentation experiments, this embodiment also uses the same settings as the ModelNet40 dataset and compares this embodiment with other methods on the ScanNet dataset. The experimental results are shown in Table 5. The experiments demonstrate that this embodiment's method also achieves excellent results on large-scale indoor scene datasets, surpassing the performance of the Transformer and Mamba methods, which are single-benchmark methods. This embodiment's method is less affected by the difficulty of fully extracting information from a single benchmark, which impacts the segmentation task. This further demonstrates the advantages of hybrid architectures for point cloud segmentation tasks.

[0165] Table 5 Comparison of mIoU indicators in ScanNet segmentation experiments

[0166]

[0167]

[0168] Voting refers to the voting strategy used in PCT, which can improve the performance of the algorithm.

[0169] Example 2

[0170] This embodiment provides a point cloud segmentation system, including:

[0171] Acquisition module: used to acquire point cloud data, build an octree for the point cloud data, enhance the information of the nodes in the octree, and obtain the point cloud octree after information enhancement;

[0172] Sequence generation module: used to generate point cloud sequence X based on the information enhanced point cloud octree and perform patch encoding on the point cloud sequence X;

[0173] Segmentation module: used to perform point cloud segmentation on the point cloud sequence X after Patch encoding.

[0174] Example 3

[0175] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the point cloud segmentation method described in the first embodiment are implemented.

[0176] Example 4

[0177] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the point cloud segmentation method described in the first embodiment are implemented.

[0178] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.

[0179] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0182] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0183] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A point cloud segmentation method, characterized by: include: Obtain point cloud data, construct an octree for the point cloud data, and perform information enhancement on the nodes in the octree to obtain an information-enhanced point cloud octree; Generate a point cloud sequence X based on the information-enhanced point cloud octree, and perform Patch encoding on the point cloud sequence X; Perform point cloud segmentation on the point cloud sequence X after Patch encoding.

2. The point cloud segmentation method according to claim 1, wherein: The method for enhancing information of nodes in an octree comprises: During the construction process, the information of the corresponding parent node to the root node is added to each node of the octree as a global association. At the same time, the information of the remaining leaf nodes at the same depth as the current node is included as a local association to obtain the information-enhanced point cloud octree.

3. The point cloud segmentation method according to claim 1, wherein: The formula for information enhancement for any node in the octree is: f f ′ eat (p i )=Concat(φ feat (p i ),f partfeat (p i(2-8) ),f overallfeat (p iI ,p iI-1 ,...,p0)) Among them, φ feat (p i ) is the node p i Self-information, φ partfeat (p i(2-8) ) for its p i The local information provided by the seven adjacent nodes, φ overallfeat (p iI ,p iI-1 ,...,p0) is the node p i The global information provided by all associated parent nodes, p iI For p i The corresponding parent node has iI<d; d is the depth of the octree, p0 is the root node, and concat is the data concatenation function.

4. The point cloud segmentation method according to claim 1, wherein: The method for generating a point cloud sequence X based on the information-enhanced point cloud octree includes: Morton code is used to construct binary key values ​​for point cloud nodes of the same depth in the octree. The scanning method of the binary key value is expressed as: key(x,y,z)=x0y0z0..x. i y i z i ..x. d-1 y d-1 z d-1 Among them, key(x,y,z) represents the key value of the octree node (x,y,z), d is the depth of the octree, (x i ,y i ,z i ) is the position code of the i-th node; The point cloud sequence X is obtained by scanning the octree nodes using binary key values, which is expressed as: Among them, L is the sequence length; M is the point cloud feature mapping dimension; x t is an element in the sequence X and x t Scanned by key(x,y,z); x t It is a spatial-semantic joint representation derived from the mapping of octree nodes and is expressed as: x t =Concat(φ coard (p t ),φ feature (f t ),φ oct (c t )) Among them, φ coord (p t ) is the node p t Spatial information, φ feature (f t ) is the node p t Attribute information, φ oct (c t ) is p t The contextual relevance information in the octree.

5. The point cloud segmentation method according to claim 1, wherein: The method for performing point cloud segmentation on the point cloud sequence X after Patch encoding includes: The point cloud sequence X after Patch encoding is feature aggregated through the Point-TransMamba architecture, and then the aggregated features are segmented by the decoder, wherein the Point-TransMamba architecture includes a first mamba layer and multiple Transformer-Mamba hybrid layers, the first mamba layer is used to perform point cloud feature expansion on the point cloud sequence X after Patch encoding, and the Transformer-Mamba hybrid layer is used to perform point cloud feature aggregation on the data after point cloud feature expansion.

6. The point cloud segmentation method according to claim 6, wherein: The Transformer-Mamba hybrid layer includes a Transformer layer, a second Mamba layer and a multi-layer perceptron MLP connected in sequence. The Transformer layer is used to extract local feature information of point cloud data, the second Mamba layer is used to extract global feature information of point cloud data, and the multi-layer perceptron MLP is used to aggregate local and global feature information of point cloud data.

7. The point cloud segmentation method according to claim 6, wherein: The first mamba layer and the second mamba layer both include a first RMSNorm layer, the first RMSNorm layer is connected to the first branch and the second branch respectively, the first branch includes a first linear layer, the first linear layer is connected to the first sub-branch and the second sub-branch respectively, the first sub-branch includes a first DW convolution, a first activation function σ, a feedforward SSM, and a second RMSNorm layer connected in sequence; the second sub-branch includes a second DW convolution, a second activation function σ, a feedback SSM, and a third RMSNorm layer connected in sequence; the second branch includes a second linear layer; Performing element-wise addition of the output of the second RMSNorm layer and the output of the second linear layer to obtain a first summation result; The output of the third RMSNorm layer is element-wise added to the output of the second linear layer to obtain a second summation result; the first summation result is element-wise added to the second summation result to obtain a third summation result; the third summation result is summed with the data input into the first RMSNorm layer after passing through the third linear layer.

8. The point cloud segmentation method according to claim 6, wherein: The first mamba layer meets: Among them, X0 is used as the point cloud sequence X and input into the first mamba layer to obtain the output sequence K is the overall transformation matrix of the state space model, and is obtained by continuously updating the state of the first mamba layer. The formula is: in, Δ is the sampling step of the learnable point cloud; I is the identity matrix; A, B, and C are projected from the input sequence X0 to the output sequence The matrix that realizes state recording, input control, and state transition in the process; k is used to record the state changes at each step of the sequence.

9. The point cloud segmentation method according to claim 6, wherein: The Transformer layer satisfies: Among them, X0 is used as the point cloud sequence X and input into the first mamba layer to obtain the output sequence is the output of the Transformer layer; Atten represents multi-head attention; OctDil is the octree expansion window constructed when constructing the point cloud sequence; LN is the normalization layer.

10. A point cloud segmentation system, characterized by: include: Acquisition module: used to acquire point cloud data, build an octree for the point cloud data, enhance the information of the nodes in the octree, and obtain the point cloud octree after information enhancement; Sequence generation module: used to generate point cloud sequence X based on the information enhanced point cloud octree and perform patch encoding on the point cloud sequence X; Segmentation module: used to perform point cloud segmentation on the point cloud sequence X after Patch encoding.

Images