A point cloud semantic segmentation method based on a state space model

Abstract

This invention provides a point cloud semantic segmentation method based on a state-space model, belonging to the field of point cloud semantic segmentation technology. It aims to address the problems of long-range information collapse, single serialization bias, and insufficient local feature aggregation capability in the downsampling stage of state-space models when modeling large-scale point cloud sequences. This invention alleviates long-range information decay by stacking sub-manifold convolutional layers after the state-space module, utilizing convolutional locality bias; simultaneously, it employs a staggered Hilbert serialization strategy, randomly switching Hilbert curves of three axis orders to reduce spatial bias and enhance spatial diversity; furthermore, it designs an adaptive downsampling module, utilizing a selective state mechanism to aggregate features and retain the last point as output, thereby comprehensively improving global and local modeling capabilities. This method ensures excellent segmentation accuracy while also offering the advantages of fast inference speed and low memory consumption, making it suitable for large-scale complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of point cloud semantic segmentation technology, and more specifically, to a point cloud semantic segmentation method based on a state space model. Background Technology

[0002] With the widespread adoption of LiDAR, RGB-D cameras, and 3D scanning equipment, point cloud data has become the core carrier for 3D scene perception, playing an irreplaceable role in fields such as autonomous driving environment understanding, robot autonomous navigation, smart city spatial modeling, remote sensing mapping and feature recognition, and 3D digital twin construction. Point cloud semantic segmentation, as a fundamental technology for parsing scene semantic information, requires assigning precise semantic categories to massive discrete points one by one. Its processing accuracy and efficiency directly affect the performance of downstream tasks. Faced with the demands of point cloud processing in large-scale, complex scenes, the industry urgently needs semantic segmentation technologies that balance high segmentation accuracy, low computational overhead, and strong generalization capabilities.

[0003] Current point cloud semantic segmentation techniques mainly develop along three paths. Projection-based methods map 3D point clouds to a 2D plane and then process them using 2D convolutional networks. Voxel-based methods transform the point cloud structure through regular meshing. Point-based methods directly construct local feature relationships on the original point set. In recent years, Transformer-based segmentation methods have introduced attention mechanisms to improve long-distance dependency modeling capabilities, controlling computational complexity through window partitioning or sparse attention strategies. Some studies have attempted to introduce state-space models to process sequential point cloud data, using space-filling curves to establish a mapping relationship from 3D points to 1D sequences, thereby reducing the computational burden of modeling long sequences.

[0004] Existing technical solutions have significant limitations. Projection methods cause irreversible loss of 3D spatial structural information, voxelization methods lead to a surge in storage and computing resources in large-scale scenes, and point-based methods rely on highly complex neighborhood query operations. While Transformer methods improve modeling capabilities, the computational complexity of the self-attention mechanism increases quadratically with the number of points, making it difficult to meet real-time requirements in terms of memory usage and inference latency. When directly applying state-space models, unidirectional sequential scanning leads to the loss of backward context information, fixed serialization methods introduce spatial relationship biases, long-distance state propagation causes local geometric feature attenuation, and traditional downsampling strategies lose local details and semantic cues due to the use of max pooling or average pooling. These problems collectively restrict the practical application of large-scale point cloud semantic segmentation technology in complex scenes. Summary of the Invention

[0005] This invention provides a point cloud semantic segmentation method based on a state-space model to improve at least one of the aforementioned technical problems.

[0006] This invention provides a point cloud semantic segmentation method based on a state space model, which includes steps S1 to S5.

[0007] S1: Acquire point cloud data, preprocess and voxelize it, embed the original attribute features to obtain initial point features.

[0008] S2, the three coordinate components of the voxel grid coordinates are Hilbert encoded in three cyclic arrangements. In the current encoding stage, the current point features are sorted according to the corresponding encoding to obtain the current point feature sequence.

[0009] S3, the current point feature sequence is input into the state space convolutional feature extraction module, and forward and reverse scanning are performed by the bidirectional state space module respectively, and the scan outputs are fused. The fused scan outputs are reverse-ordered to their three-dimensional spatial positions, enhanced by two layers of sub-manifold sparse convolution, and then fused with the residuals of the reverse-ordered scan outputs to obtain the point cloud semantic features of the current encoding stage.

[0010] S4. Input the semantic features of the point cloud into the selective state sampling module, compress and serialize the encoding, divide the points with the same encoding into local regions, reset the hidden state by the region boundary and perform state aggregation, take the end state of the region as the representative feature, and record the mapping relationship between the points and the regions.

[0011] S5, the representative feature is used as the current point feature in the next encoding stage, and S2 to S4 are repeated according to the Hilbert encoding of the next encoding stage until the encoding is completed. In the decoding stage, the features are upsampled step by step according to the mapping relationship, the upsampled features are fused with the encoded features of the same layer and enhanced by the state space convolution feature extraction module, and the recovered features are mapped to the semantic category space to output the semantic label of each point.

[0012] As a preferred aspect of the present invention, each point in the point cloud data includes three-dimensional spatial coordinates and at least one original attribute feature selected from color information, reflection intensity information, and normal vector information.

[0013] As a preferred aspect of the present invention, the preprocessing includes coordinate normalization, original attribute feature normalization, mesh sampling, random rotation, random scaling, random flipping, random jittering, and random cropping.

[0014] As a preferred aspect of the present invention, voxel meshing includes: discretizing the continuous spatial coordinates of each point according to a preset voxel size to obtain voxel mesh coordinates.

[0015] .

[0016] In the formula, Indicates the first The voxel grid coordinates corresponding to each point. Indicates the first The point is at Voxel grid coordinates along the axis. Indicates the first The point is at Voxel grid coordinates along the axis. Indicates the first The point is at Voxel grid coordinates along the axis. Indicates the first The three-dimensional spatial coordinates of each point. Indicates the first The first-dimensional spatial coordinates of each point. Indicates the first The second-dimensional spatial coordinates of each point. Indicates the first The third-dimensional spatial coordinates of each point. Represents the real number field. It represents a three-dimensional vector composed of the minimum coordinate components of the current point cloud scene on the three coordinate axes. This indicates the preset voxel size that is greater than 0. Indicates a point index. This indicates a floor operation performed on the coordinate components.

[0017] Embedding the original attribute features includes: using linear mapping layers or sparse convolutional mapping layers to embed the first attribute feature. Embed the original attribute features of each point:

[0018] .

[0019] In the formula, Indicates the first The current point features of each point. Indicates the first The original attribute features of each point. This represents the learnable weight matrix in the input feature embedding module. This represents the bias vector in the input feature embedding module.

[0020] As a preferred aspect of the present invention, S2 specifically includes: No. The voxel grid coordinates of each point are According to the first The voxel grid coordinates are Hilbert encoded using a cyclic arrangement order.

[0021] .

[0022] In the formula, Indicates the first The point at the th The Hilbert sequence encoding corresponding to a cyclic permutation order. Indicates the first The voxel grid coordinates corresponding to each point. Indicates a point index. This indicates the sequential index for a cyclic arrangement. This represents the encoding function for the three-dimensional Hilbert space filling curve. Indicates the first A cyclical arrangement order transformation operation. .

[0023] The three cyclical arrangements are as follows: .

[0024] .

[0025] .

[0026] In the formula, Indicates to maintain axis, shaft and The first type of cyclic arrangement order transformation operation for axis arrangement. Indicates according to axis, shaft and The second type of cyclic arrangement order transformation operation is performed on the axis arrangement order. Indicates according to axis, shaft and The third type of cyclic arrangement order transformation operation is performed on the axis arrangement order. Indicates the first The point is at Voxel grid coordinates along the axis. Indicates the first The point is at Voxel grid coordinates along the axis. Indicates the first The point is at Voxel grid coordinates along the axis.

[0027] According to the The Hilbert sequence encoding corresponding to the cyclic permutation order sorts the features of the current point in ascending order: .

[0028] In the formula, Indicates according to the first The current point feature sequence after being arranged in a cyclical order. Indicates the first The current point features of each point. This indicates the number of points in the current encoding stage. This represents a sorting operation that arranges point cloud features in ascending order based on serialization encoding.

[0029] As a preferred aspect of the present invention, S3 specifically includes: The bidirectional state space module performs discrete state updates along the forward and reverse order of the current point feature sequence, respectively, and fuses the forward scan output and the reverse scan output. Each direction is updated in the 6th... The state update relationship for each sequence position is as follows: .

[0030] .

[0031] In the formula, Indicates the first The hidden state corresponding to each discrete sequence position. Indicates the first The hidden state corresponding to each discrete sequence position. Represents the position index of a discrete sequence. This represents the state transition matrix after discretization. This represents the input mapping matrix after discretization. Indicates the first The input features corresponding to the positions of each discrete sequence. Indicates the first Output features corresponding to the positions of each discrete sequence. This represents the state output mapping parameter matrix. This represents the input pass-through parameter matrix.

[0032] No. Each network layer performs pointwise layer normalization and bidirectional selective state-space modeling on the input sequence features, and then fuses the modeling results with the input sequence feature residuals: .

[0033] In the formula, This represents the sequence characteristics after bidirectional selective state-space modeling. Indicates the first The sequence features input to each network layer. This represents the network layer index. Indicates to The sequence features are formed by performing point-by-point layer normalization on the features of each point. This indicates the bidirectional selective state space modeling operation performed by the bidirectional state space module.

[0034] Perform a reverse sort based on the sort index corresponding to the current cyclic arrangement used by the network layer: .

[0035] In the formula, This represents the feature representation after restoring the correspondence between three-dimensional spatial positions. Indicates by the first The first point in each of the network layers A sorted index generated by encoding a cyclic permutation order Hilbert sequence. This indicates the cyclic sorting index used by the current network layer. Indicates sorting based on index The reverse sorting operation was performed. .

[0036] As a preferred aspect of the present invention, S3 further includes: Both sub-manifold sparse convolutions have kernel sizes of 1. A 3D submanifold sparse convolutional layer with a stride of 1 and a padding size of 1 is used. The two submanifold sparse convolutional layers are connected to a batch normalization layer and a nonlinear activation function layer, respectively. The processing relationships between the first submanifold sparse convolutional layer, the second submanifold sparse convolutional layer, and residual fusion are as follows: .

[0037] .

[0038] .

[0039] In the formula, Indicates the first Point cloud features output by the first submanifold sparse convolutional layer in each network layer. Indicates the first Point cloud features output by the second submanifold sparse convolutional layer in a network layer. Indicates the first Point cloud features output by the state space convolutional feature extraction module in each network layer. This represents the sparse convolution operation on the first submanifold. This represents the sparse convolution operation on the second submanifold. This indicates batch normalization operation. This represents a non-linear activation function.

[0040] As a preferred aspect of the present invention, S4 specifically includes: No. The point cloud feature set output by each encoding stage is: .

[0041] In the formula, Indicates the first Point cloud features output from each encoding stage. Indicates the first In the first coding stage Feature representation of each point. Indicates the first The number of points output by each encoding stage. This indicates the index of the encoding stage. Indicates a point index.

[0042] make Indicates the first In the first coding stage The spatial coordinates of the points, and .make Indicates the first In the first coding stage Non-negative integer serialization encoding of points, where .in, Indicates the first The three-dimensional spatial coordinates of each point. Indicates the first The point at the th The Hilbert sequence encoding corresponding to a cyclic permutation order.

[0043] The encoding displacement depth is determined based on the preset downsampling step size: .

[0044] In the formula, It represents the non-negative integer single-dimensional spatial dimension encoded displacement depth determined by a preset downsampling step size. This represents the preset downsampling step size, which is not less than 1 and is a non-negative integer power of 2. This indicates the rounding up operation. This function returns the number of bits required for the binary representation of a non-negative integer, and returns 0 when the input is 0.

[0045] Perform a binary right shift on the serialization encoding of the current encoding stage: .

[0046] In the formula, Indicates the first In the first coding stage Each point completes the spatial sequence encoding after scale compression. This indicates a binary right shift operation.

[0047] Points with the same spatial sequence encoding after scale compression are grouped into the same local downsampling region, forming multiple non-empty and non-overlapping local downsampling regions: .

[0048] .

[0049] .

[0050] In the formula, Indicates the first A set of local downsampling regions for each coding stage. Indicates the first The first coding stage A non-empty local downsampling region. Indicates the first The number of local regions retained after downsampling in each coding stage. Indicates the first The first coding stage A non-empty local downsampling region. This represents the index of the first local downsampling region. Indicates the first The first coding stage A non-empty local downsampling region. This indicates the index of the second local downsampling region. Indicates the first The first coding stage A non-empty local downsampling region. This represents the enumeration index of the local downsampling region. Indicates by the first Spatial coordinates of each point in each coding stage The set of points that constitutes this.

[0051] As a preferred aspect of the present invention, S4 further includes: The first Point cloud features from each encoding stage are sorted according to serialized encoding, and selective state aggregation is performed by resetting the boundaries of each local downsampled region as the hidden state: .

[0052] In the formula, This represents the local aggregate sequence features after selective state modeling. Indicates the first The set of serialized codes for all points in each encoding stage. This represents a sorting operation that arranges point cloud features in ascending order based on serialization encoding. This indicates a point-by-point layer normalization operation. This refers to the state space feature aggregation process set in the selective state sampling module, using the boundaries of each local downsampling region as the hidden state reset boundary.

[0053] The state output corresponding to the end position of each local downsampled region in the current sequence order is determined as the representative feature: .

[0054] In the formula, Indicates the first In the first coding stage The local downsampling region is generated after selective state sampling and used for the first local downsampling region. Representative features of the input at each encoding stage. Indicates the first In the first coding stage The end index position of a local downsampled region in the current sequence order.

[0055] The spatial coordinates of the representative points in the next encoding stage are obtained by averaging the spatial coordinates of each point within the local downsampling area. .

[0056] In the formula, Indicates by the first In the first coding stage Spatial coordinates of each point within a local downsampling region The first obtained by aggregation Each encoding stage represents the spatial coordinates of a point. Indicates the first In the first coding stage The number of points contained in a non-empty local downsampling region.

[0057] As a preferred aspect of the present invention, S4 further includes: The scale-compressed spatial sequence encoding common to all points within each local downsampled region is determined as the encoding for that local downsampled region: .

[0058] In the formula, Indicates the first In the first coding stage Scale-compressed spatial sequence encoding of local downsampled regions. (Entering the...) After each encoding stage, the region index will be... As a point index for this encoding stage and order .

[0059] The encodings of each local downsampling region are reordered, and the corresponding inverse mapping index is generated: .

[0060] .

[0061] In the formula, This indicates the serialization sorting index required for the next encoding stage. This indicates an operation that returns an index sorted in ascending order by region encoding. This indicates the inverse mapping index required for the next encoding stage. This indicates the operation of generating an inverse mapping index based on the sorted index.

[0062] Depend on Constituting the first Point cloud features after downsampling at each encoding stage and take it as the first The current point features of the encoding stage, where the first... Number of points in each coding stage equal The mapping relationship includes the local downsampling region to which each point belongs, the serialization sorting index, and the inverse mapping index.

[0063] As a preferred aspect of the present invention, S5 specifically includes: In the first encoding stage, the current point feature sequence, arranged according to the current cyclic arrangement, is input into the state space convolutional feature extraction module. In subsequent encoding stages, the point cloud semantic features output from the previous encoding stage are sequentially input into the selective state sampling module and the state space convolutional feature extraction module of the current encoding stage. .

[0064] .

[0065] In the formula, Indicates the first Point cloud features output from each encoding stage. Indicates according to the first The current point feature sequence after being arranged in a cyclical order. This represents the state-space convolutional feature extraction process in the first encoding stage. This indicates the index of the encoding stage. This indicates the cyclic permutation order index used in the first encoding stage. Indicates the first Point cloud features output from each encoding stage. Indicates the first The state-space convolutional feature extraction process in each encoding stage. Indicates The associated point coordinates and serialization encoding are used as input and set in the first... After the coding stage and located at the , Selective state sampling process prior to each coding stage.

[0066] Let the deepest stage index of the encoder be... In the decoding stage, starting from the deepest stage of the encoder, the low-resolution decoding features are mapped to the next higher spatial scale according to the mapping relationship. The upsampled features are fused with the point cloud semantic features output from the corresponding encoding stage, and then enhanced by the state space convolutional feature extraction module on the decoding side. .

[0067] .

[0068] .

[0069] .

[0070] In the formula, This represents the initial low-resolution point features of the decoder. This represents the point cloud features output from the deepest stage of the encoder. Indicates restoration to the common level index. Point features corresponding to spatial scales. Represents hierarchical index The corresponding low-resolution decoding features. This indicates a feature upsampling operation performed based on the mapping relationship recorded during the encoding phase. This indicates the common level index of the current decoding scale and its corresponding encoding stage. Indicates a common hierarchical index The corresponding fusion decoding features. Indicates a common hierarchical index The point cloud features output during the corresponding encoding stage. This indicates a feature concatenation operation, a feature addition operation, or a fusion operation after linear projection. Indicates a common hierarchical index The recovery features are output from the corresponding decoding stage. This indicates that it is set in a common level index. This corresponds to the state-space convolutional feature enhancement process in the decoding stage. .

[0071] As a preferred aspect of the present invention, S5 further includes: The recovered features are input into the point-by-point segmentation module. The linear mapping layer obtains the prediction score for each semantic category at each point, and the softmax function is used to obtain the category prediction probability. The semantic category with the highest prediction probability is determined as the semantic label of the corresponding point. .

[0072] .

[0073] .

[0074] In the formula, Indicates the first Each point corresponds to A predicted score vector for each semantic category. This represents the learnable weight matrix in the pointwise segmentation module. The feature after restoration is the first Features of each point. This represents the bias vector in the point-by-point segmentation module. Indicates a point index. Indicates the first The point was predicted to be the [number]th [point]. The probability of the semantic category, and , Indicates the first The point is the first Predicted score for semantic category. Indicates the first The point is the first Predicted score for semantic category. This represents a semantic category summation index. This represents an exponential function. Indicates the first The final semantic prediction category for each point. This represents a semantic category index. This represents the number of semantic categories that are greater than 0. Represents the real number field.

[0075] As a preferred aspect of the present invention, during training, the labeled point cloud data is divided into a training set, a validation set, and a test set. Supervised training is performed using the training set. After each training round, the performance is verified using the validation set, and the network parameters at the optimal verification result are saved. After training is completed, the semantic labels of each point are output using the test set.

[0076] As a preferred aspect of the present invention, during training, the sum of cross-entropy loss and Lovasz loss is used as the overall supervision loss: .

[0077] .

[0078] .

[0079] In the formula, This represents the cross-entropy loss. This indicates the number of points with a value greater than 0 that are involved in the current loss calculation. Indicates by the first The true semantic category label of each point The true category indicator value obtained by one-hot encoding, when the first The point belongs to the th point When class, take 1 and not belong to the first class. The value is 0 when the class is used. Indicates the first The point was predicted to be the [number]th [point]. The probability of a semantic category. This represents the number of semantic categories that are greater than 0. This represents the natural logarithm function. This indicates the loss in Lovasz. Indicates the first The Jaccard collection functions corresponding to the class. Represents Jaccard set functions The Lovasz extension functions. Indicates according to the first The prediction error vector is constructed from the predicted probability of the class and the true class indication value. This represents the overall supervision loss of the point cloud semantic segmentation model.

[0080] By adopting the above technical solution, the present invention can achieve the following technical effects: This invention maps 3D point clouds into multiple staggered Hilbert sequences, which maintains spatial proximity while breaking the spatial bias caused by single serialization. This enables the selective state space model to efficiently process large-scale point cloud data, significantly reducing computational complexity and memory usage.

[0081] By combining a bidirectional scanning mechanism with a post-processed local sparse convolution design, this invention effectively suppresses the attenuation of local geometric structure and semantic information during long-distance context modeling, overcomes the problem of backward visibility loss caused by unidirectional state propagation, and thus obtains more refined feature representations in object boundaries and small target regions.

[0082] Furthermore, this invention introduces a selective state-space model into the downsampling process, utilizing a state aggregation mechanism to adaptively fuse effective features within local regions. Compared to traditional grid pooling or max pooling operations, this significantly reduces the number of points while preserving key semantic clues and geometric details to the greatest extent. This design enhances the model's adaptability to complex indoor and outdoor scenes, sparse point clouds, and various spatial structures. Combined with a joint optimization strategy of cross-entropy and Lovasz loss, it effectively alleviates the class imbalance problem, ultimately achieving simultaneous improvements in segmentation accuracy and processing efficiency on multiple mainstream datasets such as ScanNet, S3DIS, NuScenes, and DALES. Attached Figure Description

[0083] Figure 1 This is a schematic diagram illustrating the principle of the adaptive sampling mechanism.

[0084] Figure 2 This is a schematic diagram of the overall structure of a point cloud semantic segmentation network.

[0085] Figure 3 A visualization of the point cloud semantic segmentation results on the DALES dataset.

[0086] Figure 4 A visualization of the point cloud semantic segmentation results on the S3DIS dataset. Detailed Implementation

[0087] Please see Figures 1 to 4 This embodiment provides a point cloud semantic segmentation method based on a state-space model. The method preprocesses and voxelizes the original 3D point cloud, serializes the point cloud using Hilbert space-filling curves with three cyclic arrangements, utilizes a state-space convolutional feature extraction module for bidirectional long-range context modeling and local spatial feature enhancement, employs a selective state sampling module for local region segmentation, selective state aggregation, and representative feature generation, and progressively restores the point cloud feature resolution during the decoding stage. Finally, it predicts the semantic category of each point in the point cloud.

[0088] The point cloud semantic segmentation model constructed in this embodiment can be called the PointSSM model. For example... Figure 2 As shown, the point cloud semantic segmentation model employs a network structure combining an encoder and a decoder. The encoder includes an input feature embedding module, Hilbert point cloud serialization units based on three cyclic permutations, multiple state-space convolutional feature extraction modules, and a selective state sampling module positioned between adjacent encoding stages. The decoder includes a feature upsampling module, a skip connection feature fusion module, a decoding-side state-space convolutional feature extraction module, and a point-by-point segmentation module.

[0089] The state-space convolutional feature extraction module can be called the MambaConv module (i.e.) Figure 2 The MC module (in the context of DSamba) includes a bidirectional state space module, two sub-manifold sparse convolutional layers, and residual connection units. The bidirectional state space module can be a Bi-Mamba module. The selective state sampling module can be called a DSamba downsampling module, including a feature mapping unit, a selective state space aggregation unit, and a sampling unit.

[0090] This embodiment provides a point cloud semantic segmentation method based on a state-space model, which can be executed by a point cloud semantic segmentation device based on a state-space model (hereinafter referred to as: point cloud semantic segmentation device). Specifically, it is executed by one or more processors in the point cloud semantic segmentation device to implement steps S1 to S5.

[0091] S1: Acquire point cloud data, preprocess and voxelize it, embed the original attribute features to obtain initial point features.

[0092] The point cloud data can be point cloud scene data to be semantically segmented, or it can be labeled point cloud data used to train, validate, and test point cloud semantic segmentation models.

[0093] The point cloud data can originate from LiDAR, RGB-D cameras, 3D scanners, mobile mapping systems, or airborne LiDAR. Each point in the point cloud includes three-dimensional spatial coordinates and at least one of the following original attribute features: color information, reflection intensity information, and normal vector information. In other optional embodiments, the original attribute features may also include timestamps or other point attributes corresponding to a specific point cloud acquisition device.

[0094] Point cloud semantic segmentation datasets used for model training and performance validation may include indoor point cloud datasets and outdoor point cloud datasets. The indoor point cloud datasets may include ScanNetv2, ScanNet200, or S3DIS datasets, and the outdoor point cloud datasets may include NuScenes, SemanticKITTI, DALES, or Toronto3D datasets.

[0095] In the training implementation, the labeled point cloud data is divided into training sets. Validation set and test set Using the training set The point cloud semantic segmentation model is trained under supervision. After each training epoch, a validation set is used. Verify the semantic segmentation performance of the current model and save the network parameters corresponding to the optimal verification results. After completing model training, use the test set. The trained point cloud semantic segmentation model is tested to obtain the predicted semantic labels for each point in the test scene.

[0096] The original input point cloud sample with labels is represented as follows: In the formula, It represents a set of labeled original input point cloud samples consisting of the three-dimensional spatial coordinates of points, original attribute features, and true semantic category labels. This indicates the number of points in the current input point cloud. Indicates a point index. Indicates the first The three-dimensional spatial coordinates of each point. Indicates the first The original attribute features of each point. Indicates the first The actual semantic category label corresponding to each point.

[0097] No. The three-dimensional spatial coordinates of the points satisfy ,in, Indicates the first The first-dimensional spatial coordinates of each point Indicates the first The second-dimensional spatial coordinates of each point Indicates the first The third-dimensional spatial coordinates of each point Represents the field of real numbers. The... The original attribute features of each point satisfy ,in, This represents the original attribute feature dimension corresponding to each point.

[0098] The input point cloud of the model that does not contain true semantic category labels is represented as: In the formula, This represents the model input feature matrix, which consists of the three-dimensional spatial coordinates of each point and the original attribute features. This indicates the number of points in the current input point cloud. This represents the original attribute feature dimension corresponding to each point. This indicates that each point has three spatial coordinate components.

[0099] Before inputting point cloud data into the model, the point cloud data is preprocessed. The preprocessing may include one or more of the following: coordinate normalization, original attribute feature normalization, color normalization, mesh sampling, random discarding, random rotation, random scaling, random flipping, random jittering, elastic distortion, color perturbation, random cropping, and label sorting.

[0100] Coordinate normalization is used to adjust the coordinates in different point cloud scenes to a range suitable for model processing. Original attribute feature normalization is used to reduce numerical differences between different feature units or different acquisition devices. Random discarding, random rotation, random scaling, random flipping, random jittering, elastic distortion, color perturbation, and random pruning are used to expand the spatial morphology and attribute distribution of training samples, improving the model's adaptability to point cloud pose changes, local missing values, density changes, and acquisition noise.

[0101] After preprocessing, the continuous spatial coordinates of each point are discretized according to the preset voxel size, the point cloud is mapped to a three-dimensional isometric voxel grid, and the spatial coordinate information and original attribute feature information of each point are recorded.

[0102] No. The voxel grid coordinates of each point are determined according to the following formula: In the formula, Indicates the first The voxel grid coordinates corresponding to each point, and Indicates the first The point is at Voxel grid coordinates along the axis. Indicates the first The point is at Voxel grid coordinates along the axis. Indicates the first The point is at Voxel grid coordinates along the axis. Indicates the first The three-dimensional spatial coordinates of each point. It represents a three-dimensional vector composed of the minimum coordinate components of the current point cloud scene on the three coordinate axes. This indicates the preset voxel size that is greater than 0. Indicates a point index. This indicates a floor operation performed on the coordinate components.

[0103] The voxelization process can also be represented as a whole as: In the formula, This represents the voxelization operator that maps the input point cloud to a discrete voxel space. This represents the model input feature matrix before voxelization. This represents the discrete point cloud representation after voxelization.

[0104] After voxel meshing is completed, the original attribute features of each point are embedded using the input feature embedding module. This module can consist of a linear mapping layer, or it can consist of a sparse convolutional mapping layer, a normalization layer, and an activation function layer. When using a sparse convolutional mapping layer, feature computation is performed only at valid voxel locations to reduce the computational overhead caused by empty voxels.

[0105] No. The original attribute feature embedding relationship of each point is: In the formula, Indicates the first The current point feature of the nth point, in the first encoding stage, is the nth point. Initial point features of points. Indicates the first The original attribute features of each point. This represents the learnable weight matrix in the input feature embedding module. This represents the bias vector in the input feature embedding module. Indicates a point index.

[0106] Through the above processing, the coordinates, colors, reflection intensity, normal vectors or other original attributes in the point cloud are mapped to the feature space required by the network, thus obtaining the initial point features used in the first encoding stage.

[0107] S2, the three coordinate components of the voxel grid coordinates are Hilbert encoded in three cyclic arrangements. In the current encoding stage, the current point features are sorted according to the corresponding encoding to obtain the current point feature sequence.

[0108] Three-dimensional point clouds are characterized by disorder, sparsity, and irregular spatial distribution, while state-space models require state propagation according to a defined sequence. Therefore, this embodiment utilizes Hilbert space-filling curves to map the coordinates of a three-dimensional voxel grid into a one-dimensional serialized code. By changing the order in which the three coordinate components enter the Hilbert encoding function, three point cloud sequences with complementary spatial ergodic relationships are constructed.

[0109] In the first encoding stage, the current point feature is the initial point feature obtained in step S1. In subsequent encoding stages, the current point feature is the representative feature generated by the selective state sampling module in the previous encoding stage.

[0110] No. The voxel grid coordinates of each point are According to the first The voxel grid coordinates are Hilbert encoded using a cyclic arrangement order: In the formula, Indicates the first The point at the th The Hilbert sequence encoding corresponding to a cyclic permutation order. Indicates the first The voxel grid coordinates corresponding to each point. Indicates a point index. This indicates a cyclic sorted index, and This represents the encoding function for the three-dimensional Hilbert space filling curve. Indicates the first A cyclical arrangement order transformation operation.

[0111] The three cyclical arrangement orders are as follows: .

[0112] In the formula, Indicates to maintain axis, shaft and The first type of cyclic arrangement order transformation operation for axis arrangement. , and They represent the first The point is at axis, shaft and Voxel grid coordinates along the axis.

[0113] .

[0114] In the formula, Indicates according to axis, shaft and The second type of cyclic arrangement order transformation operation is performed on the axis arrangement order. , and They represent the first The point is at axis, shaft and Voxel grid coordinates along the axis.

[0115] .

[0116] In the formula, Indicates according to axis, shaft and The third type of cyclic arrangement order transformation operation is performed on the axis arrangement order. , and They represent the first The point is at axis, shaft and Voxel grid coordinates along the axis.

[0117] Substituting the three cyclic permutation orders into the three-dimensional Hilbert space filling curve encoding function, we can obtain: .

[0118] In the formula, Indicates the first The points are encoded in a one-dimensional Hilbert sequence under the first cyclic permutation order. This represents the encoding function for the three-dimensional Hilbert space filling curve.

[0119] .

[0120] In the formula, Indicates the first The points are encoded in a one-dimensional Hilbert sequence under the second cyclic permutation order. This represents the encoding function for the three-dimensional Hilbert space filling curve.

[0121] .

[0122] In the formula, Indicates the first The one-dimensional Hilbert sequence encoding of points in the third cyclic permutation order. This represents the encoding function for the three-dimensional Hilbert space filling curve.

[0123] In the current encoding phase, according to the first encoding phase adopted... The Hilbert sequence encoding corresponding to the cyclic permutation order is used to sort the features of the current point in ascending order: .

[0124] In the formula, Indicates according to the first The current point feature sequence after being arranged in a cyclical order. Indicates the first The current point features of each point represent the initial point features in the first encoding stage, and correspond to the representative features output by the previous encoding stage in subsequent encoding stages. This indicates the number of points in the current encoding stage. Indicates the first The point at the th The Hilbert sequence encoding corresponding to a cyclic permutation order. This represents a sorting operation that arranges point cloud features in ascending order based on serialization encoding. This indicates the cyclic permutation order index used in the current encoding stage.

[0125] The feature sequence of the current point corresponding to the first cyclic permutation order can be represented as follows: Its feature dimensions satisfy: In the formula, This represents the current point feature sequence after being arranged according to the first cyclic permutation order. This indicates the number of points in the current encoding stage. This indicates the feature channel dimension of the initial point that is greater than 0. Represents the real number field.

[0126] When the same point cloud is Hilbert encoded using different cyclic permutation orders, different one-dimensional spatial traversal paths will be formed. Points that are far apart in one cyclic permutation order may form closer sequential adjacency relationships in another cyclic permutation order. Therefore, by using different cyclic permutation orders at different encoding stages or in different network layers, feature sequences of the current point with complementary spatial adjacency relationships can be obtained, reducing the spatial relationship bias caused by a fixed single serialization method and enhancing the model's ability to express spatial structures in different directions.

[0127] S3: Input the current point feature sequence into the state space convolutional feature extraction module. Perform forward and reverse scanning through the bidirectional state space module and fuse the scan outputs. Reverse sort the fused scan outputs to their three-dimensional spatial positions, enhance them with two layers of sub-manifold sparse convolution, and then fuse them with the residuals of the reverse sorted scan outputs to obtain the point cloud semantic features of the current encoding stage.

[0128] The state-space convolutional feature extraction module includes a bidirectional state-space module, a first submanifold sparse convolutional layer, a first batch of normalization layers, a first nonlinear activation function layer, a second submanifold sparse convolutional layer, a second batch of normalization layers, a second nonlinear activation function layer, and residual connection units connected in sequence.

[0129] The bidirectional state-space module performs selective state-space modeling along both the forward and reverse order of the feature sequence of the current point. The forward scan is used to enable the current sequence position to obtain the context information corresponding to its predecessor position, and the reverse scan is used to enable the current sequence position to obtain the context information corresponding to its successor position, thereby alleviating the problem of the absence of backward information in unidirectional sequence scanning.

[0130] Before entering the bidirectional state space module, the first... Input sequence features of each network layer Perform point-by-point layer normalization. The mean of the feature channels for each point is: In the formula, Indicates the first The input of the nth network layer The mean of the feature channels of each point. Indicates the first The feature channel dimension of points with a value greater than 0 in each network layer. Indicates the feature channel index. Indicates the first The input of the nth network layer The point at the th Feature values ​​on each feature channel. This represents the network layer index. Indicates a point index.

[0131] No. The variance of the characteristic channels for each point is: In the formula, Indicates the first The input of the nth network layer The characteristic channel variance of each point. Indicates the first The point feature channel dimension of each network layer. Indicates the first The input of the nth network layer The point at the th Feature values ​​on each feature channel. This represents the mean of the characteristic channels at the corresponding point. Indicates the feature channel index.

[0132] The point-by-point layer normalization result is: In the formula, This indicates a point-by-point layer normalization operation. Indicates the first The input of the nth network layer The feature vector of each point. This represents the mean of the characteristic channels at the corresponding point. This represents the variance of the characteristic channels at the corresponding points. This represents a positive constant used to prevent the denominator from being zero. Indicates the first Learnable scaling vectors in each network layer. Indicates the first Learnable translation vectors in each network layer. This indicates element-wise multiplication performed on the characteristic channel.

[0133] As a supplementary explanation of the selective state-space modeling process, the hidden state update process of the continuous state-space model can be expressed as: In the formula, This represents a continuous-time variable. Representing continuous time variables The corresponding input features. Representing continuous time variables The corresponding hidden state. Indicates hidden state Regarding continuous time variables The derivative of . This represents the continuous state transition parameter matrix. This represents a continuous input mapping parameter matrix.

[0134] The output process of a continuous state-space model can be represented as: In the formula, Representing continuous time variables The corresponding output features. Representing continuous time variables The corresponding hidden state. Representing continuous time variables The corresponding input features. This represents the state output mapping parameter matrix. This represents the input pass-through parameter matrix.

[0135] After discretization, each scan direction of the bidirectional state-space module is in the... Each sequence position is updated discretely according to the following relationship: .

[0136] In the formula, Indicates the first The hidden state corresponding to each discrete sequence position. Indicates the first The hidden state corresponding to each discrete sequence position. Represents the position index of a discrete sequence. This represents the state transition matrix after discretization. This represents the input mapping matrix after discretization. Indicates the first The input features corresponding to the positions of each discrete sequence.

[0137] .

[0138] In the formula, Indicates the first Output features corresponding to the positions of each discrete sequence. Indicates the first The hidden state corresponding to each discrete sequence position. Indicates the first The input features corresponding to the positions of each discrete sequence. This represents the state output mapping parameter matrix. This represents the input pass-through parameter matrix.

[0139] For a forward scan, the discrete state update described above is performed sequentially from the beginning to the end according to the current point feature sequence. For a reverse scan, the discrete state update described above is performed sequentially from the end to the beginning according to the current point feature sequence, and the reverse scan output is restored to the same point order as the forward scan output.

[0140] The forward and reverse scan outputs can be fused using one or more of the following methods: summation, concatenated linear projection, gated weighting, or residual connection. The fused scan output contains sequence context information both before and after the current point.

[0141] No. Each network layer performs pointwise layer normalization and bidirectional selective state-space modeling on the input sequence features, and then fuses the modeling results with the input sequence features using residuals. .

[0142] In the formula, This represents the sequence features after bidirectional selective state-space modeling and fusion with the residual features of the input sequence. Indicates the first The sequence features input to each network layer. This represents the network layer index. Indicates to The sequence features are formed by performing point-by-point layer normalization on the features of each point. This indicates the bidirectional selective state space modeling operation performed by the bidirectional state space module, which includes forward scanning, reverse scanning, and fusion of the outputs of the two scanning directions.

[0143] because The sequence is still organized according to the one-dimensional Hilbert sequence. However, subsequent submanifold sparse convolutions require local spatial convolutions based on the three-dimensional voxel grid positions. Therefore, the fused scan output is inversely sorted according to the sorting index corresponding to the cyclic arrangement used by the current network layers. .

[0144] In the formula, This represents the feature representation after restoring the correspondence between three-dimensional spatial positions. This represents the sequence characteristics after bidirectional selective state-space modeling. Indicates by the first The first point in each of the network layers A sorted index generated by encoding a cyclic permutation order Hilbert sequence. This indicates the cyclic index used by the current network layer, and Indicates sorting based on index The reverse sorting operation was performed.

[0145] After completing the reverse sort, Input a locally sparse convolution module. Both the first and second submanifold sparse convolutional layers can use a kernel size of... A 3D submanifold sparse convolutional layer with a stride of 1 and a padding size of 1. Submanifold sparse convolution performs local convolution only at existing effective voxel locations without additionally expanding the effective voxel locations, thereby enhancing the local geometric relationships between adjacent effective points while maintaining the sparse structure of the point cloud.

[0146] The processing relationship of the first submanifold sparse convolutional layer is as follows: .

[0147] In the formula, Indicates the first Point cloud features output by the first submanifold sparse convolutional layer in each network layer. Indicates the kernel size as The first submanifold sparse convolution operation. This represents the feature representation after restoring the correspondence between three-dimensional spatial positions. This indicates batch normalization operation. This represents a non-linear activation function. This represents the network layer index.

[0148] The processing relationship of the second submanifold sparse convolutional layer is as follows: .

[0149] In the formula, Indicates the first Point cloud features output by the second submanifold sparse convolutional layer in a network layer. Indicates the kernel size as The second submanifold sparse convolution operation. This represents the point cloud features output by the sparse convolutional layer of the first submanifold. This indicates batch normalization operation. This represents a non-linear activation function.

[0150] The local enhancement features output from the sparse convolutional layer of the second submanifold are residually fused with the bidirectional state space scan output after inversion: .

[0151] In the formula, Indicates the first The point cloud features output by the state space convolutional feature extraction module in each network layer constitute the point cloud semantic features of the current encoding stage. This represents the local enhancement features of the output of the sparse convolutional layer of the second submanifold. This represents the long-range contextual features after being reverse-ordered to their three-dimensional spatial location. This represents a non-linear activation function.

[0152] Through the above processing, the bidirectional state space module can establish long-range dependencies between point clouds along a one-dimensional sequence, enabling each point to simultaneously obtain both preceding and following context. Two layers of sub-manifold sparse convolution can recover and enhance local geometric structure information in three-dimensional space. Residual connections can preserve the long-range context features obtained by the bidirectional state space module and reduce information loss caused by consecutive local convolutions.

[0153] Therefore, the state space convolutional feature extraction module can reduce the attenuation of local boundary features, local shape features, and small target features during long-distance state propagation while maintaining the advantages of linear sequence modeling in the state space model, thus alleviating the problems of local feature weakening and feature collapse in long-distance point cloud modeling.

[0154] S4. Input the semantic features of the point cloud into the selective state sampling module, compress and serialize the encoding, divide the points with the same encoding into local regions, reset the hidden state by the region boundary and perform state aggregation, take the end state of the region as the representative feature, and record the mapping relationship between the points and the region.

[0155] like Figure 1 As shown, the selective state sampling module includes a feature mapping unit, a selective state space aggregation unit, and a sampling unit. Figure 1 In this context, σ represents the Sigmoid activation function. SSM stands for State Space Model.

[0156] The feature mapping unit adjusts the channel dimension of the input point cloud features so that the representative features output by the selective state sampling module are adapted to the number of feature channels in the next encoding stage. The selective state space aggregation unit propagates the state along the serialized point cloud features and resets the hidden state between different local downsampling regions, so that the effective point cloud features in each local downsampling region are aggregated into the hidden state of the corresponding region. The sampling unit selects the end state of each local downsampling region as the representative feature and generates the representative point space coordinates and serialized code for the next encoding stage.

[0157] No. The point cloud feature set output by each encoding stage is: In the formula, Indicates the first Point cloud features output from each encoding stage. Indicates the first In the first coding stage Feature representation of each point. Indicates the first The number of points output by each encoding stage. This indicates the index of the encoding stage. Indicates a point index.

[0158] make Indicates the first In the first coding stage The spatial coordinates of the points, and .make Indicates the first In the first coding stage Non-negative integer serialization encoding of points, where .in, Indicates the first The original three-dimensional spatial coordinates of each point. Indicates the first The point at the th The Hilbert sequence encoding corresponding to a cyclic permutation order.

[0159] for , It is obtained by aggregating the spatial coordinates of each point within the corresponding local downsampling region in the previous encoding stage. It is inherited from the scale-compressed spatial sequence encoding of the local downsampling region corresponding to the previous encoding stage.

[0160] First, determine the encoding displacement depth based on the preset downsampling step size: In the formula, It represents the non-negative integer single-dimensional spatial dimension encoded displacement depth determined by a preset downsampling step size. This represents the preset downsampling step size, which is not less than 1 and is a non-negative integer power of 2. This indicates the rounding up operation. This function returns the number of bits required for the binary representation of a non-negative integer, and returns 0 when the input is 0.

[0161] Subsequently, a binary right shift is performed on the serialization encoding in the current encoding stage to compress the spatial scale corresponding to the serialization encoding: In the formula, Indicates the first In the first coding stage Each point completes the spatial sequence encoding after scale compression. Indicates the first In the first coding stage Serialization encoding of points. This indicates a binary right shift operation. This represents the depth of the encoded displacement in a single spatial dimension. Since the point cloud is in three-dimensional space, the encoded displacement length is set to... .

[0162] Points with the same spatial sequence encoding after scale compression are grouped into the same local downsampling region, forming multiple non-empty and non-overlapping local downsampling regions: In the formula, Indicates the first A set of local downsampling regions for each coding stage. to They represent the first Each non-empty local downsampling region in each coding stage. Indicates the first The number of local regions retained after downsampling in each coding stage. This indicates the index of the encoding stage.

[0163] Different local downsampling regions satisfy the following: In the formula, Indicates the first The first coding stage A non-empty local downsampling region. Indicates the first The first coding stage A non-empty local downsampling region. and Indicates the index of different local downsampling regions. This represents the empty set.

[0164] All local downsampling regions together cover the set of points in the current encoding stage: In the formula, Indicates the first The first coding stage A non-empty local downsampling region. This represents the enumeration index of the local downsampling region. This indicates the number of local downsampling regions. Indicates by the first Spatial coordinates of each point in each coding stage The set of points that constitutes this.

[0165] After completing the local region division, the first... Point cloud features from each encoding stage are sorted according to serialized encoding, and selective state aggregation is performed by resetting the boundaries of each local downsampled region as the hidden state: .

[0166] In the formula, This represents the local aggregate sequence features after selective state modeling. Indicates the first Point cloud features output from each encoding stage. Indicates the first The set of serialized codes for all points in each encoding stage. This represents a sorting operation that arranges point cloud features in ascending order based on serialization encoding. This indicates a point-by-point layer normalization operation. This refers to the state space feature aggregation process set in the selective state sampling module, using the boundaries of each local downsampling region as the hidden state reset boundary. Indicates the first The number of points output by each encoding stage.

[0167] In execution When a sequence position moves from one local downsampling region to the next, the hidden state is reset, making the state aggregation process of different local downsampling regions independent of each other. Within the same local downsampling region, the hidden state is continuously updated according to the sequence order, so that the effective features at the beginning of the region gradually converge to the end of the region.

[0168] The state output corresponding to the end position of each local downsampled region in the current sequence order is determined as the representative feature of that region: In the formula, Indicates the first In the first coding stage The local downsampling region is generated after selective state sampling and used for the first local downsampling region. Representative features of the input at each encoding stage. This represents the local aggregate sequence features after selective state modeling. Indicates the first In the first coding stage The end index position of a local downsampled region in the current sequence order.

[0169] To generate the representative point spatial locations required for the next encoding stage, the spatial coordinates of each point within the local downsampling region are averaged: In the formula, Indicates by the first In the first coding stage Spatial coordinates of each point within a local downsampling region The first obtained by aggregation Each encoding stage represents the spatial coordinates of a point. Indicates the first In the first coding stage The number of points contained in a non-empty local downsampling region. Indicates the first In the first coding stage The spatial coordinates of the points.

[0170] Entering the After each encoding stage, the region index will be... As a point index for this encoding stage use.

[0171] Since the scale-compressed spatial sequence codes are the same for all points within the same local downsampling region, the scale-compressed spatial sequence code shared by all points within each local downsampling region is determined as the code for that local downsampling region: In the formula, Indicates the first In the first coding stage Spatial sequence encoding after scale compression of local downsampled regions. Indicates the first [number]th [unit] in the local downsampling region. Spatial sequence encoding after scale compression of points. Indicates the first In the first coding stage The spatial coordinates of the points. Indicates the first A local downsampling region.

[0172] Entering the After each encoding stage, let This allows the current point in the next encoding stage to inherit the scale-compressed spatial sequence encoding of its corresponding local downsampled region.

[0173] The encoding of each local downsampling region is reordered to generate the serialization sort index required for the next encoding stage: In the formula, This indicates the serialization sorting index required for the next encoding stage. Indicates the first The scale-compressed spatial sequence encoding set of all local downsampled regions in each encoding stage. This indicates an operation that returns an index sorted in ascending order by region encoding. This indicates the number of local downsampling regions.

[0174] Generate the corresponding inverse mapping index based on the serialized sorted index: .

[0175] In the formula, This indicates the inverse mapping index required for the next encoding stage. This indicates the serialization sorting index required for the next encoding stage. This indicates the operation of generating an inverse mapping index based on the sorted index.

[0176] Depend on Constituting the first Point cloud features after downsampling at each encoding stage and will As the first The current point features of each encoding stage. Number of points in each coding stage equal to the Number of local downsampling regions in each coding stage .

[0177] The mapping relationship between points and regions is recorded during the downsampling process. This mapping relationship includes at least: The local downsampling region to which each point belongs in each encoding stage, the representative features corresponding to each local downsampling region, and the serialization sorting index for the next encoding stage. and inverse mapping index The mapping relationship is used for serialization feature extraction in the next encoding stage, and for feature position recovery and skip connection fusion in the decoding stage.

[0178] Traditional max pooling only retains the maximum value of each feature channel in a local region, average pooling calculates a fixed mean for local region features, and random sampling may directly discard points containing boundary or category discrimination information. Unlike these methods, this embodiment utilizes a selective state-space model to perform state propagation within each local downsampling region, adaptively aggregating the effective point cloud features within the region to the region's final state, and then using the final state as the representative feature. Therefore, the selective state sampling module can retain local boundary information, local geometric structure information, and category discrimination information while reducing the number of points and computational load.

[0179] S5 uses the representative feature as the current point feature for the next encoding stage, and repeats S2 to S4 based on the Hilbert encoding of the next encoding stage until encoding is completed. In the decoding stage, upsampling is performed level by level according to the mapping relationship. The upsampled features are fused with the encoded features of the same layer and enhanced by the state space convolution feature extraction module. The restored features are mapped to the semantic category space, and the semantic labels of each point are output.

[0180] The encoder comprises multiple encoding stages set according to spatial resolution levels. The first encoding stage performs Hilbert encoding and sorting on the initial point features obtained in step S1 according to the current cyclic arrangement, and inputs the resulting current point feature sequence into the state space convolutional feature extraction module.

[0181] like Figure 2As shown, in subsequent encoding stages, the point cloud semantic features output from the previous encoding stage are input into the selective state sampling module to obtain representative features of each local downsampled region, representative point spatial coordinates, compressed serialized encoding, and the mapping relationship between points and regions. These representative features are then used as the current point features in the next encoding stage. Steps S2 to S4 are repeated according to the Hilbert sequence encoding corresponding to the cyclic arrangement used in the next encoding stage, until all encoding stages are completed.

[0182] The feature extraction relationships between the first encoding stage and subsequent encoding stages are as follows: .

[0183] In the formula, Indicates the first The point cloud features output by each encoding stage. hour, This represents the semantic features of the point cloud output from the first encoding stage. Indicates according to the first The current point feature sequence after being arranged in a cyclical order. This represents the state-space convolutional feature extraction process in the first encoding stage. This indicates the index of the encoding stage. This indicates the cyclic permutation order index used in the first encoding stage.

[0184] .

[0185] In the formula, Indicates the first Point cloud features output from each encoding stage. Indicates the first Point cloud features output from each encoding stage. Indicates the first The state-space convolutional feature extraction process in each encoding stage. Indicates The associated point coordinates and serialization encoding are used as input and set in the first... After the coding stage and located at the , Selective state sampling process prior to each coding stage. This indicates the index of the encoding stage.

[0186] As the encoding stage progresses, the number of points gradually decreases while the dimension of point feature channels gradually increases, enabling the network to obtain deeper semantic representations at lower spatial resolutions. Different encoding stages can employ different permutations from three cyclic permutation sequences, allowing the model to learn complementary spatial adjacency relationships during multi-scale feature extraction.

[0187] During selective state sampling in the encoding phase, the mapping relationship between each point and its corresponding local downsampled region is preserved. Based on this mapping relationship, the decoder maps the low-resolution point features to the point set corresponding to the next higher spatial scale step by step, and performs skip connections and fusion with the point cloud semantic features of the same level in the encoder.

[0188] Let the deepest stage index of the encoder be... The point cloud features output from the deepest stage of the encoder are used as the starting low-resolution point features of the decoder: In the formula, This represents the initial low-resolution point features of the decoder. This represents the point cloud features output from the deepest stage of the encoder. This indicates the index of the deepest stage of the encoder.

[0189] For common level indexes Based on the mapping relationship recorded during the encoding stage, the hierarchical index is... The corresponding low-resolution decoded features are restored to the hierarchical index. Corresponding spatial scale: .

[0190] In the formula, Indicates restoration to the common level index. Point features corresponding to spatial scales. Represents hierarchical index The corresponding low-resolution decoding features. This indicates a feature upsampling operation performed based on the mapping relationship recorded during the encoding phase. This indicates the common level index of the current decoding scale and its corresponding encoding stage.

[0191] The upsampled features are fused with the point cloud semantic features output from the corresponding encoding stage: .

[0192] In the formula, Indicates a common hierarchical index The corresponding fusion decoding features. Indicates restoration to the common level index. Point features corresponding to spatial scales. Indicates a common hierarchical index The point cloud features output during the corresponding encoding stage. This indicates a feature fusion operation.

[0193] The feature fusion operation can employ feature concatenation, feature addition, linear projection after concatenation, gated weighting, or a combination of the above. By introducing shallow features from the encoder through skip connections, shallow local geometric details can be fused with deep semantic information.

[0194] The fused decoded features are then input into the state-space convolutional feature extraction module on the decoding side for enhancement. .

[0195] In the formula, Indicates a common hierarchical index The recovery features are output from the corresponding decoding stage. Indicates a common hierarchical index The corresponding fusion decoding features. This indicates that it is set in a common level index. This corresponds to the state-space convolutional feature enhancement process in the decoding stage.

[0196] The decoding-side state-space convolutional feature extraction module can employ the same or corresponding bidirectional state-space modeling, inverse sorting, two-layer submanifold sparse convolution, and residual fusion structure as the encoding-side state-space convolutional feature extraction module. Through progressive upsampling, skip connection fusion, and decoding feature enhancement, deep semantic information is progressively propagated to the original point cloud spatial resolution, and object boundaries, small targets, and local geometric structures preserved in the shallow layer of the encoder are reintegrated into the final point feature representation.

[0197] After decoding, the recovered point cloud features are input into the point-by-point segmentation module. This module can include one of a multilayer perceptron, a linear layer, or a convolutional layer, and can further include a normalization layer and a non-linear activation function layer. The point-by-point segmentation module maps the number of feature channels for each point to the number of semantic categories.

[0198] For the recovered features, the first The features of each point are used to obtain the prediction score for each semantic category corresponding to that point through a linear mapping layer: In the formula, Indicates the first Each point corresponds to A predicted score vector for each semantic category. This represents the learnable weight matrix in the pointwise segmentation module. The feature after restoration is the first The features of each point, in this embodiment, can be the output of the final decoding stage. The Middle Features of each point. This represents the bias vector in the point-by-point segmentation module. Indicates a point index. This represents the number of semantic categories that are greater than 0.

[0199] The predicted score vector satisfies: In the formula, Indicates the first The predicted score vector corresponding to each point. Indicates the number of semantic categories. Represents the real number field.

[0200] The predicted scores are converted into category predicted probabilities using the Softmax function. In the formula, Indicates the first The point was predicted to be the [number]th [point]. The probability of a semantic category. Indicates the first The point is the first Predicted score for semantic category. Indicates the first The point is the first Predicted score for semantic category. This represents a semantic category summation index. This represents a semantic category index. Indicates the number of semantic categories. This represents an exponential function.

[0201] The predicted probability of the category satisfies: .

[0202] In the formula, Indicates the first The point was predicted to be the [number]th [point]. The probability of a semantic category.

[0203] .

[0204] In the formula, Indicates the first The point was predicted to be the [number]th [point]. The probability of a semantic category. This represents a semantic category index. Indicates the number of semantic categories.

[0205] The semantic category with the highest predicted category probability is determined as the [number]. Semantic tags for each point: .

[0206] In the formula, Indicates the first The final semantic prediction category for each point. Indicates the first The point was predicted to be the [number]th [point]. The probability of a semantic category. This represents a semantic category index. Indicates the number of semantic categories. This indicates the semantic category index that maximizes the predicted probability of the category.

[0207] Thus, the semantic label corresponding to each point in the input point cloud scene is obtained, and the semantic segmentation result of the point cloud scene is formed.

[0208] During the training phase, the cross-entropy loss function and the Lovasz loss function are used to jointly supervise the learning of the point cloud semantic segmentation model.

[0209] The cross-entropy loss function measures the difference between the predicted class probability distribution output by the model and the true class label distribution. Its calculation formula is as follows: .

[0210] In the formula, This represents the cross-entropy loss. This indicates the number of points greater than 0 participating in the current loss calculation. The points participating in the loss calculation are ranked according to... Renumber. This represents the number of semantic categories that are greater than 0. Indicates by the first The true semantic category label of each point The true category indicator value obtained by one-hot encoding, when the first The point belongs to the th point When class, take 1 and not belong to the first class. The value is 0 when the class is used. Indicates the first The point was predicted to be the [number]th [point]. The probability of a semantic category. This represents the natural logarithm function.

[0211] The Lovasz loss function is used to directly optimize the model's performance on the intersection-union ratio (IU) metric, mitigating the impact of sample imbalance between different semantic categories. Its calculation formula is as follows: .

[0212] In the formula, This indicates the loss in Lovasz. Indicates the number of semantic categories. Indicates the first The Jaccard collection functions corresponding to the class. Represents Jaccard set functions The Lovasz extension functions. Indicates according to the first The prediction error vector is constructed from the predicted probability of the class and the true class indication value. This represents a semantic category index.

[0213] The overall supervised loss of the point cloud semantic segmentation model is: .

[0214] In the formula, This represents the overall supervision loss of the point cloud semantic segmentation model. This represents the cross-entropy loss. This indicates the loss in Lovasz.

[0215] Using the training set The point cloud semantic segmentation model is trained in multiple rounds of supervised training, and the validation set is used after each training round. Evaluate the semantic segmentation performance of the current model. When the evaluation metric on the validation set achieves the best result, save the current network parameters as the target model parameters. After training, use the test set... Input the target model to obtain the semantic category prediction results for each point in the test scene point cloud.

[0216] During the inference phase, the point cloud to be segmented, which does not contain real semantic category labels, is input into the trained PointSSM model. The model then sequentially performs preprocessing and voxel meshing, original attribute feature embedding, Hilbert encoding in three cyclic arrangements, bidirectional state space modeling, two-layer submanifold sparse convolution enhancement, selective state sampling, multi-level encoding, stepwise upsampling, skip connection fusion, and point-by-point category prediction, outputting the semantic labels corresponding to each point in the point cloud to be segmented.

[0217] To verify the effectiveness of the method in this embodiment, the PointSSM model was implemented based on the PyTorch 1.12 deep learning framework under the Ubuntu 22.04.4 LTS operating system. The development environment was PyCharm 2021.3.3, and the model training and testing were completed on an NVIDIA RTX 4090 graphics processor.

[0218] The experiments utilize two publicly available point cloud datasets, DALES and S3DIS, to verify the semantic segmentation performance and multi-scene generalization ability of the proposed method in indoor scenes, vehicle-mounted LiDAR scenes, and urban-level airborne LiDAR scenes. The DALES dataset contains urban-level macroscopic scene point clouds obtained from high-altitude airborne LiDAR scans, covering eight geographic semantic categories, including ground and buildings. The S3DIS dataset consists of 271 rooms from six regions in three different buildings, encompassing 13 semantic categories.

[0219] like Figure 3As shown, the method in this embodiment can adapt to the characteristics of vehicle-mounted lidar point clouds, such as a large number of points, large density variations, numerous dynamic targets, and a large spatial range. It can adapt to the macroscopic spatial range, non-uniform point density, and complex geographical target distribution of urban-level airborne lidar point clouds.

[0220] like Figure 4 As shown, Hilbert serialization with three cyclic permutations, bidirectional state space modeling, local submanifold sparse convolution enhancement, and selective state sampling can collectively improve the model's semantic representation ability for complex indoor scenes.

[0221] The above experiments show that this embodiment uses Hilbert space-filling curves with three cyclic permutations to map disordered 3D point clouds into one-dimensional sequences with different spatial traversal relationships. While maintaining the spatial proximity relationship of point clouds as much as possible, it enables the selective state-space model to be directly used for modeling long point cloud sequences, reducing complex neighborhood search and high-overhead self-attention computation.

[0222] The state-space convolutional feature extraction module obtains both the preceding and following contexts through bidirectional state-space modeling. After state-space modeling, it enhances the local spatial structure using two layers of sub-manifold sparse convolution, enabling the model to take into account both long-distance semantic relationships and local geometric details, thus alleviating the problems of unidirectional state propagation, long-range information decay, and local feature collapse.

[0223] The selective state sampling module divides local regions through compressed serialization encoding and resets the hidden state at the region boundaries, adaptively aggregating the effective point cloud features within the region to the final state of the region. Compared with max pooling, average pooling, or random sampling, this downsampling method can preserve local geometric structure, boundary information, and class discrimination information while reducing the number of points and the computational cost of the model.

[0224] Therefore, the method described in this embodiment can be used for point cloud point-to-point semantic segmentation tasks in scenarios such as autonomous driving, robot navigation, smart cities, remote sensing and mapping, indoor 3D scene understanding, building modeling, road facility recognition, city-level 3D reconstruction, and 3D digital twins.

[0225] It is understood that the point cloud semantic segmentation device can be an electronic device with computing power, such as a portable laptop computer, desktop computer, server, smartphone or tablet computer.

[0226] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A point cloud semantic segmentation method based on a state-space model, characterized in that, include: S1. Acquire point cloud data, perform preprocessing and voxel meshing, embed the original attribute features to obtain initial point features; S2, Hilbert encoding is performed on the three coordinate components of the voxel grid coordinates in three cyclic arrangement orders. In the current encoding stage, the current point features are sorted according to the corresponding encoding to obtain the current point feature sequence. S3, input the current point feature sequence into the state space convolutional feature extraction module, perform forward scanning and reverse scanning through the bidirectional state space module respectively and fuse the scanning output; reverse sort the fused scanning output to the three-dimensional space position, enhance it through two layers of sub-manifold sparse convolution, and fuse it with the reverse sorted scanning output residual to obtain the point cloud semantic features of the current encoding stage; S4, input the semantic features of the point cloud into the selective state sampling module, compress and serialize the encoding, divide the points with the same encoding into local regions, reset the hidden state by the region boundary and perform state aggregation, take the end state of the region as the representative feature, and record the mapping relationship between the points and the region. Among them, the first Point cloud features from each encoding stage are sorted according to serialized encoding, and selective state aggregation is performed by resetting the boundaries of each local downsampled region as the hidden state: ； In the formula, This represents the local aggregate sequence features after selective state modeling; Indicates the first The set of serialized codes for all points in each coding stage; This indicates a sorting operation that arranges point cloud features in ascending order based on serialization encoding. This indicates a point-by-point layer normalization operation; This refers to the state space feature aggregation process set in the selective state sampling module, using the boundaries of each local downsampling region as the hidden state reset boundary; Indicates the first Point cloud features output from each encoding stage; The state output corresponding to the end position of each local downsampled region in the current sequence order is determined as the representative feature: ； In the formula, Indicates the first In the first coding stage The local downsampling region is generated after selective state sampling and used for the first local downsampling region. Representative features of the input at each encoding stage; Indicates the first In the first coding stage The end index position of each local downsampled region in the current sequence order; The spatial coordinates of the representative points in the next encoding stage are obtained by averaging the spatial coordinates of each point within the local downsampling area. ； In the formula, Indicates by the first In the first coding stage Spatial coordinates of each point within a local downsampling region The first obtained by aggregation Each encoding stage represents the spatial coordinates of a point; Indicates the first In the first coding stage The number of points contained in a non-empty local downsampling region; Indicates the first In the first coding stage The spatial coordinates of the points; S5, take the representative feature as the current point feature of the next encoding stage, repeat S2 to S4 according to the Hilbert encoding of the next encoding stage until the encoding is completed; in the decoding stage, upsample step by step according to the mapping relationship, fuse the upsampled features with the same layer encoding features and enhance them through the state space convolution feature extraction module, map the recovered features to the semantic category space, and output the semantic label of each point.

2. The point cloud semantic segmentation method based on a state-space model according to claim 1, characterized in that, S2 specifically includes: No. The voxel grid coordinates of each point are According to the first The voxel grid coordinates are Hilbert encoded using a cyclic arrangement order. ； In the formula, Indicates the first The point at the th The Hilbert sequence encoding corresponding to the cyclic permutation order; Indicates the first The voxel grid coordinates corresponding to each point; Indicates a point index; Indicates the sequential index of a cyclic arrangement; This represents the encoding function for the three-dimensional Hilbert space filling curve. Indicates the first A cyclical arrangement order transformation operation; ; The three cyclical arrangements are as follows: ； ； ； In the formula, Indicates to maintain axis, shaft and The first type of cyclic arrangement order transformation operation for axis arrangement; Indicates according to axis, shaft and The second type of cyclic arrangement order transformation operation is performed on the axis arrangement order; Indicates according to axis, shaft and The third type of cyclic arrangement order transformation operation performed on the axis arrangement order; Indicates the first The point is at Voxel grid coordinates along the axis; Indicates the first The point is at Voxel grid coordinates along the axis; Indicates the first The point is at Voxel grid coordinates along the axis; According to the The Hilbert sequence encoding corresponding to the cyclic permutation order sorts the features of the current point in ascending order: ； In the formula, Indicates according to the first The current point feature sequence after cyclic permutation; Indicates the first The current point features of each point; This indicates the number of points in the current encoding stage; This represents a sorting operation that arranges point cloud features in ascending order based on serialization encoding.

3. The point cloud semantic segmentation method based on a state-space model according to claim 1, characterized in that, S3 specifically includes: The bidirectional state space module performs discrete state updates along the forward and reverse order of the current point feature sequence, respectively, and fuses the forward scan output and the reverse scan output. Each direction is updated in the 6th... The state update relationship for each sequence position is as follows: ； ； In the formula, Indicates the first The hidden state corresponding to each discrete sequence position; Indicates the first The hidden state corresponding to each discrete sequence position; Represents the position index of a discrete sequence; This represents the discretized state transition matrix; This represents the discretized input mapping matrix; Indicates the first Input features corresponding to the positions of discrete sequences; Indicates the first Output features corresponding to the positions of each discrete sequence; This represents the state output mapping parameter matrix; This represents the input pass-through parameter matrix; No. Each network layer performs pointwise layer normalization and bidirectional selective state-space modeling on the input sequence features, and then fuses the modeling results with the input sequence features using residuals. ； In the formula, This represents the sequence characteristics after bidirectional selective state-space modeling; Indicates the first Sequence features of the input to each network layer; Indicates the network layer index; Indicates to The sequence features are formed by performing layer-by-layer normalization on the features of each point in the sequence. This indicates the bidirectional selective state-space modeling operation performed by the bidirectional state-space module; Perform a reverse sort based on the sort index corresponding to the current cyclic arrangement used by the network layer: ； In the formula, This represents the feature representation after restoring the correspondence between three-dimensional spatial positions; Indicates by the first The first point in each of the network layers A sorted index generated by Hilbert sequence encoding of cyclic permutations; Indicates sorting based on index The reverse sorting operation performed; .

4. The point cloud semantic segmentation method based on a state-space model according to claim 3, characterized in that, S3 also includes: Both sub-manifold sparse convolutions have kernel sizes of 1. A 3D submanifold sparse convolutional layer with a stride of 1 and a padding size of 1; two submanifold sparse convolutional layers are connected to a batch normalization layer and a nonlinear activation function layer, respectively; the processing relationship between the first submanifold sparse convolutional layer, the second submanifold sparse convolutional layer, and residual fusion is as follows: ； ； ； In the formula, Indicates the first Point cloud features output by the first submanifold sparse convolutional layer in each network layer; Indicates the first Point cloud features output by the sparse convolutional layer of the second submanifold in each network layer; Indicates the first Point cloud features output by the state space convolutional feature extraction module in each network layer; This represents the sparse convolution operation on the first submanifold; This represents the sparse convolution operation on the second submanifold; This indicates a batch normalization operation; This represents a non-linear activation function.

5. The point cloud semantic segmentation method based on a state-space model according to claim 1, characterized in that, S4 specifically includes: No. The point cloud feature set output by each encoding stage is: ； In the formula, Indicates the first Point cloud features output from each encoding stage; Indicates the first In the first coding stage Feature representation of each point; Indicates the first The number of output points in each encoding stage; Indicates the encoding stage index; Indicates a point index; make ;make Indicates the first In the first coding stage Non-negative integer serialization encoding of points, where ;in, Indicates the first The three-dimensional spatial coordinates of each point; Indicates the first The point at the th The Hilbert sequence encoding corresponding to the cyclic permutation order; The encoding displacement depth is determined based on the preset downsampling step size: ； In the formula, This represents the non-negative integer single-dimensional spatial dimension encoded displacement depth determined by a preset downsampling step size; This represents the preset downsampling step size, which is not less than 1 and is a non-negative integer power of 2. This indicates the rounding up operation; This function returns the number of bits required for the binary representation of a non-negative integer, and returns 0 when the input is 0; Perform a binary right shift on the serialization encoding of the current encoding stage: ； In the formula, Indicates the first In the first coding stage Each point completes the spatial sequence encoding after scale compression; This represents a binary right shift operation; Points with the same spatial sequence encoding after scale compression are grouped into the same local downsampling region, forming multiple non-empty and non-overlapping local downsampling regions: ； ； ； In the formula, Indicates the first A set of local downsampling regions for each coding stage; Indicates the first The first coding stage A non-empty local downsampling region; Indicates the first The number of local regions retained after downsampling in each coding stage; Indicates the first The first coding stage A non-empty local downsampling region; Indicates the index of the first local downsampling region; Indicates the first The first coding stage A non-empty local downsampling region; Indicates the index of the second local downsampling region; Indicates the first The first coding stage A non-empty local downsampling region; The enumeration index represents the local downsampling region; Indicates by the first Spatial coordinates of each point in each coding stage The set of points that constitutes this.

6. The point cloud semantic segmentation method based on a state-space model according to claim 5, characterized in that, S4 also includes: The spatial sequence encoding with scale compression shared by all points within each local downsampled region is determined as the encoding of that local downsampled region: ； In the formula, Indicates the first In the first coding stage Spatial sequence encoding of scale-compressed local downsampled regions; proceeding to the... After each encoding stage, the region index will be... As a point index for this encoding stage and order ; The encodings of each local downsampling region are reordered, and the corresponding inverse mapping index is generated: ； ； In the formula, This indicates the serialization sorting index required for the next encoding stage; This indicates an operation that returns an index sorted in ascending order by region code; Indicates the inverse mapping index required for the next encoding stage; This indicates the operation of generating an inverse mapping index based on the sorted index; Depend on Constituting the first Point cloud features after downsampling at each encoding stage and take it as the first The current point features of the encoding stage, where the first... Number of points in each coding stage equal The mapping relationship includes the local downsampling region to which each point belongs, the serialization sorting index, and the inverse mapping index.

7. The point cloud semantic segmentation method based on a state-space model according to claim 1, characterized in that, S5 specifically includes: In the first encoding stage, the current point feature sequence, arranged according to the current cyclic arrangement, is input into the state space convolutional feature extraction module. In subsequent encoding stages, the point cloud semantic features output from the previous encoding stage are sequentially input into the selective state sampling module and the state space convolutional feature extraction module of the current encoding stage. ； ； In the formula, Indicates the first Point cloud features output from each encoding stage; Indicates according to the first The current point feature sequence after cyclic permutation; This represents the state-space convolutional feature extraction process in the first encoding stage; Indicates the encoding stage index; Indicates the cyclic permutation order index used in the first encoding stage; Indicates the first Point cloud features output from each encoding stage; Indicates the first The state-space convolutional feature extraction process in each encoding stage; Indicates The associated point coordinates and serialization encoding are used as input and set in the first... After the coding stage and located at the , Selective state sampling process prior to each coding stage; Let the deepest stage index of the encoder be... In the decoding stage, starting from the deepest stage of the encoder, the low-resolution decoding features are mapped to the next higher spatial scale according to the mapping relationship. The upsampled features are fused with the point cloud semantic features output from the corresponding encoding stage, and then enhanced by the state space convolutional feature extraction module on the decoding side. ； ； ； ； In the formula, This represents the initial low-resolution point features of the decoder; Indicates restoration to the common level index. Point features corresponding to spatial scales; Represents hierarchical index Corresponding low-resolution decoding features; This indicates a feature upsampling operation performed based on the mapping relationship recorded during the encoding phase; This indicates the common level index of the current decoding scale and its corresponding encoding stage; Indicates a common hierarchical index Corresponding fusion decoding features; Indicates a common hierarchical index The corresponding point cloud features output during the encoding stage; This indicates a feature concatenation operation, a feature addition operation, or a fusion operation after linear projection; Indicates a common hierarchical index The recovered features corresponding to the output of the decoding stage; This indicates that it is set in a common level index. This corresponds to the state-space convolutional feature enhancement process in the decoding stage; .

8. The point cloud semantic segmentation method based on a state-space model according to claim 7, characterized in that, S5 also includes: The recovered features are input into the point-by-point segmentation module. The linear mapping layer obtains the prediction score for each semantic category at each point, and the softmax function is used to obtain the category prediction probability. The semantic category with the highest prediction probability is determined as the semantic label of the corresponding point. ； ； ； In the formula, Indicates the first Each point corresponds to The predicted score vectors for each semantic category; This represents the learnable weight matrix in the pointwise segmentation module; The feature after restoration is the first Features of each point; This represents the bias vector in the pointwise segmentation module; Indicates a point index; Indicates the first The point was predicted to be the [number]th [point]. The probability of the semantic category, and , ; Indicates the first The point is the first Predicted scores for semantic categories; Indicates the first The point is the first Predicted scores for semantic categories; Represents an exponential function; Indicates the first The final semantic prediction category for each point; Represents a semantic category index; Indicates the number of semantic categories greater than 0; ; Represents the real number field.

9. The point cloud semantic segmentation method based on a state-space model according to claim 1, characterized in that, Each point in the point cloud data includes three-dimensional spatial coordinates and at least one of the original attribute features from color information, reflection intensity information, and normal vector information; Preprocessing includes coordinate normalization, original attribute feature normalization, mesh sampling, random rotation, random scaling, random flipping, random jittering, and random cropping; Voxel meshing includes: discretizing the continuous spatial coordinates of each point according to a preset voxel size to obtain voxel mesh coordinates; ； In the formula, Indicates the first The voxel grid coordinates corresponding to each point; ; Indicates the first The point is at Voxel grid coordinates along the axis; Indicates the first The point is at Voxel grid coordinates along the axis; Indicates the first The point is at Voxel grid coordinates along the axis; Indicates the first The three-dimensional spatial coordinates of each point; ; Indicates the first The first-dimensional spatial coordinates of each point; Indicates the first The second-dimensional spatial coordinates of each point; Indicates the first The third-dimensional spatial coordinates of each point; Represents the real number field; This represents a three-dimensional vector composed of the minimum coordinate components of the current point cloud scene on the three coordinate axes; Indicates the preset voxel size that is greater than 0; Indicates a point index; This indicates a floor operation performed on the coordinate components; Embedding the original attribute features includes: using linear mapping layers or sparse convolutional mapping layers to embed the first attribute feature. Embed the original attribute features of each point: ； In the formula, Indicates the first The current point features of each point; Indicates the first The original attribute features of each point; This represents the learnable weight matrix in the input feature embedding module; This represents the bias vector in the input feature embedding module; During training, the labeled point cloud data is divided into training set, validation set and test set. Supervised training is performed using the training set. After each training round, the performance is verified using the validation set and the network parameters with the best verification results are saved. After training is completed, the semantic labels of each point are output using the test set. During training, the sum of cross-entropy loss and Lovasz loss is used as the overall supervision loss: ； ； ； In the formula, Represents cross-entropy loss; This indicates the number of points that are greater than 0 and are involved in the current loss calculation; Indicates by the first The true semantic category label of each point The true category indicator value obtained by one-hot encoding, when the first The point belongs to the th point When the class takes 1, when the first The point does not belong to the first... When the class is used, it takes the value 0; Indicates the first The point was predicted to be the [number]th [point]. The probability of a semantic category; Indicates the number of semantic categories greater than 0; Represents the natural logarithm function; Indicates Lovasz loss; Indicates the first The Jaccard collection functions corresponding to the class; Represents Jaccard set functions Lovasz extension functions; Indicates according to the first The prediction error vector is constructed from the predicted probability of the class and the true class indication value. This represents the overall supervision loss of the point cloud semantic segmentation model.

Images