State space model point cloud segmentation method based on local semantic enhancement strategy

Abstract

The invention discloses a state space model point cloud segmentation method based on a local semantic enhancement strategy, and the method comprises the steps: 1, calculating the center of mass of a point cloud, translating all point coordinates to the center of mass as an original point, and zooming the whole point cloud into a unit sphere, thereby obtaining a standardized point cloud; step 2, processing the standardized point cloud to obtain an ordered local point cloud group feature tokens with spatial continuity; step 3, based on the standardized point cloud, pre-training the ordered local point cloud group feature token sequence obtained in the step 2 by using a mask auto-encoder frame to obtain a local semantic perception mask enhanced ordered local point cloud group feature token sequence; 4, extracting multi-scale features from the feature tokens sequence of the ordered local point cloud group enhanced by the local semantic perception mask, and further enhancing the spatial perception capability and feature expression of the point cloud to obtain the multi-scale features; and step 5, performing point cloud segmentation on the multi-scale features obtained in the step 4, and accurately distinguishing the road surface and the obstacle area.

Description

Technical Field

[0001] The present invention relates to the field of point cloud segmentation, and in particular to a state space model point cloud segmentation method based on a local semantic enhancement strategy. Background Art

[0002] In the field of point cloud data processing, existing technologies have problems with insufficient spatial continuity and limited semantic understanding capabilities when processing sparse point clouds. Traditional methods (such as PointNet, PointNet++, etc.) use random sorting or farthest point sampling to process point clouds, which will destroy the spatial local continuity of the point cloud and make it difficult to capture the geometric features of small targets. This problem is particularly prominent in autonomous driving scenarios. Small obstacles (such as roadblocks such as flower pots) are usually represented by only a small number of points in the point cloud and are easily affected by background interference. Traditional methods find it difficult to achieve accurate detection and segmentation. In addition, existing mask strategies lack targeted guidance for small targets, making it difficult to effectively distinguish targets from noise. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a state space model point cloud segmentation method based on local semantic enhancement strategy in response to the shortcomings of the existing technology.

[0004] In order to solve the above technical problems, the present invention discloses a state space model point cloud segmentation method based on a local semantic enhancement strategy, comprising the following steps:

[0005] Step 1: Calculate the point cloud centroid, translate all point coordinates to the centroid as the origin, determine the distance from the farthest point to the origin, and scale the entire point cloud to the unit sphere based on this distance to obtain a standardized point cloud. This process ensures that obstacle point clouds of different sizes have a consistent scale and reference frame in subsequent processing.

[0006] Step 2: Use Zigzag spatial continuity sorting to process the standardized point cloud, so that the point cloud data is sorted according to spatial continuity, and ordered local point cloud group feature tokens with spatial continuity are obtained;

[0007] Step 3: Based on the standardized point cloud, the ordered local point cloud group feature tokens sequence is pre-trained using a masked autoencoder framework to obtain the ordered local point cloud group feature tokens sequence enhanced by local semantics-aware masking. The redundant local point cloud group feature tokens are masked using a local semantics-aware masking strategy, and the masked point cloud is restored by minimizing the Chamfer distance loss, thereby learning its global and local features.

[0008] Semantic-aware masking strategy: Semantic-aware masking analyzes the similarity between feature groups in a point cloud to identify and mask semantically redundant groups while retaining key, information-rich groups. This approach first evaluates the feature relevance of all groups, then dynamically determines the masking ratio based on the task requirements (such as classification or segmentation), prioritizing the removal of duplicate or low-value semantic information. Finally, a random masking strategy is used to balance the model's learning capabilities.

[0009] Step 4: The ordered local point cloud group feature tokens enhanced by the local semantic perception mask are sequentially extracted through the PointMamba architecture to further enhance the spatial perception ability and feature expression of the point cloud to obtain the multi-scale feature F output ;

[0010] Step 5: Extract the multi-scale features F output Perform point cloud segmentation to accurately distinguish the road surface from obstacle areas.

[0011] Step 1 is as follows:

[0012] Step 1-1, given a point cloud dataset P = {p i ∈R 3 |i=1,2,...,N}, where each point p i =(x i ,y i ,z i ), i represents the index of the point, which is used to uniquely identify each point in the point cloud, and its value range is 1 to N; N represents the total number of points in the point cloud, and x i ,y i ,z i Represent the x-coordinate, y-coordinate, and z-coordinate values ​​of the i-th point in three-dimensional space, and calculate the centroid of the point cloud:

[0013]

[0014] And translate all point cloud data to the center of mass as the origin, and obtain the translated point cloud P' i :

[0015] P′ i ={p i -μ P |p i ∈P}

[0016] Among them, μ P is the centroid of the point cloud;

[0017] Steps 1-2 calculate the distance of the farthest point in the translated point cloud from the origin (the new reference center of the point cloud, i.e., coordinates (0,0,0)):

[0018]

[0019] Among them, d max is the distance from the farthest point to the origin, ||p i ||2 is point p i The Euclidean norm (distance from the origin) of the point cloud is obtained by scaling the entire translated point cloud to the unit sphere. norm :

[0020] P norm ={p′ i / d max |p' i ∈P'}.

[0021] This standardization process can ensure that the point cloud data will not cause erroneous results due to scale issues in subsequent calculations.

[0022] Step 2 is as follows:

[0023] In step 2-1, the farthest point sampling is performed on the standardized point cloud to extract the representative center point, and then the K-nearest neighbor (KNN) algorithm is used to construct a local point cloud group for each center point: In order to effectively process a large amount of data, we introduced the batch size (B) in the training process. B represents the number of point cloud samples processed each time, which is usually set to 128, that is, 128 samples are input for training each time.

[0024] Given a normalized point cloud set P norm ={p1,p2,...,p N}∈R N×3 , from which G representative center points C={c1,c2,...,c G}, that is, divide the input point cloud into G point cloud groups and obtain the center point set C∈R G ×3 :

[0025] The core idea of ​​FPS is to select the point farthest from the currently selected point set each time to ensure that the entire point cloud structure is covered.

[0026] c1=random(P)

[0027]

[0028] The first formula represents randomly selecting a point from the data set P as the first center point c1; the second formula describes how to select each subsequent center point c during the FPS sampling process i , where c j represents the selected center point set, p represents any point in the point cloud dataset, Indicates selecting the point farthest from the existing center point set. For each point p, calculate its distance from all selected center points c j The distance between them, select the point with the farthest distance as the next center point c i ;

[0029] Then for each center point c i , find its nearest M neighbor points in the original point cloud to form a local point cloud group G i , the local point cloud group is a three-dimensional tensor:

[0030] G i ={p j |p j ∈KNN(c i ,M)},i=1,...,G

[0031] G i ∈R B×G×M×3

[0032] And perform normalization operation to obtain the final local point cloud group, that is, each local group is subtracted from its center point coordinates:

[0033]

[0034] Step 2-2, feature encoding is performed on each local point cloud group through the convolutional network to obtain the initial feature representation of the local point cloud group:

[0035] Each local point cloud group is input into the convolutional neural network for feature encoding. Through the convolution operation, the spatial features of each local point cloud group are extracted to obtain the initial feature vector f of each local point cloud group. i ∈R d , where R d It is a d-dimensional real vector space, where d represents the vector of feature dimension, and a shared convolutional network is used to process each group of point clouds:

[0036] f i =ConvNet(G i ),i=1,...,G

[0037] Each local point cloud group finally generates a feature vector as the initial feature representation of the point cloud group, and the final initial feature sequence is:

[0038] F={f1,f2,...,f G}

[0039] F∈R G×d

[0040] In steps 2-3, a ZigZag path is then constructed on the three main planes XY, XZ, and YZ, so that the points in each plane are sorted according to the zigzag path, alternating between ascending and descending order to ensure that spatial neighboring points maintain an adjacent relationship in the sequence, and finally generate ordered local point cloud group feature tokens with spatial continuity.

[0041] The key advantage of the Zigzag path is that it can traverse the point cloud along the spatial neighbor direction, so that adjacent points remain adjacent in the sequence. This spatial locality is crucial for subsequent PointMamba sequence modeling.

[0042] Traditional point cloud processing methods often use strategies such as random sorting or farthest point sampling, which can easily destroy the spatial continuity of point cloud data. The Zigzag path sorting mechanism proposed in this paper can effectively maintain the local continuity of point clouds in space;

[0043] Steps 2-3 are as follows:

[0044] Step 2-3-1: The standardized point cloud is layered on the XY plane (horizontal plane), XZ plane (lateral plane), and YZ plane (vertical plane) to generate n paths Path sorting scheme;

[0045] Step 2-3-2, for n paths Path selection, specifically:

[0046] Case 1: Path selection during training: To enhance the generalization ability of the model, a valid path is randomly selected during training:

[0047] Path selected =Path random_choice(indices)

[0048] Among them, Path selected is the selected path, indices is the index set of valid paths, and random_choice() is the random selection function;

[0049] Case 2: Path selection during the testing phase: To ensure the stability and repeatability of the test results, the same path is always selected during the testing process:

[0050] Path selected =Path random_choice(0)

[0051] In step 2-3-3, ordered local point cloud group feature tokens with spatial continuity are generated according to the selected path.

[0052] Step 2-3-1 is as follows:

[0053] Step 2-3-1-1, horizontal plane Zigzag path generation:

[0054] First, sort the point cloud in ascending order according to the Z coordinate:

[0055] Z sorted ={p i ∈P norm |sorted by z i}

[0056] Among them, Z sorted is the point set sorted by Z coordinate;

[0057] Divide the sorted point cloud into n xy layer:

[0058]

[0059] Among them, n paths is the total number of paths to be generated (default is 6), n xy is the number of layers in the XY plane, Z layers is the point set after stratification, Z j is the point set of the jth layer;

[0060] For each layer Z j The points in are sorted by X coordinate:

[0061] X sorted_j ={p i ∈Z j |sorted by x i}

[0062] where X sorted_j is the point set sorted by X coordinate in the jth layer;

[0063] X sorted_j Divided into m segments,

[0064] X segments_j ={X j1 ,X j2 ,...,X jm}

[0065] Among them, m is the number of segments, w is the number of point clouds expected to be included in each path, l is the number of segments limited to each layer (or each path), X segments_j is the point set of the kth segment of the jth layer, |X sorted_j | is the number of points in the jth layer;

[0066] For each segment X jk, the sorting method of the Y coordinate is determined according to the parity of k: if k is an odd number, the Y coordinate is sorted in ascending order; if k is an even number, the Y coordinate is sorted in descending order, and the final Y coordinate is sorted_jk is the point set sorted by Y coordinates in the jth layer and kth segment;

[0067] Connect the sorted results of all segments to form the Zigzag path of this layer:

[0068] Path j_xy =[Y sorted_j1 ,Y sorted_j2 ,...,Y sorted_jm ]

[0069] Among them, Path j_xy is the Zigzag path of the j-th layer in the XY plane;

[0070] Step 2-3-1-2, XZ plane Zigzag path generation

[0071] Sort the point cloud in ascending order by Y coordinate and divide it into n xz layer, where For each layer, the points are sorted by X coordinates, and after segmentation, they are sorted in ascending and descending order by Z coordinates to generate a set of Zigzag paths in the XZ plane, where n xz is the number of layers in the XZ plane.

[0072] Step 2-3-1-2, XZ plane Zigzag path generation:

[0073] First, sort the point cloud in ascending order according to the Y coordinate:

[0074] Y sorted ={p i ∈P norm |sorted by y i}

[0075] Among them, Y sorted is the point set sorted by Y coordinate;

[0076] Divide the sorted point cloud into n xz layer:

[0077]

[0078] Among them, n paths is the total number of paths to be generated (default is 6), n xz is the number of layers in the XZ plane, Y layers is the point set after stratification, Y j is the point set of the jth layer;

[0079] For each layer Y j The points in are sorted by X coordinate:

[0080] X sorted_j ={p i ∈Y j |sorted by x i}

[0081] where X sorted_j is the point set sorted by X coordinates in the jth layer.

[0082] X sorted_j Divided into m segments

[0083] X segments_j ={X j1 ,X j2 ,...,X jm}

[0084] Among them, m is the number of segments, w is the number of point clouds expected to be included in each path, l is the number of segments limited to each layer (or each path), X segments_j is the point set of the kth segment of the jth layer, |X sorted_j | is the number of points in the jth layer;

[0085] For each segment X jk , the sorting method of Z coordinates is determined by the parity of k: if k is an odd number, the Z coordinates are sorted in ascending order; if k is an even number, the Z coordinates are sorted in descending order. Finally, Z sorted_jk is the point set sorted by Z coordinates in the kth segment of the jth layer;

[0086] Connect the sorted results of all segments to form the Zigzag path of this layer:

[0087] Path j_zy =[Z sorted_j1 ,Z sorted_j2 ,...,Z sorted_jm ]

[0088] Among them, Path j_xz is the Zigzag path of the XZ plane of the jth layer;

[0089] Step 2-3-1-3, YZ plane Zigzag path generation

[0090] Sort the point cloud in ascending order by X coordinate and divide it into n xz layer, where For each layer, the points are sorted by Y coordinates, and then segmented and sorted by Z coordinates in ascending and descending order to generate a set of Zigzag paths in the YZ plane, where n yz is the number of layers in the YZ plane.

[0091] Step 2-3-1-3, YZ plane Zigzag path generation:

[0092] First, sort the point cloud in ascending order according to the X coordinate:

[0093] X sorted ={p i ∈P norm |sorted by x i}

[0094] Among them, X sorted is the point set sorted by X coordinate;

[0095] Divide the sorted point cloud into n yz layer:

[0096]

[0097] Among them, n paths is the total number of paths to be generated (default is 6), n yz is the number of layers in the YZ plane, X layers is the point set after stratification, X j is the point set of the jth layer;

[0098] For each layer X j The points in are sorted by Y coordinate:

[0099] Y sorted_j ={p i ∈X j |sorted by y i}

[0100] where Y sorted_j is the point set sorted by Y coordinate in the jth layer.

[0101] Y sorted_j Divided into m segments

[0102] Y segments_j ={Y j1 ,Y j2 ,...,Y jm}

[0103] Among them, m is the number of segments, w is the number of point clouds expected to be included in each path, l is the number of segments limited to each layer (or each path), and Y segments_j is the point set of the kth segment of the jth layer, |Y sorted_j | is the number of points in the jth layer;

[0104] For each segment Y jk, the sorting method of Z coordinates is determined by the parity of k: if k is an odd number, the Z coordinates are sorted in ascending order; if k is an even number, the Z coordinates are sorted in descending order. Finally, Z sorted_jk is the point set sorted by Z coordinates in the kth segment of the jth layer;

[0105] Connect the sorted results of all segments to form the Zigzag path of this layer:

[0106] Path j_yz =[Z sorted_j1 ,Z sorted_j2 ,...,Z sorted_jm ]

[0107] Among them, Path j_yz is the Zigzag path of the YZ plane of the jth layer;

[0108] Step 2-3-3, specifically:

[0109] A spatial continuity path selected according to the strategy of step 2-3-2 corresponds to the initial feature f of the local point cloud group obtained in step 2-2 i The ordered local point cloud group feature tokens sequence τ is formed in the order of the path:

[0110]

[0111] Among them, i1,i2,...,i G is the path index sequence, Represents a local point cloud group feature tokens, and each local point cloud group feature tokens represents a local point cloud group.

[0112] Step 3 is as follows:

[0113] Step 3-1: Generate random mask with preset mask rate:

[0114] Given a batch size B of samples, calculate the number of groups that need to be masked:

[0115]

[0116] Among them, G is the number of point cloud groups obtained in step 2-1, mask_ratio is the mask ratio, and num_mask is the number of groups that need to be masked. represents the floor function;

[0117] Then for each sample in each batch B, a mask matrix is ​​randomly generated:

[0118] M rand ∈{0,1} B×G

[0119] Among them, M rand is a random mask matrix, B is the batch size, M rand [b,g]=1 means that the g-th group of the b-th sample is masked.

[0120] Step 3-2, calculate the normalized representation and cosine similarity of the local point cloud group feature tokens, and evaluate the redundancy of each local point cloud group feature tokens based on the similarity;

[0121] For the feature vector obtained in step 2-3-3 Normalize:

[0122]

[0123] Among them, τ norm is the normalized feature vector set, is the eigenvector The L2 norm of

[0124] Calculate the cosine similarity matrix S between feature vectors:

[0125]

[0126] Step 3-3, based on the similarity matrix, the redundancy score vector R of each local point cloud group feature tokens is calculated:

[0127]

[0128] Among them, S i,j is the element in the i-th row and j-th column of the similarity matrix;

[0129] According to the set semantic threshold θ, the proportion of unmasked local point cloud group feature tokens to be retained in each batch B is determined:

[0130]

[0131] Among them, θ is the semantic threshold, k is the number of local point cloud group feature tokens to be retained;

[0132] The k points with the lowest redundancy scores are selected and retained, and the remaining points are masked to obtain the semantic-aware mask matrix M sem :

[0133] θ k =TopK(R,k,largest=False)[-1]

[0134] M sem [i]=1 if R[i]>θ k else 0

[0135] Among them, θ k is the threshold of the redundancy score, and TopK(v,k,largest) returns the kth largest / smallest element in the vector v;

[0136] Step 3-4, merge the random mask matrix with the sentence-aware mask matrix to generate the final mask matrix M final :

[0137] M final =M rand ∨M sem

[0138] Among them, ∨ represents the matrix OR operation;

[0139] Handle the extreme case where all local point cloud group feature tokens in a batch are masked due to masking operations; ensure that each sample has at least one group with unknown mask:

[0140] For any b∈{1,...,B},

[0141] Then M final [b,g rand ]←0where g rand ~Uniform(1,G);

[0142] Among them, M final [b,g] represents the binary mask value of the local point cloud group feature tokensg in batch b. If the local point cloud group feature tokens is masked, the value is 0, and if the local point cloud group feature tokens is visible, the value is 1. The sum in the condition means that if all local point cloud group feature tokens in batch b are masked, the sum result will be 0. g rand ~Uniform(1,G) represents g rand Obey uniform distribution.

[0143] In steps 3-5, feature learning and point cloud reconstruction are achieved by pre-training using a masked autoencoder framework (MAE) based on the mask matrix and the ordered local point cloud group feature tokens sequence, wherein the masked autoencoder framework includes an encoder network and a decoder network.

[0144] The specific process of steps 3-5 is as follows:

[0145] Step 3-5-1, according to the mask matrix M final Divide the local point cloud group feature tokens sequence into visible parts and the masked part

[0146] Step 3-5-2, the visible local point cloud group feature tokens set τ visible Input encoder network, the network architecture learns the global spatial context of the point cloud through the self-attention mechanism and obtains the global context feature:

[0147] H ctx =Encoder(τ visible )∈R G×d

[0148] At the same time, record the center point coordinates C corresponding to the masked local point cloud group feature tokens masked ={c i |M final [i] = 1} as the location prior information;

[0149] Step 3-5-3, the decoder accepts the H output of the encoder ctx and C masked , both restore local set details through multi-layer deconvolution, and then complete the missing structure based on spatial continuity (Zigzag spatial continuity sorting) and mask mechanism, predict the coordinates, and gradually reconstruct the masked point cloud part

[0150] In order to enable the model to better learn the spatial structure and semantic representation of the point cloud, the Chamfer distance metric is used in the training process to minimize the error between the reconstructed point cloud and the original point cloud: it enables it to learn the intrinsic structure and semantic representation of the point cloud, and establish a robust feature foundation for subsequent segmentation and classification tasks.

[0151]

[0152] Among them, P and It represents the set of original point cloud and reconstructed point cloud. and q j are the coordinates of each point.

[0153] Step 4 is as follows:

[0154] Step 4-1: The ordered local point cloud group feature tokens sequence pre-trained by the masked autoencoder (MAE) Input into the PointMamba architecture for further feature enhancement. The PointMamba architecture is also an encoder-decoder structure, which combines transformer technology and spatial perception modules to deeply extract multi-scale features of point cloud data. In this process, the PointMamba encoder gradually learns the complex spatial relationship of the point cloud and extracts features through multiple layers of blocks. Each block contains normalization and mixer structures to ensure that the model can effectively handle long sequence dependencies and learn more levels of point cloud features. Then, the PointMamba architecture outputs a set of multi-scale features F output ={f out1 ,f out2 ,...,f outG}, where each f outi Representation and correspondence Related high-dimensional feature vectors, these features can effectively characterize the obstacle information in the point cloud and provide accurate feature support for subsequent point cloud segmentation and classification tasks.

[0155] Step 5 is as follows:

[0156] The feature F output from step 4 output ={f out1 ,f out2 ,...,f outG} and the original point cloud P = {p i ∈R 3 |i=1,2,...,N} as input, through the segmentation network, the model is for each point p i Predict its category probability and output the segmentation result of the point cloud S={s1,s2,...,s T}, where s i Represents point p i The probability of belonging to a certain category (such as obstacles or background) distinguishes the road surface from the obstacle area. The segmentation network, also known as the segmentation head, is the part of the network responsible for mapping the high-level features learned by the feature extraction network (here referring to the PointMamba architecture) to the category prediction of each point. The segmentation head can generate category predictions point by point through structures such as fully connected layers, convolutional layers, or graph convolutional networks (GCNs), and output the probability of each point belonging to a certain category. These probability values ​​represent the position of each point in the point cloud and the possibility that the point belongs to an obstacle, road surface, or background.

[0157] Beneficial effects:

[0158] The present invention significantly improves the processing effect of point cloud data through the innovative Zigzag spatial continuity sorting and local semantic-aware masking strategy. The method first uses Zigzag spatial continuity sorting to convert the disordered point cloud into an ordered sequence with spatial continuity, effectively maintaining the local geometric structure of the point cloud, so that adjacent points in the sequence still maintain a spatial proximity relationship. This feature is particularly conducive to capturing the geometric features of small targets. On this basis, the introduced local semantic-aware masking strategy analyzes feature similarity, dynamically selects mask targets, and prioritizes the retention of key semantic features, greatly improving the model's ability to extract global and local features of the point cloud. Combined with the efficient PointMamba architecture, this method achieves effective extraction of multi-level semantic information. Experimental data show that the classification accuracy on the ModelNet40 dataset is improved by 0.4%, the accuracy on the ScanObjectNN dataset is improved by 0.17%, and the mIoU of the segmentation task on the ShapeNetPart dataset is improved by 0.376%, fully verifying the superiority of the method. This method is particularly suitable for autonomous driving scenarios and can accurately distinguish between road surfaces and obstacle areas. It can also be widely used in three-dimensional reconstruction, industrial inspection and other fields. It has important engineering practice value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0159] Figure 1 It is a schematic diagram of a flow chart of the present invention.

[0160] Figure 2 This is the effect of the mask strategy after preprocessing of point cloud data. FIG3 is a diagram showing the final point cloud data processing effect of the method of the present invention. DETAILED DESCRIPTION

[0161] To address the shortcomings of the existing technology, the present invention proposes an innovative solution based on Zigzag spatial continuity sorting and local semantic perception masking, which can improve the detection and segmentation performance of small obstacles in autonomous driving scenarios.

[0162] This embodiment performs point cloud segmentation based on the obstacle detection scenario of autonomous driving:

[0163] A state-space model point cloud segmentation method based on a local semantic enhancement strategy includes the following steps:

[0164] Step 1, the original point cloud is as follows Figure 2 As shown in the figure, the center of mass of the point cloud is calculated, and the coordinates of all points are translated to the center of mass as the origin to determine the distance from the farthest point to the origin. Based on this, the entire point cloud is scaled to the unit sphere to obtain a standardized point cloud.

[0165] Step 2: Use Zigzag spatial continuity sorting to process the standardized point cloud to obtain ordered local point cloud group feature tokens with spatial continuity;

[0166] Step 3: Based on the standardized point cloud, the ordered local point cloud group feature tokens sequence obtained in step 2 is pre-trained using a masked autoencoder framework to obtain the ordered local point cloud group feature tokens sequence enhanced by local semantic perception mask;

[0167] Step 4: Extract multi-scale features from the ordered local point cloud group feature tokens enhanced by the local semantic perception mask, further enhance the spatial perception ability and feature expression of the point cloud to obtain the multi-scale feature F output ;

[0168] Step 5: The multi-scale feature F obtained in step 4 is output Perform point cloud segmentation to accurately distinguish the road surface from obstacle areas.

[0169] The results are as follows Figure 3 As shown in the figure, this patent significantly improves point cloud data processing by introducing innovative Zigzag spatial continuity sorting and local semantic enhancement masking strategies. This method enables efficient point cloud data recovery, resulting in more accurate reconstruction results with better spatial structure. This innovative improvement has significant application value in obstacle detection tasks in autonomous driving. It can effectively identify and segment obstacles in complex environments, enhance the system's perception capabilities, reduce false positives, and thus improve the safety and reliability of autonomous driving systems in complex environments.

[0170] Point cloud data standardization

[0171] Normalization of point cloud data is an important foundation for subsequent steps. In practical applications, raw point cloud data often has problems such as inconsistent size and position offset, which will affect the training effect of the model. This step normalizes the point cloud data through the following operations:

[0172] First, calculate the geometric center (center of mass) of the point cloud data and translate the coordinates of all points into a coordinate system with the center of mass as the origin. The calculation formula of the center of mass is:

[0173]

[0174] Where N is the total number of points in the point cloud, p i are the coordinates of the i-th point.

[0175] Then subtract the centroid coordinates from the coordinates of each point to achieve the centering of the point cloud:

[0176] P′ i ={p i -μP |p i ∈P}

[0177] This operation translates the point cloud, eliminating its positional offset and making subsequent processing more stable.

[0178] Find the point farthest from the origin in the translated point cloud and calculate its Euclidean distance:

[0179]

[0180] This distance is used for subsequent scaling operations.

[0181] Finally, divide the coordinates of all points by d max , scale the point cloud to fit within the unit sphere:

[0182] P norm ={p′ i / d max |p' i ∈P'}

[0183] Through this step, the point cloud data is unified into the same scale range, avoiding the degradation of model performance due to size differences.

[0184] Multi-plane collaborative Zigzag sorting

[0185] In order to make full use of the spatial information of the point cloud, this step converts the disordered point cloud data into an ordered sequence with spatial continuity through Zigzag spatial continuity sorting. The specific implementation is as follows:

[0186] The Farthest Point Sampling (FPS) algorithm is used to select G representative center points C = {c1, c2, ..., c G The nearest neighbor points around each center point form a local point cloud group:

[0187] G i ={p j |p j ∈KNN(c i ,M)},i=1,...,G

[0188] Among them, KNN(c i ,M) represents point c i The M nearest neighbors of .

[0189] Then normalize each local point cloud group and subtract the coordinates of its center point:

[0190]

[0191] This operation makes the position information of the local point cloud group more normalized.

[0192] Next, the convolutional neural network (CNN) extracts features from each local point cloud group to obtain the initial feature representation:

[0193] f i =ConvNet(G i ),i=1,...,G,F={f1,f2,...,f G}∈R G×d

[0194] Here, d is the dimension of the feature.

[0195] After processing using the above methods, the Zigzag method will layer the point cloud on the three main planes XY, XZ, and YZ, generating multiple Zigzag paths. For example, on the XY plane, the point cloud is first sorted in ascending order by Z coordinate, then evenly layered. The points in each layer are then sorted and segmented by X coordinate. Finally, the Y coordinate sorting direction (ascending or descending) is determined based on the parity of the segments. The generated Zigzag randomly selects a path during the training phase to enhance the model's generalization ability, and the path is fixed during the testing phase to ensure repeatability of the results:

[0196]

[0197] Finally, according to the selected path, the features of the local point cloud group are arranged in order to form an ordered local point cloud group feature tokens sequence

[0198] Local semantics-aware masking strategy

[0199] In order to enhance the model's ability to understand the semantic information of point clouds, this step will use the Masked Autoencoder (MAE) framework with a local semantic-aware masking strategy for pre-training. The specific process is as follows:

[0200] Generating a mask matrix includes generating a random mask matrix and generating a local semantic perception mask matrix. According to the preset mask ratio mask_ratio, some local point cloud group feature tokens are randomly selected for masking to generate a random mask matrix M rand ∈{0,1} B×G . Calculate the cosine similarity matrix S between the local point cloud group feature tokens features, and evaluate the redundancy score R of each local point cloud group feature tokens based on the similarity:

[0201]

[0202] Select local point cloud group feature tokens with higher redundancy scores for masking and generate the semantic perception mask matrix M sem .

[0203] The unmasked local point cloud group feature tokens are then input into the encoder to learn the global context features of the point cloud through the self-attention mechanism:

[0204] H ctx =Encoder(τ visible )

[0205] Combining the global features output by the encoder and the position information of the feature tokens of the masked local point cloud group, the masked point cloud part is gradually reconstructed. During the training process, the Chamfer distance is used as the loss function to measure the error between the reconstructed point cloud and the original point cloud:

[0206]

[0207] By minimizing this loss, the model is able to learn both local and global features of the point cloud.

[0208] Multi-scale feature extraction

[0209] To further enhance the feature representation capabilities of point clouds, this step feeds the pre-trained ordered sequence of local point cloud group feature tokens into the PointMamba architecture. PointMamba is a sequence modeling method based on the Selective State Space Model (Selective SSM) that efficiently captures long-range dependencies. The specific steps are as follows:

[0210] The local point cloud group feature tokens sequence τ is input into the PointMamba architecture, and multi-scale features are extracted through a multi-layer state space model:

[0211] F output ={f out1 ,f out2 ,...,f outG}

[0212] Each f outi It is a high-dimensional feature representation corresponding to the feature tokens of the local point cloud group, which integrates local and global semantic information.

[0213] Point cloud segmentation

[0214] Finally, the multi-scale feature F output And the original point cloud P is input into the segmentation network to predict its category probability for each point:

[0215] S={s1,s2,...,s T}

[0216] Among them, s i Represents point p i The probability of belonging to a certain category (such as road surface or obstacle). Segmentation networks usually use multi-layer perceptrons (MLP) or graph convolutional networks (GCN) combined with cross entropy loss function for end-to-end training.

[0217] This example uses multiple public datasets, including ModelNet40, ScanObjectNN, and ShapeNet55. These datasets contain point cloud data of varying quantities and complexities, and are primarily used to verify the robustness and accuracy of the model in point cloud classification and segmentation tasks.

[0218] It is worth noting that when this embodiment introduces the Zigzag path sorting mechanism and the local semantic-aware masking strategy during the training process, the number of Zigzags is set to 6, and the threshold of semantic-aware masking is set to 0.8 in the classification task and to 0.6 in the segmentation task.

[0219] First, for the classification task, the model of the present invention improved by 0.4% compared with the traditional method (92.7472%) on the ModelNet40 dataset. This result shows that the present invention can effectively improve performance in point cloud classification tasks, especially when processing more complex data, the optimized path sorting and semantic masking strategies have brought improvements to the accuracy of the model; on the ScanObjectNN dataset, the results of fine-tuning with different configurations also show the advantages of the model of the present invention. For example, under the configuration finetune_scan_objbg.yaml (ours), the accuracy of the model is improved by 0.17% compared with the traditional method. This result shows that the performance of the present invention in the scanning object recognition task has also been improved, especially for point cloud data with background interference, the model can better maintain accuracy, and the experimental results of different configurations of the ScanObjectNN dataset also show that the present invention can further optimize performance under different configurations by introducing Zigzag path sorting and semantic-aware masking strategies to process more complex and difficult datasets.

[0220] Next, we examine the segmentation task results. The proposed method also achieved excellent results on the ShapeNetPart dataset. Based on traditional methods, the model's best accuracy on the ShapeNetPart dataset improved by 0.376%, and both category and instance mean Intersection Over Union (MIoU) scores were also higher than those of traditional methods. This result demonstrates that the proposed model demonstrates stronger semantic understanding and detail extraction capabilities in point cloud segmentation tasks, particularly when handling component segmentation and instance recognition.

[0221] By introducing a Zigzag path sorting mechanism and a local semantic perception masking strategy, the present invention not only better maintains spatial continuity when processing structured point cloud data, but also reduces computational overhead and improves model robustness while enhancing semantic perception. Therefore, the present invention can be applied to obstacle detection in autonomous driving, distinguishing between the road surface and the obstacle area, thereby improving autonomous driving safety.

[0222] The present invention provides a state-space model point cloud segmentation method based on a local semantic enhancement strategy. There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A state space model point cloud segmentation method based on local semantic enhancement strategy, characterized in that: The steps involved are as follows: Step 1: Calculate the center of mass of the point cloud, translate all point coordinates to the center of mass as the origin, determine the distance from the farthest point to the origin, and scale the entire point cloud to the unit sphere to obtain a standardized point cloud. Step 2: Use Zigzag spatial continuity sorting to process the standardized point cloud to obtain ordered local point cloud group feature tokens with spatial continuity; Step 3: Based on the standardized point cloud, the ordered local point cloud group feature tokens sequence obtained in step 2 is pre-trained using a masked autoencoder framework to obtain an ordered local point cloud group feature tokens sequence enhanced by local semantic perception masking. Step 4: Extract multi-scale features from the ordered local point cloud group feature tokens sequence enhanced by the local semantic perception mask to further enhance the spatial perception ability and feature expression of the point cloud to obtain the multi-scale feature F output ; Step 5: The multi-scale features F extracted in step 4 are output Perform point cloud segmentation to accurately distinguish the road surface from obstacle areas.

2. The state space model point cloud segmentation method based on local semantic enhancement strategy according to claim 1 is characterized in that: Step 1 is as follows: Step 1-1, given a point cloud dataset P = {p i ∈R 3 |i=1,2,...,N}, where each point p i =(x i ,y i ,z i ), i represents the index of the point, ranging from 1 to N; N represents the total number of points in the point cloud, x i ,y i ,z i Represent the x-coordinate, y-coordinate, and z-coordinate values ​​of the i-th point in three-dimensional space, and calculate the center of mass μ of the point cloud P : And translate all point cloud data to the center of mass as the origin, and obtain the translated point cloud P' i : P′ i ={p i -μ P |p i ∈P} Step 1-2, calculate the distance of the farthest point from the origin in the translated point cloud: Among them, d max is the distance from the farthest point to the origin, ||p i ||2 is point p i The Euclidean norm of (distance from the origin); Steps 1-3: scale the entire translated point cloud to the unit sphere to obtain the standardized point cloud P norm : P norm ={p′ i / d max |p' i ∈P'}。 3. The state space model point cloud segmentation method based on local semantic enhancement strategy according to claim 1, characterized in that: Step 2 is as follows: Step 2-1: Perform farthest point sampling on the standardized point cloud to extract representative center points, and then construct a local point cloud group for each center point: Given a normalized point cloud set P norm ={p1,p2,...,p N }∈R N×3 , from which G representative center points C={c1,c2,...,c G }, get the center point set C∈R G×3 : c1=random(P) Among them, c j represents the selected center point set, p represents any point in the point cloud dataset, Indicates selecting the point farthest from the existing center point set. For each point p, calculate its distance from all selected center points c j The distance between them, select the point with the farthest distance as the next center point c i ; Then for each center point c i , find its nearest M neighbor points in the original point cloud to form a local point cloud group G i , the local point cloud group is a three-dimensional tensor: G i ={p j |p j ∈KNN(c i ,M)},i=1,...,G G i ∈R B×G×M×3 And perform normalization operation to obtain the final local point cloud group That is, each local group is subtracted from its center point coordinates: Step 2-2, feature encoding is performed on each local point cloud group through the convolutional network to obtain the initial feature representation of the local point cloud group: Each local point cloud group is input into the convolutional neural network for feature encoding, and the spatial features of each local point cloud group are extracted through the convolution operation to obtain the initial feature vector f of each local point cloud group. i ∈R d : f i =ConvNet(G i ),i=1,...,G Among them, R d It is a d-dimensional real vector space, where d represents the vector of feature dimension; Each local point cloud group finally generates a feature vector as the initial feature representation F of the point cloud group: F={f1,f2,...,f G } F∈R G×d In steps 2-3, a ZigZag path is then constructed on the three main planes XY, XZ, and YZ, and finally ordered local point cloud group feature tokens with spatial continuity are generated.

4. The state space model point cloud segmentation method based on local semantic enhancement strategy according to claim 3 is characterized in that: Steps 2-3 are as follows: Step 2-3-1: perform layered processing on the XY plane, XZ plane, and YZ plane to generate n paths Path sorting scheme; Step 2-3-2, for n paths Path selection, specifically: Case 1: Path selection during training: A valid path is randomly selected during training: Path selected =Path random_choice(indices) Among them, Path selected is the selected path, indices is the index set of valid paths, and random_choice() is the random selection function; Case 2: Path selection during the testing phase: The same path is always selected during the testing process: Path selected =Path random_choice(0) In step 2-3-3, ordered local point cloud group feature tokens with spatial continuity are generated according to the selected path.

5. The state space model point cloud segmentation method based on local semantic enhancement strategy according to claim 4, characterized in that: Step 2-3-1 is as follows: Step 2-3-1-1, horizontal plane Zigzag path generation: First, sort the point cloud in ascending order according to the Z coordinate: Z sorted ={p i ∈P norm |sorted by z i } Among them, Z sorted is the point set sorted by Z coordinate; Divide the sorted point cloud into n xy layer: Among them, n paths is the total number of paths to be generated, n xy is the number of layers in the XY plane, Z layers is the point set after stratification, Z j is the point set of the jth layer; For each layer Z j The points in are sorted by X coordinate: X sorted_j ={p i ∈Z j |sorted by x i } where X sorted_j is the point set sorted by X coordinate in the jth layer; X sorted_j Divided into m segments, X segments_j ={X j1 ,X j2 ,...,X jm } Among them, m is the number of segments, w is the number of point clouds expected to be included in each path, l is the limit on the number of segments per layer, X segments_j is the point set of the kth segment of the jth layer, |X sorted_j | is the number of points in the jth layer; For each segment X jk , the sorting method of the Y coordinate is determined according to the parity of k: if k is an odd number, the Y coordinate is sorted in ascending order; if k is an even number, the Y coordinate is sorted in descending order, and the final Y coordinate is sorted_jk is the point set sorted by Y coordinates in the jth layer and kth segment; Connect the sorted results of all segments to form the Zigzag path of this layer: Path j_xy =[And sorted_j1 ,AND sorted_j2 ,...,AND sorted_jm ] Among them, Path j_xy is the Zigzag path of the j-th layer in the XY plane; Step 2-3-1-2, XZ plane Zigzag path generation: First, sort the point cloud in ascending order according to the Y coordinate: Y sorted ={p i ∈P norm |sorted by y i } Among them, Y sorted is the point set sorted by Y coordinate; Divide the sorted point cloud into n xz layer: Among them, n paths is the total number of paths to be generated, n xz is the number of layers in the XZ plane, Y layers is the point set after stratification, Y j is the point set of the jth layer; For each layer Y j The points in are sorted by X coordinate: X sorted_j ={p i ∈Y j |sorted by x i } where X sorted_j is the point set sorted by X coordinate in the jth layer; X sorted_j Divided into m segments, X segments_j ={X j1 ,X j2 ,...,X jm } Among them, m is the number of segments, w is the number of point clouds expected to be included in each path, l is the number of segments limited to each layer (or each path), X segments_j is the point set of the kth segment of the jth layer, |X sorted_j | is the number of points in the jth layer; For each segment X jk , the sorting method of the Z coordinate is determined according to the parity of k: if k is an odd number, the Z coordinate is sorted in ascending order; if k is an even number, the Z coordinate is sorted in descending order, and the final Z coordinate is sorted_jk is the point set sorted by Z coordinates in the kth segment of the jth layer; Connect the sorted results of all segments to form the Zigzag path of this layer: Path j_zy =[Z sorted_j1 ,Z sorted_j2 ,...,Z sorted_jm ] Among them, Path j_xz is the Zigzag path of the XZ plane of the jth layer; Step 2-3-1-3, YZ plane Zigzag path generation: First, sort the point cloud in ascending order according to the X coordinate: X sorted ={p i ∈P norm |sorted by x i } Among them, X sorted is the point set sorted by X coordinate; Divide the sorted point cloud into n yz layer: Among them, n paths is the total number of paths to be generated, n yz is the number of layers in the YZ plane, X layers is the point set after stratification, X j is the point set of the jth layer; For each layer X j The points in are sorted by Y coordinate: Y sorted_j ={p i ∈X j |sorted by y i } where Y sorted_j is the point set sorted by Y coordinate in the jth layer; Y sorted_j Divided into m segments, AND segments_j ={And j1 ,AND j2 ,...,AND jm } Among them, m is the number of segments, w is the number of point clouds expected to be included in each path, l is the number of segments limited to each layer (or each path), and Y segments_j is the point set of the kth segment of the jth layer, |Y sorted_j | is the number of points in the jth layer; For each segment Y jk , the sorting method of the Z coordinate is determined according to the parity of k: if k is an odd number, the Z coordinate is sorted in ascending order; if k is an even number, the Z coordinate is sorted in descending order, and the final Z coordinate is sorted_jk is the point set sorted by Z coordinates in the kth segment of the jth layer; Connect the sorted results of all segments to form the Zigzag path of this layer: Path j_yz =[Z sorted_j1 ,Z sorted_j2 ,...,Z sorted_jm ] Among them, Path j_yz is the Zigzag path of the YZ plane of the jth layer.

6. The state space model point cloud segmentation method based on local semantic enhancement strategy according to claim 5, characterized in that: Step 2-3-3, specifically: A spatial continuity path selected according to the strategy of step 2-3-2 corresponds to the initial feature f of the local point cloud group obtained in step 2-2 i The ordered local point cloud group feature tokens sequence τ is formed in the order of the path: Among them, i1,i2,...,i G is the path index sequence, Represents a local point cloud group feature tokens, and each local point cloud group feature tokens represents a local point cloud group.

7. The state space model point cloud segmentation method based on local semantic enhancement strategy according to claim 1, characterized in that: Step 3 is as follows: Step 3-1: Generate a random mask based on the preset mask rate: Given a batch size B and a mask ratio mask_ratio, calculate the number of groups that need to be masked: Among them, G is the number of point cloud groups obtained in step 2-1, mask_ratio is the mask ratio, and num_mask is the number of groups that need to be masked. represents the floor function; Then for each sample in each batch B, a mask matrix is ​​randomly generated: M rand ∈{0,1} B×G Among them, M rand is a random mask matrix, B is the batch size, M rand [b,g]=1 means the g-th group of the b-th sample is masked; Step 3-2, calculate the normalized representation and cosine similarity of the local point cloud group feature tokens, and evaluate the redundancy of each local point cloud group feature tokens based on the similarity; For the feature vector obtained in step 2-3-3 Normalize: Among them, τ norm is the normalized feature vector set, is the eigenvector The L2 norm of Calculate the cosine similarity matrix S between feature vectors: Step 3-3, based on the similarity matrix, the redundancy score vector R of each local point cloud group feature tokens is calculated: Among them, S i,j is the element in the i-th row and j-th column of the similarity matrix; According to the set semantic threshold θ, the proportion of unmasked local point cloud group feature tokens to be retained in each batch B is determined: Among them, θ is the semantic threshold, k is the number of local point cloud group feature tokens to be retained; The k points with the lowest redundancy scores are selected and retained, and the remaining points are masked to obtain the semantic-aware mask matrix M sem : θ k =TopK(R,k,largest=False)[-1] M sem [i]=1if R[i]>θ k else0 Among them, θ k is the threshold of the redundancy score, and TopK(v,k,largest) returns the kth largest / smallest element in the vector v; Step 3-4, merge the random mask matrix with the sentence-aware mask matrix to generate the final mask matrix M final : M final =M rand ∨M sem Among them, ∨ represents the matrix OR operation; Handle the extreme case where all local point cloud group feature tokens in a batch are masked due to masking operations; For any b∈{1,...,B}, Then M final [b,g rand ←0 where g rand ~Uniform(1,G); Among them, M final [b,g] represents the binary mask value of the local point cloud group feature tokens g in batch b. If the local point cloud group feature tokens is masked, the value is 0, and if the local point cloud group feature tokens is visible, the value is 1. The sum in the condition means that if all local point cloud group feature tokens in batch b are masked, the sum result will be 0. g rand ~Uniform(1,G) represents g rand Obey uniform distribution; In steps 3-5, feature learning and point cloud reconstruction are achieved by pre-training using a masked autoencoder framework based on the mask matrix and the ordered local point cloud group feature tokens sequence, wherein the masked autoencoder framework includes an encoder network and a decoder network.

8. The state space model point cloud segmentation method based on local semantic enhancement strategy according to claim 7, characterized in that: The specific process of steps 3-5 is as follows: Step 3-5-1, according to the mask matrix M final Divide the local point cloud group feature tokens sequence into visible parts and the masked part Step 3-5-2, the visible local point cloud group feature tokens set τ visible The input encoder network learns the global spatial context of the point cloud through the self-attention mechanism and obtains the global context features: H ctx =Encoder(τ visible )∈R G×d At the same time, record the center point coordinates C corresponding to the masked local point cloud group feature tokens masked ={c i |M final [i] = 1} as the location prior information; Step 3-5-3, the decoder network accepts the H output by the encoder ctx and C masked , both restore local set details through multi-layer deconvolution, and then complete the missing structure based on spatial continuity and masking mechanism, predict the coordinates, and gradually reconstruct the masked point cloud part During the training process, the Chamfer distance metric is used to minimize the error between the reconstructed point cloud and the original point cloud: Among them, P and It represents the set of original point cloud and reconstructed point cloud. and q j are the coordinates of each point.

9. The state space model point cloud segmentation method based on local semantic enhancement strategy according to claim 1, characterized in that: Step 4 is as follows: Step 4-1: The ordered local point cloud group feature tokens sequence after pre-training of the masked autoencoder Input into the PointMamba architecture, further feature enhancement is performed to obtain a set of multi-scale features: F output ={f out1 ,f out2 ,...,f outG } in, Each f outi Representation and correspondence Related high-dimensional feature vectors.

10. The state space model point cloud segmentation method based on local semantic enhancement strategy according to claim 1, characterized in that: Step 5 is as follows: The feature F output from step 4 output ={f out1 ,f out2 ,...,f outG } and the original point cloud P = {p i ∈R 3 |i=1,2,...,N} as input, the segmentation network model is used to segment each point p i Predict its category probability and output the segmentation result of the point cloud S={s1,s2,...,s T }, where s i Represents point p i The probability of belonging to a certain category distinguishes the road surface from the obstacle area.

Images