Point cloud segmentation method and system based on point cloud serialization and Mama network

Abstract

The invention relates to the technical field of point cloud semantic segmentation, and discloses a point cloud segmentation method and system based on point cloud serialization and a Mama network. Specifically, a geometric distance matrix and a semantic distance matrix are calculated for a plurality of representative point units obtained by sampling on an original point cloud, and a path sequence is generated based on a fused comprehensive distance matrix. And traversing the path sequence by adopting a bidirectional scanning strategy to generate a forward sequence and a reverse sequence, extracting features from the forward sequence and the reverse sequence by utilizing a Mama network, cascading or fusing the extracted features, and finally performing classification based on the fused features. According to the method, better long-range dependency and local context modeling effects can be obtained, the comprehensive expression capability of detail features and global context in sparse and irregular scenes is enhanced, and the calculation and memory overhead required for global dependency capture of large-scale point clouds is greatly reduced.

Description

Technical Field

[0001] The present application relates to the technical field of point cloud semantic segmentation, and in particular to a point cloud segmentation method and system based on point cloud serialization and Mamba network. Background Art

[0002] LiDAR technology has developed rapidly in recent years, making it easier to obtain large-scale, high-density three-dimensional point clouds. However, the LiDAR point cloud itself is sparse and unevenly distributed, naturally disordered, and has high scene complexity, which brings many challenges to network modeling based on deep learning. On the one hand, unlike images, three-dimensional point clouds do not have a natural two-dimensional grid structure. Traditional convolutional or Transformer models are difficult to process disordered data directly, and additional serialization or regularization of point clouds is required, otherwise the spatial topology will be destroyed or the information will be confused. On the other hand, in urban modeling, road detection, power inspection and other practical applications, point clouds often contain a variety of ground objects, which vary greatly in scale and appearance, increasing the difficulty of global semantic modeling.

[0003] Existing point cloud segmentation methods, including the PointNet series, convolution-based PointConv, and Transformer based on the self-attention mechanism, have brought significant progress to three-dimensional scene analysis. However, they still face the problems of excessive computational and storage overhead and insufficient local-global balance when processing large-scale point clouds. In particular, for the capture of long-range dependencies, if the self-attention mechanism is used to establish global associations, its time and space complexity often grows quadratically or to a higher order, making it difficult to adapt to truly large-scale point cloud scenes. In addition, point cloud serialization itself lacks systematic design: if it is only sorted by simple spatial distance or Z-order, Hilbert and other curves, it is easy to destroy semantic consistency; if only semantic features are considered, the neighborhood relationship in the physical space may be ignored, making it difficult for the model to take into account both global and local accurate expressions. Summary of the invention

[0004] The present application provides a point cloud segmentation method based on point cloud serialization and Mamba network to solve the problems in the prior art of excessive computing and storage overhead, insufficient local-global balance, and lack of system design of point cloud serialization itself when processing large-scale point clouds.

[0005] Correspondingly, the present application also provides a point cloud segmentation system based on point cloud serialization and Mamba network, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.

[0006] To solve the above technical problems, the present application discloses a point cloud segmentation method based on point cloud serialization and the Mamba network, and the method includes:

[0007] Performing upsampling on the original point cloud to obtain multiple representative point units;

[0008] Calculating a geometric distance matrix and a semantic distance matrix for the multiple representative point units, and fusing the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix;

[0009] Generating a path sequence based on the comprehensive distance matrix, and traversing the path sequence using a bidirectional scanning strategy to generate a forward sequence and a reverse sequence;

[0010] Using the Mamba network to extract features from the forward sequence and the reverse sequence respectively, and cascading or fusing the extracted features to obtain bidirectional fusion features;

[0011] Performing classification based on the bidirectional fusion features to obtain the semantic segmentation result of the lidar point cloud.

[0012] Preferably, after performing upsampling on the original point cloud to obtain multiple representative point units, the method further includes:

[0013] Using one or more layers of multi-layer perceptrons to extract semantic features from the representative point units, and recording the spatial coordinates of the representative point units.

[0014] Preferably, calculating a geometric distance matrix and a semantic distance matrix for the multiple representative point units, and fusing the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix, includes:

[0015] Calculating a geometric distance matrix and a semantic distance matrix based on the semantic features and spatial coordinates of the multiple representative point units;

[0016] Performing weighted fusion on the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix.

[0017] Preferably, generating a path sequence based on the comprehensive distance matrix, and traversing the path sequence using a bidirectional scanning strategy to generate a forward sequence and a reverse sequence, includes:

[0018] Taking the comprehensive distance matrix as the adjacency matrix of a weighted graph, and using a preset graph traversal method to search the weighted graph to obtain a path sequence that connects all nodes and has the minimum cumulative weight;

[0019] Performing forward traversal on the path sequence to obtain a forward sequence;

[0020] Performing reverse traversal on the path sequence to obtain a reverse sequence.

[0021] Preferably, the Mamba network is used to extract features from the forward sequence and the reverse sequence respectively, and the extracted features are cascaded or fused to obtain bidirectional fusion features, including:

[0022] Input the forward sequence and the reverse sequence into the Mamba network respectively;

[0023] In the Mamba network, temporal modeling is performed on the input forward sequence and backward sequence respectively by selecting a state space model;

[0024] Use fused depthwise separable convolutions to extract local features of the forward sequence and the reverse sequence respectively, and fuse the local features with the outputs of the corresponding selected state space models to form a comprehensive representation of the forward sequence and a comprehensive representation of the reverse sequence;

[0025] Concatenate or element-wise weighted fuse the comprehensive representation of the forward sequence and the comprehensive representation of the reverse sequence to obtain bidirectional fusion features.

[0026] Preferably, before inputting the forward sequence and the reverse sequence into the Mamba network respectively, the method further includes:

[0027] Perform linear projection or normalization processing on the features of each representative point unit in the forward sequence and the reverse sequence.

[0028] Preferably, classification is performed based on the bidirectional fusion features to obtain the semantic segmentation result of the lidar point cloud, including:

[0029] Build a lightweight segmentation head at the last layer of the Mamba network;

[0030] Use the lightweight segmentation head to classify the bidirectional fusion features corresponding to each representative point unit, and obtain the probability distribution of each category through a preset activation function;

[0031] Take the category corresponding to the maximum value in the probability distribution of each category as the category label of the representative point unit;

[0032] Restore the category label of the representative point unit to the original point cloud through a preset interpolation method;

[0033] Integrate the category labels of all the original point clouds to obtain the semantic segmentation result of the lidar point cloud.

[0034] This application also discloses a point cloud segmentation system based on point cloud serialization and the Mamba network, and the system includes:

[0035] A point cloud sampling module for sampling the original point cloud to obtain a plurality of representative point units;

[0036] A sequence generation module, configured to calculate a geometric distance matrix and a semantic distance matrix for multiple representative point units, and fuse the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix;

[0037] The sequence generation module is further configured to generate a path sequence based on the comprehensive distance matrix, and traverse the path sequence by adopting a bidirectional scanning strategy to generate a forward sequence and a reverse sequence;

[0038] A sequence feature fusion module, configured to use a Mamba network to extract features from the forward sequence and the reverse sequence respectively, and cascade or fuse the extracted features to obtain bidirectional fusion features;

[0039] A classification and prediction module, configured to perform classification based on the bidirectional fusion features to obtain a semantic segmentation result of the lidar point cloud.

[0040] This application also discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, one or more methods in this application are implemented.

[0041] This application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods in this application are implemented.

[0042] This application has at least the following beneficial effects:

[0043] 1. At the serialization stage, geometric distance and semantic distance are integrated, and the minimum spanning tree algorithm is combined to generate a sequence that preserves local topology and semantic consistency, so that adjacent representative point units have higher similarity in three-dimensional coordinates and feature vectors, and thus better long-range dependence and local context modeling effects are obtained in the network.

[0044] 2. The Mamba network introducing the Selective State Space Models (SSMs) can process long sequences in a linear time manner, greatly reducing the computational and memory overhead required for capturing global dependencies of large-scale point clouds, and is more suitable for large-scale three-dimensional point cloud analysis in environments with limited hardware resources.

[0045] 3. By using the bidirectional scanning strategy to generate a forward sequence and a reverse sequence, the associations between distant representative point units can be captured from different scanning directions, and the features of the two are fused, significantly enhancing the comprehensive expression ability for detailed features and global context in sparse and irregular scenes.

[0046] 4. This application is compatible with multiple types of LiDAR data. Whether it is airborne multi-spectral lidar, airborne lidar, or vehicle-mounted mobile lidar, this application can flexibly adjust weights and feature extraction methods according to specific application scenarios, and has good versatility and scalability. Through the full-process design of sampling, serialization, Mamba network feature extraction, and semantic segmentation output, accurate and efficient segmentation results can be obtained in a variety of actual environments.

[0047] 5. In addition to three-dimensional coordinates and high-dimensional semantic features, this application can also expand diverse inputs such as laser intensity, normal vector, RGB, or spectral information according to requirements. The final segmentation results can be mapped back to the original point cloud using common interpolation and post-processing schemes to achieve more refined object recognition and three-dimensional reconstruction.

[0048] Additional aspects and advantages of this application will be given in the following description section, which will become apparent from the following description, or can be learned through the practice of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of this application will become apparent and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0050] Figure 1 is a flowchart of the point cloud segmentation method based on point cloud serialization and Mamba network provided by the embodiment of this application;

[0051] Figure 2 is a schematic diagram of the process of serialization and point cloud segmentation based on Mamba network provided by the embodiment of this application;

[0052] Figure 3 is a comparison diagram of the path structures of various point cloud sorting methods in two-dimensional grids and three-dimensional point clouds provided by the embodiment of this application;

[0053] Figure 4 is a schematic diagram of the generation process of the comprehensive distance matrix provided by the embodiment of this application;

[0054] Figure 5 is a comparison result diagram of mIoU, average F1 score, and OA of different serialization methods on three lidar datasets provided by the embodiment of this application;

[0055] Figure 6 is a calculation result diagram of relevant indicators on different serialization methods provided by the embodiment of this application;

[0056] Figure 7 is a schematic diagram of the structure of the point cloud segmentation system based on point cloud serialization and Mamba network provided by the embodiment of this application;

[0057] Figure 8Schematic diagram of the structure of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0058] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and should not be construed as a limitation to the present application.

[0059] Those skilled in the art of the present technology can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "including" used in the specification of the present application means the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their combinations. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more related listed items.

[0060] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as here.

[0061] The solution provided by the embodiment of the present application can be executed by any electronic device, such as a terminal device or a server. Among them, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, and the present application does not limit this. For the technical problems existing in the prior art, the point cloud segmentation method and system based on point cloud serialization and Mamba network provided by the present application are intended to solve at least one of the technical problems of the prior art.

[0062] The following uses specific embodiments to elaborate in detail on the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0063] In recent years, Mamba networks, represented by the Selective State Space Model (SSMs), have received attention in sequence tasks such as natural language processing for their long-range modeling ability in linear time, becoming an emerging solution for processing long sequences. Introducing Mamba networks into lidar point cloud semantic segmentation has the potential to achieve large-scale point cloud long-range dependence modeling while maintaining light weight. However, Mamba networks need to receive inputs in the form of one-dimensional sequences. If the point cloud serialization strategy is inappropriate, it may lead to the network having difficulty accurately learning the relationship between spatial topology and semantic distribution. Therefore, a serialization method that can both retain the neighboring structure of the point cloud in physical space and take into account the similarity of the same type of objects in the semantic embedding space is needed, and then combined with the advantages of the Mamba network to complete high-precision and high-efficiency large-scale three-dimensional point cloud segmentation.

[0064] The embodiments of the present application provide a possible implementation manner, as Figure 1 shown, a flowchart of a point cloud segmentation method based on point cloud serialization and Mamba network is provided. This solution can be executed by any electronic device. Optionally, it can be executed on the server side or the terminal device.

[0065] As Figure 1 shown, the method may include the following steps:

[0066] Step 101, upsample the original point cloud to obtain multiple representative point units.

[0067] Optionally, as Figure 2 shown, first input the lidar point cloud as the original point cloud, and downsample the original point cloud to obtain multiple representative point units (Tokens) to reduce subsequent calculation and storage overhead. Among them, downsampling can use distance-based or density-based methods, such as farthest point sampling (FPS) or grid sampling, and a reasonable sampling radius or threshold can be selected in combination with the actual scene.

[0068] Step 102, calculate the geometric distance matrix and the semantic distance matrix for the multiple representative point units, and fuse the geometric distance matrix and the semantic distance matrix to obtain the comprehensive distance matrix.

[0069] In the embodiments of the present application, the geometric distance matrix and the semantic distance matrix are calculated based on the relevant features and spatial coordinates of the representative units.

[0070] Step 103: Generate a path sequence based on the comprehensive distance matrix, and traverse the path sequence using the Bi-Scanning strategy to generate a forward sequence and a backward sequence.

[0071] In the embodiment of the present application, multiple representative point units obtained by sampling are serialized based on the comprehensive distance matrix and a preset path generation method, and the corresponding forward sequence and backward sequence are obtained by using the Bi-Scanning strategy. In the serialization stage, the geometric distance and the semantic distance are integrated, and the generated path sequence retains the local topology and semantic consistency, making the adjacent representative point units have higher similarity in both spatial coordinates and feature vectors, which is convenient for obtaining better long-range dependence and local context modeling effects in the Mamba network subsequently.

[0072] Step 104: Use the Mamba network to extract features from the forward sequence and the backward sequence respectively, and cascade or fuse the extracted features to obtain bidirectional fusion features.

[0073] In the embodiment of the present application, the forward sequence and the backward sequence are respectively input into the Mamba network through the Token local embedding module. The Mamba network includes a stacked Mamba network module. The Selective State Space Models (SSMs) used in the Mamba network module can process long sequences in a linear time manner, greatly reducing the computational and memory overhead required for capturing global dependencies of large-scale point clouds. Compared with the network based on the self-attention mechanism, the Mamba network used in the embodiment of the present application is more suitable for large-scale three-dimensional point cloud analysis in an environment with limited hardware resources.

[0074] Step 105: Classify based on the bidirectional fusion features to obtain the semantic segmentation result of the lidar point cloud.

[0075] As Figure 2 shown, in the embodiment of the present application, a specific semantic segmentation head is designed to classify the bidirectional fusion features extracted by the Mamba network to determine the final semantic segmentation result. For the input large-scale lidar point cloud, the three-dimensional scene segmented by the above steps includes soil, trees, cars, and buildings.

[0076] In the embodiments of the present application, a geometric distance matrix and a semantic distance matrix are calculated for multiple representative point units obtained by upsampling the original point cloud, and the two matrices are fused to obtain a comprehensive distance matrix. A path sequence is generated based on the comprehensive distance matrix. The above process of point cloud serialization combines geometric distance and semantic distance, making adjacent representative point units have higher similarity in both spatial coordinates and feature vectors, and thus achieving better long-range dependence and local context modeling effects in the network. A two-way scanning strategy is used to traverse the path sequence to generate a forward sequence and a reverse sequence, which can capture the associations between distant representative point units from different scanning directions. The Mamba network is used to extract features from the forward sequence and the reverse sequence respectively, and the extracted features are concatenated or fused to obtain two-way fusion features, significantly enhancing the comprehensive expression ability for detailed features and global context in sparse and irregular scenes. Moreover, using the Mamba network to extract sequence features can process long sequences in a linear time manner, greatly reducing the computational and memory overhead required for capturing global dependencies in large-scale point clouds. Finally, classification is performed based on the two-way fusion features to obtain a more accurate semantic segmentation result.

[0077] In an alternative embodiment, after obtaining multiple representative point units by upsampling the original point cloud, the method further includes:

[0078] One or more layers of multi-layer perceptrons (MLPs) are used to extract semantic features from the representative point units, and the spatial coordinates (X, Y, Z) of the representative point units are recorded. The MLP can be supplemented with laser intensity, RGB, spectral information, normal vectors, etc. according to specific applications to distinguish features of different classes. Through the projection and mapping of the multi-layer MLP, a more discriminative high-dimensional representation can be obtained, laying a foundation for subsequent serialization and network modeling.

[0079] In an alternative embodiment, as Figure 4 shown, calculating a geometric distance matrix and a semantic distance matrix for multiple representative point units, and fusing the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix, includes:

[0080] 1.1. Calculate the geometric distance matrix and the semantic distance matrix based on the semantic features and spatial coordinates of multiple representative point units.

[0081] In the embodiments of the present application, the sampled representative point unit is denoted as P i =(x i , f i ), where x i is the spatial coordinate and f i is the semantic feature output by the MLP. For N representative point units respectively calculate the geometric distance matrix M g and the semantic distance matrix M f :

[0082]

[0083] Among them, for any two representative point units, the geometric distance can be obtained from the Euclidean norm of the difference in their spatial coordinates, and the semantic distance comes from the difference between the output feature vectors of the MLP.

[0084] Perform appropriate normalization on the calculated geometric distance matrix and semantic distance matrix so that they are comparable on the same numerical scale, and avoid imbalance during subsequent distance fusion due to dimensional differences.

[0085] 1.2. Perform weighted fusion on the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix.

[0086] In the embodiment of the present application, the geometric distance matrix and the semantic distance matrix are fused by weighting. To balance the contribution degrees of the spatial structure constraint and the semantic information, an adjustable weight w g and w f are set. Optionally, initially, the adjustable weights w g and w f can be set to be equal, or can be adjusted according to scene experience. For example, by default according to experience, w g =2, w f =1, or iteratively optimize on the validation set. Furthermore, the geometric distance matrix M g and the semantic distance matrix M f are fused in a weighted manner to obtain a comprehensive distance matrix:

[0087] M d (i,j) = w g ·M g (i,j) + w f ·M f (i,j)

[0088] Optionally, the embodiment of the present application can perform another normalization or filtering operation on the comprehensive distance matrix M d to improve the subsequent search efficiency and robustness. The filtering operation adopts methods such as removing extreme noise distances and restricting the upper bound of distances.

[0089] In an optional embodiment, a path sequence is generated based on the comprehensive distance matrix, and a two-way scanning strategy is used to traverse the path sequence to generate a forward sequence and a reverse sequence, including:

[0090] 2.1. Take the comprehensive distance matrix M d as the adjacency matrix of a weighted graph, and use a preset graph traversal method to search the weighted graph to obtain a path sequence that connects all nodes and has the minimum cumulative weight.

[0091] Optionally, the preset graph traversal method can use the minimum spanning tree (MST), or other graph traversal algorithms (such as Prim or Kruskal) to search the weighted graph to obtain a path sequence that connects all nodes and has the minimum cumulative weight:

[0092] π = {p π(1) , p π(2) , …, p π(N)}

[0093] This path can ensure that adjacent points are close in "space and semantics" as much as possible, thereby improving the ability of the subsequent Mamba network to capture long-range dependencies and local context. If it is also necessary to combine distance metrics at different scales (such as multi-resolution features, etc.), a multiple sorting strategy can be performed before or after the MST generation to further optimize the balance between the global and local aspects of the sequence.

[0094] 2.2. Perform a forward traversal on the path sequence to obtain the forward sequence P h .

[0095] In the embodiment of the present application, after obtaining the unidirectional path sequence, the unidirectional path sequence can be directly regarded as the forward sequence P h .

[0096] 2.3. Perform a reverse traversal on the path sequence to obtain the reverse sequence P h′ .

[0097] In an optional embodiment, the Mamba network is used to extract features from the forward sequence and the reverse sequence respectively, and the extracted features are cascaded or fused to obtain bidirectional fusion features, including:

[0098] 3.1. Input the forward sequence and the reverse sequence into the Mamba network respectively;

[0099] In the embodiment of the present application, first, the Mamba network is constructed and initialized, and the forward sequence P h and the reverse sequence P h′ generated in the previous steps are respectively used as the inputs of the Mamba network, the hyperparameters in the Mamba network (such as the hidden layer dimension, the state space size, etc.) are set, and the random or pre-trained initialization of the initial weights is completed.

[0100] 3.2. In the Mamba network, perform temporal modeling on the input forward sequence and backward sequence respectively by selecting the state space model.

[0101] Inside the Mamba network, the Selective State Space Models (SSMs) are used to recursively update the state of the input sequence (forward sequence or backward sequence) in the temporal dimension (i.e., the Token sequence dimension). Each Token selectively retains or discards the information from the previous moment based on the current input and historical state, thus achieving efficient modeling of long-range dependencies. Different from the attention-based methods, the core computational cost of the Mamba network grows linearly with the sequence length, making it more suitable for large-scale point clouds.

[0102] 3.3. Use the Depth-Wise Convolution to extract the local features of the forward sequence and the backward sequence respectively, and fuse the local features with the outputs of the corresponding Selective State Space Models to form the comprehensive representations of the forward sequence and the backward sequence.

[0103] In the embodiments of this application, local enhancement is performed through the Depth-Wise Convolution. A local convolution module (DWConv) is added to the Mamba block to extract local features from adjacent channels or local neighborhoods at the same moment. This operation can enhance the perception of the local structure of the point cloud (such as details, edges, etc.) and avoid the loss of details that may occur when only using the temporal dimension of the sequence. The features obtained by DWConv are fused with the outputs of the Selective State Space Models (such as element-wise addition, feature concatenation, etc.) to form a comprehensive representation of the point cloud that has both global sequence dependencies and local spatial information.

[0104] 3.4. Concatenate or fuse the comprehensive representations of the forward sequence and the backward sequence element-wise with weights to obtain the bidirectional fusion features.

[0105] In the embodiments of this application, multiple Mamba blocks can be stacked for bidirectional feature fusion, enabling the model to perform multi-level abstractions on the point cloud sequence and enriching the multi-scale understanding of the 3D point cloud. After the forward sequence and the backward sequence pass through their respective Mamba blocks, they will be concatenated or fused at the high-level feature end to integrate the information from both scanning directions. The finally obtained bidirectional fusion features encode the context information of the point cloud in different scanning directions, enhancing the expression ability of the bidirectional association of the point cloud in space and semantics, and contributing to improving the discrimination ability for sparse and irregular scenes.

[0106] In an optional embodiment, before the forward sequence and the backward sequence are respectively input into the Mamba network, the method further includes:

[0107] Perform linear projection or normalization on the features of each representative point unit in the forward sequence and the backward sequence to match the input dimension of the subsequent Mamba block.

[0108] In an optional embodiment, classification is performed based on the bidirectional fusion features to obtain the semantic segmentation result of the lidar point cloud, including:

[0109] 4.1. Establish a lightweight segmentation head (such as a simple MLP structure composed of fully connected, BatchNorm, and ReLU) at the last layer of the Mamba network, and this lightweight segmentation head receives the Mamba feature input.

[0110] 4.2. Use the lightweight segmentation head to classify the bidirectional fusion features corresponding to each representative point unit, and obtain the probability distribution of each category through a preset activation function (such as Softmax or other logical functions), so as to provide a semantic prediction label for each Token.

[0111] 4.3. Take the category corresponding to the maximum value in the probability distribution of each category as the category label of the representative point unit. According to application requirements (such as minority class balance, confidence threshold setting, etc.), post-process or filter the classification result to improve the detection rate of specific targets (such as rare categories).

[0112] 4.4. Restore the category label of the representative point unit to the original point cloud through a preset interpolation method.

[0113] Interpolation methods such as nearest neighbor (NN) or inverse distance weighted (IDW) can be adopted to map the prediction result of the representative point unit (Token) to the surrounding unsampled points, so as to realize the semantic discrimination of all points. For each point in the original point cloud, the Token with the closest distance to it can be selected, or the final category can be determined according to the voting of several neighboring Tokens.

[0114] 4.5. Integrate the category labels of all the original point clouds to obtain the semantic segmentation result of the lidar point cloud.

[0115] In the embodiment of the present application, according to the scenario requirements (such as airborne, multispectral, vehicle-mounted, etc.), the segmented three-dimensional scene can be visualized and output, including various targets such as roads, buildings, and vegetation. If necessary, the segmentation result can be further post-processed (such as removing small connected components, smoothing, or fusing other sensor information) to form a more complete final result.

[0116] Such as Figure 3As shown in the figure, the embodiments of the present application compare the path structures of six existing typical point cloud sorting methods (Hilbert curve, radial, lexicographic order, OPTICS clustering, Z-shaped, random) in two-dimensional grids and three-dimensional point clouds. It can be seen from the figure that although some sorting methods based on spatial geometric mapping have a certain degree of continuity in two-dimensional scenarios, they generally have problems such as jumps and path distortions in three-dimensional space, seriously affecting the preservation of point cloud structure information. Clustering methods (such as the OPTICS clustering sorting method) show large fluctuations in terms of sequential coherence. In contrast, the method in the embodiments of the present application generates a path with greater continuity and aggregation by comprehensively considering the spatial proximity and semantic similarity of the point cloud, which is convenient for subsequent networks to learn the global structure and local details, and significantly improves the robustness and accuracy of modeling.

[0117] Figure 5 It shows the semantic segmentation performance of seven point cloud sorting methods (Hilbert curve, OPTICS, lexicographic order, radial, semantic sequence, Z-shaped, random) on three lidar datasets, namely DALES, MS-LiDAR, and Toronto-3D. The evaluation metrics include the mean intersection over union (mIoU), mean F1 score, and overall accuracy (OA). The results show that the performance differences of different sorting methods on different datasets are significant. Among them, the Hilbert curve and Lexicographic methods perform relatively better in DALES and MS-LiDAR, while methods such as Z-shaped and Random are weaker in multiple metrics. This phenomenon indicates that although traditional geometric or lexicographic sorting strategies can maintain spatial adjacency to a certain extent, their ability to capture semantic structures in complex scenarios is limited. It can be seen that a single-dimensional sorting strategy cannot fully balance global topology and semantic consistency, and a fusion sorting method needs to be introduced to enhance the modeling effect. This is also the core motivation for the proposed method of the present application.

[0118] As Figure 6 shown, the method of the present application is compared with seven typical sorting strategies (Hilbert curve, OPTICS, lexicographic order, radial, semantic sequence, Z-shaped, random) in terms of multiple metrics on three mainstream lidar datasets, covering three core metrics: semantic segmentation accuracy (mIoU), spatial proximity, and semantic similarity. In Figure 6 (a), the mean intersection over union (mIoU) results show that the method of the present application performs excellently on all datasets, especially with the most significant accuracy improvement in Toronto-3D, indicating that it has stronger semantic recognition ability in complex multi-class scenarios; Figure 6 (b) shows the spatial proximity comparison. The method of the present application is significantly higher than other sorting strategies, indicating that the path it constructs has greater geometric continuity and local consistency, which is beneficial to maintaining the original point cloud structure; Figure 6(c), the semantic similarity results further verify the advantage of this method in semantic feature aggregation, which can effectively reduce the breakage and interpenetration of similar targets in the path. Generally speaking, the sorting method proposed in this application is superior to the existing solutions in both structural preservation and semantic expression, providing more stable and high-quality input support for subsequent sequence modeling based on the Mamba network.

[0119] Based on the above embodiments, an embodiment of the present application provides a point cloud serialization method that fuses spatial and semantic distance metrics. Under the Bi-Scanning strategy, the selection state space models (SSMs) of the Mamba network and local convolutional enhancement units are used to jointly extract long-range dependencies and local contexts. It can not only capture long-range dependency relationships in large-scale lidar point clouds but also retain rich local details. This method has good robustness in various lidar point cloud scenarios such as airborne, aerial survey, or vehicle-mounted, can achieve high-precision and high-efficiency 3D semantic segmentation, and can still maintain a low computational overhead in the case of long sequence inputs, providing new ideas and technical support for practical applications such as road extraction, urban modeling, and power line inspection.

[0120] Based on the same principle as the method provided in the embodiments of the present application, an embodiment of the present application also provides a point cloud segmentation system based on point cloud serialization and the Mamba network, as Figure 7 shown, the system includes:

[0121] A point cloud sampling module 701, configured to sample the original point cloud to obtain a plurality of representative point units;

[0122] A sequence generation module 702, configured to calculate a geometric distance matrix and a semantic distance matrix for the plurality of representative point units, and fuse the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix;

[0123] The sequence generation module 702 is further configured to generate a path sequence based on the comprehensive distance matrix, traverse the path sequence using a bidirectional scanning strategy, and generate a forward sequence and a reverse sequence;

[0124] A sequence feature fusion module 703, configured to use the Mamba network to extract features from the forward sequence and the reverse sequence respectively, and cascade or fuse the extracted features to obtain a bidirectional fusion feature;

[0125] A classification prediction module 704, configured to perform classification based on the bidirectional fusion feature to obtain the semantic segmentation result of the lidar point cloud.

[0126] In the embodiments of the present application, a geometric distance matrix and a semantic distance matrix are calculated for multiple representative point units obtained by upsampling the original point cloud, and the comprehensive distance matrix is obtained by fusion. A path sequence is generated based on the comprehensive distance matrix. The above process of point cloud serialization combines geometric distance and semantic distance, making adjacent representative point units have higher similarity in both spatial coordinates and feature vectors, and thus obtaining better long-range dependence and local context modeling effects in the network. The path sequence is traversed using a bidirectional scanning strategy to generate a forward sequence and a reverse sequence, which can capture the associations between distant representative point units from different scanning directions. The Mamba network is used to extract features from the forward sequence and the reverse sequence respectively, and the extracted features are concatenated or fused to obtain bidirectional fusion features, which significantly enhances the comprehensive expression ability of detailed features and global context in sparse and irregular scenes. Moreover, using the Mamba network to extract sequence features can process long sequences in a linear time manner, greatly reducing the computational and memory overhead required for capturing global dependencies of large-scale point clouds. Finally, classification is performed based on the bidirectional fusion features, and a more accurate semantic segmentation result can be obtained.

[0127] The point cloud segmentation system based on point cloud serialization and the Mamba network provided by the embodiments of the present application can implement Figures 1 to 6 each process implemented in the method embodiments. To avoid repetition, it will not be elaborated here.

[0128] The point cloud segmentation system based on point cloud serialization and the Mamba network in the embodiments of the present application can execute the point cloud segmentation method based on point cloud serialization and the Mamba network provided by the embodiments of the present application. Their implementation principles are similar. The actions performed by each module and unit in the point cloud segmentation system based on point cloud serialization and the Mamba network in each embodiment of the present application correspond to the steps in the point cloud segmentation method based on point cloud serialization and the Mamba network in each embodiment of the present application. For the detailed function descriptions of each module of the point cloud segmentation system based on point cloud serialization and the Mamba network, reference can specifically be made to the descriptions in the corresponding point cloud segmentation method based on point cloud serialization and the Mamba network shown above, and it will not be elaborated here.

[0129] Based on the same principle as the method shown in the embodiments of the present application, the embodiments of the present application further provide an electronic device, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the point cloud segmentation method based on point cloud serialization and the Mamba network shown in any optional embodiment of the present application by calling the computer program. Compared with the prior art, the point cloud segmentation method based on point cloud serialization and the Mamba network provided by the present application calculates a geometric distance matrix and a semantic distance matrix for multiple representative point units obtained by upsampling the original point cloud, and fuses them to obtain a comprehensive distance matrix, and generates a path sequence based on the comprehensive distance matrix. The above process of point cloud serialization combines geometric distance and semantic distance, so that adjacent representative point units have higher similarity in both spatial coordinates and feature vectors, and thus better long-range dependence and local context modeling effects can be obtained in the network. A bidirectional scanning strategy is used to traverse the path sequence to generate a forward sequence and a reverse sequence, which can capture the associations between distant representative point units from different scanning directions. The Mamba network is used to extract features from the forward sequence and the reverse sequence respectively, and the extracted features are cascaded or fused to obtain bidirectional fusion features, which significantly enhances the comprehensive expression ability of detailed features and global context in sparse and irregular scenes. And using the Mamba network to extract sequence features can process long sequences in a linear time manner, greatly reducing the computational and memory overhead required for capturing global dependencies of large-scale point clouds. Finally, classification is performed based on the bidirectional fusion features, and a more accurate semantic segmentation result can be obtained.

[0130] In an optional embodiment, an electronic device is further provided, as Figure 8 shown Figure 8 The electronic device 800 shown may be a server, including: a processor 801 and a memory 803. Among them, the processor 801 and the memory 803 are connected, such as through a bus 802. Optionally, the electronic device 800 may further include a transceiver 804. It should be noted that in practical applications, the transceiver 804 is not limited to one, and the structure of the electronic device 800 does not constitute a limitation on the embodiments of the present application.

[0131] The processor 801 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of this application. The processor 801 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0132] The bus 802 may include a path for transmitting information between the above components. The bus 802 may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 802 may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 8 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0133] The memory 803 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or it may also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0134] The memory 803 is used to store the application program code for executing the solution of this application, and is controlled by the processor 801 to execute. The processor 801 is used to execute the application program code stored in the memory 803 to implement the content shown in the foregoing method embodiments.

[0135] Among them, the electronic device includes but is not limited to: mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not impose any restrictions on the functions and usage scope of the embodiments of this application.

[0136] The server provided by this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here.

[0137] The embodiments of this application provide a computer-readable storage medium, on which a computer program is stored. When it runs on a computer, it enables the computer to execute the corresponding content in the foregoing method embodiments.

[0138] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order restriction, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same moment, but can be executed at different moments, and their execution order is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0139] It should be noted that the above computer-readable storage medium in this application can also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. And in this application, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0140] The above computer-readable medium can be included in the above electronic device; it can also exist separately without being assembled into the electronic device.

[0141] The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to execute the method shown in the above embodiments.

[0142] According to one aspect of this application, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to execute the point cloud segmentation method and system based on point cloud serialization and the Mamba network provided in the above various alternative implementation manners.

[0143] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and this module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can also occur in a different order than that marked in the accompanying drawings. For example, two consecutively represented blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0145] The modules described in the embodiments of this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation to the module itself in some cases. For example, the point cloud sampling module can also be described as "a point cloud sampling module for sampling on the original point cloud to obtain multiple representative point units".

[0146] The above description is only the preferred embodiments of this application and the explanation of the technical principles applied. Those skilled in the art should understand that the scope of disclosure involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in this application.

Claims

1. A point cloud segmentation method based on point cloud serialization and Mamba network, characterized in that The method includes: Sampling the original point cloud to obtain a plurality of representative point units; Calculating a geometric distance matrix and a semantic distance matrix for the plurality of representative point units, and fusing the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix; Generating a path sequence based on the comprehensive distance matrix, and traversing the path sequence using a bidirectional scanning strategy to generate a forward sequence and a reverse sequence; Using a Mamba network to extract features from the forward sequence and the reverse sequence respectively, and concatenating or fusing the extracted features to obtain a bidirectional fusion feature; Classifying based on the bidirectional fusion feature to obtain a semantic segmentation result of the lidar point cloud.

2. The point cloud segmentation method based on point cloud serialization and Mamba network according to claim 1, wherein After sampling the original point cloud to obtain a plurality of representative point units, the method further includes: Using one or more multi-layer perceptrons to extract semantic features from the representative point units, and recording the spatial coordinates of the representative point units.

3. The point cloud segmentation method based on point cloud serialization and Mamba network according to claim 2, wherein, The calculating a geometric distance matrix and a semantic distance matrix for the plurality of representative point units, and fusing the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix includes: Calculating the geometric distance matrix and the semantic distance matrix based on the semantic features and spatial coordinates of the plurality of representative point units; Performing weighted fusion on the geometric distance matrix and the semantic distance matrix to obtain the comprehensive distance matrix.

4. The point cloud segmentation method based on point cloud serialization and Mamba network according to claim 1, characterized in that, The generating a path sequence based on the comprehensive distance matrix, and traversing the path sequence using a bidirectional scanning strategy to generate a forward sequence and a reverse sequence includes: Taking the comprehensive distance matrix as the adjacency matrix of a weighted graph, and using a preset graph traversal method to search the weighted graph to obtain the path sequence that connects all nodes and has the minimum cumulative weight; Performing forward traversal on the path sequence to obtain the forward sequence; Performing reverse traversal on the path sequence to obtain the reverse sequence.

5. The point cloud segmentation method based on point cloud serialization and Mamba network according to claim 1, wherein The using a Mamba network to extract features from the forward sequence and the reverse sequence respectively, and concatenating or fusing the extracted features to obtain a bidirectional fusion feature includes: Inputting the forward sequence and the reverse sequence into the Mamba network respectively; Performing temporal modeling on the input forward sequence and backward sequence respectively by selecting a state space model in the Mamba network; Using fused depthwise separable convolutions to extract local features of the forward sequence and the reverse sequence respectively, and fusing the local features with the outputs of the corresponding selected state space models to form a comprehensive representation of the forward sequence and a comprehensive representation of the reverse sequence; Concatenating or element-wise weighted fusing the comprehensive representation of the forward sequence and the comprehensive representation of the reverse sequence to obtain the bidirectional fusion feature.

6. The point cloud segmentation method based on point cloud serialization and Mamba network according to claim 4, characterized in that, Before inputting the forward sequence and the reverse sequence into the Mamba network respectively, the method further includes: Performing linear projection or normalization processing on the features of each representative point unit in the forward sequence and the reverse sequence.

7. The point cloud segmentation method based on point cloud serialization and Mamba network according to claim 1, characterized in that The classifying based on the bidirectional fusion feature to obtain a semantic segmentation result of the lidar point cloud includes: Establishing a lightweight segmentation head at the last layer of the Mamba network; Classify the two-way fusion features corresponding to each representative point unit using the lightweight segmentation head, and obtain the probability distribution of each category through a preset activation function; Take the category corresponding to the maximum value in the probability distribution of each category as the category label of the representative point unit; Restore the category label of the representative point unit to the original point cloud through a preset interpolation method; Integrate the category labels of all the original point clouds to obtain the semantic segmentation result of the lidar point cloud.

8. A point cloud segmentation system based on point cloud serialization and the Mamba network, characterized in that, The system includes: A point cloud sampling module for sampling multiple representative point units on the original point cloud; A sequence generation module for calculating a geometric distance matrix and a semantic distance matrix for multiple representative point units, and fusing the geometric distance matrix and the semantic distance matrix to obtain a comprehensive distance matrix; The sequence generation module is further configured to generate a path sequence based on the comprehensive distance matrix, traverse the path sequence using a two-way scanning strategy, and generate a forward sequence and a reverse sequence; A sequence feature fusion module for using a Mamba network to extract features from the forward sequence and the reverse sequence respectively, and concatenating or fusing the extracted features to obtain two-way fusion features; A classification prediction module for classifying based on the two-way fusion features to obtain the semantic segmentation result of the lidar point cloud.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the method according to any one of claims 1-7.

Images