Serialization-based point cloud over-segmentation method, device and equipment and medium

Abstract

The invention discloses a serialization-based point cloud over-segmentation method, device and equipment and a medium, and relates to the technical field of point cloud over-segmentation. The point cloud over-segmentation method comprises the following steps: obtaining point cloud data set preprocessing; serializing the preprocessed point cloud data into a Hilbert curve; dividing the serialized point cloud data into a plurality of initial segments based on spatial continuity, and obtaining a multi-scale segment set; the feature similarity between adjacent segments is calculated, and a similarity matrix is obtained and established to provide a basis for subsequent super-point clustering; carrying out super-point clustering on the initial segments based on a self-adaptive updating algorithm; updating super-point features through a cross attention mechanism, establishing dynamic association between points and super-points, and obtaining a new super-point structure; and according to the updated super-point features, constructing a super-point graph, performing enhancement processing through a graph convolutional network, fusing features from the backbone point cloud network and corresponding multi-level super-point features, then transmitting the fused features to a segmentation head for segmentation, and obtaining semantic segmentation output.

Description

Technical Field

[0001] This invention relates to the field of point cloud over-segmentation technology, and more specifically, to a point cloud over-segmentation method, apparatus, device, and medium based on serialization. Background Technology

[0002] 3D point clouds, as an important carrier of scene geometric information, have wide applications in fields such as autonomous driving, robot navigation, and smart city construction.

[0003] 3D point cloud semantic segmentation, by assigning semantic labels to each point, enables accurate identification and understanding of objects in a scene, serving as a crucial prerequisite for tasks such as environmental perception and path planning. Compared to 2D image data, 3D point clouds are unaffected by changes in lighting and viewpoint, and can more comprehensively describe the spatial structure and geometric features of objects, providing a unique advantage for the analysis of complex scenes.

[0004] Large-scale point cloud datasets typically contain millions or even tens of millions of irregularly arranged points. Their inherent sparsity and non-uniformity pose significant challenges to segmentation tasks. Traditional methods often struggle to balance computational efficiency and segmentation accuracy when processing such data: on the one hand, complex spatial query operations (such as nearest neighbor search) significantly increase the computational burden; on the other hand, simple downsampling or voxelization operations lose detailed information, resulting in blurred object boundaries. While existing deep learning models (such as PointNet and DGCNN) can extract local features, they have limitations in modeling global semantic consistency, making it difficult to effectively aggregate spatially adjacent and semantically consistent points.

[0005] Furthermore, existing methods suffer from significant drawbacks when processing large-scale point clouds: oversegmentation methods based on spatial partitioning (such as VCCS and SPG) can guarantee spatial proximity but ignore the inherent semantic relationships within the point cloud. While deep learning-based semantic segmentation models (such as MinkUNet) can capture local geometric features, their limited receptive field prevents them from effectively modeling long-distance dependencies. This contradiction leads to poor performance on key metrics such as boundary recall (BR) and boundary precision (BP), particularly in complex scenarios like wall interfaces and thin structures where maintaining boundary clarity and semantic consistency is difficult. Therefore, a segmentation scheme that balances efficient computation with semantic aggregation capabilities is urgently needed. Summary of the Invention

[0006] The present invention provides a method, apparatus, device and medium for over-segmentation of point clouds based on serialization, so as to improve at least one of the above-mentioned technical problems.

[0007] Firstly, the present invention provides a point cloud over-segmentation method based on serialization, which includes steps S1 to S7.

[0008] S1. Obtain the point cloud dataset and perform preprocessing.

[0009] S2. Serialize the preprocessed point cloud data into Hilbert curves.

[0010] S3. Based on spatial continuity, the serialized point cloud data is divided into multiple initial segments to construct a hierarchical segmentation structure and obtain a multi-scale segment set.

[0011] S4. Calculate the feature similarity between adjacent segments and obtain a similarity matrix to provide a basis for subsequent super-point clustering.

[0012] S5. Perform super-point clustering on the initial segments based on the adaptive update algorithm to ensure semantic consistency within the superpoints.

[0013] S6. Update the superpoint features through the cross-attention mechanism, establish dynamic relationships between points and superpoints, and obtain new superpoint structures.

[0014] S7. Based on the updated superpoint features, construct a superpoint map and enhance it through a graph convolutional network. Then, fuse the features from the backbone point cloud network with the corresponding multi-level superpoint features, and pass it to the segmentation head for segmentation to obtain semantic segmentation output.

[0015] Preferably, step S1 specifically includes steps S11 to S12.

[0016] S11. Perform voxelization on the input 3D point cloud data to obtain a sparse tensor format suitable for sparse convolution operations.

[0017] S12. Input the voxelized point cloud into a U-Net architecture constructed using a sparse convolutional network SpConv to extract local detail features. The bottleneck part of the U-Net architecture integrates a Mamba module to enhance the overall structural perception of the point cloud through global context interaction, compensating for the limitations of the local receptive field of sparse CNNs.

[0018] Preferably, step S2 specifically involves: transforming the preprocessed point cloud data into a one-dimensional sequence using Hilbert space-filling curve mapping, denoted as... .

[0019] Preferably, step S3 specifically includes steps S31 to S32.

[0020] S31. Padded sequence to obtain padded sequence. .

[0021] .

[0022] .

[0023] In the formula, Represent real numbers, For the length after filling, The total dimension of the feature vector, For the number of points, The number of segments is the finest granularity, to ensure that the finest granularity segments can be... Perform uniform division.

[0024] S32, using recursive quadrilateral partitioning to... Divided into Each segment is used to construct a hierarchical segmentation structure and obtain a multi-scale segment set.

[0025] .

[0026] .

[0027] .

[0028] .

[0029] In the formula, For multi-scale segment sets, For the first The first in the layer Each segment Indicates the first The length of each segment in the layer, For the number of segments, The index of the point It is a preset constant. This is the index for the current level.

[0030] Preferably, step S4 specifically includes steps S41 to S42.

[0031] S41. Constructing fine-grained layers based on hierarchical relationships with its parent layer The correspondence. Wherein: Each coarse-grained segment A set of points containing four fine-grained segments.

[0032] S42. For each original point in the coarse-grained superpoints, calculate the similarity matrix based on the cosine similarity between its features and the features of each fine-grained superpoint.

[0033] The similarity matrix model is as follows: .

[0034] The elements of the similarity matrix are defined as follows: .

[0035] In the formula, For similarity matrix, For the elements of the similarity matrix, Represent real numbers, Indicates the first The number of original points contained in the coarse-grained superpoint layer. This indicates the number of fine-grained superpoints under the current coarse-grained superpoint. Features corresponding to the original point For fine-grained superpoint features, For the first point cloud One original point.

[0036] Preferably, step S5 specifically involves: selecting the superpoint with the highest similarity to complete the point-to-superpoint assignment. Wherein, each original point... Allocation index set It is determined by calculating its similarity to the superpoint, the mapping function. Represents the origin point The assigned superpoint index: .

[0037] In the formula, The index of the original point, Indicates the first The number of original points contained in the coarse-grained superpoint.

[0038] Preferably, step S6 specifically includes: The cross-attention mechanism adjusts the feature updates between the superpoint and its member points through dynamic adaptive relationships. The superpoint features are used as the query, while the member point features are used as keys and values, and the superpoint features are updated by calculating a weighted sum.

[0039] The updated formula is: .

[0040] In the formula, For the updated super-point features, For cross-attention mechanism, For the first Original features of each superpoint Features corresponding to the original point For the first point cloud One original point This is a mapping function.

[0041] Attention weights are calculated using scaled dot products, allowing each superpoint to focus on the most informative original point in its local neighborhood. The specific formula is as follows: .

[0042] .

[0043] .

[0044] .

[0045] .

[0046] In the formula, For the Softmax activation function, For query, For key, For value, Indicates transpose, For the query learnable projection matrix, The learnable projection matrix of the key, Learnable projection matrix with value Represent real numbers, Indicates the first The first in the layer The number of points in each segment This represents the total dimension of the feature vector.

[0047] Through multiple iterations, the final superpoint feature update result is obtained, and a new superpoint structure is constructed. .

[0048] Preferably, step S7 specifically includes steps S71 to S75.

[0049] S71. Construct a superpoint graph based on the updated superpoint features.

[0050] S72. Calculate the edge features of the super-point graph.

[0051] .

[0052] In the formula, Features of the edges For a linear mapping function, and They represent the first Super point and the The coordinates of the superpoint.

[0053] S73. After constructing the superpoint graph and its edge features, use a graph convolutional network to enhance the superpoint graph.

[0054] S74. After processing by the graph convolutional network, the features from the backbone point cloud network are fused with the corresponding multi-level superpoint features.

[0055] S75. Pass these fused features to the segmentation head for final semantic segmentation output.

[0056] Secondly, the present invention provides a point cloud over-segmentation device based on serialization, which includes a preprocessing module, a serialization module, a hierarchical module, a similarity module, a clustering module, an update module, and a segmentation module.

[0057] The preprocessing module is used to acquire point cloud datasets and perform preprocessing.

[0058] The serialization module is used to serialize preprocessed point cloud data into Hilbert curves.

[0059] The hierarchical module is used to divide the serialized point cloud data into multiple initial segments based on spatial continuity, so as to construct a hierarchical segmentation structure and obtain a multi-scale segment set.

[0060] The similarity module is used to calculate the feature similarity between adjacent segments and obtain a similarity matrix to provide a basis for subsequent super-point clustering.

[0061] The clustering module is used to perform super-point clustering on the initial segments based on an adaptive update algorithm to ensure semantic consistency within the super-points.

[0062] The update module is used to update the superpoint features through a cross-attention mechanism, establish dynamic relationships between points and superpoints, and obtain new superpoint structures.

[0063] The segmentation module is used to construct a superpoint map based on the updated superpoint features, enhance it through a graph convolutional network, fuse features from the backbone point cloud network with the corresponding multi-level superpoint features, and then pass it to the segmentation head for segmentation to obtain semantic segmentation output.

[0064] Thirdly, the present invention provides a serialization-based point cloud over-segmentation device, which includes a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a serialization-based point cloud over-segmentation method as described in any paragraph of the first aspect.

[0065] Fourthly, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute a serialization-based point cloud over-segmentation method as described in any paragraph of the first aspect.

[0066] By adopting the above technical solution, the present invention can achieve the following technical effects: This embodiment of a point cloud over-segmentation method based on serialization utilizes serialization strategies and super-point clustering to simplify the processing flow and reduce computational pressure by aggregating point clouds into an ordered super-point structure. It also generates high-quality feature representations through a cross-attention mechanism, effectively avoiding complex spatial query operations, significantly improving segmentation efficiency, and more efficiently dividing point clouds into multiple regions to identify the point cloud features of each item. Attached Figure Description

[0067] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the specific embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some specific embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0068] Figure 1 This is a logical block diagram of a point cloud oversegmentation method based on serialization.

[0069] Figure 2 This is a logical block diagram of an end-to-end point cloud semantic segmentation method.

[0070] Figure 3 This is a visualization of the point cloud oversegmentation method on the ScanNet V2 and S3DIS Area5 datasets.

[0071] Figure 4 This is a flowchart illustrating a point cloud over-segmentation method based on serialization. Detailed Implementation

[0072] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0073] Example 1, please refer to Figures 1 to 4 The first embodiment of the present invention provides a point cloud over-segmentation method based on serialization, which can be executed by a point cloud over-segmentation device based on serialization (hereinafter referred to as: point cloud over-segmentation device). In particular, it is executed by one or more processors in the point cloud over-segmentation device to implement steps S1 to S7.

[0074] S1. Obtain the point cloud dataset and perform preprocessing.

[0075] In this invention, the input point cloud data is defined as... . In the formula, Represent real numbers, The number of points, For each point, there are feature dimensions (e.g., normal vector, RGB color values, etc.), and the spatial coordinates of each point are obtained through... express.

[0076] To preprocess point cloud datasets, this invention designs a point cloud processing method based on a sparse convolutional neural network (CNN) and U-Net architecture. This method mainly includes steps S11 to S12: S11. The input 3D point cloud data is voxelized to obtain a sparse tensor format suitable for sparse convolution operations. Specifically, the point cloud data will be processed by operators... Voxelization is performed to obtain discretized point cloud data. In this way, the original point cloud data... It is converted into a format suitable for further processing.

[0077] S12. Input the voxelized point cloud into a U-Net architecture constructed using a sparse convolutional network SpConv to extract local detail features. The bottleneck part of the U-Net architecture integrates a Mamba module to enhance the overall structural perception of the point cloud through global context interaction, compensating for the limitations of the local receptive field of sparse CNNs.

[0078] Based on voxelized point cloud data, this invention uses a U-Net architecture based on sparse convolutions (SpConv) for processing. The shallow part of this U-Net architecture utilizes SpConv layers to capture fine local geometric information. Through these operations, the generated feature representation not only maintains high spatial resolution but also extracts detailed geometric structures that are crucial for accurate object boundary delineation.

[0079] Furthermore, considering the limitations of local receptive field in traditional sparse CNNs when processing point clouds, this invention proposes an improvement: In the bottleneck part of U-Net, the inventors enhanced the network structure by integrating the Mamba module to improve global context interaction capabilities. The Mamba module possesses global modeling capabilities with linear complexity, effectively compensating for local inductive biases in convolutional layers, thereby ensuring the consistency of superpoints in the global structure.

[0080] The Mamba module enhances the model's ability to perceive features from different regions through the interaction of global context information. Furthermore, it addresses the inherent limitation of local receptive fields in sparse CNNs at deeper levels. Through this improvement, the U-Net architecture can enhance its understanding of global features while maintaining its detail extraction capabilities, thereby improving the model's point cloud processing performance.

[0081] It is understood that the point cloud oversegmentation device can be an electronic device with computing power, such as a portable laptop computer, desktop computer, server, smartphone, or tablet computer.

[0082] S2. The preprocessed point cloud data is serialized into Hilbert curves to avoid complex spatial query operations using serialization technology. Specifically, in this invention, the preprocessed point cloud data is first transformed into a one-dimensional sequence through Hilbert space-filling curve mapping, denoted as... In the formula, This represents the total dimension of the feature vector.

[0083] By leveraging the spatial locality preservation property of Hilbert space-filling curves, neighboring point sets in 3D space are mapped to continuous positions in a one-dimensional sequence using Hilbert space-filling curves, thus achieving hierarchical segmentation of point clouds at different granularities. The ability of Hilbert space-filling curves to map neighboring point sets in 3D Euclidean space to continuous positions in a one-dimensional sequence simplifies 3D segmentation tasks based on spatial proximity.

[0084] S3. Based on spatial continuity, the serialized point cloud data is divided into multiple initial segments to construct a hierarchical segmentation structure and obtain a multi-scale segment set. Preferably, step S3 specifically includes steps S31 to S32.

[0085] S31. Padded sequence to obtain padded sequence. .

[0086] .

[0087] .

[0088] In the formula, Represent real numbers, For the length after filling, The total dimension of the feature vector, For the number of points, The number of superpoints in the finest-grained layer is determined to ensure that the finest-grained segments can be uniformly divided.

[0089] S32, using recursive quadrilateral partitioning to... Divided into Each segment is used to construct a hierarchical segmentation structure and obtain a multi-scale segment set.

[0090] .

[0091] .

[0092] .

[0093] .

[0094] In the formula, For multi-scale segment sets, For the first The first in the layer Each segment Indicates the first The length of each segment in the layer, The index of the original point, It is a preset constant. This is the index of the current level (i.e., the sequence number of the level).

[0095] In this embodiment, and , .in The number of levels. Represents natural numbers.

[0096] Specifically, the filled point cloud is recursively processed through quadrilateral partitioning at each layer. Lieutenant General Divided into The paragraph is described in detail as follows: In the formula, Indicates the first The length of each segment in the layer. (During...) After this process, a hierarchical partitioning structure is obtained. Each recursive partition establishes a fixed parent-child index relationship between levels.

[0097] For each segment in each layer, an aggregation operation that permutes its local features is performed. To generate features, specifically: In the formula, This represents a permutation-invariant aggregation operation (such as max pooling). Indicates the first The first in the layer The number of points in each segment . For is the dimension eigenvectors, For the first point cloud One original point.

[0098] This invention can efficiently generate hierarchical point cloud segmentation through recursive quadrilateral segmentation and permutation-invariant aggregation operations, thereby effectively capturing local and global information in the point cloud and achieving accurate point cloud segmentation at multiple scales.

[0099] Specifically, given the point cloud as input and its corresponding feature set Features of each point It is generated from the backbone of the network. This invention constructs a hierarchical partitioning structure. To perform recursive quadrilateral segmentation on the serialized point cloud. Each layer It contains a complete serialized point cloud, recursively divided into multiple segments, represented as: In the formula, Indicates the first The first in the layer Segments, number of segments Allocation is performed according to geometric progression, satisfying... .

[0100] S4. Calculate the feature similarity between adjacent segments and obtain a similarity matrix to provide a basis for subsequent super-point clustering.

[0101] Based on the hierarchical structure, the core objective of this invention is to partition the heuristic space into a hierarchical structure. and the original features of the corresponding superpoints Transform into semantically consistent superpoint representation .in, Indicates the first Each segment contains superpoints within precise geometric boundaries. Specifically, these refined features are generated through dynamic modeling of local semantic associations, thereby more accurately representing the geometric and semantic information in the point cloud.

[0102] Preferably, step S4 specifically includes steps S41 to S42.

[0103] S41. Constructing fine-grained layers based on hierarchical relationships with its parent layer The correspondence. Wherein: Each coarse-grained segment A set of points containing four fine-grained segments.

[0104] First, consider two adjacent levels. and its coarser-grained parent layer ,in Indicates the number of superpoints in the finest-grained layer and the coarse-grained parent layer. This represents the number of superpoints in the coarse-grained parent layer. The hierarchical relationship shows that the number of superpoints in a coarser-grained layer is one-quarter the number of superpoints in a finer-grained layer. The specific relationship is as follows: .

[0105] For each layer, each superpoint Include The original point. And in the coarser-grained layer... Each super point It contains four times the number of original points of its corresponding level, reaching Each coarse-grained superpoint contains features of the original point, which are further updated through interactions with neighboring superpoints.

[0106] S42. For each original point in the coarse-grained superpoints, calculate the similarity matrix based on the cosine similarity between its features and the features of each fine-grained superpoint.

[0107] The similarity matrix model is as follows: .

[0108] The elements of the similarity matrix are defined as follows: .

[0109] In the formula, For similarity matrix, For the elements of the similarity matrix, Represent real numbers, Indicates the first The number of original points contained in the coarse-grained superpoint layer. This indicates the number of fine-grained superpoints under the current coarse-grained superpoint. Features corresponding to the original point For fine-grained superpoint features, For the first point cloud One original point.

[0110] Specifically, in order to establish a point-to-superpoint association between the coarse-grained layer and the fine-grained layer, this invention first establishes a point-to-superpoint association for each original point. The most similar initial superpoint features are identified in the fine-grained layer. Then, neighboring superpoint members are further considered to expand the receptive field of the original points, ensuring more accurate feature updates. This process involves calculating a similarity matrix. To achieve this, where each element Measure the first The original point and the first Cosine similarity between superpoint features: .

[0111] This process generates each level. The similarity matrix.

[0112] S5. Perform super-point clustering on the initial segments based on the adaptive update algorithm to ensure semantic consistency within the superpoints.

[0113] In this embodiment, each original point is assigned to the most similar superpoint through a mapping function. Establish the correspondence between points and superpoints. The point-to-superpoint assignment is completed by selecting the superpoint with the highest similarity. Each original point... Allocation index set It is determined by calculating its similarity to the superpoint, the mapping function. Represents the origin point The assigned superpoint index: .

[0114] In the formula, The index of the original point, Indicates the first The number of original points contained in the coarse-grained superpoint layer. Specifically, by comparing the feature similarity between each point and potential superpoints, the point is assigned to the most similar superpoint, and the superpoint boundary is dynamically adjusted according to the assignment result to make it more closely match the actual object boundary.

[0115] S6. Update the superpoint features through the cross-attention mechanism, establish dynamic relationships between points and superpoints, and obtain new superpoint structures.

[0116] Preferably, the cross-attention mechanism adjusts the feature updates between the superpoint and its member points through dynamic adaptive relationships. The superpoint features are used as queries, while the member point features are used as keys and values, and the superpoint features are updated by calculating a weighted sum.

[0117] The updated formula is: .

[0118] In the formula, For the updated super-point features, For cross-attention mechanism, For the first Original features of each superpoint Features corresponding to the original point For the first point cloud One original point This is a mapping function.

[0119] Attention weights are calculated using scaled dot products, allowing each superpoint to focus on the most informative original point in its local neighborhood. The specific formula is as follows: .

[0120] .

[0121] .

[0122] .

[0123] .

[0124] In the formula, For the Softmax activation function, For query, For key, For value, Indicates transpose, For the query learnable projection matrix, The learnable projection matrix of the key, Learnable projection matrix with value Represent real numbers, Indicates the first The first in the layer The number of points in each segment This represents the total dimension of the feature vector.

[0125] .

[0126] Through multiple iterations, the final superpoint feature update result is obtained, and a new superpoint structure is constructed. .

[0127] For coarser-grained superpoints, the same update logic is applied, but the interaction occurs between fine-grained superpoint features and their corresponding coarse-grained initialized superpoint features. The difference is that coarse-grained superpoint features are updated by using the aggregation of fine-grained superpoint features. The specific relationship can be calculated using the inter-layer index relationship formula.

[0128] The adaptive update algorithm and hierarchical segmentation structure designed in this embodiment can accurately capture object boundaries, ensure semantic consistency within superpoints, and preserve local geometric details and global semantic information.

[0129] S7. Based on the updated superpoint features, construct a superpoint map and enhance it through a graph convolutional network. Then, fuse the features from the backbone point cloud network with the corresponding multi-level superpoint features, and pass it to the segmentation head for segmentation to obtain semantic segmentation output.

[0130] Updated superpoint features By encoding locally uniform geometric semantic information, an effective representation is provided for each local point set. These superpoint features are used for semantic segmentation tasks and are combined with oversegmentation networks through a superpoint interaction module to form an end-to-end framework.

[0131] Each superpoint captures different semantic details through a hierarchical design. Fine-grained superpoints capture local information, while coarse-grained superpoints represent object-level semantic information. To capture the relationships between superpoints, step S7 preferably includes steps S71 to S75.

[0132] S71. Construct a superpoint graph based on the updated superpoint features.

[0133] Super point graph is defined ,in Denotes the set of all superpoints Each superpoint is represented by its position in the feature space. Next, to connect every two superpoints in the superpoint graph, their similarity is calculated, and the nearest neighbor method is used to connect each superpoint to its nearest neighbor. Connect the nearest neighbor superpoints.

[0134] S72. Calculate the edge features of the super-point graph.

[0135] .

[0136] In the formula, Features of the edges For a linear mapping function, and They represent the first Super point and the The coordinates of the superpoint.

[0137] Specifically, the characteristics of each edge It is defined using relative position encoding. In this way, edge features can capture the spatial relationships between nodes.

[0138] S73. After constructing the superpoint graph and its edge features, use a graph convolutional network to enhance the superpoint graph.

[0139] Graph Convolutional Networks (GCNs) enable local feature interactions between supernodes through a message-passing mechanism. By passing messages multiple times, GCNs can effectively enhance the feature representation between adjacent supernodes, thereby achieving better semantic segmentation.

[0140] S74. After processing by the graph convolutional network, the features from the backbone point cloud network are fused with the corresponding multi-level superpoint features.

[0141] The fusion can utilize various existing fusion methods, such as channel-attention weighted concatenation, weighted summation, or attention-based fusion; this invention does not impose specific limitations on these methods. The backbone point cloud network refers to the "point cloud processing method based on sparse convolutional neural network (CNN) and U-Net architecture" in step S1. The multi-level superpoint features are the updated superpoint features.

[0142] S75. Pass these fused features to the segmentation head for final semantic segmentation output.

[0143] Specifically, the segmentation head uses an existing point cloud segmentation model, but this invention does not impose any specific limitations on it.

[0144] This embodiment of a point cloud over-segmentation method based on serialization utilizes serialization strategies and super-point clustering to simplify the processing flow and reduce computational pressure by aggregating point clouds into an ordered super-point structure. It also generates high-quality feature representations through a cross-attention mechanism, effectively avoiding complex spatial query operations, significantly improving segmentation efficiency, and more efficiently dividing point clouds into multiple regions to identify the point cloud features of each item.

[0145] Based on the above embodiments, in an optional embodiment of the present invention, a loss function is designed for the above-mentioned point cloud over-segmentation method based on serialization.

[0146] The first step is to construct the superpoint generation process.

[0147] By introducing a superpoint aggregation loss function, this invention aims to optimize the superpoint generation process and ensure that superpoints possess good semantic segmentation characteristics. This loss function includes multiple sub-loss terms, with a compact loss function being the first to be used. Strengthening the cohesion of features within superpoints minimizes the distance between superpoints. This part of the loss can be expressed by the following formula: .

[0148] in, Representing point features, Indicates super-point features, For the first A set of points with super-points It is the size of the superpoint. It represents the number of superpoints in the point cloud.

[0149] The second step is to enhance the separability between classes.

[0150] Subsequently, a discriminant loss function is used. Enhance the separability of superpoints between different classes. This loss function improves inter-class separability by increasing the distance between superpoints from different classes. The label of each superpoint is determined by the majority vote of its constituent points. The mathematical expression of the discriminative loss function is: .

[0151] in, A collection representing classes. It is a predefined minimum separation distance (set to 0.5 in this paper). and They represent the first and the Features of a superpoint It represents Euclidean distance.

[0152] The third step is to maintain label consistency.

[0153] To ensure label consistency within each superpoint, a purity loss function is used. This constrains the variance of the label distribution, ensuring the homogeneity of labels within the superpoint. The mathematical expression of this loss function is: .

[0154] in, It's over the limit. Belongs to the The probability of a class It is the total number of points exceeding the limit. It represents the number of categories.

[0155] The fourth step is to calculate the total loss.

[0156] Ultimately, the super-point aggregation loss It is the weighted sum of the compactness loss, distinguishability loss, and purity loss mentioned above. Its total loss function is: .

[0157] This loss function optimizes the superpoint generation process by weighted summation of the above sub-loss terms, thereby achieving high-quality point cloud semantic segmentation.

[0158] Through the above steps, the present invention can effectively optimize the superpoint generation process, enhance the inter-class separability between superpoints, and maintain the label consistency within superpoints, thereby improving the performance of point cloud semantic segmentation, especially when dealing with complex scenes, it can better capture fine-grained semantic information.

[0159] Finally, the aforementioned loss function, combined with the cross-entropy loss function, is used as the loss function for training the point cloud oversegmentation and semantic segmentation networks to supervise network learning. The cross-entropy loss function is a common loss function used for point cloud semantic segmentation tasks.

[0160] Specifically, the cross-entropy loss function measures the difference between the predicted class distribution and the true class distribution. Its function definition is as follows: .

[0161] In the formula, It is the sample size. It is the number of categories. It is a sample Category The actual label (using one-hot encoding). It is a sample Predicted as category The probability of the predicted distribution is calculated using the cross-entropy loss function. In practical experiments, the cross-entropy loss function is often used in conjunction with the softmax activation function. The softmax function transforms the network output logits into a probability distribution, and then the cross-entropy loss is used to calculate the difference between the predicted distribution and the true distribution.

[0162] The superpoint aggregation loss function optimization scheme in this embodiment greatly enhances the distinguishability of features, making superpoints of different categories more separated in the feature space.

[0163] The following describes the model training process for "A Point Cloud Over-Segmentation Method Based on Sequence": Obtain the point cloud semantic segmentation dataset and divide it into training and training sets. test set and verification set Using the training set Complete point cloud segmentation network Supervised parameter training is performed in rounds. To ensure basic training effectiveness, In this experiment, ScanNet V2 was configured with a training epoch number of rounds. Each round uses a validation set Perform partial segmentation performance verification and save the network parameters corresponding to the rounds that yielded the best verification results. Use the test set. The performance of the trained point cloud segmentation network was tested to obtain the prediction results of points in each scene.

[0164] Before the point cloud data is input into the point cloud scene semantic segmentation model, preprocessing operations are performed on it. Preprocessing includes discarding, rotating, scaling, flipping, dithering, distorting and cropping. After processing, it is converted into npy and pth data formats and used as input data for the model.

[0165] The input dimension of the raw point cloud data is ,in, This indicates the number of points in the point cloud. This represents the original feature vector of each point. include coordinate, color.

[0166] The specific definitions of the linear layer, layer normalization, depthwise separable convolution (DWConv), and selective state-space model (SelectiveSSM) used in the model are as follows: A linear layer uses a linear layer to map point cloud coordinate information to feature information. .

[0167] In the formula, It is a weight matrix. It is the bias vector.

[0168] LayerNorm is a normalization technique used to accelerate the training of neural networks and improve their performance. The basic idea of ​​LayerNorm is to normalize the features of each input sample, rather than normalizing the features of mini-batch samples.

[0169] To verify the effectiveness of this solution, the following explanation is based on experimental data: In evaluating segmentation performance, in addition to visual perception, quantitative evaluation is even more important. This involves considering both training efficiency and segmentation effectiveness.

[0170] Regarding training efficiency: determined by training loss ( ) and training accuracy ( )Analyze training efficiency as the number of training sessions changes.

[0171] Regarding segmentation performance: the most classic evaluation metric, intersection-union ratio (IU), was selected. ), select Evaluation of segmentation performance for each semantic category.

[0172] Use the average crossover ratio (CLORD) ), and evaluate the overall segmentation effect.

[0173] The Intersection over Union (IoU) ratio represents the ratio of the intersection to the union of the prediction set and the label set. The formula is as follows: .

[0174] The confusion matrix, also known as the error matrix, is represented by an n*n matrix. An example is shown below.

[0175]

[0176] Precision (P) represents the ratio of the number of samples correctly classified into a certain class to the number of samples correctly classified into that class. .

[0177] Recall rate represents the proportion of correctly classified positive samples out of the total number of positive samples.

[0178] .

[0179] TP represents a sample with a positive label and a positive prediction, FN represents a sample with a positive label and a negative prediction, FP represents a sample with a negative label and a positive prediction, and TN represents a sample with a negative label and a negative prediction. Additionally, yes The average value. The experiment was tested and run on Ubuntu 22.04.4LTS, implemented using the PyTorch 1.12 deep learning framework, developed in PyCharm 2021.3.3, and computed on an Nvidia RTX 4090. The experimental environment configuration is shown in Table 1.

[0180] Table 1: Experimental Environment Configuration for the Invention

[0181] To implement this invention, the mainstream publicly available point cloud indoor scene dataset ScanNet V2 is used as the validation dataset. Experimental results on this dataset illustrate the specific implementation process. The ScanNet V2 dataset contains 1201 training scenes and 312 validation scans reconstructed from RGB-D frames. The model uses the reconstructed grid sampled point cloud as input, and each point is assigned a semantic label from one of 20 categories.

[0182] Table 2 presents the oversegmentation performance comparison results of the method of this invention with other state-of-the-art algorithms on the S3DIS Area5, nuScene, and SemanticKITTI datasets. The inventors used Oracle Overall Accuracy (OOA), Boundary Recall (BR), Boundary Precision (BP), and F1 score to evaluate superpoints. BR and BP are used to evaluate boundary quality, while OOA represents the upper limit of accuracy when using superpoints for semantic segmentation. The F1 score is used to balance recall and precision, and is defined as follows: In semantic segmentation tasks for indoor and outdoor scenes, the inventors used the mean intersection-union ratio (mIoU) as the evaluation metric.

[0183] Table 2: Comparison of the method of the present invention with other methods on the S3DIS Area5, nuScene, and SemanticKITTI datasets.

[0184] To more comprehensively evaluate the effectiveness of this invention, the inventors conducted experiments on three datasets and retained the ScanNet dataset for ablation studies to ensure that all methods had a consistent number of superpoints. Specifically, on the S3DISArea 5 dataset, methods such as VCCS, SPG, SSP, and SPNet generated approximately 1050 superpoints, matching the 1024 Hilbert partitions used by the inventors. The deep learning-based method proposed by the inventors outperforms traditional methods due to the ability of the specialized backbone network to extract rich geometric features that can capture complex local geometric structures. Unlike methods such as SPNet, which rely on FPS+KNN for presampling and soft assignment mapping and do not limit the clustering range of the original point set, resulting in blurred boundaries (e.g., ... Figure 3 (d) shows the method of the present invention. By utilizing the locality of Hilbert curves, the method limits the range of original point clustering, avoids the incorrect clustering of semantically similar but spatially distant points, and finally optimizes the superpoints through the cross-attention module, reducing noise and maintaining accurate and clear boundaries.

[0185] like Figure 3 As shown, this method can accurately maintain object boundaries while preserving superpoint aggregation within objects, especially in challenging scenarios such as wall whiteboard interfaces, thus achieving better performance in metrics such as BR (boundary recall) and BP (boundary precision). Furthermore, the inventors' method can effectively merge semantically continuous regions (e.g., floors, desktops), thereby reducing erroneous superpoint boundaries and significantly improving the BP metric.

[0186] Table 3: Comparison of the method of this invention with other methods on ScanNet V2, S3DIS Area5, nuScene, and SemanticKITTI datasets.

[0187]

[0188] Table 3 shows the evaluation of the inventors' method on four indoor and outdoor datasets, and compares it with point-based and superpoint-based methods. The inventors' optimized oversegmentation strategy shows a significant improvement: on S3DISArea 5, the inventors achieved a 3.2% improvement in mIoU compared to SPT. Compared with state-of-the-art point-based methods, the inventors' method is also competitive, outperforming PTv3 by 0.3% (reaching 77.8%) on ScanNet and PTv2 by 0.5% on S3DISArea 5. Furthermore, the inventors achieved scores of 71.9% and 80.2% on SemanticKITTI and nuScenes, respectively, outperforming SuperLiDAR by 1.3% and 1.7%, respectively. These results demonstrate that the inventors' superpoints can effectively capture both local and global semantic information. Furthermore, the concise backbone network incorporating superpoint features performs comparably to more complex point-based methods, validating the superior quality of the generated superpoints.

[0189] Table 4: Performance ratio of the method of the present invention on different backbone network architectures on the ScanNet validation set.

[0190]

[0191] Table 5: The impact of expanding the neighborhood segment range on Superpoint's performance on the ScanNet validation set.

[0192]

[0193] Tables 4 and 5 analyze the impact of backbone network design and neighborhood expansion on superpoint quality. The inventors evaluated three backbone network configurations: (1) using only the basic 3D sparse U-Net, (2) a U-Net based entirely on Mamba, and (3) the inventors' default hybrid design, which introduces a Mamba module in the bottleneck layer. The results show that, compared with the inventors' default configuration, the BR, BP, and OOA of the basic 3D sparse U-Net decreased by 7.90%, 1.70%, and 1.19%, respectively. Similarly, the U-Net based entirely on Mamba also performed poorly because Mamba smooths out high-frequency boundary information when modeling the global context in shallow networks. The inventors' hybrid architecture, "shallow convolution + deep Mamba," achieves optimal performance by using convolution to preserve local details in shallow layers and using Mamba modules to model the global context in deep layers.

[0194] In neighborhood expansion experiments (similar to k-parameter adjustment in KNN), the inventors observed that moderately expanding the candidate superpoint set of the original points significantly improves sampling accuracy during feature similarity-based assignment. The model performs best when K=2. Excessive expansion (K=4, 6) leads to performance degradation. This may be because a larger receptive field introduces noise, and the hard assignment strategy introduces more uncertainty as the number of candidate targets increases, causing early assignment errors to propagate to subsequent processing stages.

[0195] Experimental results on multiple large-scale indoor and outdoor datasets demonstrate that the method provided in this invention outperforms existing point cloud oversegmentation and semantic segmentation methods, highlighting its potential to improve segmentation performance in various practical applications and providing a new technical path for point cloud scene understanding.

[0196] Example 2: The present invention provides a point cloud over-segmentation device based on serialization, which includes a preprocessing module, a serialization module, a hierarchical module, a similarity module, a clustering module, an update module, and a segmentation module.

[0197] The preprocessing module is used to acquire point cloud datasets and perform preprocessing.

[0198] The serialization module is used to serialize preprocessed point cloud data into Hilbert curves.

[0199] The hierarchical module is used to divide the serialized point cloud data into multiple initial segments based on spatial continuity, so as to construct a hierarchical segmentation structure and obtain a multi-scale segment set.

[0200] The similarity module is used to calculate the feature similarity between adjacent segments and obtain a similarity matrix to provide a basis for subsequent super-point clustering.

[0201] The clustering module is used to perform super-point clustering on the initial segments based on an adaptive update algorithm to ensure semantic consistency within the super-points.

[0202] The update module is used to update the superpoint features through a cross-attention mechanism, establish dynamic relationships between points and superpoints, and obtain new superpoint structures.

[0203] The segmentation module is used to construct a superpoint map based on the updated superpoint features, enhance it through a graph convolutional network, fuse features from the backbone point cloud network with the corresponding multi-level superpoint features, and then pass it to the segmentation head for segmentation to obtain semantic segmentation output.

[0204] Example 3: This invention provides a serialization-based point cloud over-segmentation device, which includes a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a serialization-based point cloud over-segmentation method as described in any paragraph of Example 1.

[0205] Example 4: The present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute a point cloud over-segmentation method based on serialization as described in any paragraph of Example 1.

[0206] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0207] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0208] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0209] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0210] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0211] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0212] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0213] The terms "first" and "second" used in the embodiments are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0214] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A point cloud over-segmentation method based on serialization, characterized in that, Include: S1. Obtain the point cloud dataset and perform preprocessing; S2. Serialize the preprocessed point cloud data into Hilbert curves; S3. Based on spatial continuity, the serialized point cloud data is divided into multiple initial segments to construct a hierarchical segmentation structure and obtain a multi-scale segment set. S4. Calculate the feature similarity between adjacent segments and obtain a similarity matrix to provide a basis for subsequent super-point clustering; S5. Perform super-point clustering on the initial segments based on the adaptive update algorithm to ensure semantic consistency within the super-points; S6. Update the superpoint features through the cross-attention mechanism, establish dynamic relationships between points and superpoints, and obtain new superpoint structures; S7. Based on the updated superpoint features, construct a superpoint map and enhance it through a graph convolutional network. Then, fuse the features from the backbone point cloud network with the corresponding multi-level superpoint features, and pass it to the segmentation head for segmentation to obtain semantic segmentation output.

2. The point cloud over-segmentation method based on serialization according to claim 1, characterized in that, Step S1 specifically includes steps S11 to S12; S11. Perform voxelization on the input 3D point cloud data to obtain a sparse tensor format suitable for sparse convolution operations. S12. Input the voxelized point cloud into the U-Net architecture constructed using the sparse convolutional network SpConv to extract local detail features; wherein, the bottleneck part of the U-Net architecture integrates the Mamba module to enhance the overall structural perception of the point cloud through global context interaction, thereby compensating for the limitations of the local receptive field of sparse CNN. Step S2 specifically involves: transforming the preprocessed point cloud data into a one-dimensional sequence using Hilbert space-filling curve mapping, denoted as... .

3. The point cloud over-segmentation method based on serialization according to claim 1, characterized in that, Step S3 specifically includes steps S31 to S32; S31. Padded sequence to obtain padded sequence. ; ； ； In the formula, Represent real numbers, For the length after filling, The total dimension of the feature vector, For the number of points, The number of segments is the finest granularity, to ensure that the finest granularity segments can be... Perform uniform division; S32, using recursive quadrilateral partitioning to... Divided into Each segment is used to construct a hierarchical segmentation structure and obtain a multi-scale segment set; ； ； ； ； In the formula, For multi-scale segment sets, For the first The first in the layer Each segment Indicates the first The length of each segment in the layer, For the number of segments, The index of the point It is a preset constant. This is the index for the current level.

4. The point cloud over-segmentation method based on serialization according to claim 1, characterized in that, Step S4 specifically includes steps S41 to S42; S41. Constructing fine-grained layers based on hierarchical relationships with its parent layer The correspondence; where: Each coarse-grained segment A set of points containing four fine-grained segments; S42. For each original point in the coarse-grained superpoints, calculate the similarity matrix based on the cosine similarity between its features and the features of each fine-grained superpoint. The similarity matrix model is as follows: ; The elements of the similarity matrix are defined as follows: ; In the formula, For similarity matrix, For the elements of the similarity matrix, Represent real numbers, Indicates the first The number of original points contained in the coarse-grained superpoint layer. This indicates the number of fine-grained superpoints under the current coarse-grained superpoint. Features corresponding to the original point For fine-grained superpoint features, For the first point cloud One original point.

5. The point cloud over-segmentation method based on serialization according to claim 1, characterized in that, Step S5 specifically involves selecting the superpoint with the highest similarity to complete the point-to-superpoint assignment; wherein, each original point... Allocation index set It is determined by calculating its similarity to the superpoint, the mapping function. Represents the origin point The assigned superpoint index: ； In the formula, The index of the origin point, Indicates the first The number of original points contained in the coarse-grained superpoint.

6. The point cloud over-segmentation method based on serialization according to claim 1, characterized in that, Step S6 is as follows: The cross-attention mechanism adjusts the feature updates between the superpoint and its member points through dynamic adaptive relationships; where the superpoint features are used as queries, and the member point features are used as keys and values, and the superpoint features are updated by calculating a weighted sum. The updated formula is: ； In the formula, For the updated super-point features, For cross-attention mechanism, For the first Original features of each superpoint Features corresponding to the original point For the first point cloud One original point For mapping functions; Attention weights are calculated using scaled dot products, allowing each superpoint to focus on the most informative original point in its local neighborhood; the specific formula is as follows: ； ； ； ； ； In the formula, For the Softmax activation function, For query, For key, For value, Indicates transpose, For the query learnable projection matrix, The learnable projection matrix of the key, Learnable projection matrix with value Represent real numbers, Indicates the first The first in the layer The number of points in each segment The total dimension of the feature vector; Through multiple iterations, the final superpoint feature update result is obtained, and a new superpoint structure is constructed. .

7. The point cloud over-segmentation method based on serialization according to claim 1, characterized in that, Step S7 specifically includes steps S71 to S75; S71. Construct a superpoint graph based on the updated superpoint features; S72. Calculate the edge features of the super-vertex graph; ； In the formula, Features of the edges For a linear mapping function, and They represent the first Super point and the The coordinates of the superpoint; S73. After constructing the superpoint graph and its edge features, use a graph convolutional network to enhance the superpoint graph. S74. After processing by the graph convolutional network, the features from the backbone point cloud network are fused with the corresponding multi-level superpoint features. S75. Pass these fused features to the segmentation head for final semantic segmentation output.

8. A point cloud over-segmentation device based on serialization, characterized in that, Include: The preprocessing module is used to acquire point cloud datasets and perform preprocessing. The serialization module is used to serialize preprocessed point cloud data into Hilbert curves; The hierarchical module is used to divide the serialized point cloud data into multiple initial segments based on spatial continuity in order to construct a hierarchical segmentation structure and obtain a multi-scale segment set. The similarity module is used to calculate the feature similarity between adjacent segments and obtain a similarity matrix to provide a basis for subsequent super-point clustering. The clustering module is used to perform super-point clustering on the initial segments based on an adaptive update algorithm to ensure semantic consistency within the super-points. The update module is used to update the superpoint features through the cross-attention mechanism, establish dynamic relationships between points and superpoints, and obtain new superpoint structures. The segmentation module is used to construct a superpoint map based on the updated superpoint features, enhance it through a graph convolutional network, fuse features from the backbone point cloud network with the corresponding multi-level superpoint features, and then pass it to the segmentation head for segmentation to obtain semantic segmentation output.

9. A point cloud over-segmentation device based on serialization, characterized in that, It includes a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a serialization-based point cloud over-segmentation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform a serialization-based point cloud over-segmentation method as described in any one of claims 1 to 7.

Images