Point cloud segmentation method based on adaptive weight octree

Abstract

The invention discloses a point cloud segmentation method based on an adaptive weight octree, and the method specifically comprises the steps: firstly, preparing a point cloud data set containing any number and size, and carrying out the preprocessing of the point cloud data; at the same time, an octree data structure is expanded into a self-adaptive weight octree, and a self-adaptive weight octree model is designed. Thirdly, the adaptive weight octree model is trained; then, training is repeated until the preset number of iterations is reached, all scene point clouds are traversed in each round, and model parameters are stored for verification; and finally, loading the optimal model parameters selected in the training stage, and applying the optimal model parameters to a test data set or new point cloud input. In the reasoning stage, the model is used for outputting a segmentation result. The point cloud segmentation model based on the adaptive weight octree has excellent segmentation precision and capability of efficiently processing large-scale point cloud data, and is suitable for various practical application scenes.

Description

Technical Field

[0001] The present invention belongs to the field of 3D vision, and particularly relates to a point cloud segmentation method based on an adaptive weight octree. Background Art

[0002] With the continuous development of 3D radar sensing technology and data acquisition technology, 3D point cloud segmentation methods, which aim to divide a collected 3D point cloud into multiple parts or regions, each part usually corresponding to a specific object, surface, or semantic category, have become an important technology for understanding real scenes. They have shown broad application prospects in fields such as autonomous driving, robot environmental perception, and augmented reality and virtual reality (AR / VR).

[0003] PointNet is a pioneering work on point-based semantic segmentation methods. It first proposed directly learning features from raw point clouds, aggregating global features using permutation invariant operators, and retaining rich geometric detail information.

[0004] Subsequently, PointTransformer introduced a vector attention mechanism on this basis. Through position information encoding based on a multi-layer perceptron and a KNN downsampling module, it effectively achieved a reduction in point resolution and context feature modeling.

[0005] Although these methods perform well in 3D point cloud segmentation tasks and can effectively segment object categories, they still face the challenges of irregular and unordered point cloud data. This data characteristic results in insufficient accuracy in segmenting object edge details, and at the same time, the segmentation process is also time-consuming, limiting their application in scenarios with high-precision and real-time requirements. Summary of the Invention

[0006] Aiming at the deficiencies of several current point cloud segmentation methods and focusing on solving the problem of irregular and unordered point clouds, the present invention proposes a point cloud segmentation method based on an adaptive weight octree.

[0007] The main steps of the present invention are as follows:

[0008] Step 1: Prepare a point cloud data set containing any number and size, and preprocess the point cloud data.

[0009] Step 2: Expand the octree data structure into an adaptive weight octree and design an adaptive weight octree model.

[0010] Step 3: Train the adaptive weight octree model.

[0011] Step 4: Repeatedly execute Step 3 until a preset number of iterations is reached. Traverse all scene point clouds in each round and save the model parameters for verification.

[0012] Step 5: Load the optimal model parameters selected during the training phase and apply them to the test dataset or new point cloud input; during the inference phase, use the model output to obtain the segmentation result.

[0013] Based on the above technical solutions, the present invention has the following technical effects:

[0014] The present invention is applicable to scenarios and objects of different scales. Whether it is the overall segmentation of large objects or the fine extraction of small objects, it can be reasonably processed through the adaptive partitioning strategy, with strong adaptability and versatility.

[0015] Secondly, it can adjust the partitioning scale according to local geometric information, avoiding the boundary blurring problem that may be caused by fixed-level partitioning, being able to better maintain the object contour, and improving the accuracy of point cloud segmentation.

[0016] Finally, the present invention achieves the current optimal segmentation performance on the Scannet dataset, exceeding existing methods in key metrics such as mIoU (mean intersection over union), and showing stronger robustness especially in point cloud segmentation tasks in complex environments. (Scannet is a standard dataset widely used for indoor scene point cloud segmentation, containing various indoor object categories and being the most influential dataset for point cloud segmentation currently.) BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the architecture diagram of the adaptive weight octree model of the embodiment of the present application;

[0018] Figure 2 It is the extended label leaf node diagram of the embodiment of the present application;

[0019] Figure 3 It is the extended dynamic weight trunk diagram of the embodiment of the present application;

[0020] Figure 4 It is the TreeMamba semantic perception module diagram of the embodiment of the present application;

[0021] Figure 5 It is the model training flowchart of the embodiment of the present application;

[0022] Figure 6 It is the index comparison diagram of the embodiment of the present application;

[0023] Figure 7 It is the point cloud segmentation visualization diagram of the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] The implementation steps of the present invention will be further described below in conjunction with the accompanying drawings and specific examples:

[0026] As Figures 1-5 shown, the example of this application provides a point cloud segmentation method based on an adaptive weight octree, which specifically includes the following steps:

[0027] Step 1: Prepare a point cloud data set containing any number and size, and preprocess the point cloud data. Specifically:

[0028] Prepare a point cloud data set containing any number and size, and use Python to load the point cloud data set in.npz format, ensuring that each file contains the three-dimensional coordinates and color information of the point cloud.

[0029] After data loading, perform standardization of the point cloud size, uniformly sample the number of points in each point cloud data set to a fixed number of points to ensure data consistency. In particular, for the case where the number of points in the point cloud exceeds the specified number, a certain number of points can be selected through random sampling to ensure that the number of points in each point cloud is the same. If the number of points in the point cloud data is less than the specified number, the number of points can be supplemented by interpolation or other methods.

[0030] Normalize the coordinate and color information, and map its value to the range [0, 1] to reduce the impact of numerical fluctuations on training.

[0031] Align the point cloud coordinates along the X-axis to ensure the consistency of the geometric direction, and construct an octree data structure based on the point cloud distribution to provide efficient spatial index support for subsequent feature extraction.

[0032] Step 2: Expand the octree data structure into an adaptive weight octree and design an adaptive weight octree model. Specifically:

[0033] Select the dominant label from the multiple point label data contained in the leaf nodes of the original octree, and make this node its label leaf node.

[0034] Design a dynamic weight tree trunk, introduce a learnable weight mechanism, and automatically evaluate the importance of different input features through the semantic guiding role of the label leaf nodes, so as to ensure that the leaf nodes focus on the most semantically valuable feature information and increase the spatial perception ability of the point cloud data.

[0035] Design a TreeMamba semantic perception module, make full use of the linear complexity selection mechanism, not only greatly expands the scale of the local window, but also overcomes the computational bottleneck brought by the disorder of the point cloud, and can efficiently capture large-range up and down semantic features, making up for the limitations of existing methods in dealing with large-scale point cloud data.

[0036] Specifically, in order to make full use of the structured properties of the spatial data structure octree, the example of this application designs an adaptive weight octree model, and uses the adaptive weight octree structure combined with the TreeMamba semantic perception module to segment different point cloud data sets.

[0037] As Figure 1 shown, the adaptive weight octree model consists of an adaptive weight octree and a TreeMamba semantic perception module, where the adaptive weight octree consists of a label leaf node module and a dynamic weight trunk module.

[0038] Furthermore, as Figure 2 shown, regarding the label leaf node module, for the class label l of the point cloud ∈ R N , N is the number of point cloud samples. Assuming there are C classes, each label l i ∈ {0, 1,..., C - 1}, it is converted into matrix form using One - Hot encoding:

[0039] L = OneHot(l) ∈ R N×C

[0040] where L i,j = 1 indicates that the i - th point belongs to class j, otherwise it is 0.

[0041] Then, aggregate the class counts by voxel. Let I ∈ {0, 1,..., M - 1} N represent the voxel index of the points in the point cloud, with a total of M voxels. Aggregate the One - Hot label matrix L according to the voxel index to obtain the count matrix of each class in each voxel:

[0042]

[0043] where A ∈ R M×C is the class count matrix of the voxel, and A m,j represents the number of times class j appears in the m - th voxel.

[0044] Finally, calculate the dominant class of each voxel. For each voxel m, the dominant class is the class j with the largest class count.

[0045]

[0046] where y ∈ R M represents the dominant class label of each voxel.

[0047] Furthermore, as Figure 3As shown, regarding the dynamic weight trunk module, multi-level feature fusion of top-down and bottom-up is applied simultaneously to learn the complex spatial information of the point cloud, and learnable weights are introduced to learn the importance of different input features, so as to screen out the semantic features useful for the leaf nodes. The calculation formula for updating the weights of the adaptive weight octree:

[0048]

[0049] where new_ω i represents the weight coefficient after the update of each layer of the trunk, and Relu(old_ω i ) means applying the Relu function to the weight value old_ω i of the previous stage to ensure that the weights are non-negative and avoid the negative impact of the weights in the weighting operation. eps is an extremely small value, and in the experiment, eps = 0.0001 is mainly for numerical stability and to prevent abnormal situations.

[0050] Furthermore, the adaptive weight octree model mainly adopts the hierarchical feature fusion strategy of top-down in stage one and bottom-up in stage two. The feature fusion formula of top-down in stage one:

[0051]

[0052] where represents the h-layer feature of the initial input, ω 1 and ω 2 represent the learnable weight parameters of the trunk, represents the feature reconstruction by downsampling the (h + 1)-th layer in stage one, represents the h-layer feature obtained in stage one.

[0053] In the process of bottom-up feature fusion in stage two, an attempt is made to fit the virtual leaf nodes and make predictions with the labeled leaf nodes, dynamically adjusting the weight values of the weight tree trunk. This requires more semantic information fusion. Therefore, in this application example, not only the top-down hierarchical feature information obtained in stage one is considered, but also the original feature information in the decoding process is considered crucial and is added to the bottom-up feature fusion process in stage two. The feature fusion formula of bottom-up in stage two:

[0054]

[0055] where represents the h-layer feature of the initial input, ω 1 , ω 2 and ω 3 represent the learnable weight parameters of the trunk, represents the h-layer feature obtained after feature fusion in stage one, It represents the feature reconstruction by upsampling the (h - 1)-th layer in stage two. It represents the features of the h-th layer obtained in stage two.

[0056] Furthermore, as Figure 4 shown, the TreeMamba semantic perception module contains two feature extraction sub-modules: the LocalTreeMamba (LTM) module for local object information and the GlobalTreeMamba (GTM) module for global scene information. The original point cloud is sorted by the space-filling curve Z-Order and mapped into an adaptive-weight octree structure, thereby preserving the spatial locality of the point cloud and achieving its serialized representation. Define the mapping function AWOctree(·) as follows:

[0057] X AWOctree = AWOctree(X)

[0058] where represents the input point cloud data, and N is the number of points in the point cloud. When constructing the adaptive-weight octree, only the coordinate information of the point cloud is used, and X AWOctree represents the index of the point cloud data after being mapped by the adaptive-weight octree structure. Based on the ordered index X AWOctree of the tree structure, the local window division can be efficiently performed.

[0059] To achieve the GPU parallel computing efficiency, first perform the padding operation on the point cloud data. According to the structure of the adaptive-weight octree, at the current depth depth, define the number n num_t of valid non-empty nodes as:

[0060] n num_t = X AWOctree .n num_nempty [depth]

[0061] The goal is to organize these points into several windows according to the window division rule, with each window containing W points and padding to the number that meets the partition requirements. Therefore, the target number of points n num_a [depth] after padding can be calculated as follows:

[0062]

[0063] Subsequently, calculate the number of points num to be padded:

[0064] num = n num_a [depth] - n num_t [depth]

[0065] Next, construct a padding matrix T of size num×C, where all elements are fill_value, and concatenate the input data X and the padding matrix T along the 0th dimension to obtain the padded data:

[0066] X fill = cat(X, T, dim = 0)

[0067] According to the structure index of the adaptive weight octree, adjust the padded data to an array of shape Localsequence:

[0068] Localsequence = X AWOctree .reshape(X fill , [M, W, C])

[0069] where M represents the number of windows after partitioning, W is the window size, and C is the number of channel features. Next, the local sequence is input into the LocalTreeMamba module for item-level semantic perception processing:

[0070]

[0071] When considering the perception of the overall scene information, the GlobalTreeMamba module treats all voxels as a single group and sorts these voxels into a single sequence Globalsequence according to the structure of the adaptive weight octree:

[0072]

[0073] where B = 1 indicates that there is only one global sequence, L is the sequence length, and C is the number of channel features. Finally, the overall sequence Globalsequence is input into the GlobalTreeMamba module for scene-level semantic perception processing:

[0074]

[0075] Step 3: Train the adaptive weight octree model, specifically:

[0076] Augment the point cloud data to improve the generalization ability and robustness of the model, including adding coordinate non-linear transformation, random rotation, scaling, and position jitter to simulate the distortion and noise in the point cloud acquisition process.

[0077] Map the augmented point cloud data into the adaptive weight octree spatial data structure, extract features under the hierarchical division of the octree, and select the label leaf nodes to store the key semantic information.

[0078] The point cloud data passes through the TreeMamba semantic perception module, which uses multi-scale feature integration and long-distance context modeling strategies to overcome the computational bottleneck caused by the disorder of the point cloud and extract the global point cloud feature map information.

[0079] These features are fed into the dynamic weight trunk, and the weight values of the trunk are dynamically adjusted through the semantic guiding role of the labeled leaf nodes to strengthen the information expression of key regions, further refine the feature information, and improve the model's understanding ability of the point cloud spatial distribution.

[0080] The decoder performs layer-by-layer decoding and fusion on the extracted multi-scale features, and finally generates a per-point classification segmentation result. The model uses the Cross-Entropy Loss function to calculate the error between the prediction result and the ground truth:

[0081]

[0082] where p c is the probability that the model predicts to belong to class c, and y c is the ground truth prediction probability.

[0083] Furthermore, in order to comprehensively consider the semantic losses of the original points and the leaf nodes, the example of this application uses weighted summation to calculate the total loss function. Let the loss functions of the point task L point and multiple labeled leaf node tasks L leaf,m , m = 1,..., M be L point and L leaf,m respectively, where M represents the number of labeled leaf node tasks. To balance the contributions of each task to the total loss, a dynamic weight ω point and ω leaf,m are assigned to each task. These weights are dynamically adjusted according to the loss values of each task, so that tasks with larger losses can receive more attention during training. The total loss function L total can be expressed in the following weighted summation form:

[0084]

[0085] where L point is the loss of the point task, L leaf,m is the loss of the m-th labeled leaf node task, and ω point and ω leaf,m are the dynamic weights of the point task and the labeled leaf node task respectively. Specifically, the dynamic weight ω point of the point task and the dynamic weight ω leaf,m of each subtask of the labeled leaf node task are calculated through the following formulas respectively:

[0086]

[0087] Through this weighted summation method, the model can adaptively adjust its learning weights according to the importance of each weighted trunk, ensuring that during the training process, tasks with larger losses can guide the model to pay more attention to their features, thereby effectively improving the performance and generalization ability of the model.

[0088] Finally, all parameters of the model are optimized through error backpropagation, enabling it to continuously learn and improve its classification performance during the training process.

[0089] Step 4: Repeat Step 3 until the preset number of iterations is reached. In each round, traverse all scene point clouds and save the model parameters for verification. After each round of training, use an independent validation set to evaluate the performance of the model. By calculating the mean Intersection over Union (mIoU) metric of the generated segmentation results, measure the performance of the model in the point cloud segmentation task. The validation stage utilizes the multi-scale feature extraction advantages of the dynamic weight octree and the TreeMamba module to ensure that the model still has accurate semantic segmentation ability in high-complexity scenarios. Save the model parameters of each round according to the validation results, and finally select the set of parameters with the highest mIoU metric as the best model to avoid overfitting and optimize the generalization ability.

[0090] In the example of this application, the entire training process consists of 700 training epochs. In each epoch, traverse the point cloud data of all scenes and regularly save the model parameters for subsequent verification and testing to ensure the generalization ability and robustness of the model. At the same time, divide a part of the original dataset into a validation set, and after each round of training, use the validation set for prediction and calculate the mean Intersection over Union (mIoU) metric of the generated point cloud labels as the evaluation criterion for the model performance. This metric is widely used in semantic segmentation tasks. By measuring the ratio of the intersection to the union between the prediction results of each category and the ground truth labels, the overall accuracy is then calculated. Specifically, mIoU is obtained by summing and averaging the Intersection over Union (IoU) values of each category. The calculation formula is as follows:

[0091]

[0092] Among them, TP represents the correct prediction (true positive) of the true category i, FN is the misprediction of the true category i as other categories (false negative), FP is the misprediction of other categories as i (false positive), and C represents the total number of categories.

[0093] Step 5: Load the best model parameters selected in the training stage and apply them to the test dataset or new point cloud input; in the inference stage, use the model to output the segmentation results. Specifically:

[0094] After completing the model training and evaluating the performance of each round, the mean intersection over union (mIoU) parameter set with the best performance on the validation set is selected and loaded into the model. Through the dynamic weight octree structure and the TreeMamba module, the model can quickly process large-scale point cloud data and generate per-point classification results. Subsequently, different point cloud input data are used to infer the model, generating corresponding segmentation results. This process utilizes the efficient computing and adaptive characteristics of the model to output segmentation results with high resolution and accuracy. By precisely tuning the model parameters, it is ensured that the model can achieve the best segmentation effect on unseen data, thereby improving the generalization ability and accuracy of the model in practical applications.

[0095] According to the application requirements, if it is necessary to improve the performance of specific tasks, the training process or module design can be adjusted, and the above steps can be repeated to achieve better segmentation performance.

[0096] Specific experimental data and parameters:

[0097] The specific comparison results with other current point cloud segmentation models are as Figure 6 shown. The first column represents the current point cloud segmentation model, the second column represents the mIoU value of point cloud segmentation Val, and the third column represents the mIoU value of point cloud segmentation Test. The results show that the segmentation effect of this application is better than the previous methods. The specific experimental details are as follows:

[0098] 1) The training data used in this experiment comes from the ScanNetV2 point cloud dataset, and the test set is a part divided from this dataset for evaluating the model performance. The ScanNetV2 dataset is a large-scale 3D reconstruction and semantic segmentation dataset, aiming to promote the understanding of indoor scenes. The dataset contains 2.5 million views generated from more than 1500 scans, with a total of 1513 scenes. The dataset covers 21 different categories, among which 1201 scenes are used for training and 312 scenes are used for testing, providing rich diversity and complexity for the model. In terms of data processing and encoding, the point cloud data is first normalized with a scale factor of 0.01m, and then encoded using an octree with a depth of 11. The deepest 11 layers are used as the leaf nodes of the labels. The data augmentation process includes the following methods: random rotation in the interval [-180°, 180°], random scaling in the interval [0.75, 1.25], and random translation in the interval [-0.1, 0.1].

[0099] 2) The model parameters that achieved the highest mIoU value during training were selected in this experiment and comprehensively evaluated on the ScanNetV2 dataset. The experimental results show that the mIoU values of the validation set and the test set reached 76.7 and 76.8 respectively. Compared with other existing point cloud segmentation methods, the model described in this paper demonstrated a significant improvement in performance, indicating its strong advantages in processing complex point cloud data.

[0100] 3) Figure 7 The visualization results of the point cloud segmentation method proposed in this experiment are shown, which are significantly better than other existing models. The experimental results show that in different indoor scenes, the model can accurately capture the details of local objects, such as tables, chairs, sofas, etc., indicating that the model has excellent generalization ability. Good generalization is crucial for the practical application of point cloud segmentation models, especially when dealing with unknown point cloud scenarios. It can be seen that this application not only demonstrated excellent performance within the dataset but also showed outstanding generalization ability in the segmentation task of external point cloud data, further proving its wide applicability in practical applications.

[0101] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention and are not limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or variations can be made based on the above description. It is impossible to enumerate all the implementation manners here. Any obvious changes or variations derived from the technical solutions of the present invention still fall within the protection scope of the present invention.

Claims

1. A point cloud segmentation method based on adaptive weighted octree, characterized in that: The following steps are involved: Step 1: Prepare a point cloud dataset containing any number and size, and preprocess the point cloud data; Step 2: Expand the octree data structure to an adaptive weighted octree and design an adaptive weighted octree model; Step 3: training the adaptive weighted octree model; Step 4: Repeat step 3 until the preset number of iterations is reached, traverse all scene point clouds in each round, and save the model parameters for verification; Step 5: Load the best model parameters selected in the training phase and apply them to the test dataset or new point cloud input; in the inference phase, use the model to output the segmentation results.

2. The point cloud segmentation method based on adaptive weighted octree according to claim 1, characterized in that: The step 1 comprises: Prepare a point cloud dataset containing any number and size, each file contains the 3D coordinates and color information of the point cloud; Standardize the point cloud data and uniformly sample the point cloud to a fixed number of points; Normalize the coordinate and color information; Align the point cloud coordinates on the X-axis and build an octree data structure based on the point cloud distribution.

3. The point cloud segmentation method based on adaptive weighted octree according to claim 1, characterized in that: The adaptive weighted octree model includes an adaptive weighted octree and a TreeMamba semantic perception module; The adaptive weighted octree is used to aggregate semantic information at leaf nodes through the interaction of labeled leaf nodes and dynamic weighted trunks to correctly refine the interpolation features of a single point cloud; The TreeMamba semantic perception module is used to expand the scale of the local window and capture the semantic features of a wide range of contexts by utilizing the linear complexity selection mechanism of Mamba.

4. The point cloud segmentation method based on adaptive weighted octree according to claim 3, characterized in that: The adaptive weighted octree includes a label leaf node module and a dynamic weighted trunk module; The label leaf node module is used to select a dominant label according to multiple point label data contained in the original octree leaf node, and make the node its label leaf node; The dynamic weight tree trunk module is used to introduce a learnable weight mechanism, automatically evaluate the importance of different input features through the semantic guidance of label leaf nodes, and screen semantic features that are useful for leaf nodes.

5. The point cloud segmentation method based on adaptive weighted octree according to claim 4, characterized in that: The label leaf node module performs the following operations: Use One-Hot encoding to convert the category labels of the point cloud into matrix form; Aggregate category counts by voxel, aggregate the One-Hot label matrix by voxel index, and obtain the count matrix of each category in each voxel; Compute the dominant class for each voxel.

6. A point cloud segmentation method based on adaptive weighted octree according to claim 3 or 5, characterized in that: The TreeMamba semantic perception module includes a LocalTreeMamba module and a GlobalTreeMamba module; The LocalTreeMamba module is used to perform item-level semantic perception processing on local sequences; The GlobalTreeMamba module is used to perform scene-level semantic perception processing on the entire sequence.

7. The point cloud segmentation method based on adaptive weighted octree according to claim 1, characterized in that: The adaptive weighted octree model adopts a top-down hierarchical feature fusion strategy in stage one and a bottom-up hierarchical feature fusion strategy in stage two; In the bottom-up feature fusion process of stage 2, virtual leaf nodes are fitted and predicted with label leaf nodes, and the weight values ​​of the weight tree trunk are dynamically adjusted.

8. The point cloud segmentation method based on adaptive weighted octree according to claim 1, characterized in that: The step 3 comprises: Enhance point cloud data to simulate distortion and noise during point cloud acquisition; Map the enhanced point cloud data to the adaptive weighted octree spatial data structure, extract features, and select label leaf nodes to store key semantic information; The point cloud data passes through the TreeMamba semantic perception module to extract the global point cloud feature map information; These features are passed into the dynamic weight tree trunk, and the weight value of the tree trunk is dynamically adjusted through the semantic guidance of the label leaf nodes to further refine the feature information; The decoder decodes and fuses the extracted multi-scale features layer by layer to generate point-by-point classified segmentation results; The learning weights are adaptively adjusted using the loss function, and the model parameters are optimized through back-propagation.

9. The point cloud segmentation method based on adaptive weighted octree according to claim 8, characterized in that: The loss function is calculated by weighted summation: Among them, L point is the loss of the task, L leaf,m is the loss of the mth label leaf node task; ω point is the dynamic weight of the point task, ω leaf,m is the dynamic weight of the mth label leaf node task, calculated by the following formula:

10. The point cloud segmentation method based on adaptive weighted octree according to claim 1, characterized in that: The step 4 comprises: Repeat step 3 until the preset number of iterations is reached, traversing the point cloud data of all scenes in each round and saving the model parameters regularly; Divide a part of the original data set into a validation set, and use the validation set to evaluate the performance of the model after each round of training; The performance of the model in the point cloud segmentation task is measured by calculating the average intersection-over-union ratio of the generated segmentation results.

Images