A fast point cloud instance segmentation method for open disaster scenarios
By using a sparse point cloud decoder and a continuous learning framework, the problems of slow point cloud instance segmentation and insufficient generalization ability in open disaster scenarios are solved, achieving fast and accurate point cloud instance segmentation, supporting disaster assessment and other 3D vision tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING AEROSPACE INST FOR METROLOGY & MEASUREMENT TECH
- Filing Date
- 2025-11-21
- Publication Date
- 2026-06-05
AI Technical Summary
Existing methods are slow in segmenting point cloud instances in open disaster scenarios, making it difficult to meet urgent needs, and their generalization ability is insufficient, especially when facing unknown categories.
A sparse point cloud decoder with center-oriented assistance is adopted, which combines sparse convolutional networks and Transformer structures. The sparse point cloud decoder is designed through self-attention and asymmetric attention interaction. A learnable embedding scoring filter is used to improve training convergence speed and prediction efficiency. The model's generalization ability is enhanced by combining a continuous learning fine-tuning framework.
It achieves rapid point cloud instance segmentation in open disaster scenarios, improves the training convergence speed and prediction efficiency of the model, can accurately generate instance masks and classification information, supports disaster assessment, and has the potential for transfer and application to other open-world 3D vision tasks.
Smart Images
Figure CN122156598A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of three-dimensional data processing, specifically relating to a method for rapid point cloud instance segmentation for open disaster scenarios. Background Technology
[0002] With the rapid development of disaster prevention and mitigation technologies, 3D point cloud data, due to its comprehensive advantages such as high accuracy, all-weather operation, and anti-interference, can extract high-precision disaster scene information, effectively supporting disaster prevention and mitigation applications. However, because the original disaster 3D point cloud data is large in volume and contains many types of objects, direct manual analysis and decision-making is inefficient and ineffective. Utilizing computer algorithms to process point cloud data can provide efficient and accurate assistance for disaster assessment and decision-making. As a key technology for identifying object types and segmenting instance shapes, point cloud instance segmentation is an important part of this process.
[0003] Currently, point cloud instance segmentation research mainly falls into two categories: traditional methods and deep learning-based methods. Traditional methods primarily rely on manually designed feature extraction methods, i.e., segmentation based on low-dimensional features such as color. These methods can achieve some success in general scenarios (such as indoor environments), but disaster scenarios, as a special type of scenario, are characterized by complex environments, diverse elements, and variable features, making it difficult for traditional methods to achieve good results in such open scenarios. Deep learning-based point cloud instance segmentation typically uses implicit fitting functions in neural networks, leveraging point cloud data to drive the network, self-learning complex patterns and high-dimensional features to generate item classifications and instance masks. However, most deep learning-based methods are implemented in closed scenarios, only on indoor and street scene data with known categories. Their model generalization ability and robustness remain significantly insufficient when facing unknown classes in open disaster scenarios. While some methods have been proposed for point cloud instance segmentation in open scenarios, they still have considerable drawbacks. It uses submanifold sparse convolution for feature extraction, generating features that are typically highly sparsity. However, it lacks methods for fast decoding that leverage this sparsity, and it is also difficult to converge, often requiring numerous training epochs and consuming significant training time. These speed limitations pose challenges to the practical use of the algorithm.
[0004] In summary, point cloud instance segmentation methods for open disaster scenarios are still in their early stages and have not yet met the urgent needs of real-world disaster scenarios in terms of running speed. Summary of the Invention
[0005] To address the problems and shortcomings of existing methods, this invention proposes a fast point cloud instance segmentation method for open disaster scenarios. The aim is to solve the problem of rapid point cloud instance segmentation in open disaster scenarios. Based on the sparsity of point cloud features, a sparse point cloud decoder with center-location assistance is constructed to accelerate network training convergence and prediction inference speed, enabling the application of advanced technologies such as point cloud instance segmentation in emergency disaster scenarios.
[0006] The technical solution for implementing the present invention is as follows: A fast point cloud instance segmentation method for open disaster scenarios includes the following steps: The input point cloud data is voxelized to generate a multidimensional tensor; The tensor is input into a submanifold sparse convolutional network to extract point cloud features and predict center positions. The point cloud features and center position predictions are input into a sparse point cloud decoder for decoding. The sparse point cloud decoder maintains and iteratively updates a set of feature embeddings and a set of center position embeddings, and uses an attention mechanism to perform interactive calculations between the feature embeddings, center position embeddings, point cloud features and center position predictions, and outputs the final feature embeddings and final center position embeddings after decoding. Based on the final feature embedding and the final center position embedding, predict the center point coordinates, instance category, and instance mask for each instance; The network prediction results are supervised and trained using a combined loss function until the model converges.
[0007] Furthermore, the sparse point cloud decoder is composed of multiple stacked layers, each layer comprising: A self-attention decoding layer is used to perform self-attention calculations on the feature embedding and the center position embedding respectively, and interact with the point cloud features and center position prediction based on the calculation results to complete the initial embedding update; An asymmetric attention decoding layer is used to enable the feature embedding to interact with the point cloud features and to enable the center position embedding to interact with the center position prediction. Then, the result of the feature interaction is used to query the result of the position interaction to complete the second update of the feature embedding. A learnable embedding scoring and filtering layer is used to score the importance of the updated embeddings and select some important embeddings based on the scoring results to pass to the next layer.
[0008] Furthermore, the operation of the self-attention decoding layer includes: Calculate the self-attention output of the center location embedding and the self-attention output of the feature embedding; The two self-attention outputs are concatenated to generate a combined query vector; The predicted center location is concatenated with the point cloud features to generate a combined key-value pair; Attention is calculated on the combined key-value pairs using the combined query vector, and the embedding is updated through a feedforward network.
[0009] Furthermore, the operation of the asymmetric attention decoding layer includes: Calculate the attention output between the center location embedding and the center location prediction as the location interaction result; The attention output between the feature embedding and the linearly transformed point cloud features is calculated as the feature interaction result; Using the feature interaction results as a query, attention is calculated on the location interaction results to update the feature embedding.
[0010] Furthermore, the operation of the learnable embedding scoring filtering layer includes: Use one or more perceptrons to calculate an importance score for each embedded element; The importance scores are processed using the Gumbel-Softmax technique to achieve differentiable Top-K selection during training. The selected K important embeddings are added to their corresponding positional codes and then output to the next decoding layer.
[0011] Furthermore, the prediction step is implemented in the following way: The center point coordinates of the instance are obtained by adding the final center position embedding to the final feature embedding after it has been transformed by a multilayer perceptron. The instance mask is obtained by performing matrix multiplication and Sigmoid activation on the result of the final feature embedding and the point cloud features transformed by another multilayer perceptron. The instance category is obtained by transforming the final feature embedding through a third multilayer perceptron.
[0012] Furthermore, the combined loss function is a weighted sum of the center point localization loss, the instance classification loss, and the mask prediction loss, wherein the mask prediction loss includes both the Dice loss and the binary cross-entropy loss.
[0013] Furthermore, before applying it to open disaster scenarios, there is a continuous learning fine-tuning stage. In this stage, regular scene point cloud data and disaster scene point cloud data are mixed to fine-tune the pre-trained network, so as to enhance the model's ability to identify unknown categories in disaster scenarios.
[0014] Beneficial effects: 1. This invention effectively solves the problem of insufficient generalization ability of existing methods when facing unknown and multi-category objects in disaster scenarios by introducing a fine-tuning framework based on continuous learning and designing a dedicated network structure. It can accurately generate instance masks and classification information of various objects at the disaster site, providing key technical support for disaster assessment.
[0015] 2. To address the sparsity of point cloud data, this invention designs a sparse point cloud decoder that includes a learnable embedding scoring filter. This design can dynamically identify and retain important feature embeddings, significantly reducing the computational load of subsequent decoding layers, thereby accelerating the convergence process of network training and significantly improving the predictive inference efficiency of the model during deployment, better meeting the stringent timeliness requirements of emergency disaster relief.
[0016] 3. The core of this invention, a center-oriented sparse point cloud decoder, achieves efficient and deep interaction between instance location information and feature information through the collaborative design of self-attention and asymmetric attention. This unique "feature query location" mechanism enables the model to more accurately locate the instance center and perceive its boundary, thereby obtaining a clearer and more accurate segmentation mask in complex backgrounds.
[0017] 4. The technical solution proposed in this invention is not limited to disaster scenarios. It is pre-trained using a conventional urban point cloud dataset, and then fine-tuned to adapt to specific scenarios. Through a modular decoder design, this method has the potential to be transferred to other open-world 3D vision tasks (such as autonomous driving and robot navigation), demonstrating broad application prospects. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating the method of the present invention. Detailed Implementation
[0019] This invention provides a fast point cloud instance segmentation method for open disaster scenarios, such as... Figure 1 As shown, the detailed technical solution is as follows: Step 1: Pre-train the network using a standard city point cloud dataset and its corresponding labels. The point cloud space is then voxelized to obtain a six-dimensional tensor for input to the network. ,in, Represents voxel coordinates, The color represents the voxel. Typical urban point cloud datasets can generally be categorized into several types, such as roads, buildings, vegetation, vehicles, and other urban components. Therefore, existing 3D urban point cloud datasets, such as UrbanBIS, should be used.
[0020] Step 2: Input the tensor into the submanifold sparse convolutional network, which only focuses on non-empty voxels during computation. Let the weight of a certain convolutional kernel be denoted as... The input is The output is The calculation formula is as shown in (1).
[0021]
[0022] Step 3: Obtain features and predict the center location using a convolutional network, denoted as follows: and The input is fed into a center-oriented sparse point cloud decoder, which consists of a multi-layer Transformer structure and maintains a set of feature embeddings. Embedded with a set of central positions The specific calculation steps are as follows: 1) Self-attention layer: In the self-attention decoding layer, the two embeddings calculate self-attention separately, concatenate them to generate a query, and then use the position. With features Generate key-value computation attention, the first The formula for calculating the layer is as shown in (2).
[0023]
[0024]
[0025]
[0026]
[0027]
[0028] In the formula, Dimensions representing keys and values, It's splicing. For the softmax function, It is a feedforward network.
[0029] 2) Asymmetric Attention Layer: The asymmetric attention decoding layer calculates attention using features and feature embeddings, and positions and position embeddings respectively. It then uses feature embeddings to generate queries and position embeddings to generate keys and values, and calculates and updates the feature embeddings. The formula for calculating the layer is as shown in (3).
[0030]
[0031]
[0032]
[0033]
[0034]
[0035] In the formula, It is a linear transformation matrix used to compress eigenvectors.
[0036] 3) Learnable Embedding Scorer: This function scores the embeddings and selects the top k highest-scoring embedding elements for subsequent decoding layers. The scorer uses a multilayer perceptron to calculate the score for each embedding element. After calculation, a gumble-softmax algorithm is used to generate soft probabilities. First, to ensure that the distribution after extreme value sampling is consistent with the distribution before sampling, gumble noise is added to the embedding elements. The formula is shown in equation (4).
[0037]
[0038] in, It follows a uniform distribution. For embedded elements with added noise, soft probabilities are calculated using softmax. The first k samples obtained by the soft probabilities are used in the forward propagation process, while the soft probabilities are used instead in the backward propagation process. The selected elements are added to the original position encoding to form a new embedding, which is then input into the next decoding layer.
[0039] Stacking the above processes forms a sparse point cloud decoder with center-location assistance. After decoding, the feature and center location predictions are obtained as the final feature embedding and center location embedding.
[0040] Step 4: Use feature embedding and center position embedding to calculate and obtain the center position, mask, and instance category, as shown in Equation (5):
[0041]
[0042]
[0043] In the formula, These represent the predicted center point location, mask, and instance category, respectively. Represents a multilayer perceptron. This represents the sigmoid function.
[0044] Step 5: For the prediction results in step 4, the network uses a loss function for backpropagation, as shown in equation (6).
[0045]
[0046]
[0047]
[0048]
[0049]
[0050] in, These represent the actual center point location, mask, and instance type, respectively.
[0051] Step 6: Repeat steps 1-5 until the network converges, completing the pre-training.
[0052] Step 7: Fine-tune the network using a continuous learning framework based on example-based point cloud instance segmentation, using a disaster scenario point cloud dataset. Mix regular urban point cloud data with disaster scenario data to create a new dataset. Use Equation (6) as the loss for iteration until the network converges, completing the network's continuous learning in the disaster scenario. At this point, the network directly obtains the type and instance number of each point from the point cloud input and saves it to a file. In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Contents not described in detail in this specification are common knowledge to those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A fast point cloud instance segmentation method for open disaster scenarios, characterized in that, Includes the following steps: The input point cloud data is voxelized to generate a multidimensional tensor; The tensor is input into a submanifold sparse convolutional network to extract point cloud features and predict center positions. The point cloud features and center position predictions are input into a sparse point cloud decoder for decoding. The sparse point cloud decoder maintains and iteratively updates a set of feature embeddings and a set of center position embeddings, and uses an attention mechanism to perform interactive calculations between the feature embeddings, center position embeddings, point cloud features and center position predictions, and outputs the final feature embeddings and final center position embeddings after decoding. Based on the final feature embedding and the final center position embedding, predict the center point coordinates, instance category, and instance mask for each instance; The network prediction results are supervised and trained using a combined loss function until the model converges.
2. The method according to claim 1, characterized in that, The sparse point cloud decoder is composed of multiple stacked structures, each layer comprising: A self-attention decoding layer is used to perform self-attention calculations on the feature embedding and the center position embedding respectively, and interact with the point cloud features and center position prediction based on the calculation results to complete the initial embedding update; An asymmetric attention decoding layer is used to enable the feature embedding to interact with the point cloud features and to enable the center position embedding to interact with the center position prediction. Then, the result of the feature interaction is used to query the result of the position interaction to complete the second update of the feature embedding. A learnable embedding scoring and filtering layer is used to score the importance of the updated embeddings and select some important embeddings based on the scoring results to pass to the next layer.
3. The method according to claim 2, characterized in that, The operations of the self-attention decoding layer include: Calculate the self-attention output of the center location embedding and the self-attention output of the feature embedding; The two self-attention outputs are concatenated to generate a combined query vector; The predicted center location is concatenated with the point cloud features to generate a combined key-value pair; Attention is calculated on the combined key-value pairs using the combined query vector, and the embedding is updated through a feedforward network.
4. The method according to claim 2, characterized in that, The operations of the asymmetric attention decoding layer include: Calculate the attention output between the center location embedding and the center location prediction as the location interaction result; The attention output between the feature embedding and the linearly transformed point cloud features is calculated as the feature interaction result; Using the feature interaction results as a query, attention is calculated on the location interaction results to update the feature embedding.
5. The method according to claim 2, characterized in that, The operations of the learnable embedded scoring filtering layer include: Use one or more perceptrons to calculate an importance score for each embedded element; The importance scores are processed using the Gumbel-Softmax technique to achieve differentiable Top-K selection during training. The selected K important embeddings are added to their corresponding positional codes and then output to the next decoding layer.
6. The method according to any one of claims 1-5, characterized in that, The prediction step is implemented in the following way: The center point coordinates of the instance are obtained by adding the final center position embedding to the final feature embedding after it has been transformed by a multilayer perceptron. The instance mask is obtained by performing matrix multiplication and Sigmoid activation on the result of the final feature embedding and the point cloud features transformed by another multilayer perceptron. The instance category is obtained by transforming the final feature embedding through a third multilayer perceptron.
7. The method according to claim 1, characterized in that, The combined loss function is a weighted sum of centroid localization loss, instance classification loss, and mask prediction loss, wherein the mask prediction loss includes both Dice loss and binary cross-entropy loss.
8. The method according to claim 1, characterized in that, Before being applied to open disaster scenarios, a continuous learning fine-tuning stage is also included. In this stage, point cloud data from regular scenarios are mixed with point cloud data from disaster scenarios to fine-tune the pre-trained network, thereby enhancing the model's ability to identify unknown categories in disaster scenarios.