Construction and segmentation method of point cloud instance segmentation model based on non-local attention
By using a point cloud instance segmentation model based on non-local attention, and leveraging SE-enhanced sparse 3D U-Net and a non-local attention decoder for long-range dependency modeling, this approach solves the problems of high computational complexity and poor segmentation results caused by occlusion in traditional point cloud instance segmentation methods, achieving efficient instance segmentation.
Patent Information
- Application Number
- CN202511600846.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-04
- Publication Date
- 2026-03-03
AI Technical Summary
Existing point cloud instance segmentation methods suffer from high computational complexity and poor segmentation results due to object occlusion.
A point cloud instance segmentation model based on non-local attention is adopted. Point-level features are extracted by SE-enhanced sparse 3D U-Net, and long-range dependency modeling is performed by combining group pooling layers and non-local attention decoders. Instance segmentation is performed using learnable query vectors and dynamic attention masks.
The data structure is simplified, key information is preserved, the segmentation effect is improved, the computational complexity is reduced, and the problem of incomplete segmentation or incorrect merging caused by object occlusion is solved.
Smart Images
Figure CN121600255A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and 3D data processing technology, and relates to a point cloud instance segmentation method, specifically a point cloud instance segmentation model construction and segmentation method based on non-local attention. Background Technology
[0002] Point cloud instance segmentation is a key technology in the field of 3D vision, playing a crucial role in applications such as autonomous driving and robot navigation. Current mainstream methods mainly rely on local feature aggregation or global attention mechanisms. The former struggles to model long-distance geometric relationships, while the latter suffers from high computational complexity. Especially in real-world scenarios, object occlusion limits the segmentation performance of existing methods.
[0003] Especially in real-world scenarios, when dealing with occluded objects, the visible parts are spatially discontinuous, preventing the model from correctly associating them using the overall geometric priors of the object. This leads to problems such as incomplete instance segmentation or incorrect merging. Therefore, it is evident that semantic association between instances in complex scenes and improving processing efficiency are of great value.
[0004] To address this situation, there is an urgent need to propose a method for constructing and segmenting point cloud instance segmentation models based on non-local attention. Through dynamic range constraints and feature optimization strategies, efficient long-range dependency modeling is achieved. Summary of the Invention
[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method for constructing and segmenting a point cloud instance segmentation model based on non-local attention, thereby solving the technical problems of high computational complexity and poor segmentation results caused by object occlusion in existing point cloud instance segmentation methods.
[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: On the one hand, a method for constructing a point cloud instance segmentation model based on non-local attention is provided, which includes the following steps: Step 1: Collect all raw point cloud data in a scene and preprocess the raw point cloud data to obtain voxelized point cloud and superpoint labels; The raw point cloud data includes geometric information and instance labels; Step 2: Construct a point cloud instance segmentation model; The point cloud instance segmentation model comprises a sparse 3D U-Net based on SE enhancement, a group pooling layer, and a decoder layer connected in sequence; the decoder layer comprises a non-local attention decoder and a prediction head. The SE-enhanced sparse 3D U-Net is used to extract features from all voxelized point clouds to obtain point-level features. The grouped pooling layer is used to group point-level features according to superpoint labels and perform average pooling to obtain multiple superpoint-level features. The nonlocal attention decoder creates a set of learnable query vectors and performs multi-level attention calculations on the super-point level features and the learnable query vectors to obtain optimized query vectors. The prediction head is used to decode the optimized query vector to obtain the instance class probability distribution of the voxelized point cloud, select the instance class with the highest probability, and use the instance class corresponding to the highest probability as the segmentation result. Step 3: Take the voxelized point cloud obtained in Step 1 as input, take the instance class corresponding to the maximum probability of the voxelized point cloud in the class probability distribution as output, take the instance label as the real label for supervision, and train the point cloud instance segmentation model constructed in Step 2 with binary cross-entropy loss and DICE as joint loss functions to obtain the trained point cloud instance segmentation model. This invention also includes the following technical features: In step 1, the preprocessing specifically includes: 1.1 The original point cloud data is enhanced by geometric transformation and color perturbation to obtain enhanced point cloud data; 1.2. Based on geometric information, perform superpoint segmentation on the enhanced point cloud data to obtain superpoint labels, and perform voxelization on the enhanced point cloud data to obtain voxelized point cloud.
[0007] The SE-enhanced sparse 3D U-Net includes an initial module, a downsampling module, an upsampling module, and a feature fusion module connected in sequence. The initial module includes two residual blocks.
[0008] The nonlocal attention decoder comprises six sequentially connected layers, each layer including a connected cross attention layer, a self attention layer, a feedforward network layer, and a nonlocal attention layer. The feedforward network layer consists of two linear layers and an activation function; The nonlocal attention layer consists of three one-dimensional convolutional layers and a softmax normalization layer arranged sequentially. Secondly, a point cloud instance segmentation method based on non-local attention includes the following steps: Step 1: Obtain the point cloud data to be segmented, and preprocess the point cloud data to be segmented to obtain voxelized point cloud. Step 2: Input the voxelized point cloud obtained in Step 1 into the trained point cloud instance segmentation model obtained in the construction method of the point cloud instance segmentation model based on non-local attention, and obtain the segmentation result.
[0009] Thirdly, a point cloud instance segmentation device based on non-local attention is provided to implement the point cloud instance segmentation method based on non-local attention.
[0010] Fourthly, a computer-readable storage medium storing a computer program that, when executed by a processor, implements the point cloud instance segmentation method based on non-local attention.
[0011] Fifthly, a computer program product includes a computer program / instructions that, when executed by a processor, implement the point cloud instance segmentation method based on non-local attention.
[0012] Compared with the prior art, the beneficial technical effects of this invention are: (I) The point cloud instance segmentation model in this invention can first convert the preprocessed data into a format suitable for sparse convolution processing, then use SE-enhanced sparse 3D U-Net to extract point-level features, deeply mine the geometric and semantic information of point cloud data, and then aggregate point-level features into super-point features through group pooling layers, simplifying the data structure while retaining key information; then use the non-local attention mechanism in the decoder layer to perform global modeling, capture the long-range dependencies between super-point features, and combine learnable query vectors and dynamic attention masks to obtain instance segmentation results, which solves the technical problems of high computational complexity and poor segmentation effect caused by object occlusion in existing point cloud instance segmentation methods.
[0013] (II) The SE-enhanced sparse 3D U-Net proposed in this invention can adaptively enhance the channel features of point cloud data and extract point-level features with rich semantic and detailed information. The proposed decoder layer introduces a non-local attention mechanism, which breaks through the dependence of traditional methods on local information and captures the long-range dependencies between super-point features from a global perspective. Attached Figure Description
[0014] Figure 1 This is a schematic diagram of the method flow of the present invention.
[0015] The specific content of the present invention will be further explained in detail below with reference to the embodiments. Detailed Implementation
[0016] It should be noted that, unless otherwise specified, all components in this invention are those known in the art.
[0017] The following are specific embodiments of the present invention. It should be noted that the present invention is not limited to the following specific embodiments. All equivalent modifications made based on the technical solutions of this application fall within the protection scope of the present invention.
[0018] This invention presents a method for constructing a point cloud instance segmentation model based on non-local attention, the method comprising the following steps: Step 1: Collect all raw point cloud data in a scene and preprocess the raw point cloud data to obtain voxelized point cloud and superpoint labels; The raw point cloud data includes geometric information and instance labels; Step 2: Construct a point cloud instance segmentation model; The point cloud instance segmentation model comprises a sparse 3D U-Net based on SE enhancement, a group pooling layer, and a decoder layer connected in sequence; the decoder layer comprises a non-local attention decoder and a prediction head. The SE-enhanced sparse 3D U-Net is used to extract features from all voxelized point clouds to obtain point-level features. The grouped pooling layer is used to group point-level features according to superpoint labels and perform average pooling to obtain multiple superpoint-level features. The nonlocal attention decoder creates a set of learnable query vectors and performs multi-level attention calculations on the super-point level features and the learnable query vectors to obtain optimized query vectors. The prediction head is used to decode the optimized query vector to obtain the instance class probability distribution of the voxelized point cloud, select the instance class with the highest probability, and use the instance class corresponding to the highest probability as the segmentation result. Step 3: Take the voxelized point cloud obtained in Step 1 as input, take the instance class corresponding to the maximum probability of the voxelized point cloud in the class probability distribution as output, take the instance label as the real label for supervision, and train the point cloud instance segmentation model constructed in Step 2 with binary cross-entropy loss and DICE as joint loss functions to obtain the trained point cloud instance segmentation model. In the above technical solution, the point cloud instance segmentation model first converts the preprocessed data into a format suitable for sparse convolution processing, then uses SE-enhanced sparse 3D U-Net to extract point-level features, deeply mining the geometric and semantic information of the point cloud data. Next, the point-level features are aggregated into super-point features through a group pooling layer, simplifying the data structure while retaining key information. Then, the non-local attention mechanism in the decoder layer is used for global modeling to capture the long-range dependencies between super-point features, and the instance segmentation result is obtained by combining the learnable query vector and dynamic attention mask. This solves the technical problems of high computational complexity and poor segmentation effect caused by object occlusion in existing point cloud instance segmentation methods.
[0019] In SE-enhanced sparse 3D U-Net, the initial module converts the voxelized point cloud into a sparse convolutional format and extracts initial features. Then, it uses sparse 3D convolution to downsample and extract high-level semantic features. Finally, it uses deconvolution to upsample and restore the resolution. Skip connections are used to fuse shallow detail features, and finally, it outputs point-level features containing rich semantic information.
[0020] Grouped pooling layer: The point-level features are divided into multiple super-point units according to geometric information. Then, average pooling is applied to each super-point unit. That is, the mean of the point-level feature group belonging to the same super-point unit is calculated point by point, thereby transforming the point-level features of dimension (N×C, where N is the total number of points and C is the feature dimension) into super-point-level features of dimension (M×C, where M is the number of super-points).
[0021] Non-local attention decoder: The non-local attention decoder randomly creates multiple learnable query vectors to transform super-point features into super-point feature vectors. It calculates attention using a cross-attention mechanism to obtain a weighted sum of the similarity weights between the query vector and all super-point feature vectors. This weighted sum is then randomly discarded, residual connected, and normalized by layers. The results of each batch are then concatenated to obtain the relevant information between the query vector and the super-point feature vectors. Subsequently, the query vector's feature representation capability is enhanced through a self-attention layer and a feedforward network layer to obtain a feature-enhanced query vector. After the feature-enhanced query vector is input into the non-local attention layer, it is first passed through three one-dimensional convolutional layers, g, theta, and phi, and then projected onto a feature space with an intermediate channel number of inter_channels (inter_channels = d_model / / reduction, where reduction refers to a preset reduction ratio), resulting in g, theta, and phi tensors of shape (B, inter_channels, n_q).
[0022] Next, the matrix product of the transposes of theta and phi is calculated to obtain the similarity matrix sim, with dimensions (B, n_q, n_q). The similarity matrix sim is normalized using the softmax function to obtain the attention weight of each position to other positions. The normalized sim is then multiplied with the transpose of g to obtain the weighted sum y, with dimensions (B, n_q, inter_channels). The dimensions of the weighted sum y are then adjusted to (B, inter_channels, n_q), and then mapped through the output convolutional layer to obtain the weighted sum y with the number of channels d_model. Finally, the weighted sum y with the number of channels d_model is residually connected with the feature-enhanced query vector output by the feedforward network layer to obtain the optimized query vector. In step 1, the preprocessing specifically includes: 1.1 The original point cloud data is enhanced by geometric transformation and color perturbation to obtain enhanced point cloud data; 1.2. Based on geometric information, perform superpoint segmentation on the enhanced point cloud data to obtain superpoint labels, and perform voxelization on the enhanced point cloud data to obtain voxelized point cloud.
[0023] In the above scheme, the geometric information includes coordinates, color, and normal vector.
[0024] The SE-enhanced sparse 3D U-Net consists of an initial module, a downsampling module, an upsampling module, and a feature fusion module connected in sequence. The initial module includes two residual blocks. In the above technical solution, the SE-enhanced sparse 3D U-Net first converts the input voxelized point cloud into a sparse convolution format. The initial module, downsampling, upsampling and feature fusion module are connected by skip connections to avoid gradient vanishing and enhance the expressive power of key feature channels to complete the initial feature extraction. Then, sparse 3D convolution is used to downsample and extract high-level semantic features, and deconvolution is used to upsample to restore the resolution. Skip connections are used to fuse shallow detail features, and finally, point-level features containing rich semantic information are output.
[0025] The nonlocal attention decoder consists of six sequentially connected layers, each of which includes a connected cross attention layer, a self attention layer, a feedforward network layer, and a nonlocal attention layer. The feedforward network layer consists of two linear layers and one activation function; The nonlocal attention layer consists of three one-dimensional convolutional layers and a softmax normalization layer arranged sequentially. In the above technical solution, the nonlocal attention decoder transforms the superpoint-level features into K and V, and the query vector as Q. It calculates attention to obtain a weighted output by means of a cross-attention mechanism. After random discarding, residual connection and normalization processing, the output is spliced together with the results of each batch to obtain the relevant information of the query vector and the superpoint features. Then, it passes through a self-attention layer, a feedforward network layer and a nonlocal attention layer to further improve the feature representation capability of the query vector.
[0026] This invention also provides a point cloud instance segmentation method based on non-local attention, comprising the following steps: Step 1: Obtain the point cloud data to be segmented, and preprocess the point cloud data to be segmented to obtain voxelized point cloud. Step 2: Input the voxelized point cloud obtained in Step 1 into the trained point cloud instance segmentation model obtained in the construction method of the point cloud instance segmentation model based on non-local attention, and obtain the segmentation result.
[0027] Comparative example: This comparative example presents the segmentation results of the ScanNetv2 dataset using different point cloud instance segmentation methods. Average precision (AP) and average class average precision (mAP) are used as evaluation metrics. The evaluation results are shown in Table 1. Table 1 Experimental results on the ScanNetv2 dataset
[0028] As shown in Table 1, the segmentation accuracy of this method is better. Compared with the SPFormer model, on the ScanNetv2 dataset, when the intersection-union ratio is greater than or equal to 25% and 50%, the mAP25 is improved by 1.1% and the mAP50 is improved by 1.0%. Compared with other existing models, the model performance is improved under different intersection-union ratios.
Claims
1. A method for constructing a point cloud instance segmentation model based on non-local attention, the method comprising the following steps: Step 1: Collect all raw point cloud data in a scene and preprocess the raw point cloud data to obtain voxelized point cloud and superpoint labels; The raw point cloud data includes geometric information and instance labels; Step 2: Construct a point cloud instance segmentation model; The point cloud instance segmentation model comprises a sparse 3D U-Net based on SE enhancement, a group pooling layer, and a decoder layer connected in sequence; the decoder layer comprises a non-local attention decoder and a prediction head. The SE-enhanced sparse 3D U-Net is used to extract features from all voxelized point clouds to obtain point-level features. The grouped pooling layer is used to group point-level features according to superpoint labels and perform average pooling to obtain multiple superpoint-level features. The nonlocal attention decoder creates a set of learnable query vectors and performs multi-level attention calculations on the super-point level features and the learnable query vectors to obtain optimized query vectors. The prediction head is used to decode the optimized query vector to obtain the instance class probability distribution of the voxelized point cloud, select the instance class with the highest probability, and use the instance class corresponding to the highest probability as the segmentation result. Step 3: Take the voxelized point cloud obtained in Step 1 as input, take the instance class corresponding to the maximum probability of the voxelized point cloud in the class probability distribution as output, take the instance label as the real label for supervision, and train the point cloud instance segmentation model constructed in Step 2 with binary cross-entropy loss and DICE as joint loss functions to obtain the trained point cloud instance segmentation model.
2. The method for constructing a point cloud instance segmentation model based on non-local attention as described in claim 1, characterized in that, In step 1, the preprocessing specifically includes: 1.1 The original point cloud data is enhanced by geometric transformation and color perturbation to obtain enhanced point cloud data; 1.
2. Based on geometric information, perform superpoint segmentation on the enhanced point cloud data to obtain superpoint labels, and perform voxelization on the enhanced point cloud data to obtain voxelized point cloud.
3. The method for constructing a point cloud instance segmentation model based on non-local attention as described in claim 1, characterized in that, The SE-enhanced sparse 3D U-Net includes an initial module, a downsampling module, an upsampling module, and a feature fusion module connected in sequence. The initial module includes two residual blocks.
4. The method for constructing a point cloud instance segmentation model based on non-local attention as described in claim 1, characterized in that, The nonlocal attention decoder comprises six sequentially connected layers, each layer including a connected cross attention layer, a self attention layer, a feedforward network layer, and a nonlocal attention layer. The feedforward network layer consists of two linear layers and an activation function; The nonlocal attention layer consists of three one-dimensional convolutional layers and a softmax normalization layer arranged sequentially.
5. A point cloud instance segmentation method based on non-local attention, characterized in that, Includes the following steps: Step 1: Obtain the point cloud data to be segmented, and preprocess the point cloud data to be segmented to obtain voxelized point cloud; Step 2: Input the voxelized point cloud obtained in Step 1 into the trained point cloud instance segmentation model obtained in the construction method of the point cloud instance segmentation model based on non-local attention as described in any one of claims 1 to 4, and obtain the segmentation result.
6. A point cloud instance segmentation device based on non-local attention, characterized in that, This method is used to implement the point cloud instance segmentation method based on nonlocal attention as described in claim 5.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the point cloud instance segmentation method based on non-local attention as described in claim 5.
8. A computer program product, characterized in that, Includes a computer program / instruction, which, when executed by a processor, implements the point cloud instance segmentation method based on non-local attention as described in claim 5.