PointGroup point cloud segmentation method fusing multiple attention mechanisms

By introducing Transformer encoding and SEAttention attention mechanisms in the PointGroup network, combining offset point sets and original coordinate sets for point cloud clustering, the point cloud segmentation algorithm in the existing technology is solved. The accuracy and robustness of point cloud segmentation algorithms in indoor scenarios are low in the existing technology. The problem of this achieves more efficient and accurate point cloud segmentation.

CN118941787APending Publication Date: 2024-11-12WUXI UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410914493.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing point cloud segmentation algorithm has low accuracy and poor robustness in indoor scenarios, making it difficult to effectively process point cloud data in dense and sparse areas.

Method used

Improve the UNet network of feature extraction part in the PointGroup network, add Transformer encoding and SEAttention attention mechanism, capture long-distance dependencies and allocate channel attention weights, and perform point cloud clustering through the combination of offset point sets and original coordinate sets , optimize the feature extraction and segmentation process.

Benefits of technology

It improves the accuracy and robustness of point cloud segmentation, maintains high accuracy in dense and sparse areas, and significantly improves the overall segmentation performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941787A_ABST
    Figure CN118941787A_ABST
Patent Text Reader

Abstract

The invention discloses a PointGroup point cloud segmentation method fusing a multi-head attention mechanism, the PointGroup network structure is composed of an improved UNet network, a point cloud clustering module and a ScoreNet scoring module, and the following steps are carried out in sequence: selecting a ScanNetV2 data set, and using the point cloud data in the ScanNetV2 data set as a training set, a verification set and a test set; building an improved feature extraction network model; inputting the extracted features into semantics and offset branches to generate semantic tags and offset vectors; introducing a clustering method, and grouping points into candidate clusters on the original coordinate set and the shift coordinate set; inputting a clustering result into ScoreNet to evaluate candidate examples, removing repeated examples in combination with non-maximum suppression, and outputting a segmentation result; training and testing the model; the robustness of the segmentation network model is improved, the generalization ability is improved, and rapid and accurate segmentation of the indoor scene point cloud instance is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of point cloud segmentation technology, and in particular to a PointGroup point cloud segmentation method integrating a multi-head attention mechanism. Background Art

[0002] In recent years, with the continuous development of artificial intelligence, computer vision technology has played an increasingly important role in intelligent manufacturing and production. How to use machines to perceive and understand complex real-world scenes through computer vision technology has always been a research focus. With the support of various advanced artificial intelligence technologies, researchers have achieved good performance in various tasks such as image classification, image segmentation, and object detection in the field of two-dimensional images.

[0003] Compared with two-dimensional images, point clouds, as three-dimensional data, contain more information. More and more researchers have also noticed the potential of point cloud data and invested in point cloud-related research. For example, with the development of technology in the field of autonomous driving, lidar has been widely used in vehicles. It has advantages over traditional solutions such as cameras and millimeter-wave radars in terms of detection accuracy, reliability, and anti-interference ability. In addition to outdoor scenes, point clouds also have great potential in indoor scenes. Through the point cloud instance segmentation task of indoor scenes, we can identify objects in the scene only by relying on point cloud data and related algorithms, which makes it possible to realize related applications, such as instructing visually impaired people to obtain designated items through voice interaction in indoor scenes. However, the accuracy achieved by the current instance segmentation task of indoor scenes still has a lot of room for improvement. For example, traditional point cloud segmentation algorithms often have low accuracy and poor robustness, such as SGPN (Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation) and 3D-BEVIS (3D Bird's-Eye-View Instance Segmentation). In this context, an efficient and accurate point cloud segmentation algorithm is urgently needed to improve the practicality of lidar. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a PointGroup point cloud segmentation method integrating a multi-head attention mechanism, comprising the following steps:

[0005] S1. Select the ScanNetV2 public dataset, use the 3D point cloud data captured by the RGB-D sensor, and divide it into training set, validation set, and test set; preprocess the acquired point cloud data, read the point cloud data, extract the coordinates and color information, and save it as a tensor file of PyTorch;

[0006] S2. Build a feature extraction network model. The feature extraction network model is set as a UNet network consisting of an encoder and a decoder with a symmetrical structure. Add Transformer encoding to the bottleneck layer of the UNet network to capture long-distance dependencies, and add SEAttention mechanism to the UNet network encoding and decoding block to assign channel attention weights. The input variable of the UNet network is a point set with color and coordinate information, and the output variable is the feature vector of each point after feature extraction.

[0007] S3, input the feature F of each point after feature extraction into the semantic branch and the offset branch, use the multi-layer perceptron to process the feature of each point, and generate a semantic label and an offset vector;

[0008] S4, grouping the points into candidate clusters on the dual coordinate set, i.e., the original coordinate set and the offset coordinate set, by point cloud clustering, to achieve point group clustering and generate candidate instances;

[0009] S5. Take the union of the clustering results and input them into ScoreNet to evaluate the candidate instances. Generate feature vectors for each candidate point group. Use a small UNet to encode the features of the clustered point groups. Use NMS to remove duplicate instances on the candidate point groups. Finally, output the instance segmentation results.

[0010] S6, training model, initializing the improved overall network, importing the training set scene into the model, training, and obtaining a trained network model;

[0011] S7. Test the trained model, use Python script to read and process the prediction results generated by the model, combine it with the point cloud data and save it, and use CloudComapre software to visualize the results of point cloud instance segmentation.

[0012] The beneficial effects of the present invention are:

[0013] (1) In the present invention, the feature extraction part UNet network is improved in the PointGroup network structure, and Transformer encoding is added to the bottleneck layer to optimize the capture of long-distance dependencies and improve the global context information of features to address the problem of limited receptive field of sparse convolution; and the SEAttention mechanism is added to the UNet encoding and decoding block to allocate channel attention weights so that the extraction process focuses on more important feature parts; compared with the traditional UNet network, which usually shows limitations in explicitly modeling long-range dependencies due to the inherent locality of convolution operations, the improved network has obvious advantages in feature capture and feature fusion, providing better feature materials for subsequent instance segmentation;

[0014] (2) In the present invention, the offset point set and the original coordinate set are combined. By predicting the offset of each point relative to the centroid of its instance and using the offset coordinates for clustering, points belonging to different instances can be effectively separated, thereby avoiding misclassification of adjacent instances. In addition, the offset points move the point cloud data to a more concentrated and denser location, thereby reducing the sparsity problem caused by uneven point cloud density during the clustering process, so that the clustering algorithm can maintain a high accuracy when processing both dense and sparse areas, thereby improving the robustness of the overall segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall structural principle diagram of the present invention;

[0016] Figure 2 This is a diagram of the improved UNet network structure in an embodiment of the present invention;

[0017] Figure 3 It is a structural diagram of SEAttention in an embodiment of the present invention;

[0018] Figure 4 This is a comparison diagram of the segmentation results before and after the improvement of the restaurant scene in the data set in an embodiment of the present invention;

[0019] Figure 5 This is a comparison diagram of the segmentation results before and after the improvement of the living room table scene in the data set in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] This embodiment provides a PointGroup point cloud segmentation method that integrates a multi-head attention mechanism. The improved PointGroup network structure consists of an improved UNet feature extraction network, a point cloud clustering module and a ScoreNet scoring module. By improving the feature extraction part of the UNet network, Transformer encoding is added to the bottleneck layer to capture long-distance dependencies and improve the global context information of the features. The SEAttention mechanism is added to the UNet encoding and decoding block to allocate channel attention weights, so that the extraction process focuses on more important feature parts, and finally achieves efficient segmentation of point cloud instances.

[0021] This embodiment combines Transformer encoding and SEAttention mechanism to propose an improved UNet network to achieve more accurate point cloud segmentation. The input variable of the improved UNet network is a point set with color and coordinate information, and the output variable is the feature vector of each point after feature extraction. The structure principle of the segmentation method is as follows: Figure 1 shown.

[0022] A PointGroup point cloud segmentation method integrating a multi-head attention mechanism includes the following steps:

[0023] S1. Select the ScanNetV2 public dataset, use the 3D point cloud data captured by the RGB-D sensor, and divide it into training set, validation set, and test set; preprocess the acquired point cloud data, read the point cloud data, extract the coordinate and color information, and save it as a PyTorch tensor file.

[0024] S2. Build a feature extraction network model. The feature extraction network model is set as a UNet network consisting of an encoder and a decoder with a symmetrical structure. Add Transformer encoding to the bottleneck layer of the UNet network to capture long-distance dependencies, and add SEAttention mechanism to the UNet network encoding and decoding block to assign channel attention weights. The input variable of the UNet network is a point set with color and coordinate information, and the output variable is the feature vector of each point after feature extraction.

[0025] The improved UNet network structure is as follows Figure 2 As shown, step S2 specifically includes the following sub-steps:

[0026] S2.1. The preprocessed point cloud data is a point set P with N points. The point set P is used as the input of the feature extraction network. Each point has a color f i =(r i , g i , b i ) and 3D coordinates pi =(x i ,y i , z i ), where i = (1, 2, 3, ..., N).

[0027] S2.2. The encoder extracts features from the input point cloud in the network through a series of convolutional layers. The encoder reduces the spatial dimension of the input point cloud and increases the number of channels of its feature points. The encoder is called a UNet block in the UNet network. The UNet block contains a residual block and a sparse 3D convolution. The residual block contains a shortcut connection block, a convolution block, and a SEAttention mechanism block, which can efficiently extract features. The structure of the SEAttention mechanism block is as follows: Figure 3 As shown in the figure, the convolution block consists of two sub-convolution blocks, and the sub-convolution block consists of a normalization layer, a ReLU activation function, and a 3x3 sparse convolution layer.

[0028] Step S2.2 specifically includes the following sub-steps:

[0029] S2.2.1. Input is a sparse tensor X Encoder , the shape is (BS, C in , D, H, W), where BS represents the number of point cloud samples in a batch; after entering the residual block of the encoder, it first passes through the identity branch to ensure that the number of input channels is the same as the number of output channels. The process is expressed by the following formula:

[0030] When C in =C out When I(X)=X Encoder ;

[0031] When C in ≠C out When I(X) = SubMConv3d 1×1 (X Encoder );

[0032] Among them, SubMConv3d 1×1 Represents a 3D submanifold sparse convolution of size 1*1; when the number of input and output channels is the same, the input is itself, otherwise the number of input channels and output channels are equal through 1*1 convolution.

[0033] S2.2.2, then through the first subconvolution block and the second subconvolution block in the residual block, expressed by the following formula:

[0034] Y1=ReLU(BatchNorm(SubMConv3d 3×3 (X Encoder )))

[0035] Y2=BatchNorm(SubMConv3d 3×3 (Y1))

[0036] Among them, SubMConv3d 3×3 It represents a 3D submanifold sparse convolution of size 3*3, BatchNorm is the batch normalization function, and ReLU is the activation function.

[0037] S2.2.3. Input the output Y2 into the SEAttention module, and recalibrate the weight of each channel through the two steps of compression and excitation to enhance the feature expression ability, which is expressed by the following formula:

[0038] Y se =SEAttention(Y2)

[0039] Among them, Y se Represents the output after the input passes through the SEAttention module.

[0040] Step S2.2.3 specifically includes the following sub-steps:

[0041] S2.2.3.1. First, perform the Squeeze operation and perform global average pooling to obtain the global features of each channel. The formula is as follows:

[0042]

[0043] Among them, X[:, c, d, h, w] represents the value of the cth channel, dth depth position, hth height position, and wth width position in the feature map; D represents depth, H represents height, and W represents width.

[0044] S2.2.3.2, then perform the Excitation operation, process each channel through two fully connected layers with ReLU and Sigmoid activation functions, generate the weight of each channel, and implement the channel attention mechanism. The formula is as follows:

[0045] e=σ(W2·ReLU(W1·s))

[0046] Among them, s is the global feature of each channel, W1 is the weight matrix of the first fully connected layer, and the shape is (C in , C in / r); W2 is the weight matrix of the second fully connected layer, with a shape of (C in / r,C in ), r is the dimensionality reduction ratio, and σ is the Sigmoid activation function.

[0047] S2.2.3.3. Then perform channel weight redistribution and reapply the obtained channel weight to the original point cloud feature tensor. The formula is as follows:

[0048] X′=[:,c,:,:,:]=X[:,c,:,:,:]×e[:,c]

[0049] Among them, X[:, c,:,:,:] represents the value of the position of all depths, heights and widths of each sample in the cth channel in the feature map; e[:, c] represents the weight of each sample in the cth channel; X′[:, c,:,:,:] represents the output feature map after recalibrating the channel weights, and its shape is the same as X.

[0050] S2.2.4. The last step of the residual block is to implement the residual connection, adding the original input and the feature-enhanced output to obtain the final residual feature Y Re for:

[0051] Y Re =Y se +I(X Encoder )

[0052] The overall residual block process is simplified to the following formula:

[0053] Y Re =ResidualBlock(X Encoder )

[0054] Among them, Y Re Represents the output after the input passes through ResidualBlock.

[0055] S2.2.5, use 3D sparse convolution to achieve downsampling, increase the number of channels, and obtain a single encoding block output feature Z Encoder for:

[0056] Z Encoder =ReLU(BatchNorm(SparseConv3d(Y Re ))).

[0057] S2.3. The Transformer module uses a multi-head attention mechanism to capture the relationship between different positions in the input sequence, explicitly introduces position information through position encoding, performs nonlinear transformation and feature enhancement through a feedforward neural network, and finally stabilizes the training process through a normalization layer. The combination of these components enables the Transformer to perform well in processing global context information and long-distance dependencies, and is particularly suitable for application in the bottleneck layer of UNet to improve the overall model performance.

[0058] Step S2.3 specifically includes the following sub-steps:

[0059] S2.3.1. Input point cloud coordinate matrix Input eigenvector matrix Where N is the number of points, d model is the feature dimension, each point has three coordinates (x, y, z). In order to calculate the relative position distance between point pairs, the difference matrix between point pairs is first calculated. The formula is as follows:

[0060] pairwise_dist ij =xyz i -xyz j

[0061] Among them, xyz i and xyz j The coordinates of point i and point j are 1*3, pairwise_dist ij is the relative position distance between point i and point j;

[0062] Calculate the average distance, take the average of the relative position distance in dimension 1, and get the average relative position distance of each point. The formula is as follows:

[0063]

[0064] Among them, pairwise_dist i represents the average relative position distance of point i;

[0065] The average relative position distance is mapped to the feature dimension d through a linear layer. model For the same space, the formula is as follows:

[0066] position_embedding i =W p pairwise_dist i +b p

[0067] Among them, position_embedding i represents the position code of point i, W p is the weight matrix of linear transformation, b p is the bias vector of the linear transformation;

[0068] Finally, the feature encoding is added to the input features:

[0069] X i =features i +positional_embedding i ;

[0070] Among them, X i is the final feature representation of point i, features i is the input feature of point i.

[0071] S2.3.2. Transform the feature vector matrix X with feature coding through the weight transformation matrix W Q , W K , The Q, K, and V of the multi-head self-attention module are obtained as:

[0072] Q=XW Q

[0073] K=XW K

[0074] V=XW V

[0075] Among them, d k For each head dimension, Q, K, and V are used to calculate the attention score. The formula is as follows:

[0076]

[0077] in, is the scaling factor to ensure that the dot product is not too large. The dot product range before scaling is [-d k , d k ], the range of the dot product after scaling is

[0078] S2.3.3. Concatenate the results of multiple heads and perform linear transformation. The formula is as follows:

[0079]

[0080] Among them, head i =Attention(Q, K, V), is the linear transformation matrix, and h0 is the number of heads.

[0081] S2.3.4. Output Q, K, V to the multi-head attention mechanism and obtain feature X1 as:

[0082] X1=laynorm(X+MultiHead(Q,K,V))

[0083] Among them, laynorm represents the layer normalization operation.

[0084] S2.3.5. Input the feature into the feedforward neural network and obtain the feature X2 as:

[0085] X2=laynorm(X1+FFN(X1))

[0086] Among them, FFN represents the feed-forward neural network operation.

[0087] S2.4. The decoder increases the spatial dimension and reduces the number of channels through upsampling convolution, combines it with the jump connection features of the encoder through splicing operation, and finally extracts features through the residual block containing SEAttention to effectively retain and fuse features while restoring the spatial resolution.

[0088] The decoder is an inverse UNet block, which contains a residual block and an inverse sparse 3D convolution; step S2.4 specifically includes the following sub-steps:

[0089] S2.4.1. Input sparse tensor X Decoder , the shape is (BS, C in , D / 32, H / 32, W / 32), enters the residual block, and the output is expressed as:

[0090] Y Re =ResidualBlock(X Decoder ).

[0091] S2.4.2. By concatenating the skip connection feature vector Y from the encoder skip , get the concatenated feature Z skip for:

[0092] Z skip =Concat(Y Re , Y skip )

[0093] S2.4.3. Upsampling the number of channels is achieved through three-dimensional inverse sparse convolution SparseInverseConv3d to obtain a single decoder output feature Z Decoder for:

[0094] Z Decoder =ReLU(BatchNorm(SparseInverseConv3d(Z skip )))

[0095] S2.4.4, after several inverse UNet blocks, X Decoder The shape is (BS, C out , D, H, W), and finally through the output layer, the output feature F is obtained as:

[0096] F = Linear(ReLU(BatchNorm(Z Decoder )))

[0097] in, i represents the point cloud index, N represents the number of point clouds, and k represents the number of channels.

[0098] S3, input the feature F of each point after feature extraction into the semantic branch and the offset branch, use the multi-layer perceptron MLP to process the feature of each point, and generate a semantic label and an offset vector; step S3 specifically includes the following sub-steps:

[0099] S3.1. Input the extracted feature F into the MLP of the semantic branch to obtain N class N point clouds on categories generate semantic scores SC:

[0100] SC=MLP semantic (F) = {sc1, ..., sc N}

[0101] Among them, MLP semantic It consists of a linear layer that maps the input feature F to the semantic category space; the semantic scores sc of point i in different categories i Take the maximum value to get the predicted semantic label s for each point i for:

[0102] S i = argmax(sc i )

[0103] Through the cross entropy loss L sem Regularize the result:

[0104]

[0105] Where N is the number of points, s i is the predicted semantic label, is the true semantic label.

[0106] S3.2. Input the extracted feature F into the MLP of the offset branch to obtain the semantic score O of each point:

[0107] O=MLP offset (F)

[0108] Among them, MLP offset It consists of a linear layer and an activation function; the offset vector o of each point is obtained through the linear layer i for:

[0109] o i =(Δx i , Δy i , Δz i )=Linear(O i )

[0110] Among them, Oi is the offset feature of point i;

[0111] For points belonging to the same instance, their learned offsets are constrained by the regression loss as:

[0112]

[0113] Among them, p i are the coordinates of the point; m i is a binary mask, if m i =1, then point i is on the instance, otherwise it is not; is the centroid of the instance, expressed as:

[0114]

[0115] Among them, g(i) maps from point i to the index of its corresponding instance, that is, the instance to which point i belongs; I g(i) Indicates that instance g(i) contains the set of all points; p j are the coordinates of point j in instance g(i); is the number of points in instance g(i).

[0116] For large-sized instances, the boundary points are far away from the centroid, and the predicted offset vector is not accurate enough. Moreover, only the distance constraint cannot guarantee that all points move accurately toward the centroid, especially for the case where the predicted offset vector is long. In order to further optimize the moving direction of the points and make them move more accurately toward the centroid, the offset direction loss function is proposed as follows:

[0117]

[0118] S4. A clustering method is introduced to ensure that points of the same instance in the point cloud can be accurately clustered together without being mis-segmented into different instances. The specific implementation method includes the following steps:

[0119] S4.1. Initialization parameters: coordinate set Semantic tag collection Cluster radius r and minimum cluster point threshold N θ ; Initialize a length of N l An array v with all elements set to 0 is used to record whether each point has been visited; an empty clustering set C0 is initialized.

[0120] S4.2. Traverse all points. If the semantic label of the point is not clusterable (such as wall background class), mark it as visited, that is, v[i] = 1.

[0121] S4.3. Traverse all points again and check whether point i has not been visited, that is, whether v[u] is 0. If point u has not been visited, initialize an empty queue Q0 and an empty cluster C0, and add point i to queue Q0 and cluster C0, that is, v[i] = 1; use the breadth-first search method, when queue Q0 is not empty, take out a point k, traverse all points j, if the distance between point j and point k is less than r, and the semantic labels are the same, then mark point j as visited, and add point j to queue Q0 and cluster C0; when the number of points in cluster C0 is greater than the threshold N θ , then cluster C0 is added to the final cluster set.

[0122] S4.4, apply the above algorithm steps to the original coordinate set P = {p i} and offset coordinate set Q offset = {q i}, and get two clustering result sets C P and C Q They are:

[0123] C P = Clustering(P, S, r, N θ )

[0124]

[0125] Finally, C P and C Q Take the union and get the output set:

[0126]

[0127] Among them, C final Represents the final output set.

[0128] S5. Take the union of the clustering results and input them into ScoreNet to evaluate the candidate instances. Generate a feature vector for each candidate point group. Use a small UNet to encode the features of the clustered point groups. Use NMS to remove duplicate instances on the candidate point groups. Finally, output the instance segmentation results.

[0129] Compared with the Unet network mentioned above, the small UNet is shallower in depth. Specifically, the UNet network contains 7 layers and requires 6 upsampling and 6 downsampling respectively, while the small UNet network contains only 2 layers and only requires 1 upsampling and 1 downsampling respectively. This also makes the small UNet less computationally intensive and suitable for small-scale feature extraction tasks.

[0130] Step S5 specifically includes the following sub-steps:

[0131] S5.1. For each candidate cluster Ci , extract the features of the points belonging to the cluster from the point feature set F

[0132]

[0133] Among them, h is a mapping function that transforms the cluster C i The index of the midpoint is mapped to the point index in the original point cloud; N i is cluster C i The number of points in; similarly, extract cluster C i Coordinates of the midpoint:

[0134]

[0135] S5.2. Re-voxelize the points in the cluster and encode them using a small UNet. The voxel features are obtained by averaging the features of the points within the voxel:

[0136]

[0137] Where v represents a voxel, N v is the number of points in the voxel, Belongs to cluster C i And the features of the points located in voxel v; these voxel features are input into a small UNet for encoding, and the feature vector of each cluster is obtained as:

[0138]

[0139] S5.3. Use a multi-layer perceptron and Sigmoid activation function to process the feature vector of each cluster and generate a cluster score. for:

[0140]

[0141] Where σ is the Sigmoid function.

[0142] S5.4. Use soft labels to train ScoreNet, which is defined as follows:

[0143]

[0144] Among them, IoU (C i ) is cluster C i The maximum intersection-over-union ratio with all real instances, θ l and θ h are the lower threshold and the upper threshold respectively; the binary cross entropy loss function is used to supervise the learning of clustering scores, and the formula is as follows:

[0145]

[0146] Where M is the number of candidate clusters, is cluster C i Soft label.

[0147] S5.5. Use non-maximum suppression (NMS) to select the final instance from the candidate clusters with the highest score: First, select the cluster score Sort all candidate clusters, start with the cluster with the highest score, select all low-scoring clusters that overlap with it (i.e., IoU exceeds 0.3), and finally delete these low-scoring clusters and output the final instance from the candidate cluster with the highest score.

[0148] S6. Training model: Initialize the improved overall network and import the training set scene into the model for training to obtain the trained network model.

[0149] S7. Test the trained model, use Python script to read and process the prediction results generated by the model, combine it with the point cloud data and save it, and use CloudComapre software to visualize the results of point cloud instance segmentation.

[0150] By improving the PointGroup of the UNet network, the original point cloud in the ScanNetV2 dataset is input, and the segmentation results are as follows Figures 4 to 5 shown.

[0151] Table 1 below is a performance comparison of different methods on the ScanNetV2 test set (AP 50 %), this embodiment selected 9 comparison methods, and counted the accuracy of point cloud segmentation on a total of 18 categories such as cabinets, beds, chairs, etc.; It can be seen from Table 1 that the improved PointGroup method proposed in this embodiment achieved the highest segmentation accuracy among all comparison methods on cabinets, chairs, tables, shower curtains, washrooms, bathtubs and other furniture; and compared with the original PointGroup algorithm, the overall average segmentation accuracy was improved by 2.8%.

[0152] Table 1: Performance comparison of different methods on the ScanNetV2 test set (AP 50 %):

[0153]

[0154] Table 2 below is the ablation experiment of average precision under different IoU. It can be seen from Table 2 that the SEAttention module makes the improved network better than the original network in AP and AP 50 ,AP 25The three indicators were improved by 0.4%, 1.9%, and 0.5% respectively; the Transformer multi-head attention module makes the improved network better than the original network in AP and AP 50 ,AP 25 The three indicators were improved by 0.2%, 1.3%, and 0.2% respectively; the improved overall network structure is better than the original network in AP, AP 50 ,AP 25 The three indicators were improved by 0.8%, 2.8% and 0.8% respectively. This ablation experiment demonstrates the effectiveness of adding modules in the improved PointGroup point cloud segmentation method of this embodiment, and achieves accurate point cloud segmentation of indoor scenes.

[0155] Table 2: Comparison of average accuracy under different IoU in ablation experiments:

[0156]

[0157] Note: AP 50 and AP 25 Refers to the average precision AP when IoU is 50% and 25% respectively

[0158] Table 3 below is an ablation experiment of time cost. This embodiment randomly selects four scenes in ScanNetV2 for running time test. It can be seen from Table 3 that the SEAttention module reduces the time cost of the improved network by 3.35ms compared with the original network; the Transformer multi-head attention module increases the time cost of the improved network by 6.7ms compared with the original network; the improved overall network structure increases the time cost by 1.6ms compared with the original network; this ablation experiment shows that the improved PointGroup point cloud segmentation method of this embodiment does not increase too much time cost, and realizes the rapid segmentation of indoor scene point cloud.

[0159] Table 3: Comparison of time cost of ablation experiment:

[0160]

[0161] Note: Scene 1 has a total of 230,672 points, Scene 2 has a total of 47,709 points, Scene 3 has a total of 182,281 points, and Scene 4 has a total of 61,388 points. Four experimental schemes were used to run each scene 50 times, and the running time of each scene and the total running time are shown in the table above.

[0162] In summary, the PointGroup point cloud segmentation method integrating the multi-head attention mechanism proposed in this embodiment achieves accurate and efficient segmentation of indoor scene point clouds with almost no time cost compared to the original PointGroup method, and also improves the robustness of the point cloud segmentation network model and increases the generalization ability.

[0163] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A PointGroup point cloud segmentation method integrating a multi-head attention mechanism, characterized by: The following steps are involved: S1. Select the ScanNetV2 public dataset, use the 3D point cloud data captured by the RGB-D sensor, and divide it into training set, validation set, and test set; preprocess the acquired point cloud data, read the point cloud data, extract the coordinates and color information, and save it as a tensor file of PyTorch; S2. Build a feature extraction network model. The feature extraction network model is set as a UNet network consisting of an encoder and a decoder with a symmetrical structure. Add Transformer encoding to the bottleneck layer of the UNet network to capture long-distance dependencies, and add SEAttention mechanism to the UNet network encoding and decoding block to allocate channel attention weights. The input variable of the UNet network is a point set with color and coordinate information, and the output variable is the feature vector of each point after feature extraction; S3, input the feature F of each point after feature extraction into the semantic branch and the offset branch, use the multi-layer perceptron to process the feature of each point, and generate a semantic label and an offset vector; S4, grouping the points into candidate clusters on the dual coordinate set, i.e., the original coordinate set and the offset coordinate set, by point cloud clustering, to achieve point group clustering and generate candidate instances; S5. Take the union of the clustering results and input them into ScoreNet to evaluate the candidate instances. Generate feature vectors for each candidate point group. Use a small UNet to encode the features of the clustered point groups. Use NMS to remove duplicate instances on the candidate point groups. Finally, output the instance segmentation results. S6, training model, initializing the improved overall network, importing the training set scene into the model, training, and obtaining a trained network model; S7. Test the trained model, use Python script to read and process the prediction results generated by the model, combine it with the point cloud data and save it, and use CloudComapre software to visualize the results of point cloud instance segmentation.

2. The PointGroup point cloud segmentation method integrating the multi-head attention mechanism according to claim 1 is characterized in that: The step S2 specifically includes the following sub-steps: S2.

1. The preprocessed point cloud data is a point set P with N points. The point set P is used as the input of the feature extraction network. Each point has a color f i =(r i , g i , b i ) and 3D coordinates p i =(x i ,y i , z i ), where i = (1, 2, 3, ..., N); S2.2, the encoder extracts features from the input point cloud in the network through a series of convolutional layers, reduces the spatial dimension of the input point cloud through the encoder, and increases the number of channels of its feature points; S2.3, the Transformer module uses a multi-head attention mechanism to capture the relationship between different positions in the input sequence, explicitly introduces position information through position encoding, performs nonlinear transformation and feature enhancement through a feedforward neural network, and finally stabilizes the training process through a normalization layer; S2.4, the decoder increases the spatial dimension and reduces the number of channels through upsampling convolution, combines it with the jump connection features of the encoder through splicing operation, and finally extracts features through the residual block containing SEAttention.

3. The PointGroup point cloud segmentation method integrating the multi-head attention mechanism according to claim 2 is characterized in that: In the step S2.2, the encoder is called a UNet block in the UNet network, and the UNet block contains a residual block and a sparse 3D convolution; the residual block contains a shortcut connection block, a convolution block and a SEAttention mechanism block; the convolution block consists of two sub-convolution blocks, and the sub-convolution block consists of a normalization layer, a ReLU activation function and a 3x3 sparse convolution layer.

4. The PointGroup point cloud segmentation method integrating the multi-head attention mechanism according to claim 2 is characterized in that: The step S2.2 specifically includes the following sub-steps: S2.2.

1. Input is a sparse tensor X Encoder , the shape is (BS, C in , D, H, W), where BS represents the number of point cloud samples in a batch; after entering the residual block of the encoder, it first passes through the identity branch to ensure that the number of input channels is the same as the number of output channels. The process is expressed by the following formula: When C in =C out When I(X)=X Encoder ; When C in ≠C out When I(X) = SubMConv3d 1×1 (X Encoder ); Among them, SubMConv3d 1×1 Represents a 3D submanifold sparse convolution of size 1*1; when the number of input and output channels is the same, the input is itself, otherwise the number of input channels and output channels are equal through 1*1 convolution; S2.2.2, then through the first subconvolution block and the second subconvolution block in the residual block, expressed by the following formula: Y1=ReLU(BatchNorm(SubMConv3d 3×3 (X Encoder ))) Y2=BatchNorm(SubMConv3d 3×3 (Y1)) Among them, SubMConv3d 3×3 It represents a 3D submanifold sparse convolution of size 3*3, BatchNorm is the batch normalization function, and ReLU is the activation function; S2.2.

3. Input the output Y2 into the SEAttention module, and recalibrate the weight of each channel through two steps of compression and excitation, which is expressed by the following formula: Y se =SEAttention(Y2) Among them, Y se Represents the output after the input passes through the SEAttention module; S2.2.

4. The last step of the residual block is to implement the residual connection, adding the original input and the feature-enhanced output to obtain the final residual feature Y Re for: AND Re =And se +I(X Encoder ) The overall residual block process is simplified to the following formula: Y Re =ResidualBlock(X Encoder ) Among them, Y Re Represents the output after the input passes through ResidualBlock; S2.2.5, use 3D sparse convolution to achieve downsampling, increase the number of channels, and obtain a single encoding block output feature Z Encoder for: Z Encoder =ReLU(BatchNorm(SparseConv3d(Y Re )))。 5. The PointGroup point cloud segmentation method integrating the multi-head attention mechanism according to claim 4 is characterized in that: The step S2.2.3 specifically includes the following sub-steps: S2.2.3.

1. First, perform compression operation and global average pooling to obtain the global features of each channel. The formula is as follows: Where X[:, c, d, h, w] represents the value of the cth channel, dth depth position, hth height position, and wth width position in the feature map; D represents the depth, H represents the height, and W represents the width; S2.2.3.2, then perform the excitation operation, process each channel through two fully connected layers with ReLU and Sigmoid activation functions, generate the weight of each channel, and implement the channel attention mechanism. The formula is as follows: e=σ(W2·ReLU(W1·s)) Among them, s is the global feature of each channel, W1 is the weight matrix of the first fully connected layer, and the shape is (C in , C in / r); W2 is the weight matrix of the second fully connected layer, with a shape of (C in / r,C in ), r is the dimensionality reduction ratio, σ is the Sigmoid activation function; S2.2.3.

3. Then perform channel weight redistribution and reapply the obtained channel weight to the original point cloud feature tensor. The formula is as follows: X′=[:,c,:,:,:]=X[:,c,:,:,:]×e[:,c] Among them, X[:, c,:,:,:] represents the value of the position of all depths, heights and widths of each sample in the cth channel in the feature map; e[:, c] represents the weight of each sample in the cth channel; X′[:, c,:,:,:] represents the output feature map after recalibrating the channel weights, and its shape is the same as X.

6. The PointGroup point cloud segmentation method integrating the multi-head attention mechanism according to claim 2 is characterized in that: The step S2.3 specifically includes the following sub-steps: S2.3.

1. Input point cloud coordinate matrix Input feature vector matrix X∈ Where N is the number of points, d model is the feature dimension, each point has three coordinates (x, y, z). In order to calculate the relative position distance between point pairs, the difference matrix between point pairs is first calculated. The formula is as follows: pairwise_dist ij =xyz i -xyz j Among them, xyz i and xyz j The coordinates of point i and point j are 1*3, pairwise_dist ij is the relative position distance between point i and point j; Calculate the average distance, take the average of the relative position distance in dimension 1, and get the average relative position distance of each point. The formula is as follows: Among them, pairwise_dist i represents the average relative position distance of point i; The average relative position distance is mapped to the feature dimension d through a linear layer. model For the same space, the formula is as follows: position_embedding i =W p pairwise_dist i +b p Among them, position_embedding i represents the position code of point i, W p is the weight matrix of linear transformation, b p is the bias vector of the linear transformation; Finally, the feature encoding is added to the input features: X i =features i +positional_embedding i ; Among them, X i is the final feature representation of point i, features i is the input feature of point i; S2.3.

2. Transform the feature vector matrix X with feature coding through the weight transformation matrix W Q , W K , The Q, K, and V of the multi-head self-attention module are obtained as: Q=XW Q K=XW K V=XW V Among them, d k For each head dimension, Q, K, and V are used to calculate the attention score. The formula is as follows: in, is the scaling factor; S2.3.

3. Concatenate the results of multiple heads and perform linear transformation. The formula is as follows: Among them, head i =Attention(Q, K, V), is the linear transformation matrix, h0 is the number of heads; S2.3.

4. Output Q, K, V to the multi-head attention mechanism and obtain feature X1 as: X1=laynorm(X+MultiHead(Q,K,V)) Among them, laynorm represents the layer normalization operation; S2.3.

5. Input the feature into the feedforward neural network and obtain the feature X2 as: X2=laynorm(X1+FFN(X1)) Among them, FFN represents the feed-forward neural network operation.

7. The PointGroup point cloud segmentation method integrating the multi-head attention mechanism according to claim 2 is characterized in that: In step S2.4, the decoder is an inverse UNet block, which includes a residual block and an inverse sparse 3D convolution; step S2.4 specifically includes the following sub-steps: S2.4.

1. Input sparse tensor X Decoder , the shape is (BS, C in , D / 32, H / 32, W / 32), enters the residual block, and the output is expressed as: Y Re =ResidualBlock(X Decoder ) S2.4.

2. By concatenating the skip connection feature vector Y from the encoder skip , get the concatenated feature Z skip for: Z skip =Concat(And Re ,AND skip ) S2.4.

3. Upsampling the number of channels through three-dimensional inverse sparse convolution to obtain a single decoder output feature Z Decoder for: With Decoder =ReLU(BatchNorm(SparseInverseConv3d(Z skip ))) S2.4.4, after several inverse UNet blocks, X Decoder The shape is (BS, C out , D, H, W), and finally through the output layer, the output feature F is obtained as: F=Linear(ReLU(BatchNorm(Z Decoder ))) in, i represents the point cloud index, N represents the number of point clouds, and k represents the number of channels.

8. The PointGroup point cloud segmentation method integrating the multi-head attention mechanism according to claim 1, characterized in that: The step S3 specifically includes the following sub-steps: S3.

1. Input the extracted feature F into the MLP of the semantic branch to obtain N class N point clouds on categories generate semantic scores SC: SC=MLP semantic( F)={sc1,...,sC N } Among them, MLP semantic It consists of a linear layer that maps the input feature F to the semantic category space; the semantic scores Sc of point i in different categories i Take the maximum value to get the predicted semantic label s for each point i for: s i =argmax(sc i ) Through the cross entropy loss L sem Regularize the result: Where N is the number of points, s i is the predicted semantic label, is the true semantic label; S3.

2. Input the extracted feature F into the MLP of the offset branch to obtain the semantic score O of each point: O=MLP offset (F) Among them, MLP offset It consists of a linear layer and an activation function; the offset vector o of each point is obtained through the linear layer i for: o i =(Δx i ,Δy i ,Δz i )=Linear(O i ) Among them, O i is the offset feature of point i; For points belonging to the same instance, their learned offsets are constrained by the regression loss as: Among them, p i are the coordinates of the point; m i is a binary mask, if m i =1, then point i is on the instance, otherwise it is not; is the centroid of the instance, expressed as: Among them, g(i) maps from point i to the index of its corresponding instance, that is, the instance to which point i belongs; I g(i) Indicates that instance g(i) contains the set of all points; p j are the coordinates of point j in instance g(i); is the number of points in instance g(i); In order to further optimize the moving direction of the point and make it more accurately towards the center of mass, the offset direction loss function is proposed as:

9. The PointGroup point cloud segmentation method integrating the multi-head attention mechanism according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S4.

1. Initialization parameters: coordinate set Semantic tag collection Cluster radius r and minimum cluster point threshold N θ ; Initialize a length of N l The array v of , all elements are set to 0, used to record whether each point has been visited; initialize an empty clustering set C0; S4.2, traverse all points, if the semantic label of the point is not clusterable, mark it as visited, that is, v[i] = 1; S4.

3. Traverse all points again and check whether point i has not been visited, that is, whether v[i] is 0. If point i has not been visited, initialize an empty queue Q0 and an empty cluster C0, and add point i to queue Q0 and cluster C0, that is, v[i] = 1; use the breadth-first search method, when queue Q0 is not empty, take out a point k, traverse all points j, if the distance between point j and point k is less than r, and the semantic labels are the same, then mark point j as visited, and add point j to queue Q0 and cluster C0; when the number of points in cluster C0 is greater than the threshold N θ , then cluster C0 is added to the final cluster set; S4.4, apply the methods described in steps S4.1 to S4.3 to the original coordinate set P = {p i } and offset coordinate set Q offset = {q i }, and get two clustering result sets C P and C Q They are: C P =Clustering(P,S,r,N θ ) Finally, C P and C Q Take the union and get the output set: Among them, C final Represents the final output set.

10. The PointGroup point cloud segmentation method integrating multi-head attention mechanism according to claim 1, characterized in that: The step S5 specifically includes the following sub-steps: S5.

1. For each candidate cluster C i , extract the features of the points belonging to the cluster from the point feature set F Among them, h is a mapping function that transforms the cluster C i The index of the midpoint is mapped to the point index in the original point cloud; N i is cluster C i The number of points in; similarly, extract cluster C i Coordinates of the midpoint: S5.

2. Re-voxelize the points in the cluster and encode them using a small UNet. The voxel features are obtained by averaging the features of the points within the voxel: Where v represents a voxel, N v is the number of points in the voxel, Belongs to cluster C i And the features of the points located in voxel v; these voxel features are input into a small UNet for encoding, and the feature vector of each cluster is obtained as: S5.

3. Use a multi-layer perceptron and Sigmoid activation function to process the feature vector of each cluster and generate a cluster score. for: Where σ is the Sigmoid function; S5.

4. Use soft labels to train ScoreNet, which is defined as follows: Among them, IoU (C i ) is cluster C i The maximum intersection-over-union ratio with all real instances, θ l and θ h are the lower threshold and the upper threshold respectively; the binary cross entropy loss function is used to supervise the learning of clustering scores, and the formula is as follows: Where M is the number of candidate clusters, is cluster C i Soft label; S5.

5. Use non-maximum suppression (NMS) to select the final instance from the candidate clusters with the highest score: First, select the cluster score Sort all candidate clusters, start with the cluster with the highest score, select all low-scoring clusters that overlap with it, and finally delete these low-scoring clusters and output the final instance from the candidate cluster with the highest score.

Citation Information

Cited By

  • Point cloud instance segmentation model based on dynamic convolution and SLAM map construction method

    CN119832254A

  • Point cloud segmentation method and system for crop population

    CN119832431A