Point cloud semantic segmentation method based on bilateral cross-enhancement and self-attention compensation

Abstract

The application discloses a point cloud semantic segmentation method based on bilateral cross enhancement and self-attention compensation, characterized by inputting point cloud data to be segmented into a semantic segmentation model, wherein the semantic segmentation model comprises a preprocessing module, a spatial aggregation module based on bilateral cross enhancement and self-attention compensation, a multi-scale feature fusion module and a full connection layer; the preliminary geometric context information and the preliminary semantic context information are obtained by the preprocessing module; the feature maps of different resolutions are obtained by the spatial aggregation module based on bilateral cross enhancement and self-attention compensation; the comprehensive feature map used for semantic segmentation is obtained from the multi-scale feature fusion module and reaches the full connection layer, so that the semantic segmentation result of the point cloud data to be segmented is obtained; the advantage is that the local context information of each point in the point cloud is enhanced; the singleness of the local context information is compensated, and the completeness of the features is enhanced, so that the purpose of improving the local fine-grained segmentation precision of the point cloud is achieved.

Description

Technical Field

[0001] This invention relates to point cloud semantic segmentation methods, and in particular to point cloud semantic segmentation methods based on bilateral cross-enhancement and self-attention compensation. Background Technology

[0002] Early point cloud semantic segmentation methods were mainly divided into projection-based point cloud semantic segmentation methods and voxelization-based point cloud semantic segmentation methods. These two point cloud semantic segmentation methods have made great progress, but they inevitably lose geometric information, resulting in poor segmentation accuracy.

[0003] To address the shortcomings of the aforementioned methods, scholars have proposed a semantic segmentation method that directly processes points in point clouds, which can obtain better spatial geometric information of point clouds and improve the segmentation accuracy of point clouds. For example, Qi et al. proposed the PointNet method (Reference: Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, pp. 77-85, 2017.), which directly processes the points in the point cloud. It learns point features point-by-point through a shared multi-layer perceptron (MLP), solving the problems of sparsity, disorder, and permutation invariance in point clouds. However, it cannot effectively extract the local contextual information of the point cloud. Komarichev et al. proposed a circular convolution operator (Reference: Artem Komarichev, Zichun Zhong, and Jing Hua. A-CNN: Annularly Convolutional Neural Networks on Point Clouds. Conference on Computer Vision and Pattern Recognition). Recognition (CVPR), CVF / IEEE, pp.7421-7430, 2019.) This operator specifies a ring structure and orientation in the calculation to better capture the local neighborhood geometry of each point in the point cloud, but it does not comprehensively consider the relationship between local geometric information and global information, resulting in inaccurate local fine-grained segmentation of the point cloud. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a point cloud semantic segmentation method based on bilateral cross-enhancement and self-attention compensation, which can not only obtain local context information well, but also improve the local fine-grained segmentation accuracy of point clouds by compensating local context information with global context information.

[0005] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a point cloud semantic segmentation method based on bilateral cross-enhancement and self-attention compensation, comprising the following steps:

[0006] ① Input the point cloud data to be segmented into the semantic segmentation model. The semantic segmentation model includes a preprocessing module, a spatial aggregation module based on bilateral cross-enhancement and self-attention compensation, a multi-scale feature fusion module, and a fully connected layer. The spatial aggregation module based on bilateral cross-enhancement and self-attention compensation consists of five cascaded spatial aggregation blocks. Each spatial aggregation block consists of a bilateral cross-enhancement unit and a Transformer self-attention mechanism unit. The output of the bilateral cross-enhancement unit is the input of the Transformer self-attention mechanism unit.

[0007] ② In the preprocessing module, the original coordinates of the point cloud data to be segmented in three-dimensional space are used as preliminary geometric context information; feature extraction is performed on the point cloud data to be segmented using a multilayer perceptron to obtain preliminary semantic context information;

[0008] ③ The preliminary geometric context information and preliminary semantic context information are input into the bilateral cross-enhancement unit of the first spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after one downsampling are input into the bilateral cross-enhancement unit of the second spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after two downsamplings are input into the bilateral cross-enhancement unit of the third spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after three downsamplings are input into the bilateral cross-enhancement unit of the fourth spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after four downsamplings are input into the bilateral cross-enhancement unit of the fifth spatial aggregation block. The feature maps of different resolutions are output from the Transformer self-attention mechanism units of the five spatial aggregation blocks.

[0009] ④ Input feature maps of different resolutions into the multi-scale feature fusion module to obtain a comprehensive feature map for semantic segmentation;

[0010] ⑤ Input the comprehensive feature map used for semantic segmentation into the fully connected layer to obtain the semantic segmentation result of the point cloud data to be segmented.

[0011] The specific process for obtaining the output of the aforementioned bilateral cross-enhancing unit is as follows:

[0012] A point is randomly selected from the point cloud data as the center point. The 3D-KNN algorithm is used to search for and obtain the nearest neighbors of the center point. The formula is then used to... Calculate the local geometric context information Among them, P i Represents the absolute coordinates of the center point. Represents the absolute coordinates of the nearest neighbor. This represents the relative coordinates of the nearest neighbor points with respect to the center point. This represents the concatenation operation symbol, k=16, obtained through the formula. Calculate the local semantic context information Where Fi represents the semantic feature corresponding to the center point in the semantic feature space. This represents the adjacent semantic features corresponding to the nearest neighbor points in the semantic feature space. The semantic features of nearest neighbors relative to the center point are represented by the formula. Offsets that provide local geometric context information Through formula Obtain the offset of local semantic context information Through formula The enhanced local context information Le is calculated and used as the output of the bilateral cross-enhancement unit, where concat(·) represents the connection operation and M(·) represents the multilayer perceptron operation. The absolute coordinates of the point cloud can enhance the model's spatial perception and localization capabilities, while the relative coordinates can enhance the model's ability to cope with the disorder of the point cloud. By cross-learning offsets, the local geometric context information and local semantic context information of the point cloud are enhanced, thereby achieving the effect of enhancing the local context information.

[0013] The specific process for obtaining the feature map output by the Transformer self-attention mechanism's working unit is as follows:

[0014] The enhanced local context information Le is converted into a sequence F consisting of local features arranged in descending order of resolution. in For sequence F in The first three-dimensional tensor, H×W×C, is obtained by performing average pooling downsampling on the local features at medium to high resolution. This tensor is then used to process the sequence F. inThe local features at medium to high resolution are downsampled using max pooling to obtain a second 3D tensor of H×W×C. The first and second 3D tensors are then superimposed to obtain an input tensor of (2×H×W)×C, and a positional encoding is embedded in it. This input tensor with the embedded positional encoding is then fed into the Transformer module to obtain a first feature map of H×W×c and a second feature map of H×W×C. The first feature map is upsampled using bilinear interpolation to obtain a first original resolution, and the second feature map is upsampled using bilinear interpolation to obtain a second original resolution. The first original resolution is then compared with the sequence F. in The first splicing feature is obtained by splicing, and the second original resolution is combined with sequence F. in The first concatenation feature is obtained by concatenating the second concatenation feature, and the second concatenation feature is then concatenated with the first concatenation feature to obtain the output feature map F. out Where H represents height, W represents width, and C represents channel.

[0015] The output of the bilateral cross-enhancement unit of the first spatial aggregation block is local context information with a feature vector dimension of 32, and the output of the Transformer self-attention mechanism unit of the first spatial aggregation block is a 32-dimensional feature map.

[0016] The output of the bilateral cross-enhancement unit of the second spatial aggregation block is local context information with a feature vector dimension of 128, and the output of the Transformer self-attention mechanism unit of the second spatial aggregation block is a 128-dimensional feature map.

[0017] The output of the bilateral cross-enhancement unit of the third spatial aggregation block is local context information with a feature vector dimension of 256, and the output of the Transformer self-attention mechanism unit of the third spatial aggregation block is a 256-dimensional feature map.

[0018] The output of the bilateral cross-enhancement unit of the fourth spatial aggregation block is local context information with a feature vector dimension of 512, and the output of the Transformer self-attention mechanism unit of the fourth spatial aggregation block is a 512-dimensional feature map.

[0019] The output of the bilateral cross-enhancement unit of the fifth spatial aggregation block is local context information with a feature vector dimension of 1024, and the output of the Transformer self-attention mechanism unit of the fifth spatial aggregation block is a 1024-dimensional feature map.

[0020] The specific process of step ④ is as follows:

[0021] Feature maps of different resolutions are upsampled to obtain feature maps of different scales. These feature maps of different scales are then input into a multilayer perceptron to obtain point-level information. The point-level information is then normalized using the softmax function to obtain normalized point-level information. This normalized point-level information is then fused with feature maps of different scales to obtain a comprehensive feature map for semantic segmentation.

[0022] Compared with existing technologies, the advantages of this invention lie in processing the preliminary geometric context information P and preliminary semantic context information F through a spatial aggregation module based on bilateral cross-enhancement and self-attention compensation to obtain feature maps of different resolutions. This spatial aggregation module consists of five cascaded spatial aggregation blocks, each composed of a bilateral cross-enhancement unit and a Transformer self-attention mechanism unit. The bilateral cross-enhancement unit utilizes the preliminary geometric and semantic context information of the point cloud as bilateral input information to enhance the local context information of each point in the point cloud. The Transformer self-attention mechanism unit extracts global context information, redistributes feature weights in local regions to closely link them to the global context, compensates for the singularity of local context information, and enhances the completeness of features, thereby improving the accuracy of fine-grained segmentation of the point cloud. This approach comprehensively considers not only the relationship between local geometric information and global features but also the relationship between local semantic information and global features. The present invention achieves a mean IoU (mean Intersection over Union) of 70.2%, a mean class accuracy (mAcc) of 81.7%, and an overall accuracy (OA) of 88.3% on the S3DIS dataset after six-fold cross-validation. Compared with the PointNet method, mIoU is improved by 22.6%, mAcc by 15.5%, and OA by 9.7%. Compared with the RandLA-Net method, mIoU is improved by 2.4%, mAcc by 2%, and OA by 1%. Therefore, the present invention has better segmentation performance. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0024] Figure 2 This is a schematic diagram of the spatial aggregation block in this invention;

[0025] Figure 3 This is a schematic diagram comparing the experimental results of the present invention with the RandLA-Net method; Detailed Implementation

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0027] like Figure 1and 2 As shown, the point cloud semantic segmentation method based on bilateral cross-enhancement and self-attention compensation includes the following steps:

[0028] ① Input the point cloud data to be segmented into the semantic segmentation model. The semantic segmentation model includes a preprocessing module, a spatial aggregation module based on bilateral cross-enhancement and self-attention compensation, a multi-scale feature fusion module, and a fully connected layer. The spatial aggregation module based on bilateral cross-enhancement and self-attention compensation consists of five cascaded spatial aggregation blocks. Each spatial aggregation block consists of a bilateral cross-enhancement unit and a Transformer self-attention mechanism unit. The output of the bilateral cross-enhancement unit is the input of the Transformer self-attention mechanism unit.

[0029] ② In the preprocessing module, the original coordinates of the point cloud data to be segmented in three-dimensional space are used as preliminary geometric context information; feature extraction is performed on the point cloud data to be segmented using a multilayer perceptron to obtain preliminary semantic context information;

[0030] ③ The preliminary geometric context information and preliminary semantic context information are input into the bilateral cross-enhancement unit of the first spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after one downsampling are input into the bilateral cross-enhancement unit of the second spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after two downsamplings are input into the bilateral cross-enhancement unit of the third spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after three downsamplings are input into the bilateral cross-enhancement unit of the fourth spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after four downsamplings are input into the bilateral cross-enhancement unit of the fifth spatial aggregation block. The feature maps of different resolutions are output from the Transformer self-attention mechanism units of the five spatial aggregation blocks.

[0031] ④ Input feature maps of different resolutions into the multi-scale feature fusion module to obtain a comprehensive feature map for semantic segmentation;

[0032] ⑤ Input the comprehensive feature map used for semantic segmentation into the fully connected layer to obtain the semantic segmentation result of the point cloud data to be segmented;

[0033] In this embodiment, the specific process of obtaining the output of the bilateral cross-reinforcement unit is as follows:

[0034] A point is randomly selected from the point cloud data as the center point. The 3D-KNN algorithm is used to search for and obtain the nearest neighbors of the center point. The formula is then used to... Calculate the local geometric context information Among them, P i Represents the absolute coordinates of the center point. Represents the absolute coordinates of the nearest neighbor. This represents the relative coordinates of the nearest neighbor points with respect to the center point. This represents the concatenation operation symbol, k=16, obtained through the formula. Calculate the local semantic context information Where Fi represents the semantic feature corresponding to the center point in the semantic feature space. This represents the adjacent semantic features corresponding to the nearest neighbor points in the semantic feature space. The semantic features of nearest neighbors relative to the center point are represented by the formula. Offsets that provide local geometric context information Where M(·) represents the multilayer perceptron operation, as shown by the formula Obtain the offset of local semantic context information Through formula The enhanced local context information Le is calculated and used as the output of the bilateral cross-enhancement unit, where concat(·) represents the connection operation and M(·) represents the multilayer perceptron operation; if in the bilateral cross-enhancement unit of the first spatial aggregation block, Figure 2 In this context, P represents the preliminary geometric context information of the center point, and F represents the preliminary semantic context information of the center point; if in the bilateral cross-enhancing unit of the second spatial aggregation block... Figure 2 In this context, P represents the initial geometric context information of the center point, and F represents the initial semantic context information of the center point after one downsampling. If in the bilateral cross-enhancement unit of the third spatial aggregation block... Figure 2 In this context, P represents the initial geometric context information of the center point, and F represents the initial semantic context information of the center point after two downsampling steps; if in the bilateral cross-enhancement unit of the fourth spatial aggregation block... Figure 2 In this context, P represents the initial geometric context information of the center point, and F represents the initial semantic context information of the center point after three downsampling steps; if in the bilateral cross-enhancement unit of the fifth spatial aggregation block... Figure 2 In this context, P represents the initial geometric context information of the center point, and F represents the initial semantic context information of the center point after four downsampling steps.

[0035] In this embodiment, the specific process of obtaining the feature map output by the Transformer self-attention mechanism unit is as follows:

[0036] The enhanced local context information Le is converted into a sequence F consisting of local features arranged in descending order of resolution. in For sequence F inThe first three-dimensional tensor, H×W×C, is obtained by performing average pooling downsampling on the local features at medium to high resolution. This tensor is then used to process the sequence F. in The local features at medium to high resolution are downsampled using max pooling to obtain a second 3D tensor of H×W×C, where H represents height, W represents width, and C represents channels. The first and second 3D tensors are superimposed to obtain an input tensor of (2×H×W)×C, and a positional encoding is embedded. The input tensor with the embedded positional encoding is input into the Transformer module to obtain a first feature map of H×W×C and a second feature map of H×W×C. The first feature map is upsampled using bilinear interpolation to obtain a first original resolution, and the second feature map is upsampled using bilinear interpolation to obtain a second original resolution. The first original resolution is then compared with the sequence F. in The first splicing feature is obtained by splicing, and the second original resolution is combined with sequence F. in The first concatenation feature is obtained by concatenating the second concatenation feature, and the second concatenation feature is then concatenated with the first concatenation feature to obtain the output feature map F. out The embedded positional encoding enables the semantic segmentation model to understand the spatial relationships between different tags when training it.

[0037] In this embodiment, the output of the bilateral cross-enhancement unit of the first spatial aggregation block is local context information with a feature vector dimension of 32, and the output of the Transformer self-attention mechanism unit of the first spatial aggregation block is a 32-dimensional feature map.

[0038] The output of the bilateral cross-enhancement unit of the second spatial aggregation block is local context information with a feature vector dimension of 128, and the output of the Transformer self-attention mechanism unit of the second spatial aggregation block is a 128-dimensional feature map.

[0039] The output of the bilateral cross-enhancement unit of the third spatial aggregation block is local context information with a feature vector dimension of 256, and the output of the Transformer self-attention mechanism unit of the third spatial aggregation block is a 256-dimensional feature map.

[0040] The output of the bilateral cross-enhancement unit of the fourth spatial aggregation block is local context information with a feature vector dimension of 512, and the output of the Transformer self-attention mechanism unit of the fourth spatial aggregation block is a 512-dimensional feature map.

[0041] The output of the bilateral cross-enhancement unit of the fifth spatial aggregation block is local context information with a feature vector dimension of 1024, and the output of the Transformer self-attention mechanism unit of the fifth spatial aggregation block is a 1024-dimensional feature map.

[0042] In this embodiment, the specific process of step ④ is as follows: upsampling feature maps of different resolutions to obtain feature maps of different scales, inputting feature maps of different scales into a multilayer perceptron to obtain point-level information, normalizing the point-level information using the softmax function to obtain normalized point-level information, and fusing the normalized point-level information with feature maps of different scales to obtain a comprehensive feature map for semantic segmentation.

[0043] The experiment used the Ubuntu 16.04 operating system and implemented the invention based on the Python and Tensorflow platforms.

[0044] The semantic segmentation model in this invention was trained for 100 epochs on a single NVIDIA TITAN RTX GPU using the Adam optimization algorithm in an end-to-end manner. The training set was the S3DIS dataset, with a batch size of 16. The S3DIS dataset was collected from indoor work environments and consisted of six large indoor areas (area1, area2, area3, area4, area5, and area6), containing a total of 271 rooms. Each room was composed of a dense point cloud, and all points had 3D coordinate information and RGB information. The entire dataset was labeled with 13 different categories (including: ceiling, floor, wall, beam, column, window, door, table, chair, sofa, bookcase, wooden board, and clutter).

[0045] Table 1. Results of six-fold cross-validation and area 5 individual test on the S3DIS dataset (%)

[0046]

[0047] As shown in Table 1, the present invention achieves a mean IoU (mean intersection-union ratio) of 70.2%, a mean class accuracy (mAcc) of 81.7%, and an overall accuracy (OA) of 88.3% on the S3DIS dataset after six-fold cross-validation. Compared with the PointNet method, the mIoU is improved by 22.6%, the mAcc by 15.5%, and the OA by 9.7%. Compared with the RandLA-Net method, the mIoU is improved by 2.4%, the mAcc by 2%, and the OA by 1%. Therefore, the present invention has better segmentation performance.

[0048] In the S3DIS dataset, area 5 is more challenging to segment due to the complexity of the point cloud scene. Therefore, semantic segmentation is performed separately on area 5. As shown in Table 1, the mIoU of the present invention in area 5 is 66.2%, and the segmentation effect of the present invention is significantly better than that of the existing methods.

[0049] The process of six-fold cross-validation is as follows: The S3DIS dataset contains six regions. First, it is trained on areas 2-6 and tested on area 1; then it is trained on areas 1 and 3-6 and tested on area 2; and so on, until each region is tested, and the average test result is calculated.

[0050] like Figure 3 As shown, even in complex scenes such as offices, meeting rooms, and corridors, the present invention can segment the input scene with high quality, demonstrating its effectiveness. Visual analysis selected four scene graphs; compared to the RandLA-Net method, the results produced by the present invention are more accurate and consistent, especially for categories such as walls, blackboards, and clutter. Figure 3 A medium-thick black border indicates a zoomed-in scene, making it easier to observe subtle segmentation effects. Scene 1 clearly demonstrates the superior segmentation of walls and beams. The RandLA-Net method incorrectly segments some beams as walls, while this invention correctly distinguishes between them. Scene 1 was selected from area 3 of the S3DIS dataset. By testing the average intersection-union ratio (mIoU) of walls and beams in area 3, the mIoU of the RandLA-Net method for walls and beams were 80.02 and 69.77, respectively, while the mIoU of this invention were 82.60 and 71.89, respectively. The data clearly shows the superiority of our algorithm over the RandLA-Net algorithm in segmenting walls and beams. Scenes 2, 3, and 4 illustrate subtle segmentations between objects, such as between beams and blackboards, and between tables and chairs. Compared to the RandLA-Net method, the objects segmented by this invention have smoother edges and better local segmentation. This is mainly due to the cross-learning of geometric and semantic context information, which enhances local contextual information and improves the accuracy of fine-grained local segmentation.

[0051] Figure 3 In this context, Seene 1 represents Scene 1, Seene 2 represents Scene 2, Seene 3 represents Scene 3, Seene 4 represents Scene 4, Input represents input, Ground Truth represents the original annotation, and Proposed represents the present invention.

[0052] Definitions of terms in this patent:

[0053] 3D-KNN algorithm (K-Nearest Neighbor algorithm based on 3D Euclidean space) Reference: GitHub-Crowbar97 / 3D-KNN: Chamfer distance-based KNN algorithm for point cloud models comparison.

[0054] Reference for Transformer self-attention mechanism: Yang Jun, Li Bozan. Semantic segmentation of 3D point clouds based on self-attention feature fusion group convolutional neural network [J]. Optics and Precision Engineering, 2022, 30(07): 840-853.

[0055] Multi-scale feature fusion module references adaptive fusion module:

[0056] https: / / blog.csdn.net / u014311125 / article / details / 121979997?ops_request_misc=%257B%2522request%255Fid%2522%253A%2 522167651603116782425148936%2522%252C%2522scm%2522%253A%252220140713.130102334..%2522%257D&request_id=1676516031 16782425148936&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~sobaiduend~default-2-1219799 97-null-null.142^v73^pc_new_rank, 201^v4^add_ask, 239^v1^insert_chatgpt&utm_term=baaf-net&spm=1018.2226.3001.4187.

[0057] References for the PointNet method: Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J.Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, pp.77-85, 2017.

[0058] References for the DeepGCN method: Guohao Li, Matthias Müller, Guocheng Qian, Itzel C. Delgadillo, Abdulellah Abualshour, Ali K. Thabet and Bernard Ghanem. DeepGCNs: Making GCNs Go as Deep as CNNs. Transactions on Pattern Analysis and Machine Intelligence, IEEE, 2021.

[0059] References for the SPG method: Landrieu, and Martin Simonovsky. Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs. Conference on Computer Vision and Pattern Recognition (CVPR), CVF / IEEE, pp. 4558 - 4567, 2018.

[0060] References for the PAT method: Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling Point Clouds with Self-Attention and Gumbel Subset Sampling. Conference on Computer Vision and Pattem Recognition (CVPR), CVF / IEEE, pp. 3323 - 3332. 2019.

[0061] References for the PointCNN method: Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic Graph CNN for Learnning on Point Clouds. Acm Transactions on Graphics (tog), pp: 1 - 12, 2019.

[0062] References for the PointWeb method: Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and JiayaJia. PointWeb: Enhancing local neighborhood features for point cloudprocessing. Conference on ComputerVision and Pattem Recognition (CVPR), CVF / IEEE, pp.5565-5573, 2019.

[0063] References for the SSP+SPG method: Landrieu and M o hamed Boussaha. Point cloudoversegmentation with graph-structured deepmetric learning. Conference on Computer Vision and Pattern Recognition (CVPR), CVF / IEEE, pp.7440-7449, 2019.

[0064] References for the RandLA-Net method: Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. RandLA-Net: EfficientSemantic Segmentation of Large-Scale Point C10uds. Conference on ComputerVision and Pattem Recognition(CVPR), CVF / IEEE, pp.11105-11114, 2020.

Claims

1. A point cloud semantic segmentation method based on bilateral cross-enhancement and self-attention compensation, characterized in that... Includes the following steps: ① Input the point cloud data to be segmented into the semantic segmentation model. The semantic segmentation model includes a preprocessing module, a spatial aggregation module based on bilateral cross-enhancement and self-attention compensation, a multi-scale feature fusion module, and a fully connected layer. The spatial aggregation module based on bilateral cross-enhancement and self-attention compensation consists of five cascaded spatial aggregation blocks. Each spatial aggregation block consists of a bilateral cross-enhancement unit and a Transformer self-attention mechanism unit. The output of the bilateral cross-enhancement unit is the input of the Transformer self-attention mechanism unit. ② In the preprocessing module, the original coordinates of the point cloud data to be segmented in three-dimensional space are used as preliminary geometric context information; Feature extraction is performed on the point cloud data to be segmented using a multilayer perceptron to obtain preliminary semantic context information; ③ The preliminary geometric context information and preliminary semantic context information are input into the bilateral cross-enhancement unit of the first spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after one downsampling are input into the bilateral cross-enhancement unit of the second spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after two downsamplings are input into the bilateral cross-enhancement unit of the third spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after three downsamplings are input into the bilateral cross-enhancement unit of the fourth spatial aggregation block. The preliminary geometric context information and the preliminary semantic context information after four downsamplings are input into the bilateral cross-enhancement unit of the fifth spatial aggregation block. The feature maps of different resolutions are output from the Transformer self-attention mechanism units of the five spatial aggregation blocks. ④ Input feature maps of different resolutions into the multi-scale feature fusion module to obtain a comprehensive feature map for semantic segmentation; ⑤ Input the comprehensive feature map used for semantic segmentation into the fully connected layer to obtain the semantic segmentation result of the point cloud data to be segmented; The specific process for obtaining the output of the bilateral cross-enhancing unit is as follows: A point is randomly selected from the point cloud data as the center point. The 3D-KNN algorithm is used to search for and obtain the nearest neighbors of the center point. The formula is then used to... Calculate the local geometric context information Among them, P i Represents the absolute coordinates of the center point. Represents the absolute coordinates of the nearest neighbor. This represents the relative coordinates of the nearest neighbor points with respect to the center point. This represents the concatenation operation symbol, k=16, obtained through the formula. Calculate the local semantic context information Among them, F i This represents the semantic features corresponding to the center point in the semantic feature space. This represents the adjacent semantic features corresponding to the nearest neighbor points in the semantic feature space. The semantic features of nearest neighbors relative to the center point are represented by the formula. Offsets that provide local geometric context information Through formula Obtain the offset of local semantic context information Through formula The enhanced local context information Le is calculated and used as the output of the bilateral cross-enhancement unit, where concat(·) represents the connection operation and M(·) represents the multilayer perceptron operation.

2. The point cloud semantic segmentation method based on bilateral cross-enhancement and self-attention compensation according to claim 1, characterized in that... The specific process for obtaining the feature map output by the Transformer self-attention mechanism's working unit is as follows: The enhanced local context information Le is converted into a sequence F consisting of local features arranged in descending order of resolution. in For sequence F in The first three-dimensional tensor, H×W×C, is obtained by performing average pooling downsampling on the local features at medium to high resolution. This tensor is then used to process the sequence F. in The local features at medium to high resolution are downsampled using max pooling to obtain a second 3D tensor of H×W×C. The first and second 3D tensors are then superimposed to obtain an input tensor of (2×H×W)×C, and a positional encoding is embedded in it. This input tensor with the embedded positional encoding is then fed into the Transformer module to obtain a first feature map of H×W×C and a second feature map of H×W×C. The first feature map is upsampled using bilinear interpolation to obtain a first original resolution, and the second feature map is upsampled using bilinear interpolation to obtain a second original resolution. The first original resolution is then compared with the sequence F. in The first splicing feature is obtained by splicing, and the second original resolution is combined with sequence F. in The first concatenation feature is obtained by concatenating the second concatenation feature, and the second concatenation feature is then concatenated with the first concatenation feature to obtain the output feature map F. out Where H represents height, W represents width, and C represents channel.

3. The point cloud semantic segmentation method based on bilateral cross-enhancement and self-attention compensation according to claim 2, characterized in that... The output of the bilateral cross-enhancement unit of the first spatial aggregation block is local context information with a feature vector dimension of 32, and the output of the Transformer self-attention mechanism unit of the first spatial aggregation block is a 32-dimensional feature map. The output of the bilateral cross-enhancement unit of the second spatial aggregation block is local context information with a feature vector dimension of 128, and the output of the Transformer self-attention mechanism unit of the second spatial aggregation block is a 128-dimensional feature map. The output of the bilateral cross-enhancement unit of the third spatial aggregation block is local context information with a feature vector dimension of 256, and the output of the Transformer self-attention mechanism unit of the third spatial aggregation block is a 256-dimensional feature map. The output of the bilateral cross-enhancement unit of the fourth spatial aggregation block is local context information with a feature vector dimension of 512, and the output of the Transformer self-attention mechanism unit of the fourth spatial aggregation block is a 512-dimensional feature map. The output of the bilateral cross-enhancement unit of the fifth spatial aggregation block is local context information with a feature vector dimension of 1024, and the output of the Transformer self-attention mechanism unit of the fifth spatial aggregation block is a 1024-dimensional feature map.

4. The point cloud semantic segmentation method based on bilateral cross-enhancement and self-attention compensation according to claim 1, characterized in that... The specific process of step ④ is as follows: Feature maps of different resolutions are upsampled to obtain feature maps of different scales. These feature maps of different scales are then input into a multilayer perceptron to obtain point-level information. The point-level information is then normalized using the softmax function to obtain normalized point-level information. This normalized point-level information is then fused with feature maps of different scales to obtain a comprehensive feature map for semantic segmentation.

Images