Point cloud segmentation method based on global feature learning and local feature discriminative aggregation

Abstract

The application introduces a point cloud segmentation method based on global feature learning and local feature discriminant aggregation, which comprises the following steps: step S1, designing an FPE module to enrich the features of each point cloud and expand the global receptive field; step S2, designing an LFDA module to learn more distinctive local features; step S3, designing a GFC module based on a capsule network; step S4, designing a GCM module for building global correlation; step S5, through cascading multiple global and local information encoding modules, the receptive field of the points is constantly expanded while being down-sampled; step S6, improving the weighted cross-entropy loss function to solve the problem of uneven distribution of shield tunnel typical element category data; step S7, obtaining the final semantic prediction through three fully connected layers and a Dropout layer. The application directly performs semantic segmentation on large-scale 3D shield tunnel point cloud data, avoids any preprocessing or postprocessing, and further realizes rapid segmentation of shield tunnel typical elements.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a point cloud segmentation method based on global feature learning and local feature discriminative aggregation. BACKGROUND

[0002] As important urban infrastructure, shield metro tunnels need rapid and efficient detection methods to maintain normal operation of the tunnel. In shield metro tunnels, some typical elements such as longitudinal and circumferential joints, bolt holes and the like are more prone to deformation, cracking and other problems due to tunnel structure design, thus threatening the safety of the tunnel and becoming the focus of attention in tunnel safety monitoring. In the current traditional metro tunnel measurement and monitoring tasks, instruments such as total stations and profile meters or photogrammetry are mainly used to obtain tunnel 3D point clouds. This method of obtaining tunnel point clouds has the following defects: (1) uneven density, the manual operation mode of the traditional scanner makes the generated point cloud density uneven; (2) low efficiency, the fixed acquisition mode of the traditional scanner makes the generation efficiency of the tunnel point cloud low; (3) huge workload, the traditional instrument usually needs to be manually moved to change the measurement location.

[0003] Compared with the traditional method, the mobile laser scanning technology has the advantages of high work efficiency, large acquisition scale and high data precision, and overcomes the shortcomings of the traditional method, and is thus widely used in related work of metro tunnel monitoring.

[0004] Meanwhile, in recent years, deep learning has received extensive attention from various research fields. It greatly improves the fitting speed of deep models by using the backpropagation algorithm to discover complex patterns in large datasets. Although there are many deep learning models for 3D point clouds, and they have achieved very good results on public datasets, these models have poor segmentation effects in shield tunnel 3D point cloud data. With the rapid development of three-dimensional acquisition technology, many researchers have applied deep learning to 3D data-related tasks. Typical deep learning methods for 2D images cannot be directly applied to three-dimensional point clouds due to their inherent irregularity. PointNet(Qi et al., 2017a) directly takes point clouds as input and achieves permutation invariance through a symmetric function. Specifically, PointNet learns point-wise features independently through multiple MLP layers and extracts global features through a max-pooling layer. However, due to the structure of PointNet itself, it cannot capture local neighborhood information. Therefore, Qi et al.(2017b) proposed a multi-level neural network, PointNet++, which combines sampling layers, grouping layers, and uses PointNet to capture fine neighborhood features from the geometric neighborhood of each point. In PointConv(Wu et al., 2019), convolution is defined as a Monte Carlo estimation of 3D convolution with respect to importance sampling. The convolution kernel is composed of a weighting function (learned through an MLP layer) and a density function (learned through kernel density estimation and an MLP layer). To optimize memory and improve computational efficiency, the authors further optimize 3D convolution through matrix multiplication and 2D convolution. Under the same parameter settings, memory consumption can be reduced by about 64 times. By establishing a local coordinate system for neighbor points, PointCNN(Li et al., 2018) proposes a χ-conv (implemented through an MLP) operator, which transforms input points into a latent standard order through χ-conv transformation, and then applies a typical convolution operator to the transformed features. To efficiently and quickly segment large-scale point clouds, (Hu et al., 2020) proposed a high-efficiency, lightweight network, RandLA-Net, for 3D point cloud segmentation. This network uses random sampling technology to reduce memory consumption and improve computational efficiency.In addition, the network further proposes a local feature aggregation module to capture local geometric features and enhance the receptive field. Similarly, in order to quickly process large-scale point clouds, FG-Net (Liu et al., 2020) processes noise and outliers through a new filtering method, and in order to further learn the potential features of the input point cloud, the authors design a correlated feature mining module and a deformable convolution based geometric-aware modelling to fully exploit local feature dependencies and geometric patterns. Zhao et al. (2019) proposed 3D capsule networks, the input data first passes through independent convolutional layers with different weight matrices to mine deep features, then through pooling and splicing operations to combine into primary point capsules. Finally, the primary point capsules are mapped to latent capsules through a dynamic routing mechanism. The high-level capsules are a higher level of summary of the feature information in the primary capsules. This method can be used in multiple fields, such as object classification, object reconstruction, and partial segmentation. In PointASNL (Yan et al. 2020), the authors use an adaptive sampling module (Adaptive Sampling (AS) module) to adaptively adjust the coordinates and features of the points sampled by the furthest point sampling (FPS) algorithm. And through the local-nonlocal (L-NL) module to capture the local features and long range dependencies of these sampled points. Engel et al. (2021) proposed Point Transformer, the authors established local vector attention and used position encoding at the same time to distinguishably learn the neighbor point features, this model achieved state-of-the-art results on Stanford 3D Indoor Scene Dataset (S3DIS) (Armeni et al., 2016). The Point Cloud Transformer (PCT) proposed by Tsinghua University research team (Guo et al., 2021), this method extends the Transformer (Vaswani et al., 2017) which has been applied to Nature Lanuage Process (NLP) and achieved remarkable results to 3D point cloud data tasks, and combined with the characteristics of 3D data for improvement, this method also achieved good results.To simultaneously use the coordinate information and feature information of 3D point cloud, BAAF-Net (Qiu et al., 2021) proposes a bilateral context module that learns the offset from each other between coordinate information and feature information to expand the local context of each point. To construct a feature map that can comprehensively explain the information of each point, the paper also proposes an adaptive fusion module that obtains different expressions of original point information by upsampling from different resolutions, and then uses attention mechanism to fuse different expressions of original point information to obtain the final output of original point information. SCF-Net (Fan et al., 2021) constructs a local polar coordinate system representation that is not affected by Z-axis rotation for the local neighborhood, and aggregates neighbor information through a dual-distance attentive pooling module. Although deep learning network models applied to point cloud data have been widely used, the most advanced network models currently cannot well solve the element segmentation task of real point cloud of shield tunnel.

[0005] Chinese patent CN111489358A discloses a three-dimensional point cloud semantic segmentation method based on deep learning, belonging to the field of three-dimensional point cloud and pattern recognition. It includes training a semantic segmentation neural network model using a three-dimensional point cloud training set, with labels being real semantic categories. The semantic segmentation neural network model includes a feature extraction network and a semantic segmentation network. The feature extraction network is used to extract global and local features of the three-dimensional point cloud. The semantic segmentation network is used to fuse the global and local features of the point cloud, and the output feature map corresponds to the probability of each point belonging to each semantic category. The method inputs the point cloud to be measured into the trained semantic segmentation neural network model to obtain the segmentation result of the point cloud. The designed method uses a local feature extraction module to extract multiple scale local features of the point cloud, a channel attention enhancement module to enhance the attention of important feature channels and suppress unimportant feature channels, and a weighted multi-class loss function to optimize the training effect, thereby improving the accuracy of the semantic segmentation method.

[0006] Chinese patent CN113449744A discloses a three-dimensional point cloud semantic segmentation method based on deep feature expression, which comprises the following steps: step 1: collecting three-dimensional point cloud related data sets for processing, which are used for model training; step 2: during the model training process, the data is preprocessed; step 3: a deep network model is established, the point cloud data is input into the edge feature module to obtain the edge feature, the obtained feature tensor is input into the attention weight module; the above operation is repeated multiple times, and the obtained results are integrated and input into the semantic segmentation module to obtain the final semantic label; step 4: the data obtained after step 2 is processed is sent to the network parameters established in step 3 for training; step 5: the model is tested. The designed method realizes sufficient extraction of the implicit information in the point cloud, solves the problem of local information loss in the three-dimensional point cloud feature processing process, reduces the required parameter amount, has high precision and fast speed.

[0007] However, although many deep learning models have been applied to 3D data, and many models have achieved good results on challenging large public data sets, these models cannot be directly applied to shield tunnel 3D point cloud data. Two challenges are mainly faced: a network architecture needs to be designed to efficiently capture the hidden features and geometric patterns of shield tunnel 3D data; and there is a serious imbalance problem between typical elements of shield tunnels, so this problem needs to be solved in network design. SUMMARY

[0008] To solve the above problems, an efficient deep learning network structure is designed, which can directly perform semantic segmentation on large-scale 3D shield tunnel point cloud data, and then realize that no pre-processing or post-processing is needed, and finally realize fast segmentation of typical elements of shield tunnels.

[0009] To achieve the above effects, the present application designs a point cloud segmentation method based on global feature learning and local feature discriminative aggregation.

[0010] The point cloud segmentation method based on global feature learning and local feature discriminative aggregation comprises the following steps:

[0011] Step S1, a feature perception enhancement FPE module is designed to enrich the features of each point cloud and expand the global receptive field;

[0012] Step S2, a local information mining LFDA module is designed to learn more discriminative local features;

[0013] Step S3, a global feature capture GFC module is designed based on a capsule network;

[0014] Step S4, a global correlation modeling (GCM) module based on self-attention is designed to construct global correlation;

[0015] Step S5, the global and local information capturing module is composed of FPE, LFDA and GFC, and by cascading multiple global and local information encoding modules, the receptive field of the point is constantly expanded while down-sampling;

[0016] Step S6, the weighted cross-entropy loss function is improved to solve the problem of uneven distribution of shield tunnel typical element category data;

[0017] Step S7, the final semantic prediction is obtained through three fully connected layers and a Dropout layer.

[0018] Preferably, in the step S1, the FPE module comprises a similarity feature encoding module and a hybrid pooling module.

[0019] Preferably, the network input of the similarity feature encoding module is an original point cloud P∈R N×(3+d) , which contains 3D coordinates and feature information,

[0020] The 3D coordinates are (S∈R N×3 , S={s1,…,s i ,…,s N}, s i ∈R 1×3 ),

[0021] The feature information is (F∈R N×d , F={f1,…,f i ,…,f N}, f i ∈R 1×d ),

[0022] In the formula, P represents all input point clouds, N represents the number of input points, and d is the dimension of the feature information of the input point cloud; the i-th point in P can be regarded as p i =(s i ,f i ) T .

[0023] Preferably, the method using the similarity feature encoding module is as follows:

[0024] Step S101, in the similarity feature encoding module, first, the similarity degree of points based on feature space and Euclidean space is measured by using Euclidean distance similarity, and a similarity matrix is constructed, which is shown in formula (1):

[0025]

[0026] In the formula, Θ represents pi p j x i , y i represents the value of the vector p i and p j in the i-th dimension;

[0027] Step S102, selecting the most similar κ neighboring points for each point according to the similarity matrix

[0028] In order to make the neighboring points have rich spatial information, the features of the neighboring points are re-encoded in the form of formula (2):

[0029]

[0030] In the formula, represents the splicing operation, represents the encoded feature of the k-th neighboring point of the center point i, s i represents the spatial coordinates of the center point i, represents the input feature of the k-th neighboring point of the center point i, represents the spatial coordinates of the k-th neighboring point of the center point i;

[0031] Step S103, learning the high-dimensional representation of the neighboring point features through a multi-layer perception machine (MLP) operation: In the formula, represents the feature of the k-th neighboring point of the center point i after learning.

[0032] Preferably, the method using the hybrid pooling module comprises:

[0033] Step S111, directly using the maximum value pooling to obtain the significant features H max of the neighbor points;

[0034] Step S112, using the average value pooling to obtain the overall features H mean of the neighbor points;

[0035] Step S113, starting from the neighbor point features H itself, obtaining the attention score of each feature of the neighbor points by learning the high-dimensional mapping of the neighbor point features and through the standardization function Softmax The calculation process is as shown in formula (3):

[0036]

[0037] In the formula, W ∈ R d’×d’ , d' represents the dimension of, which is the learnable weight in the full connection layer;

[0038] ​Step S114: Utilize attention scores We obtain the weighted summation of neighbor features.

[0039] Step S115: Connect the components using residual joins. and Connect, get

[0040] Step S116: Obtain the aggregated feature H by summing channel by channel. w To learn the local differences of neighboring points;

[0041] Step S117: Concatenate the pooling results of the three parts to obtain the final output H. out =[Η w ,Η max ,Η mean ].

[0042] Preferably, in step S2, the principle of the local information mining module is as follows:

[0043] Step S21: For the input point cloud P, first use the K-Nearest Neighbors (KNN) algorithm to find the K nearest neighbor points in Euclidean space;

[0044] Step S22: Treat the center point and its neighboring points as a graph structure: G = {V, E}, where V represents a node, E represents an edge connecting the center point and its neighboring points, and V = {1, 2, ..., N}.

[0045] Step S23: Then, obtain the features of adjacent edges using the spatial coordinates and features of the vertices. The neighboring edge feature can be regarded as the connection relationship between the center point and the neighboring points. The neighboring edge feature is encoded according to formula (4):

[0046]

[0047] In the formula The neighboring edge characteristics of the k-th neighbor of the center point i are represented. Let represent the inverse distance coefficient of the k-th neighbor of center point i. This coefficient decreases as the distance increases. The inverse distance coefficient is calculated as shown in formula (5-6):

[0048]

[0049]

[0050] In formula (5) s i and Let i and k represent the spatial coordinates of the center point i and the kth neighbor point, respectively. wherein denotes the feature of the k-th neighbor point of the center point i; wherein denotes the k th -dimensional coordinates of the neighbor point of the center point i, p i denotes the center point i; meanwhile, the new neighbor point feature is obtained

[0051] Step S24, for the neighbor edge feature set A shared function g(·) is designed to learn the attention score s of each feature, and the shared function g(·) is composed of a linear transformation followed by a softmax function, and its definition is as formula (7):

[0052]

[0053] wherein W is a learnable weight; the learned attention score can be regarded as a mask that can automatically select the neighbor point feature, which represents the relevance between the center point i and the k th neighbor point in the corresponding feature dimension;

[0054] Step S25, the features are weighted and summed according to formula (8):

[0055]

[0056] wherein · denotes point multiplication;

[0057] Step S26, in order to retain the features of each point itself while learning the features of the neighbor points, the extracted features are output by the following operation: wherein f i denotes the input feature of the center point i in this module, f out denotes the output feature of the center point i in this module.

[0058] Preferably, in the step S3, the method for capturing global features based on the capsule network is that a Global Feature Capture module is designed based on the capsule network;

[0059] The specific method is:

[0060] Step S31, the high-dimensional representation of the input feature is extracted point by point through the MLP, and these feature maps are input into multiple independent convolutional layers with different weights, and each layer can learn the input point cloud feature map from different angles;

[0061] Step S32, Max Pooling is used to obtain their global latent representation;

[0062] Step S33, these global latent representations are spliced into a feature vector set named primary point capsules (Primary Point Capsules), F ∈ R v×μ ; the size of F depends on the output dimension μ of each independent convolutional layer and the number v of independent convolutional layers;

[0063] Step S34, the primary point capsules are embedded into high-level point capsules F' of higher feature levels using a dynamic routing algorithm (as shown in Algorithm 1), F' ∈ R ζ×η .

[0064] Preferably, in the step S4, a self-attention-based global correlation modeling is designed, and a specific method for constructing global correlation is as follows:

[0065] Step S41, a self-attention-based global correlation modeling is designed to calculate the global semantic correlation of the input feature F G and establish global dependency relationships;

[0066] After the global correlation is established by the global correlation modeling in step S42, the global relationship in the features and geometry will be automatically captured, and then the feature representation with local and global semantic context correlation is adaptively obtained to facilitate the subsequent segmentation task;

[0067] Step S43, for the input feature F G ∈ R N’×L , where L = 1024. After linear transformation, respectively d1 = 256, and v ∈ R N’×L ;

[0068] Step S44, the correlation score matrix G R ∈ R N’×N’ is obtained by transpose dot product of l1 and l2, as formula (9):

[0069]

[0070] Where @ represents matrix multiplication; each element in G R gives the correlation score between l1 and l2;

[0071] Step S45, then the same scaling operation is performed on each element in G R , as formula (10):

[0072]

[0073] Where d k = 256;

[0074] Step S46, then the correlation score matrix G after scaling is standardized by a Softmax function s to obtain the latent representation F G The final autocorrelation weight G A ∈R N’×N’ ; G A Each element g y,z is calculated according to formula (11):

[0075]

[0076] In the formula, s y,z represents an element in G A , g y,z embodies the degree of correlation between element y and element z;

[0077] Step S47, finally, the input feature F G is calculated according to formula (12) by attention score

[0078]

[0079] Finally, the correlation between elements F G is calculated and the global dependency relationship is established, and the correlation weighted result The semantic segmentation result can be directly obtained by upsampling the aggregated latent feature .

[0080] Preferably, in step S6, the receptive field of the point is continuously expanded while being down-sampled by cascading multiple modules. The specific method is: first, calculate the proportion of the amount of data of each category according to formula (13), and then calculate the final weight according to formula (14);

[0081]

[0082] w i = exp (-r i ) (14);

[0083] In the formula, n i represents the number of point clouds of category i, r i represents the proportion of category i in the total amount of all categories, and w i represents the weight coefficient of category i.

[0084] The final weighted cross-entropy loss function l w is shown in formula (15);

[0085]

[0086] where c is the number of classes, y is the label, y = 1 when the class is i, and y = 0 otherwise. i i i represents the probability of class i output by the neural network.

[0087] Advantages and effects of the present application are as follows:

[0088] (1) The present application proposes a network model for semantic segmentation of 3D point cloud data of a shield tunnel. Global and local feature encoder blocks are used to learn global descriptors while differentially aggregating local features, and then a GCM block is used to establish global correlation. A weighted cross-entropy loss function is improved for the problem of unbalanced number of shield tunnel classes, and direct semantic segmentation of large-scale 3D shield tunnel point cloud data is performed, so that any preprocessing or postprocessing is no longer needed, and finally fast segmentation of typical elements of a shield tunnel is realized.

[0089] (2) The present application labels typical elements of 3D point cloud data of a shield metro tunnel, and constructs a shield tunnel dataset, which is convenient for data arrangement in the later stage.

[0090] (3) The present application compares with the existing most advanced semantic segmentation network model, and GL-Net obtains the optimal result in the task of semantic segmentation of 3D point cloud data of a shield tunnel. In the current experiment, it is improved by at least 14.5%.

[0091] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, so as to implement according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following will be described in detail with the preferred embodiments of the present application and the accompanying drawings.

[0092] The above and other purposes, advantages and characteristics of the present application will be more apparent to those skilled in the art from the following detailed description of specific embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0093] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings without creative labor on the basis of these drawings. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual proportions.​​

[0094] Figure 1 The structure is feature-aware enhancement; the left is a similar feature encoding module, and the right is a hybrid pooling module;

[0095] Figure 2 The feature enhancement diagram from FPE to LFDA;

[0096] Figure 3 The local information mining flowchart;

[0097] Figure 4 The global feature capture flowchart;

[0098] Figure 5 The global correlation modeling flowchart;

[0099] Figure 6 The shield tunnel provided by the application;

[0100] Figure 7 The original shield tunnel point cloud, the shield tunnel labeled point cloud, and the GL-Net segmentation result provided by the application;

[0101] Figure 8 The comparison experimental result diagram of Kappa and K;

[0102] Figure 9 The true value and the prediction diagram of different structures;

[0103] Figure 10 The semantic segmentation result diagram of different networks on shield tunnel data, and the green square represents a segmentation failure case;

[0104] Figure 11 The overall design diagram of the application. DETAILED DESCRIPTION

[0105] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. In the following description, specific details such as specific configurations and components are provided only to help a comprehensive understanding of the embodiments of the present application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, in order to be clear and concise, the description of known functions and structures is omitted in the embodiments.

[0106] It should be understood that every embodiment mentioned in the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "one embodiment" or "the embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0107] In addition, the reference numbers and / or letters can be repeated in different examples in the present application. Such repetition is for the purpose of simplification and clarity, and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0108] The term "and / or" herein is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, B alone, and A and B together. The term "and" herein is a description of another association relationship of the associated objects, which means that there can be two relationships, for example, A and B, which means that there are two cases of A alone and A and B together. In addition, the character " / " herein generally means that the associated objects before and after are in an "or" relationship.

[0109] The term "at least one" herein is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, at least one of A and B, which means that there are three cases of A alone, A and B together, and B alone.

[0110] It should also be noted that the relationship terms such as first and second used in the present application are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion.

[0111] Embodiment 1

[0112] This embodiment mainly introduces the basic design of the point cloud segmentation method based on global feature learning and local feature discrimination aggregation. For specific design, please refer to Figure 11 which includes the following steps:

[0113] Step S1, design a feature perception enhancement FPE module to enrich the features of each point cloud and expand the global receptive field;

[0114] Step S2, design a local information mining LFDA module to learn more distinctive local features;

[0115] Step S3, design a global feature capture GFC module based on a capsule network;

[0116] Step S4, a self-attention-based global correlation modeling GCM module is designed for constructing global correlation;

[0117] Step S5, the global and local information capturing module is composed of FPE, LFDA and GFC, and by cascading multiple global and local information encoding modules, the receptive field of the point is constantly expanded while down-sampling;

[0118] Step S6, the weighted cross-entropy loss function is improved to solve the problem of uneven distribution of shield tunnel typical element category data;

[0119] Step S7, the final semantic prediction is obtained through three fully connected layers and a Dropout layer.

[0120] Further, in the step S1, the FPE module comprises a similarity feature encoding module and a hybrid pooling module.

[0121] Further, the network input of the similarity feature encoding module is original point cloud P∈R N×(3+d) contains 3D coordinates and feature information,

[0122] The 3D coordinates are (S∈R N×3 ,S={s1,…,s i ,…,s N},s i ∈R 1×3 ),

[0123] The feature information is (F∈R N×d ,F={f1,…,f i ,…,f N},f i ∈R 1×d ),

[0124] In the formula, P represents all input point clouds, N represents the number of input points, and d is the dimension of the feature information of the input point cloud; the i-th point in P can be regarded as p i =(s i ,f i ) T .

[0125] Further, the method using the similarity feature encoding module is:

[0126] Step S101, in the similarity feature encoding module, first, the similarity degree of the points based on the feature space and the Euclidean space is measured by using the Euclidean distance similarity, and a similarity matrix is constructed, which is shown in formula (1):

[0127]

[0128] where Θ denotes p i ,p j dimension, x i , y i denotes the value of vector p i and p j in the i-th dimension;

[0129] Step S102, selecting the most similar κ neighboring points for each point according to the similarity matrix

[0130] In order to make the neighboring points have rich spatial information, the features of the neighboring points are re-encoded in the manner of formula (2):

[0131]

[0132] where [ ] denotes the splicing operation, denotes the encoded feature of the k-th neighboring point of the center point i, s i denotes the spatial coordinates of the center point i, denotes the input feature of the k-th neighboring point of the center point i, denotes the spatial coordinates of the k-th neighboring point of the center point i;

[0133] Step S103, learning the high-dimensional representation of the neighboring point features through a multi-layer perception (MLP) operation: where denotes the learned feature of the k-th neighboring point of the center point i.

[0134] Further, the method using the hybrid pooling module comprises:

[0135] Step S111, directly using the maximum value pooling to obtain the significant features H max of the neighbor points;

[0136] Step S112, using the average value pooling to obtain the overall features H mean of the neighbor points;

[0137] Step S113, starting from the neighbor point features itself, obtaining the attention score of each feature of the neighbor points through learning the high-dimensional mapping of the neighbor point features and through the standardization function Softmax The calculation process is as shown in formula (3):

[0138]

[0139] where W ∈ R d’×d’ , d' denotes the dimension, and is the learnable weight in the full connection layer;

[0140] Step S114: Utilize attention scores We obtain the weighted summation of neighbor features.

[0141] Step S115: Connect the components using residual joins. and Connect, get

[0142] Step S116: Obtain the aggregated feature H by summing channel by channel. w To learn the local differences of neighboring points;

[0143] Step S117: Concatenate the pooling results of the three parts to obtain the final output H. out =[Η w ,Η max ,Η mean ].

[0144] Furthermore, in step S2, the principle of the local information mining module is as follows:

[0145] Step S21: For the input point cloud P, first use the K-Nearest Neighbors (KNN) algorithm to find the K nearest neighbor points in Euclidean space;

[0146] Step S22: Treat the center point and its neighboring points as a graph structure: G = {V, E}, where V represents a node, E represents an edge connecting the center point and its neighboring points, and V = {1, 2, ..., N}.

[0147] Step S23: Then, obtain the features of adjacent edges using the spatial coordinates and features of the vertices. The neighboring edge feature can be regarded as the connection relationship between the center point and the neighboring points. The neighboring edge feature is encoded according to formula (4):

[0148]

[0149] In the formula This represents the neighbor edge characteristics of the k-th neighbor point of center point i. Let represent the inverse distance coefficient of the k-th neighbor of center point i. This coefficient decreases as the distance increases. The inverse distance coefficient is calculated as shown in formula (5-6):

[0150]

[0151]

[0152] In formula (5) s i and respectively represent the spatial coordinate values of the center point i and the k-th neighbor point; represent the features of the k-th neighbor point of the center point i; represent the k-th neighbor point of the center point i th three-dimensional coordinates of the neighbor point, p i represent the center node i; and obtain new neighbor point features

[0153] Step S24, for the neighbor edge feature set A shared function g(·) is designed to learn the attention score s of each feature, and the shared function g(·) is composed of a linear transformation followed by a softmax function, and its definition is as formula (7):

[0154]

[0155] where W is a learnable weight; the learned attention score can be regarded as a mask that can automatically select neighbor point features, which represents the relevance of the center point i and the k th neighbor point in the corresponding feature dimension;

[0156] Step S25, the features are weighted and summed according to formula (8):

[0157]

[0158] where · represents point multiplication;

[0159] Step S26, in order to retain the features of each point itself while learning the features of the neighbor points, the extracted features are output by the following operation: where f i represent the input features of the center point i in this module, f out represent the output features of the center point i in this module.

[0160] Further, in the step S3, the method for capturing global features based on the capsule network is to design a Global Feature Capture module based on the capsule network;

[0161] The specific method is:

[0162] Step S31, the high-dimensional representation of the input features is extracted point by point through the MLP, and these feature maps are input into multiple independent convolutional layers with different weights, and each layer can learn the input point cloud feature map from different angles;

[0163] ​​Step S32, obtain their global latent representations by Max Pooling;

[0164] Step S33, concatenate these global latent representations into a feature vector set F named as Primary Point Capsules, F ∈ R ν×μ The size of F depends on the output dimension μ of each independent convolutional layer and the number of independent convolutional layers v.

[0165] Step S34, use a dynamic routing algorithm (as shown in Algorithm 1) to embed the primary point capsules into higher feature levels of advanced point capsules F', F' ∈ R ζ×η .

[0166] Further, in the step S4, a global correlation modeling based on self-attention is designed, and a specific method for constructing global correlation is as follows:

[0167] Step S41, design a global correlation modeling based on self-attention to calculate the global semantic correlation of the input feature F G and establish global dependency relationships;

[0168] Step S42, after establishing the global correlation by the global correlation modeling, the global relationship in the features and geometry will be automatically captured, and then the feature representation with local and global semantic context correlation is adaptively obtained to facilitate the subsequent segmentation task;

[0169] Step S43, for the input feature F G ∈ R N’×L , where L = 1024. After linear transformation, respectively obtain d1 = 256, and v ∈ R N’×L ;

[0170] Step S44, obtain the correlation score matrix G R ∈ R N’×N’ by transpose dot product of l1 and l2, as formula (9):

[0171]

[0172] Where @ represents matrix multiplication; each element in G R gives the correlation score between l1 and l2;

[0173] Step S45, then perform the same scaling operation on each element in G R , as formula (10):

[0174]

[0175] where d k = 256;

[0176] Step S46, then the normalized correlation score matrix G s is obtained by the Softmax function G The final autocorrelation weight G A ∈ R N’×N’ ; G A Each element g y,z in G

[0177]

[0178] where s y,z represents an element in G A g y,z embodies the degree of correlation between element y and element z;

[0179] Step S47, finally, the input feature F G is calculated according to formula (12) by the attention score

[0180]

[0181] Finally, the correlation between elements F G is calculated and the global dependency relationship is established, and the correlation weighted result The semantic segmentation result can be directly obtained by upsampling the aggregated latent feature .

[0182] Further, in step S6, by cascading multiple modules, the receptive field of the point is constantly expanded while being down-sampled. The specific method is: first, calculate the proportion of the amount of data of each category according to formula (13), and then calculate the final weight according to formula (14);

[0183]

[0184] w i = exp(-r i ) (14);

[0185] where n i represents the number of point clouds of category i, r i represents the proportion of category i in the total amount of all categories, and w i represents the weight coefficient of category i.

[0186] The final weighted cross-entropy loss function l w is shown in formula (15);

[0187]

[0188] where c is the number of classes, y is the label, and y i = 1 when the class is i, otherwise y i = 0, p i represents the probability of class i output by the neural network.

[0189] The application proposes a network model for semantic segmentation of 3D point cloud data of a shield tunnel. The Global and local feature encoder block is used to learn global descriptors while differentially aggregating local features, and then the GCM block is used to establish global correlation. The weighted cross-entropy loss function is improved for the problem of unbalanced number of shield tunnel classes, and direct semantic segmentation of large-scale 3D shield tunnel point cloud data is performed, so that any preprocessing or postprocessing is no longer needed, and finally fast segmentation of typical elements of the shield tunnel is realized.

[0190] Embodiment 2

[0191] Based on the above embodiment 1, this embodiment is mainly aimed at semantic segmentation of shield tunnel point cloud, and typical elements in the tunnel point cloud are segmented out, mainly including longitudinal and ring joints, bolt holes, etc. Due to the design of the tunnel structure, the joints are more prone to deformation, cracks and other problems than other places, which threatens the safety of the tunnel, and thus becomes an object that needs to be focused on in tunnel safety monitoring. Among them, the tunnel joint is a prone position for water leakage in the tunnel, and the bolt hole is prone to stress concentration damage and concrete leakage, which will seriously affect the quality of the tunnel and reduce the service life of the tunnel. Therefore, accurate segmentation of typical elements of the tunnel is very important for tunnel health monitoring, deformation monitoring, crack positioning analysis, joint analysis, tunnel section extraction and establishment of three-dimensional visual shield tunnel data model, etc. Therefore, an efficient deep learning network structure is designed, which can directly perform semantic segmentation on large-scale 3D shield tunnel point cloud data without any preprocessing or postprocessing.

[0192] Specifically, the following method is included:

[0193] 1. Feature perception enhancement (FPE)

[0194] Before global feature and local feature extraction, a FPE block is designed to enrich the features of each point and expand the global receptive field according to the characteristics of the shield tunnel point cloud data, such as Figure 1As shown, the FPE module consists of two parts: Similarity Feature Encoding (SFE) and Hybrid Pooling (HP).

[0195] 1.1 Similarity Feature Encoding (SFE)

[0196] The SFE module is shown in the left figure. The raw point cloud input of the network Figure 1 contains 3D coordinates and feature information where P represents all input point clouds, N represents the number of input points, and d is the dimension of the feature information of the input point cloud. The i-th point in P can be regarded as p i = (s i ,f i ) T In SFE, the similarity between points in the feature space is first measured using the Euclidean distance similarity metric, and a similarity matrix is constructed, as shown in equation (1).

[0197]

[0198] where Θ represents the dimension of p i ,p j , x i , y i represents the value of the vector p i and p j in the i-th dimension. Then, according to the similarity matrix, the κ most similar neighboring points In order to make the neighboring points have rich spatial information, the features of the neighboring points are re-encoded, and the encoding method is as shown in equation (2).

[0199]

[0200] where [ ] represents the concatenation operation, represents the encoded feature of the k-th neighboring point of the center point i, s i represents the spatial coordinates of the center point i, represents the input feature of the k-th neighboring point of the center point i, represents the spatial coordinates of the k-th neighboring point of the center point i. Then, a high-dimensional representation of the neighboring point feature is learned through a multi-layer perception (MLP) operation: where ​the learned feature of the k-th neighbor point of the center point i. Finally, the neighbor features are aggregated by the designed hybrid pooling module, which learns the local difference while preserving as much as possible the salient features of the neighbor points.

[0201] 1.2 Hybrid Pooling (HP)

[0202] Qi et al. (2017a) proposed to aggregate neighbor points' features with symmetric functions such as Max-Pooling and Mean-Pooling. Qi et al. (2021) proposed that the accurate extraction of point-wise features is very important for semantic segmentation tasks. Although parameter-free symmetric functions can effectively aggregate the local information of each point, they cannot effectively reflect the local difference, which is particularly evident when the point cloud distribution tends to be concentrated. In Figure 2 this problem is explained in (c), Figure 2 the feature learning flowchart of the FPE module to the LFDA, Figure 2 (a) represents the input point cloud; Figure 2 (b) represents the k points filtered by the similarity matrix, and different colors represent different neighbors of the points (c) shows the aggregation of local information by LFDA. In addition, when the distribution of points tends to be dense, the nearest points will find similar neighbor points, such as the green and blue points in subgraph (c); at this time, the same features will be learned by the symmetric function.

[0203] In order to capture sufficient local differences and salient features of the point cloud, a hybrid local information aggregation method is designed to effectively capture local features. As shown in Figure 1 the right figure, first, the maximum value pooling is directly used to obtain the salient features of the neighbor points max . Then, the average value pooling is used to obtain the overall features of the neighbor points mean . Then, from the neighbor point features itself, the attention score of the neighbor point feature is obtained by learning the high-dimensional mapping of the neighbor point feature and by the standardization function Softmax The calculation process is shown in formula (3).

[0204]

[0205] In the formula, d' represents the dimension of, which is the learnable weight in the full connection layer. Then the attention score is used to weight and sum the neighbor features to obtain Then and are connected through a residual connection to obtain Then the aggregated features H are obtained by summing up each channel w To learn the local difference of neighbor points. Finally, the three parts of pooling results are spliced together to get the final output H out = [H w , H max , H mean ]. H out contains both the pooling information reflecting local differences obtained by attention and the pooling information reflecting significant features and overall features.

[0206] 2. Local information discriminative aggregation (LFDA)

[0207] Correct learning of local information helps the network to learn more discriminative features. In order to capture more discriminative local information, such as Figure 3 , a local information mining module considering the correlation degree of neighbor points and center points is designed: Local Feature Discriminative Aggregation (LFDA). The neighbor points closer to the center point have greater influence on the center point, and the feature difference and spatial distribution pattern of the center point and the neighbor points will also affect the relationship between the center point and the neighbor points from different angles. Based on this, this module is designed. For the input point cloud P, first find the K nearest neighbor points in the Euclidean space by K-Nearest Neighbors (KNN) algorithm. The center point and the neighbor point are regarded as a graph: G = {V, E}, where V represents the node, E represents the edge connecting the center point and the neighbor point, V = {p1, p2, …, p N}, Then the edge feature of the neighbor is obtained by the spatial coordinates and features of the vertex The edge feature of the neighbor can be regarded as the connection relationship between the center point and the neighbor point, and the edge feature of the neighbor is obtained by encoding according to formula (4):

[0208]

[0209] In formula (5), s represents the edge feature of the kth neighbor point of the center point i, represents the inverse distance coefficient of the kth neighbor point of the center point i, which decreases with the increase of the distance, and the calculation of the inverse distance coefficient is as formula (5-6):

[0210]

[0211]

[0212] In formula (5), si and denote the spatial coordinate values of the center point i and the k-th neighbor point, respectively. where denote the feature of the k-th neighbor point of the center point i. where denote the k th dimensional coordinates of the k i denote the center point i. The new neighbor point feature Then, for the neighbor edge feature set A shared function g(·) is designed to learn the attention score s of each feature, which is composed of a linear transformation followed by a softmax function, defined as equation (7):

[0213]

[0214] where W is the learnable weight. The learned attention score can be considered as a mask that can automatically select the neighbor point features, indicating the strength of the correlation between the center point i and the k-th neighbor point in the corresponding feature dimension. Then, the features are weighted and summed according to equation (8):

[0215]

[0216] where · denotes the dot product. Finally, in order to retain the features of each point itself while learning the features of the neighbor points, the extracted features are output by the following operation: where f i denote the input features of this module, f out denote the output features of this module.

[0217] 3. Global Feature Capture (GFC)

[0218] 3.1. Capsule Network (Capsule Network)

[0219] Capsules (Hinton et al., 2011) are composed of a set of vectors. These vector sets are obtained by performing internal operations and encapsulation on input data, which can represent different attribute information of the same entity (Ahmad et al., 2018). Generally, CNNs are invariant to features, but not equivariant. Equivariance refers to detecting features of objects that can be transformed into each other. For example, if an object is rotated at a certain angle, the capsule can recognize the object rotated at that angle through the equivariant property without the need to train a separate modification. This is a unique property of capsules compared to CNNs, as CNNs need to train for all variations of an object, such as scale, pixel intensity, direction, etc. Capsules have two main components. One is the local invariant probability of the existence of an entity. The second is a set of parameters describing the equivariance of the entity, also known as pose. Having these two components is important because they help to effectively identify entities.

[0220] 3.2, Squashing Function

[0221] The squashing function is also important for capsules, which is applied to the output of the capsule to normalize the vectors of the capsule. Like ReLU (Krizhevsky et al., 2012) and Sigmoid (Mount, 2011), it is also a non-linear function. However, unlike ReLU, which can handle scalars well, the squashing function has been shown to handle the vectors of the capsule output better. If the input vector is short in length, the function will compress that vector close to 0, and if the vector is long in length, it will try to constrain the output vector close to 1. The squashing function is defined in step 5 of Algorithm 1.

[0222] 3.3, Dynamic Routing Algorithm

[0223] The dynamic routing algorithm (Sabour et al., 2017) is applied to the transformation process from the master capsule to the advanced capsule, primarily addressing how the advanced capsule uses vectors from the primary capsule and helping to improve predictions through dynamic routing. Lower-level capsules send their inputs to higher-level capsules that correspond to their inputs. The weight matrix is ​​updated via a protocol between the two capsules. The routing algorithm performs a function similar to max pooling. However, while max pooling eliminates unimportant features when selecting the most important ones, the dynamic routing algorithm does not do this, but instead assigns all features from the master capsule to the correct features in the advanced capsule. Algorithm 1 describes the computation process of the dynamic routing algorithm. The inputs of the algorithm and represent the vectors in the master capsule, the number of iterations, and the layer, respectively. is a temporary variable in the computation process, initialized to 0. In step 1, a high-level representation of the vectors in the primary capsule is first obtained through a learnable matrix. Steps 3 through 7 iterate multiple times to continuously assign features from the master capsule. In step 3, coupling coefficients for feature assignment are calculated using the softmax function, and then in step 4, the high-level representation is weighted using these coupling coefficients. It is then normalized using a squeezing function. Finally, and are used for updating. When r is set to 3 in this invention, the loop ends and the result is output as the final result.

[0224] 3.4 Capsule-based Global Feature Capture (GFC)

[0225] To better compute global description information, a Global Feature Capture module was designed based on capsule networks. For example... Figure 4 First, a high-dimensional representation of the input features is extracted point-by-point using an MLP. To enable the network to comprehensively learn the features of the input point cloud, these feature maps are fed into multiple independent convolutional layers with different weights, each learning from a different perspective. Then, Max Pooling is used to obtain their global latent representations. Next, these global latent representations are concatenated into a feature vector set F called Primary Point Capsules. The size of F depends on the output dimension μ of each independent convolutional layer and the number of independent convolutional layers v. Then, a dynamic routing algorithm (as shown in Algorithm 1) is used to embed the primary point capsules into advanced point capsules (F') at a higher feature level. Latent point capsules are higher-dimensional summaries of primary point capsules, containing rich global features. Finally, the features of latent point capsules are further summarized into global feature descriptors, X, using HP.

[0226] 4. Globally relevant modeling (GCM)

[0227] Due to limitations in the previously described method for calculating the receptive field, global correlation cannot be accurately obtained, resulting in a lack of interdependencies between points. Before upsampling, it is necessary to construct the global correlation and interdependencies between points. Therefore, a self-attention-based global correlation modeling approach is designed to calculate the input feature F. G The Global Semantic Relevance and Global Dependency Relationships are established. After establishing global relevances through the GCM module, global relationships in features are automatically captured, and then feature representations with local and global semantic context associations are adaptively obtained to facilitate subsequent segmentation tasks. Figure 5 As shown, firstly, for the input features In the formula L = 1024. After linear transformation, the following were obtained: d1 = 256, and The correlation score matrix is ​​obtained by multiplying the transposes of l1 and l2. As in formula (9):

[0228]

[0229] In the formula, @ denotes matrix multiplication. G R Each element in G gives a correlation score between l1 and l2. Then, for G... R Each element in the formula undergoes the same scaling operation, as shown in formula (10):

[0230]

[0231] In the formula d k =256. Then, the scaled correlation score matrix G is standardized using the Softmax function. s To obtain the latent representation F G Final autocorrelation weights G A Each element g in y,z Calculate according to formula (11):

[0232]

[0233] In the formula s y,z G represents A The elements in g. y,z This reflects the degree of correlation between elements y and z. Even in non-neighborhood environments, it can effectively capture more relevant distributed feature relationships and assign greater attention weights to g. y,z This enhances the similarity of their semantic contexts. Finally, the input features F are evaluated using attention scores.G c is calculated according to formula (12).

[0234]

[0235] Finally, F is calculated G The correlation between elements is calculated and the global dependency relationship is established, and the correlation weighted result is obtained The semantic segmentation result can be obtained directly by upsampling the aggregated latent features.

[0236] 5、Optimized weighted cross-entropy loss function

[0237] The data imbalance problem exists in various data sets, and the large amount of data samples will dominate the network training process, resulting in poor recognition of small sample data. To solve this problem, a greater penalty can be imposed on small data samples through the loss function to optimize the imbalance of data quantity. The weighted cross-entropy loss function is a common method to solve the data imbalance problem. It measures the amount of data of each class and gives each class a weight coefficient, so that large data samples will get a smaller penalty weight, and small data samples will get a larger penalty weight, so that the network can focus on small sample categories. In this invention, a weight calculation method suitable for shield tunnel point cloud data with uneven data quantity of various categories is designed. As shown in formulas (13-14), first, calculate the proportion of the data quantity of each category according to formula (13), and then calculate the final weight through formula (14). When the data quantity difference between different categories is too large, such calculation method will make the weight remain between 0-1, and will not be too different.

[0238]

[0239] w i = exp(-r i ) # (14)

[0240] w i2 = 1 / (r i + 0.02) # (15)

[0241] In the formula, n i represents the point cloud quantity of category i, r i represents the proportion of category i in the total quantity of all categories, and w i represents the weight coefficient of category i. The final weighted cross-entropy loss function l w is shown in formula (16).

[0242]

[0243] ​In the formula, c is the number of categories, and y is the label. When the category is i, y i =1, otherwise y i =0, p i This represents the probability of category i in the output of the neural network.

[0244] This application constructs a shield tunnel dataset by annotating typical elements of 3D point cloud data of real-world shield-tunnel subway scenes.

[0245] By comparing with existing state-of-the-art semantic segmentation network models, this application demonstrates that GL-Net achieves state-of-the-art results on the semantic segmentation task of 3D point cloud data for shield tunnels, with an improvement of at least 14.5% in the current experiments.

[0246] Example 3

[0247] Based on the above embodiments 1-2, this embodiment mainly introduces the specific implementation and comparative experiment of the point cloud segmentation method based on global feature learning and local feature discrimination aggregation.

[0248] 1. Experimental Preparation

[0249] 1.1 Dataset

[0250] The shield tunnel dataset consists of 3D point cloud data collected from real shield tunnel scenarios, such as... Figure 6 As shown, raw point cloud scanning data was obtained using a FARO laser scanning device in the Chengdu Metro shield tunnel. The data includes three-dimensional coordinate information and intensity information. The tunnel's design radius is 2.7m, and its total length is approximately 1000m. Noise points and track sections were removed before annotation. Furthermore, the dataset was labeled into seven categories: transverse joints, circumferential joints, bolt holes, grouting holes, wire-1, wire-2, and miscellaneous. Approximately 80% of the dataset was used as training data, and 20% as test data. Table 1 shows a comparison of the number of points in each category of STDS. As can be seen from Table 1, the categories other than miscellaneous account for approximately 7.73% of the total data, and there are significant differences in the number of points in each category; for example, the number of points in the circumferential joint is approximately 40 times that of the wire-1.

[0251] Table 1 Details of the shield tunnel

[0252] Transverse joint Annular joint Bolt hole Grouting hole Wire-1 Wire-2 Miscellaneous Class number 2251 475 15375 2107 22 80 13 Point number 5050581 12661172 5963842 1286557 296302 1550687 341521213 Relative proportion 0.01479 0.03707 0.01746 0.00377 0.00087 0.00454 1.000

[0253] 1.2 Experimental Environment

[0254] To train the semantic segmentation of shield tunnel point cloud using GL-Net, the experiment was deployed in an environment equipped with an Intel Core i9-9900K central processing unit (CPU), 32 GB of random access memory (RAM), and an NVIDIA RTX 2080 Ti GPU (16 GB graphics processor) for training and testing. The algorithm running environment is Python 3.6.10, CUDA 9.0.176, and cuDNN 6.0. During the training process, the original learning rate is set to 5e-3, decaying at a rate of 5% per epoch. The Adam optimizer is used with the default settings, and the model is trained on the GPU using synchronous stochastic gradient descent (SGD) (LeCun et al., 1998).

[0255] 1.3. Metrics of segmentation algorithm

[0256] Accuracy and IoU (Shelhamer et al., 2015 referred to as IU) are commonly used metrics in the task of point cloud semantic segmentation. However, due to the uneven number of classes in the shield tunnel dataset, Accuracy cannot truly reflect the effect of semantic segmentation, and the present application only uses IoU as a metric. Mean IoU (mIoU) is the average IoU of each class. When calculating the IoU of a single class, the point cloud is divided into the target class, i.e., the point cloud belonging to that class, and the background class, i.e., the point cloud of all other classes. The event of correctly segmenting the target class into the target class is true positive (TP), the event of correctly segmenting the background class into the background class is true negative (TN), the event of segmenting the background class as the target class is false positive (FP), and the event of segmenting the target class into the background class is marked as false negative (FN). IoU is obtained according to formula (17):

[0257]

[0258] where TP, FP and FN represent the number of TPs, FPs and FNs.

[0259] 2. Comparative experiment

[0260] To obtain the best effect of GL-Net, some optimal settings of parameters were obtained through experimental comparison. These experiments were conducted on the shield tunnel dataset.

[0261] (1) Kappa in FPE

[0262] Firstly, the setting of the number of nearest neighbors was studied. In FPE, the number of nearest neighbors is represented as the number of nearest neighbors in the similarity matrix. In this experiment, k = 6, which is the nearest neighbor in LFDA. The results are as follows: Figure 8The best performance can be achieved when it is set to 6. When it is smaller, the model can not have enough similar points to enhance the features of the center point, resulting in lower semantic segmentation performance. When the similarity is larger, the model can use points with lower similarity to reduce the performance of the model.

[0263] (2) K in LFDA

[0264] As before, the setting of the number of nearest neighbors K is investigated, denoted as the number of nearest neighbors in Euclidean space in LFDA. When the neighborhood is smaller, the model can not have enough context to make predictions. When the neighborhood is larger, each LFDA uses a large number of data points, many of which can be less relevant, which can introduce too much noise in processing and reduce the accuracy of the model.

[0265] (3) υ, μ, ζ and η in GFC

[0266] Comparative experiments were conducted on the parameter settings of GFC. Denotes the shape of PPC, and denotes the shape of APC. Four sets of experiments were set considering the time cost and memory consumption. As shown in Table 3. The best performance is achieved. The increase of PPC and APC dimensions (such as) will result in poorer model performance, which can be caused by the introduction of too much useless information.

[0267] Table 3 Comparison of experimental results of GFC and loss function

[0268] Paremeters mIoU (%) 16×512,16×64 71.87 8×512,32×64 72.21 8×512,16×64 73.02 16×512,32×64 72.36 divide 65.65 Focal Loss(gamma=1) 68.74 Focal Loss(gamma=2) 71.5 Focal Loss(gamma=3) 69.34 ours(exp) 73.02

[0269] (4) Loss function

[0270] In this experiment, the influence of different loss functions on the network segmentation ability is compared. The results are shown in Table 3. Denotes the weight calculated according to Eq. (13) and Eq. (15), and denotes the weight calculated according to Eq. (13) and Eq. (14). Alpha and gamma are the parameters of focal loss (Lin et al., 2020), and alpha and gamma are used to solve the class imbalance problem and the segmentation difficulty problem respectively. Since the class imbalance is the same in all experiments, alpha consistency is maintained in all experiments, and the gamma value is 1, 2 and 3 respectively. The weighted loss function with weight calculated by exp has the best performance. Using other loss functions will reduce mIoU by 1.5%~7.4%, which shows that the weighting function with weight calculated by exp is crucial to the performance of the model.

[0271] The segmentation performance of the network is tested on the shield tunnel dataset. In addition, some state-of-the-art point cloud semantic segmentation networks are also tested on the shield tunnel dataset. In testing these networks, the number of training data and parameters are consistent with the original text, but the learning rate of the corresponding network will be adjusted according to the actual training results to achieve the best segmentation performance. The quantitative results of segmentation are shown in Table 2. Obviously, although many methods have achieved state-of-the-art results on datasets such as Stanford Large-Scale 3D Indoor Spaces (S3DIS) (Armeni et al., 2016) and semantic3D (Hacker et al., 2017), the results on the shield tunnel are not superior. On the one hand, the feature learning method of these methods may not be suitable for the tunnel scene, because the typical elements in the tunnel scene are evenly distributed, but the features are more difficult to capture. On the other hand, these methods may not be able to handle the imbalance in the number of classes in the shield tunnel dataset, because there is a serious data imbalance between the elements in the shield tunnel dataset. As shown in Table 2, the network of the present application is obviously superior.

[0272] Table 2 Shield tunnel semantic segmentation results

[0273] Method mIoU Transverse joint Annular joint Bolt hole Grouting hole Unsupported cable Weak cable Miscellaneous RandLANet 54.59 26.48 43.96 58.26 33.01 64.44 65.98 89.98 BAAFNet 51.99 33.25 36.53 50.27 30.81 62.07 63.68 87.33 SCFNet 58.48 40.65 46.28 63.04 35.68 63.71 68.33 91.68 GL-Net 73.02 51.76 59.12 76.29 60.08 82.95 84.97 95.98

[0274] In Figure 9 , the segmentation results of the network / network without the part structure are qualitatively shown. As shown in Figure 9 , the network can clearly segment the typical elements. The segmented typical elements are clear and complete, and there is no obvious invasion of adjacent elements. When some modules are removed, especially the LFDA, the segmentation ability of the network is obviously reduced, and the completeness of the segmented typical elements is reduced. In Figure 10 , the segmentation effects of different networks on the shield tunnel dataset are qualitatively shown. Compared with GL-Net, the other three networks have obvious erosion on adjacent elements, which may cause the position shift of longitudinal joints and circumferential joints and the connection of adjacent bolt holes, which is not conducive to further use. At the same time, the completeness of some typical elements is also poor.

[0275] 3. Ablation experiment

[0276] Due to the modular design of the network, each module can be easily combined with other modules or removed from the overall architecture. The purpose of the ablation study is to verify the effectiveness and necessity of each module, mixed pooling and the edge feature acquisition method adopted.

[0277] 4. Model effectiveness

[0278] To verify the effectiveness of each module on network segmentation, the following ablation experiments are designed to verify the effectiveness of the modules. In the actual experiment, the input and output of each module will be closed one by one. The test scheme is as follows:

[0279] (1) only GFC, FDA and GCM, no FPE;

[0280] (2) only FPE, LFDA and GCM, no GFC;

[0281] (3) only GFC, FPE and GCM, no LDFA;

[0282] (4) only GFC, FPE and LFDA, no GCM;

[0283] Table 4 Network structure ablation experiment

[0284]

[0285] As shown in Table 4, when closing the LFDA in the network structure, the performance of the model will decrease by about 18.6%, which shows that the aggregation of significant features of adjacent points is very important to the segmentation result of the whole network, and it can gradually expand the receptive field when distinguishing the features of adjacent points. In addition, when closing the FPE and GCM modules, the network segmentation mIoU decreases by 1.5% and 2% respectively, which shows that increasing the global receptive field and establishing the global dependency can effectively improve the effect of the network segmentation task. Finally, the closure of GFC leads to a decrease in the segmentation ability of most classes, especially the longitudinal joint and the grouting hole. The segmentation performance decreases by 3.3% and 2.1% respectively. At the same time, the closure of GFC also leads to a decrease of about 1% in mIoU. This shows that the addition of global descriptors helps to improve the network's segmentation ability for some typical elements.

[0286] 3.2, Boundary features in LFDA

[0287] In section 5.2, how to obtain edge features is introduced. Further research is made on the influence of different encoding methods in the framework. In particular, the following more ablation experiments are carried out on edge features:

[0288] (1) only encode relevant neighborhood features f ik ;

[0289] (2) encode inverse distance coefficient and relevant position p ik ;

[0290] (3) encode and p ik ;

[0291] (4) encode f ik and pik ;

[0292] (5) Encoding related f ik , p ik and

[0293] Table 5 LFDA ablation experiment: mIoU results of different edge feature encoding with GL-Net

[0294]

[0295] Table 5 compares the scores of mIoU under different edge feature combinations. It can be seen that 1) synchronous encoding, and leads to the best performance of 73.02%. 2) plays a very important role in the effective aggregation of neighbor information in LFDA. The inverse distance coefficient can reflect the influence of different distance neighbors on the center point, and removing the inverse distance coefficient will reduce the mIoU score by about 1.1%. The relative neighborhood feature can reflect the feature difference between the neighborhood points and the center point, and removing the relative neighborhood feature will reduce the score by about 1.7%. The relative position plays an important role in this component, mainly because the relative point position enables the network to perceive local geometric patterns. 3) When the code edge feature is only, the segmentation mIoU score decreases by about 3.3%, because only cannot accurately reflect the influence of neighboring points on the center point.

[0296] 3.3, Mixed pooling

[0297] Now the necessity of choosing mixed pooling is studied. According to the following experimental ideas, the experiment is designed:

[0298] (1) Only use max pooling;

[0299] (2) Only use average pooling;

[0300] (3) Only use weight pooling;

[0301] (4) Use max pooling and weight pooling;

[0302] (5) Use average pooling and weight pooling;

[0303] (6) Use weight pooling, max pooling and average pooling;

[0304] Table 6 Ablation experiment of mixed pooling

[0305] Pooling method mIoU (%) Only max pooling 71.76 Only mean pooling 71.56 Only weighted pooling 71.01 Max pooling+weighted pooling 72.74 Mean pooling+weighted pooling 72.09 ours 73.02

[0306] Table 6 compares the mloU scores of all ablated networks on the STDS dataset. It can be seen that: 1) Explicitly encoding all pooling methods can achieve the best mloU performance. 2) Weighted pooling plays an important role in this component, mainly because weighted pooling enables the network to perceive local differences. 3) Using only symmetric functions such as max pooling and mean pooling is unlikely to improve performance, as symmetric functions cannot learn unique local features.

[0307] The above only describes the preferred embodiments of the present application, and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any changes, modifications, replacements, integrations and parameter changes made to these embodiments within the spirit and principles of the present application, which can achieve the same functions without departing from the principles and spirit of the present application, all fall within the protection scope of the present application.

Claims

1. A point cloud segmentation method based on global feature learning and local feature discriminative aggregation, characterized in that, It comprises the following steps: Step S1, a feature perception enhancement FPE module is designed to enrich the features of each point cloud and expand the global receptive field; Step S2, a local feature digging LFDA module is designed to learn more distinctive local features; Step S3, a global feature capture GFC module is designed based on a capsule network; Step S4, a global correlation modeling GCM module based on self-attention is designed to build global correlation; Step S5, the global and local information capture module is composed of FPE, LFDA and GFC, and by cascading multiple global and local information coding modules, the receptive field of the points is constantly expanded while being down-sampled; Step S6, the weighted cross-entropy loss function is improved to solve the problem of uneven distribution of shield tunnel typical element category data; Step S7, the final semantic prediction is obtained through three fully connected layers and a Dropout layer; In the step S2, the principle of the local feature digging module is: Step S21, for the input point cloud P, first find the K nearest neighbor points in the Euclidean space through the K-NearestNeighbors (KNN) algorithm; Step S22, regarding the center point and the neighbor point as a graph structure: , wherein V represents a node, and E represents an edge connecting the center point and the neighbor point, ; Step S23, then get the edge feature of the adjacent edge through the spatial coordinates of the vertex and the feature, ), the edge feature can be regarded as the connection relationship between the center point and the neighbor point, and the edge feature is encoded according to formula (4): ； wherein denotes the center point denotes the edge feature of the kth neighbor point of the center point i, denotes the inverse distance coefficient of the kth neighbor point of the center point i, which decreases with the increase of the distance, and the inverse distance coefficient is calculated as formula (5-6): ； ； In formula (5) and They represent the center points respectively. and the Spatial coordinates of neighboring points; In the formula Represent the characteristics of the k-th neighbor of center point i; In the formula Represents the center point i The three-dimensional coordinates of neighboring points Represent the central node i; simultaneously obtain the features of new neighboring nodes. ; Step S24, for the adjacent edge feature set , a shared function is designed to learn the attention score s of each feature, and the shared function is composed of a linear transformation followed by a softmax function, which is defined as formula (7): ； Where W is a learnable weight; learned attention scores may be considered as a mask that can automatically select neighbor point features, indicating the strength of the correlation between the center point i and neighbor points in the corresponding feature dimension; Step S25, the features are weighted and summed according to formula (8): ； Where · represents point multiplication; Step S26, in order to retain the features of each point itself while learning the features of the neighbor points, the extracted features are output through the following operations: ， wherein represents the input feature of the center point i of the module, represents the output feature of the center point i of the module. 2.The point cloud segmentation method based on global feature learning and local feature discriminative aggregation of claim 1, characterized in that, In the step S1, the FPE module comprises a similarity feature coding module and a hybrid pooling module. 3.The point cloud segmentation method based on global feature learning and local feature discriminative aggregation of claim 2, characterized in that, network input raw point cloud of the similarity feature encoding module comprising 3D coordinates and feature information, The 3D coordinates: , The feature information: , In the formula, P represents all input point clouds, N represents the number of input points, d is the dimension of input point cloud feature information; the i-th point in P can be regarded as . .

4. The point cloud segmentation method based on global feature learning and local feature discriminative aggregation according to claim 3, characterized in that, The method using the similarity feature coding module is: Step S101, in the similarity feature coding module, the similarity degree of the points is measured based on the feature space and the Euclidean space, a similarity matrix is constructed, and the calculation formula (1) is shown: ； wherein represents the dimension of represents the value of the vector and in the i-th dimension; Step S102, selecting the most similar k neighboring points for each point according to the similarity matrix ; In order to make the neighboring points have rich spatial information, the features of the neighboring points are re-encoded, and the encoding method is as formula (2): ； wherein denotes a stitching operation, denotes a center point encoded features of the k-th neighbor point of the center point, denotes the spatial coordinates of the center point i, denotes the input features of the k-th neighbor point of the center point i, denotes the spatial coordinates of the k-th neighbor point of the center point i; Step S103, learning a high-dimensional representation of the features of the neighboring points through a multi-layer perceptron (MLP) operation: , where denotes the learned features of the kth neighboring point of the center point i.

5. The point cloud segmentation method based on global feature learning and local feature discriminative aggregation according to claim 2, characterized in that, The method of the hybrid pooling module comprises: Step S111, directly using maximum value pooling to obtain the salient features of the neighbor points ; Step S112, obtaining overall features of the neighbor points using average value pooling ; Step S113, obtaining neighbor point features By learning the high-dimensional mapping of neighbor point features and obtaining the attention score of each feature of the neighbor point by the standardization function Softmax The calculation process is as shown in formula (3): ； In the formula represents dimension of the fully connected layer is a learnable weight Step S114, utilizing the attention score The neighbor features are weighted and summed to obtain , ; Step S115, connecting by means of a residual connection and to obtain ; Step S116, get the aggregated features by summing up each channel to learn the local difference of neighbor points; Step S117, splicing the three-part pooling results together to obtain the final output .

6. The point cloud segmentation method based on global feature learning and local feature discriminative aggregation according to claim 1, characterized in that, In the step S3, the method of global feature capture based on a capsule network is that a GlobalFeatureCapture module is designed based on a capsule network; The specific method is: Step S31, the high-dimensional representation of the input features is extracted point by point through an MLP, and these feature maps are input into multiple independent convolutional layers with different weights, and each layer can learn the input point cloud feature map from different angles; Step S32, MaxPooling is used to obtain their global latent representation; Step S33, concatenating these global latent representations into a feature vector set named Primary Point Capsules ; The size depends on the output dimension of each individual convolutional layer and the number of individual convolutional layers ; Step S34, using a dynamic routing algorithm (as shown in Algorithm 1) to embed the primary point capsule into a higher feature level of a higher level point capsule .

7. The point cloud segmentation method based on global feature learning and local feature discriminative aggregation according to claim 1, characterized in that, In the step S4, the specific method of designing global correlation modeling based on self-attention for building global correlation is: Step S41, design global correlation modeling based on self-attention to calculate the global semantic correlation of the input features and establish global dependencies; Step S42, after establishing global correlation through global correlation modeling, the global relationship in features and geometry is automatically captured, and then a feature representation with local and global semantic context association is adaptively obtained to facilitate subsequent segmentation tasks; Step S43, for the input feature , where are obtained by linear transformation respectively , and ; Step S44, by and Point multiplication obtains the correlation score matrix As formula (9): ； wherein represents matrix multiplication; Each element in gives the correlation score between and and Step S45, then the same scaling operation is performed on each element in as formula (10): ； In the formulae ; Step S46, the correlation score matrix after scaling is then normalized by a Softmax function to obtain latent representations ; ; ; ; is calculated according to formula (11): ； wherein represents the element in embodies the degree of correlation between the element y and the element z; Step S47, finally, the input features are scored by attention scores According to formula (12) is calculated ; ； Finally, the correlation between elements is calculated and the global dependency is established, resulting in a correlation-weighted result ; semantic segmentation results can be directly obtained by upsampling the aggregated latent features . 8.The point cloud segmentation method based on global feature learning and local feature discriminative aggregation of claim 1, wherein, In the step S5, the specific method of constantly expanding the receptive field of the points while being down-sampled by cascading multiple modules is: first, calculate the proportion of each category data amount according to formula (13), and then calculate the final weight through formula (14); ； ； In the formula Indicates category The number of point clouds, This represents the proportion of category i in the total number of all categories. Indicates category Weighting coefficients; Final weighted cross-entropy loss function As shown in equation (15); ； where c is the number of classes, y is the label, and , otherwise , represents the probability of class i output by the neural network.

Images