Point cloud semantic segmentation method based on double-scale feature extraction and self-attention

Abstract

A point cloud semantic segmentation method based on double-scale feature extraction and self-attention, which is composed of the steps of selecting a data set, preprocessing a point cloud, constructing a point cloud semantic segmentation network, training the point cloud semantic segmentation network, and testing the point cloud semantic segmentation network. Since the application adopts the method of dividing large-scale point cloud data into 1m*1m*1m point cloud blocks and inputting the point cloud blocks into the point cloud semantic segmentation network, fusing the features of two scales of the local area of the point cloud, and optimizing the fused vector by using a self-attention vector, the semantic label of the point cloud can be more accurately predicted, and the artificial labeling label is closer. The simulation comparison experiment shows that the method has better point cloud semantic label accuracy, and the semantic boundary is smoother; the segmented point cloud is used as the input of the point cloud semantic segmentation network, the required computing resources are smaller while the training data amount is increased, and the method can be used in the field of automatic driving.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of point cloud processing, and particularly relates to a point cloud semantic segmentation method. BACKGROUND

[0002] Point cloud is a discrete point set describing the surface characteristics of an object, which is composed of three-dimensional information of geometric position and attribute information, including color, intensity, reflectivity, etc., and can accurately record and express the spatial geometric information of the target object. With the development of point cloud acquisition technology and processing technology, the cost of obtaining point cloud is more affordable, and point cloud has been successfully applied to many fields such as autonomous driving, power inspection, surveying and mapping engineering, and as-built building information modeling. However, point cloud is difficult to understand due to its unordered nature, rotation invariance, and uneven density. Therefore, the point cloud semantic segmentation method has important application value for scene understanding of point cloud.

[0003] The point cloud semantic segmentation method is divided into a projection-based method, a voxel-based method, and a point-based method. The projection-based method needs to project three-dimensional data to a two-dimensional view, and uses a mature two-dimensional depth image learning network to realize semantic segmentation, but the error of inverse projection in the projection domain is inevitable. The voxel-based method divides point cloud into volume grids with specific size and discrete coordinates, so that the point cloud has a position relationship similar to a two-dimensional image, and a three-dimensional convolution network can be used to realize semantic segmentation. However, with the decrease of voxels, the data volume will increase by a geometric multiple. The point-based method directly inputs point cloud, which reduces information loss and has smaller calculation amount compared with the other two methods. Therefore, the point-based point cloud semantic segmentation method has important research significance.

[0004] In recent years, the point-based point cloud semantic segmentation method technology has become a research hotspot and has achieved certain results. Most of the existing methods improve the semantic segmentation performance by improving the learning ability of the network to the local neighborhood features of the point cloud. However, a smaller scale is easy to cause semantic boundary segmentation error, a larger scale introduces larger noise, and the correlation relationship between local neighborhoods is ignored. SUMMARY

[0005] The technical problem to be solved by the application is to overcome the shortcomings of the prior art, and to provide a point cloud semantic segmentation method based on double-scale feature extraction and self-attention, which has strong anti-aliasing performance for semantic segmentation boundaries, accurate semantic label prediction, and high robustness to point cloud sparsity.

[0006] The technical solution adopted to solve the above technical problems is composed of the following steps:

[0007] (1) Selecting a data set

[0008] The point clouds in the point cloud database are divided into a training set and a test set at a ratio of 5:1, and there is no overlap between the training set and the test set.

[0009] (2) Point cloud preprocessing

[0010] The point cloud is segmented into point cloud blocks with a size of 1m x 1m x 1m cubes;

[0011] (3) Constructing a point cloud semantic segmentation network

[0012] The point cloud semantic segmentation network is composed of a feature encoding module 1, a feature decoding module 2, and a normalization prediction module 3 connected in sequence.

[0013] The feature encoding module 1 is composed of a first layer encoding block, a second layer encoding block, and a third layer encoding block connected in sequence.

[0014] The first layer encoding block is composed of a double-scale feature extraction module and a feature self-attention module connected in sequence; the second layer encoding block and the third layer encoding block have the same structure as the first layer encoding block.

[0015] The feature decoding module 2 is composed of a first layer decoding block, a second layer decoding block, and a third layer decoding block connected in sequence.

[0016] (4) Training the point cloud semantic segmentation network

[0017] 1) Constructing a loss function

[0018] The loss function L1 is constructed as follows:

[0019]

[0020] where x represents the number of points in the point cloud, H x is the label value of the xth point, S x is the predicted label value of the xth point.

[0021] 2) Training the point cloud semantic segmentation network

[0022] The software condition is CentOS 7.5 operating system, based on pytorch1.7.0 framework, the training set is input into the point cloud semantic segmentation network for training, the parameter learning rate is 0.5, the momentum decay is set to 0.7, the weight decay rate is 0.0001, the batch processing amount is 32, the training round number is 50, the server is Huawei server RH2288H V2 equipped with two Nvidia Tesla v100 graphics cards, and the training is stopped until the loss function converges.

[0023] (5) Testing the point cloud semantic segmentation network

[0024] The test set is input into the trained point cloud semantic segmentation network to obtain the final semantic label of each point cloud.

[0025] In the step (3) of constructing the point cloud semantic segmentation network, the double-scale feature extraction module of the application is composed of a feature extraction first branch module and a feature extraction second branch module in parallel.

[0026] In the step (3) of constructing the point cloud semantic segmentation network, the feature extraction first branch module of the application is composed of a shared multi-layer perception and a max pooling layer in series; the structure of the feature extraction second branch module is the same as that of the feature extraction first branch module. The construction method of the feature extraction first branch module of the application is as follows:

[0027] 1) The shared multi-layer perception F is constructed according to the following formula: i The method is as follows:

[0028]

[0029]

[0030]

[0031]

[0032] x 1×m =(ω(F 1×n ) T ) T

[0033] λ=(λ wv ) W×V

[0034] τ=(τ wv ) W×V

[0035] ω=(ω mn ) M×N

[0036] Wherein, ReLu(.) is an activation function, λ is a weight matrix of the shared multi-layer perception, τ is a bias matrix of the shared multi-layer perception, and ω is a weight coefficient matrix of different dimension features of the point cloud.

[0037] 2) The max pooling layer F is determined according to the following formula: i_max :

[0038] F i_max =MaxPooling(F i )

[0039] The construction method of the feature extraction second branch module is the same as that of the feature extraction first branch module.

[0040] In step 3, when constructing the point cloud semantic segmentation network, the kernel size of the shared multilayer perceptron of the present invention is 1x1, and the stride of the sliding window is 1.

[0041] In step 3, when constructing the point cloud semantic segmentation network, the first layer decoding block of the present invention is composed of a shared multilayer perceptron and interpolation upsampling connected in series; the structures of the second layer decoding block and the third layer decoding block are the same as the structure of the first layer decoding block.

[0042] In step (3) of constructing the point cloud semantic segmentation network, the feature self-attention module of the present invention is composed of linear layer 1, linear layer 2, linear layer 3, shared multilayer perceptron 1, shared multilayer perceptron 2, and normalization function layer connected together. The output ends of linear layer 1 and linear layer 2 are connected to the input end of shared multilayer perceptron 1, the output ends of shared multilayer perceptron 1 and shared multilayer perceptron 2 are connected to the input end of normalization function layer, and the output end of linear layer 3 is connected to normalization function layer.

[0043] The method for constructing the feature self-attention module of the present invention is as follows:

[0044] The output F of the dual-scale feature extraction module i_max The inputs are fed into linear layer 1 and linear layer 2 to obtain the input to the feature self-attention module, as shown in the following equation:

[0045] F p =Liner(F i_max )

[0046] The characteristic linear transformation F of a point p The linear feature transformation F with nearest neighbors q The self-attention vector ATT is obtained by inputting a shared multilayer perceptron 1, as shown in the following equation:

[0047] ATT=γ(F q -F p )+δ

[0048] Where γ represents the shared multilayer perceptron 1, and δ is determined by the position information input into the linear layer 3, as shown in the following equation:

[0049]

[0050] Where, q i This indicates the current location information. This represents the location information of the K nearest neighbors of the current point. Indicates feature concatenation, ||.|| 2 This represents the Euclidean distance between two points;

[0051] The characteristic linear transformation F of a point pThe input is input to the shared multi-layer perception 2, and the feature S optimized by the feature self-attention module is obtained according to formula (1) iq :

[0052]

[0053] Wherein, p is a softmax function, the mapping function a represents the multi-layer perception 2, the symbol represents the XOR operation, and K is 8-32.

[0054] Advantages of the present application:

[0055] Since the present application adopts the method of dividing large-scale point cloud data into 1m*1m*1m point cloud blocks and inputting the point cloud blocks into the point cloud semantic segmentation network, fusing the features of two scales of local regions of the point cloud, and optimizing the fused vector by using the self-attention vector, the semantic label of the point cloud can be more accurately predicted, and the artificial labeling label is closer. The simulation comparison experiment shows that the method has better point cloud semantic label accuracy, and the semantic boundary is smoother; the segmented point cloud is used as the input of the point cloud semantic segmentation network, the amount of training data is increased, and the required operation resources are smaller. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is the flowchart of embodiment 1 of the present application.

[0057] Figure 2 is a structural schematic diagram of the feature encoding module 1.

[0058] Figure 3 is a structural schematic diagram of the double-scale feature extraction module in Figure 2 .

[0059] Figure 4 is a structural schematic diagram of the feature decoding module 2.

[0060] Figure 5 is a structural schematic diagram of the feature self-attention module in Figure 2 . DETAILED DESCRIPTION

[0061] The present application will be further described in detail below in combination with the drawings and embodiments, but the present application is not limited to the following embodiments.

[0062] Embodiment 1

[0063] The point cloud semantic segmentation method based on double-scale feature extraction and self-attention of the present embodiment is composed of the following steps (see Figure 1 ):

[0064] (1) Select the data set

[0065] The point cloud in the S3DIS point cloud dataset is divided into a training set and a test set at a ratio of 5:1, and there is no overlap between the training set and the test set.

[0066] (2) Point cloud preprocessing

[0067] The point cloud is segmented into point cloud blocks with a size of 1m x 1m x 1m cubes.

[0068] (3) Construction of point cloud semantic segmentation network

[0069] The point cloud semantic segmentation network is composed of a feature encoding module 1, a feature decoding module 2, and a normalization prediction module 3 connected in sequence.

[0070] Figure 2 The structural diagram of the feature encoding module 1 is given. In the Figure 2 , the feature encoding module 1 of the embodiment is composed of a first layer encoding block, a second layer encoding block, and a third layer encoding block connected in sequence.

[0071] The first layer encoding block of the embodiment is composed of a double-scale feature extraction module and a feature self-attention module connected in sequence; the structures of the second layer encoding block and the third layer encoding block are the same as that of the first layer encoding block. The double-scale feature extraction module of the embodiment is composed of a feature extraction first branch module and a feature extraction second branch module connected in parallel.

[0072] In Figure 3 , the feature extraction first branch module of the embodiment is composed of a shared multi-layer perceptron and a max pooling layer connected in sequence. The convolution kernel size of the shared multi-layer perceptron of the embodiment is 1x1, and the step size of the sliding window is 1. The structure of the feature extraction second branch module is the same as that of the feature extraction first branch module. The construction method of the feature extraction first branch module of the embodiment is as follows:

[0073] 1) The shared multi-layer perceptron F is constructed according to the following formula: i Method:

[0074]

[0075]

[0076]

[0077]

[0078] x 1×m =(ω(F 1×n ) T ) T

[0079] λ=(λ wv ) W×V

[0080] τ = (τ wv ) W×V

[0081] ω = (ω mn ) M×N

[0082] where ReLu(.) is an activation function, λ is a weight matrix shared by the multi-layer perceptron, τ is a bias matrix shared by the multi-layer perceptron, and ω is a weight coefficient matrix of different dimension features of the point cloud;

[0083] 2) Determine the maximum pooling layer F i_max :

[0084] F i_max = MaxPooling(F i )

[0085] The construction method of the feature extraction second branch module is the same as that of the feature extraction first branch module.

[0086] In Figure 4 , the feature decoding module 2 of the embodiment is composed of a first layer decoding block and a second layer decoding block and a third layer decoding block connected in sequence. The first layer decoding block of the embodiment is composed of a shared multi-layer perceptron and an interpolation upsampling connected in series. The structures of the second layer decoding block and the third layer decoding block are the same as that of the first layer decoding block.

[0087] In Figure 5 , the feature self-attention module of the embodiment is composed of a linear layer 1, a linear layer 2, a linear layer 3, a shared multi-layer perceptron 1, a shared multi-layer perceptron 2, and a normalization function layer connected. The output ends of the linear layer 1 and the linear layer 2 are connected to the input end of the shared multi-layer perceptron 1. The output ends of the shared multi-layer perceptron 1 and the shared multi-layer perceptron 2 are connected to the input end of the normalization function layer. The output end of the linear layer 3 is connected to the normalization function layer.

[0088] The construction method of the feature self-attention module of the embodiment is as follows:

[0089] The output F i_max of the double-scale feature extraction module is input into the linear layer 1 and the linear layer 2 to obtain the input of the feature self-attention module, as follows:

[0090] F p = Linner(F i_max )

[0091] The linear transformation F p of the feature of a point and the linear feature transformation F q of the neighboring points are input into the shared multi-layer perceptron 1 to obtain the self-attention vector ATT, as follows:

[0092] ATT = γ (F q -F p )+δ

[0093] Wherein, γ represents a shared multi-layer perception 1, and δ is determined by inputting position information into a linear layer 3, as follows:

[0094]

[0095] Wherein, q i represents position information of a current point, represents position information of K nearest neighbor points of the current point, represents feature splicing,.| 2 represents the Euclidean distance between two points.

[0096] Linearly transform the feature F p of a point into a shared multi-layer perception 2, and obtain the feature S iq optimized by the feature self-attention module according to formula (1):

[0097]

[0098] Wherein, ρ is a softmax function, the mapping function α represents a multi-layer perception 2, the symbol ⊙ represents an XOR operation, and K is 8-32, and K is 16 in the embodiment.

[0099] Since the application adopts the method of dividing large-scale point cloud data into 1m*1m*1m point cloud blocks and inputting the point cloud blocks into a point cloud semantic segmentation network, fusing features of two scales in a local area of the point cloud, and optimizing the fused vector by using a self-attention vector, the semantic label of the point cloud can be more accurately predicted, and the predicted label is closer to the artificial labeled label.

[0100] (4) Training the point cloud semantic segmentation network

[0101] 1) Constructing a loss function

[0102] The loss function L1 is constructed according to the following formula:

[0103]

[0104] Wherein, x represents the number of points in the point cloud, H x is the label value of the xth point, and S x is the predicted label value of the xth point.

[0105] 2) Training the point cloud semantic segmentation network

[0106] The software condition is a CentOS7.5 operating system, based on a pytorch1.7.0 framework, the training set is input into the point cloud semantic segmentation network for training, the parameter learning rate is 0.5 during the training process, the momentum attenuation is set to 0.7, the weight attenuation rate is 0.0001, the batch processing amount is 32, the training round number is 50, the server is a Huawei server RH2288H V2 equipped with two Nvidia Tesla v100 graphics cards, and the training is stopped until the loss function converges.

[0107] (5) Testing the point cloud semantic segmentation network

[0108] The test set is input into the trained point cloud semantic segmentation network to obtain the final semantic label of each point cloud.

[0109] The point cloud semantic segmentation method based on double-scale feature extraction and self-attention is completed.

[0110] Embodiment 2

[0111] The point cloud semantic segmentation method based on double-scale feature extraction and self-attention of the embodiment comprises the following steps:

[0112] (1) Selecting a data set

[0113] This step is the same as that in Embodiment 1.

[0114] (2) Point cloud preprocessing

[0115] This step is the same as that in Embodiment 1.

[0116] (3) Constructing a point cloud semantic segmentation network

[0117] In formula (1), wherein, p is a softmax function, the mapping function a represents a multilayer perceptron 2, represents an exclusive or operation, K takes a value of 8-32, and K takes a value of 8 in this embodiment.

[0118] The other steps of this step are the same as those in Embodiment 1.

[0119] The other steps are the same as those in Embodiment 1, and the point cloud semantic segmentation method based on double-scale feature extraction and self-attention is completed.

[0120] Embodiment 3

[0121] The point cloud semantic segmentation method based on double-scale feature extraction and self-attention of the embodiment comprises the following steps:

[0122] (1) Selecting a data set

[0123] This step is the same as that in Embodiment 1.

[0124] (2) Point cloud preprocessing

[0125] The step is the same as that in Example 1.

[0126] (3) Constructing a point cloud semantic segmentation network

[0127] In formula (1), wherein, p is a softmax function, the mapping function a represents a multi-layer perceptron 2, represents an exclusive or operation, K is 8-32, and K is 32 in the embodiment.

[0128] The other steps of the step are the same as those in Example 1.

[0129] The other steps are the same as those in Example 1, and the point cloud semantic segmentation method based on double-scale feature extraction and self-attention is completed.

[0130] In order to verify the beneficial effects of the application, the method of Example 1 of the application is compared with PointNet, PointNet++, RSNet, A-SCN, DGCNN, KGCN, KVGCN, PF+LF, and A-CNN methods in a comparative simulation experiment, and in the simulation experiment, the average intersection over union mIOU, the overall accuracy oAcc, and the average accuracy mAcc are determined according to the following formula:

[0131]

[0132] Wherein, k represents the number of categories, p ii represents points correctly classified, p ji represents points that belong to class j but are predicted to be class i, and p ij represents points that belong to class i but are predicted to be class j.

[0133] The overall accuracy oAcc is calculated according to the following formula:

[0134]

[0135] The average accuracy mAcc reflects the average of the accuracy of all semantic categories, and the evaluation of the segmentation performance of the network is more objective, and the formula is as follows:

[0136]

[0137] The experimental and calculation results are shown in Tables 1 and 2.

[0138] Table 1 Simulation experiment results of the method of Example 1 and the comparative experiment method

[0139]

[0140] As shown in Table 1, the method of Example 1 improves the mIou by 2.2% compared with KVGCN, 8.3% compared with PF+LF, 4.1% compared with KGCN, 0.2% compared with A-CNN, 10.4% compared with A-SCN, 7% compared with DGCNN, 11.2% compared with RSNet, 8.6% compared with PointNet++, and 22% compared with the baseline network PointNet.

[0141] Table 2 Simulation experiment results of the method of Example 1 and comparative experiment methods (13 semantic categories)

[0142] Method celling floor wall beam column window door chair table bookcase sofa board clutter PointNet 88.6 97.3 69.8 0.05 3.92 46.3 10.8 52.6 58.9 40.3 5.9 26.4 33.2 PointNet++ 90.2 91.7 73.1 42.7 21.2 49.7 42.3 59.0 62.7 45.8 19.6 48.2 45.6 G+RCU 90.3 92.1 67.9 44.7 24.2 52.3 51.2 58.1 41.9 6.9 47.4 39.0 30.0 SEGcloud 90.1 96.1 69.9 0.0 18.4 38.4 23.1 78.6 70.4 58.4 40.9 13.0 41.1 RsNet 93.3 98.3 79.2 0.0 15.7 45.4 50.1 65.5 67.9 22.4 52.4 41.0 43.6 PF+LF 93.9 98.6 74.5 35.4 31.9 59.5 65.9 61.3 55.0 44.3 18.2 37.4 53.4 Ours 92.1 98.5 78.3 0.0 0.09 61.1 45.0 82.6 74.2 64.9 52.6 64.8 47.7

[0143] As shown in Table 2, the application shows better semantic segmentation effects on 7 of the 13 individual semantic categories, which are wall 78.3%, window 61.1%, chair 82.6%, table 74.2%, bookcase 64.9%, sofa 52.6%, and board 64.8%.

Claims

1. A point cloud semantic segmentation method based on double-scale feature extraction and self-attention, characterized in that Comprise the following steps: (1) Select data set The point cloud in the point cloud database is divided into training set and test set by 5:1, and there is no overlap between the training set and the test set; (2) Point cloud preprocessing segmenting a point cloud into cubically sized point cloud blocks; (3) Constructing point cloud semantic segmentation network The point cloud semantic segmentation network is composed of feature encoding module 1, feature decoding module 2 and normalization prediction module 3 connected in sequence; The feature encoding module 1 is composed of first layer encoding block, second layer encoding block and third layer encoding block connected in sequence; The first layer encoding block is composed of double scale feature extraction module and feature self attention module connected in sequence; the structure of the second layer encoding block and the third layer encoding block is the same as that of the first layer encoding block; The feature decoding module 2 is composed of first layer decoding block, second layer decoding block and third layer decoding block connected in sequence; The double scale feature extraction module is composed of feature extraction first branch module and feature extraction second branch module connected in parallel; The feature extraction first branch module is composed of shared multilayer perception and maximum pooling layer connected in sequence; the structure of the feature extraction second branch module is the same as that of the feature extraction first branch module; The construction method of the feature extraction first branch module is as follows: 1) A method of constructing a shared multi-layer perceptron according to the following formula: ​ wherein ReLu(.) is an activation function, is a weight matrix of the shared multi-layer perceptron, is a bias matrix of the shared multi-layer perceptron, is a weight coefficient matrix of different dimension features of the point cloud; 2) The max-pooling layer is determined as follows : The construction method of the feature extraction second branch module is the same as that of the feature extraction first branch module; The first layer decoding block is composed of shared multilayer perception and interpolation upsampling connected in sequence; the structure of the second layer decoding block and the third layer decoding block is the same as that of the first layer decoding block; The feature self attention module is composed of linear layer 1, linear layer 2, linear layer 3, shared multilayer perception 1, shared multilayer perception 2 and normalization function layer connected, the output ends of the linear layer 1 and the linear layer 2 are connected with the input end of the shared multilayer perception 1, the output ends of the shared multilayer perception 1 and the shared multilayer perception 2 are connected with the input end of the normalization function layer, and the output end of the linear layer 3 is connected with the normalization function layer; The construction method of the feature self attention module is as follows: the output of the bi-scale feature extraction module is input into linear layer 1 and linear layer 2 to obtain the input of the feature self-attention module as follows: Linearly transforming a point's features Linearly transforming a point's features An input shared multi-layer perceptron 1 yields a self-attention vector ATT, as follows: wherein, represents a shared multi-layer perceptron 1, is determined by the position information input into the linear layer 3, as follows: wherein, represents position information of a current point, represents position information of K nearest neighbor points of the current point, represents feature stitching, represents Euclidean distance between two points; Linearly transforming a point's features The input is fed into the shared multi-layer perceptron 2, and the features are optimized by the feature self-attention module according to equation (1) : （1） wherein is a softmax function, mapping function represents a multi-layer perception 2, represents an XOR operation, K takes values from 8 to 32; (4) Training point cloud semantic segmentation network 1) Constructing loss function The loss function L1 is constructed as follows: wherein x represents the number of points in the point cloud, H x is the label value of the xthpoint, S x is the predicted label value of the xthpoint; 2) Training point cloud semantic segmentation network The software condition is CentOS7.5 operating system, based on pytorch1.7.0 framework, the training set is input into the point cloud semantic segmentation network for training, the parameter learning rate is 0.5, the momentum attenuation is set to 0.7, the weight decay rate is 0.0001, the batch processing amount is 32, the training round number is 50, the server is Huawei server RH2288H V2 equipped with two Nvidia Tesla v100 graphics cards, and the training is stopped until the loss function converges; (5) Test point cloud semantic segmentation network The test set is input into the trained point cloud semantic segmentation network to obtain the final semantic label of each point cloud.

2. The point cloud semantic segmentation method based on double-scale feature extraction and self-attention according to claim 1, characterized in that: The convolution kernel size of the shared multilayer perception is 1x1, and the step of the sliding window is 1.

Images