A point cloud semantic segmentation method based on feature fusion and attention mechanism

Abstract

The application discloses a point cloud semantic segmentation method based on feature fusion and attention mechanism, wherein a local feature fusion module, a multi-layer attention pool module and a jump connection are stacked together to form a residual module as an encoding module in an encoder-decoder structure; a nearest neighbor interpolation method is used as a feature propagation method in a decoder; and when the number of point clouds is restored to the original number, a semantic label is assigned to each point cloud through a full connection layer. The network of the application can achieve competitive results in the average intersection over union (mIoU), and the network segmentation accuracy is improved by 23% than PoinNet, 5.2% than PointCNN and 4.3% than ELGS in the evaluation of the average intersection over union.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of 3D point cloud image processing, and in particular discloses a point cloud semantic segmentation method based on feature fusion and multi-layer attention mechanism. BACKGROUND

[0002] The rapid development of depth sensors and laser scanning technology makes it easy for people to collect a large amount of point cloud data, and under the driving of application scenarios such as autonomous driving, robots, virtual / mixed reality, three-dimensional point cloud perception technology has been widely used. Point cloud semantic segmentation is an important part of three-dimensional perception and has become a research hotspot in the field of three-dimensional perception.

[0003] 3D point cloud is essentially a series of three-dimensional coordinate points, which can be supplemented with color, normal vector, etc. to form a more dimensional point cloud dataset. Traditional two-dimensional image segmentation traditional convolutional neural network cannot be directly used for point cloud semantic segmentation technology, because point cloud has the characteristics of disorder, sparsity and non-uniform density.

[0004] Common methods for processing point clouds include voxel-based methods, multi-view-based methods, and direct point-based methods. Voxel-based or multi-view-based methods normalize the shape of the input point cloud, converting irregular point clouds or mesh data into regular 3D voxel grids or multi-views, which are provided to deep network architecture. However, problems such as loss of geometric structure information and data sparsity limit the development of multi-view methods and voxelization methods.

[0005] The network proposed in the document Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. 2017 can directly process unordered point cloud data for classification and segmentation tasks, which is a pioneering work using point-by-point MLP method to process point clouds. However, the farthest-point sampling downsampling strategy used by this algorithm consumes a lot of computing resources, so the above method often occupies a large amount of memory or computing resources, making it difficult to be applied in real scenarios with millions or hundreds of millions of points. In addition, existing point cloud semantic segmentation networks lack the use of global context information and ignore the local geometric relationship of point cloud space, resulting in loss of spatial geometric information and low segmentation accuracy.

[0006] Therefore, it is a technical problem to be solved to realize point cloud semantic segmentation in large-scale scenarios and ensure the accuracy of segmentation. SUMMARY

[0007] To address the aforementioned problems encountered in point cloud semantic segmentation, this invention provides a point cloud semantic segmentation method based on feature fusion and attention mechanisms. A residual module, consisting of a local feature fusion module, a multi-layer attention pool module, and skip connections stacked together, serves as the encoding module in the encoder-decoder structure. The decoder uses nearest neighbor interpolation as the feature propagation method. When the number of points in the cloud is restored to its original value, a semantic label is assigned to each point cloud through a fully connected layer.

[0008] The technical solution adopted by the present invention to solve the aforementioned technical problem is as follows:

[0009] A point cloud semantic segmentation method based on feature fusion and attention mechanisms includes a point cloud semantic segmentation network, which comprises a decoder and an encoder. The encoder employs a semantic segmentation residual module in each coding layer. In the semantic segmentation residual module, the point cloud input features are processed by an MLP layer and then input together with the point cloud spatial coordinate information into a feature fusion module. The output of the feature fusion module is sequentially connected to a multi-layer attention pool module and an MLP layer. The point cloud feature information is then processed by another MLP layer and summed with the output of the MLP layer connected to the multi-layer attention pool module to obtain the output of the semantic segmentation residual module.

[0010] The feature fusion module includes a feature encoding unit and a position encoding unit. The input of the feature encoding unit is the feature information f of the center point. i The KNN algorithm is used to collect feature information from j neighboring points and feature information f from the center point. i The feature information of the center point and the features of the neighboring points f ij The difference is then concatenated with the feature information of the center point to obtain the output of the feature encoding unit;

[0011] The location encoding unit uses point cloud spatial coordinate information to calculate the spatial feature p of the center point based on the KNN algorithm. i The neighborhood point information is used to calculate the spatial features p of the center point. i Spatial features p of j neighboring points ij Relative features c in Euclidean coordinates ij and relative features a in polar coordinates ij ;

[0012] The relative feature c in the Euclidean coordinates of j neighboring points ij and relative features a in polar coordinates ij After performing pooling operations separately, the features are concatenated with themselves, and finally, the updated feature C is obtained using a shared multilayer perceptron (MLP). ij A ij The updated feature C ij A ij Spatial features of the center point pi Spatial features p of j neighboring points ij relative position coordinates p i -p ij The output of the position encoding unit is obtained by concatenation;

[0013] The outputs of the feature coding unit and the position coding unit are used to obtain local features H using MLP and the ReLU activation function, respectively. ij and local geometric features G ij The fused feature e is obtained by multiplying the two elements element by element. ij , fusion feature e ij This is the output of the feature fusion module;

[0014] The multi-layer attention pool module includes local attention pools and cascaded attention pools. The input to the local attention pool is the fused feature e. ij The input to the cascaded attention pool is the result f' of the local attention pool. i After processing by the KNN algorithm, the spatial features p of the j neighboring points of the center point are... ij After concatenation, the vectors are processed by an MLP layer to obtain the enhanced feature vector λ. ij Then for λ ij Apply the softmax operation to calculate the attention weights s ij Then, λ ij and attention weights ij Perform element-wise multiplication to obtain the updated feature λ' ij The feature λ' will be updated. ij With λ ij After addition, the next level of feature enhancement is performed through the MLP layer, and then the next level of attention weights is calculated through the softmax operation. The enhanced feature vector of the next level of feature enhancement and the next level of attention weights are multiplied by elements to obtain the updated features. After summation, the output features of the cascaded attention pool are obtained.

[0015] The output f' of the local attention pool is obtained by using an MLP shared multilayer perceptron. i The output features of the cascaded attention pool are mapped to the same dimension, and then additive fusion is used to obtain the output of the multi-layer attention pool module;

[0016] Semantic segmentation of point cloud scenes is performed using a point cloud semantic segmentation network.

[0017] The updated feature C ij A ij The expression is:

[0018] C ij =MLP(c ij ||(α(c ij )))

[0019] A ij = MLP(a ij || η(a ij ))

[0020] where the function a is an average pooling function, the function η is a sum pooling function, MLP is a shared multi-layer perceptron, and || is a concatenation operation;

[0021] Local geometric feature G ij and local feature H ij respectively represent:

[0022] G ij = MLP(ReLU(a ij || C ij || p i -p ij )

[0023] H ij = MLP(ReLU((f i -f ij )|| f i )).

[0024] In the feature extraction process of the encoder, the number of point clouds of the input point cloud scene is N, and the feature dimension is 8 dimensions. After four times of random sampling for downsampling, the number of scene point clouds is sequentially reduced to N / 4, N / 16, N / 64, and N / 128, and the feature dimension is sequentially increased to 32, 128, 256, and 512. A semantic segmentation residual module is connected after each downsampling;

[0025] In the decoder, first, the point cloud with a number of N / 128 and a feature dimension of 512 is subjected to four times of nearest neighbor difference value, the down-sampled point cloud is restored to the original number, that is, the number of point clouds is increased from N / 128 to N / 64, N / 16, N / 4, N, and the feature dimension of the point cloud is sequentially reduced from 512 to 256, 128, 32, and 8. The features of the corresponding feature layers of the encoder and the decoder are spliced;

[0026] The output of the decoder is allocated a semantic label for each point cloud through three fully connected layers, and finally the semantic segmentation of the point cloud scene is realized.

[0027] For the input point cloud with a dimension of N*d, wherein N is the total number of all points in this batch, and d is the feature dimension of each point in the input point cloud; the point cloud is input to a fully connected layer, which is used to extract more dimensional features of the point cloud. In the present application, after the fully connected layer, the dimension of the input point cloud is changed to 8 dimensions.

[0028] The process of the local attention pool is as follows: first, the fused feature eij The fusion features are normalized by using a softmax function to obtain attention weights σ ij The product of the neighbor point features f ij and the attention weights σ ij is summed, specifically as follows:

[0029]

[0030] where f' i is the output result of the local attention pool, and is the updated feature of the center point.

[0031] Compared with the prior art, the present application has the beneficial effects including:

[0032] The present application first uses a feature fusion module to construct features for each point using the features in the neighborhood and the spatial geometric relationship, and learns the local geometric structure information by fusing spatial and semantic feature information. Then, through a multi-layer attention pool module, the local attention pool and the cascaded attention pool are used in turn to learn more discriminative local geometric features and to mine more detailed global context features. Finally, the features are embedded into an encoder-decoder architecture to ultimately realize semantic segmentation of the point cloud scene. The method of the present application optimizes the extraction of local geometric relationships and global context information of the point cloud, and can be applied to real scene point cloud segmentation tasks, with accurate segmentation results and wide application.

[0033] Compared with existing point cloud semantic segmentation networks, the network of the present application can achieve competitive results in the average intersection over union (mIoU), with the network segmentation accuracy improved by 23% compared with PoinNet, by 5.2% compared with PointCNN, and by 4.3% compared with ELGS in the evaluation of the average intersection over union. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a structural schematic diagram of a feature fusion module of an embodiment of the present application.

[0035] Figure 2 is a structural schematic diagram of a multi-layer attention pool module of an embodiment of the present application.

[0036] Figure 3 is a structural schematic diagram of a semantic segmentation residual module of an embodiment of the present application.

[0037] Figure 4 is a structural schematic diagram of a point cloud semantic segmentation network of an embodiment of the present application.

[0038] Figure 5 is a semantic segmentation effect diagram of an embodiment of the segmentation method of the present application. DETAILED DESCRIPTION

[0039] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0040] This invention provides a point cloud semantic segmentation method based on feature fusion and attention mechanisms, the method comprising the following steps:

[0041] Step 1: For a point cloud with an input dimension of N*d, where N is the total number of points in the batch and d is the feature dimension of each point in the input point cloud, the point cloud will be input into a fully connected layer. This fully connected layer is used to extract more dimensional features from the point cloud. In this embodiment of the invention, after passing through the fully connected layer, the dimension of the input point cloud becomes 8-dimensional.

[0042] Step Two: Dot Clouds Figure 1 As shown, the point cloud input points are first processed by the K-nearest neighbors (KNN) algorithm to collect spatial features p of j neighboring points. ij The collected neighborhood point cloud is then processed by the feature fusion module to establish accurate feature information for each point using the feature relationships in the neighborhood, including location coding units and feature coding units.

[0043] Step 2.1 In point clouds, spatial coordinate information is crucial and can be used to analyze the regional geometry of the point cloud. Therefore, position encoding units are first constructed in the feature fusion module using Euclidean and polar coordinates;

[0044] Calculate the spatial features p of j neighboring points ij Spatial features of the input center point p i The relative relationship is used to extract the relative features c in the Euclidean coordinates of the center point and neighboring points of the point cloud. ij :

[0045] c ij =x ij ||y ij ||z ij ||d ij

[0046] x ij y ij , z ij d is the relative coordinates of the center point and its j neighboring points in the Cartesian coordinate system. ij It represents the relative distance between the center point and j adjacent domains in the Cartesian coordinate system; || represents the concatenation operation;

[0047] Extract the relative features a in the polar coordinate system between the center point of the point cloud and its j neighboring points. ij It can be represented as:

[0048]

[0049] and θ ij are the polar radius and polar angle of the j-neighborhood points relative to the center point respectively

[0050]

[0051] The updated features C are obtained by applying the pooling operation to the relative position relationship of the j-neighborhood points in the two coordinate systems ij , A ij , the process is as follows:

[0052] C ij = MLP(c ij || (a ij (c ij )))

[0053] A ij = MLP(a ij || (η(a ij )))

[0054] Where the function a is the average pooling function, the function η is the sum pooling function, and MLP is a shared multi-layer perceptron that learns high-dimensional feature relationships.

[0055] The above features A ij , C i , and the relative position coordinates p ij -p i of the center point spatial feature p ij and the j-neighborhood point spatial feature p ij are concatenated to obtain the output of the position encoding unit, which encodes the point cloud features using position information, and then the output of the position encoding unit is processed using MLP and the activation function ReLU to obtain the local geometric feature G ij , which is represented as:

[0056] G ij = MLP(A ij || C i || p ij -p i )

[0057] Step 2.2 Figure 1 The input of the feature fusion module is the feature information f ij of the center point and the j-neighborhood point feature f i collected by the K-nearest neighbors (KNN) algorithm, and the feature encoding unit can be represented as (f ij -f i )|| f ij , and then the output of the feature encoding unit is processed using MLP and the activation function ReLU to obtain the local feature HRepresented as:

[0058] H ij =MLP(ReLU((f i -f ij )||f i ))

[0059] Among them, f i It is the feature information of the center point, f ij It represents the features of neighboring points, and ReLU is the activation function;

[0060] Step 2.3 Transfer local features H ij and local geometric features G ij The fused feature e is obtained by multiplying the two elements element by element. ij , fusion feature e ij This is the output of the feature fusion module;

[0061] Step 3 utilizes the multi-layer attention pool module to process the results generated by the feature fusion module in Step 1, enhancing features and extracting contextual information. The multi-layer attention pool module includes local attention pools and cascaded attention pools.

[0062] Step 3.1 Local attention pooling, used for point cloud semantic segmentation; such as Figure 2 As shown, the local attention pooling process first processes the fused feature e output by the feature fusion module. ij The softmax function is used to fuse the features e. ij Normalization is performed to obtain the attention weights σ of the fused features. ij Calculate the features f of all neighboring points ij With attention weight σ ij The product of these products is then summed, as shown below:

[0063]

[0064] Where f' i It is the update feature of the center point, and N(i) is the neighborhood set of point i.

[0065] Step 3.2 Cascade attention pools, such as Figure 2 As shown, the input to the cascaded attention pool is the output f' of the local attention pool. i After processing by the KNN algorithm, the spatial features p of the j neighboring points of the center point are... ij After concatenation, the vectors are processed by an MLP layer to obtain the enhanced feature vector λ. ij Then for λ ij Apply the softmax operation to calculate the attention weights s ij Then, λ ij and attention weights ijThe element multiplication is performed to obtain the updated feature λ' ij ; the updated feature λ' ij is added to λ ij , and then the next level of feature enhancement is continued through the MLP layer, and the next level of attention weight is calculated through the softmax operation; the enhanced feature vector of the next level of feature enhancement and the next level of attention weight are subjected to element multiplication to obtain the updated feature, and the output feature of the cascaded attention pool is obtained through summation;

[0066] Step 3.3 uses MLP to map the output results of the local attention pool and the output feature of the cascaded attention pool to the same dimension, and then uses addition to fuse the output feature of the local attention pool and the output feature of the cascaded attention pool, to obtain the output of the multi-layer attention pool module.

[0067] Step four: connect the feature fusion module and the multi-layer attention pool module together to form a semantic segmentation residual module, as shown in Figure 3 The specific process of the semantic segmentation residual module is as follows: first, the input feature of the point cloud is subjected to feature extraction through the MLP shared multi-layer perceptron, then the feature and the spatial feature are input into the feature fusion module and the multi-layer attention pool module in turn, and the feature extraction is continued through the MLP, and finally the input feature of the point cloud is directly input into the MLP shared multi-layer perceptron, and the output of the MLP after being processed by the multi-layer attention pool module is summed up to form the semantic segmentation residual module. Then, the semantic segmentation residual module is embedded as an encoding module into a standard encoder-decoder architecture, thereby generating a point cloud semantic segmentation network;

[0068] The structure of the point cloud semantic segmentation network is shown in Figure 4 , which includes a decoder and an encoder. The encoder has four encoding layers. The point cloud is input into the full connection layer to obtain a point cloud input of a specified dimension. The point cloud input is input into the semantic segmentation residual module at each level to obtain intermediate encoding features. The encoding features are obtained by four times of random sampling and feature extraction of the semantic segmentation residual module. The encoding features are input into the decoder, and the corresponding features corresponding to the encoding-decoding layers are spliced to obtain decoding features. The decoding features are output through three full connection layers to obtain the final point cloud segmentation result. Each encoding layer adopts the structure of random down-sampling RS and the semantic segmentation residual module in series. In the feature extraction process of the encoding layer, the number of scene point clouds input is N, the feature dimension is 8, and the number of scene point clouds is reduced to N / 4, N / 16, N / 64, and N / 128 through four times of random sampling. The feature dimension is increased to 32, 128, 256, and 512 in turn;

[0069] The decoder has four decoding layers, in the decoding layer feature propagation, firstly, the point cloud with the number of N / 128 and the feature dimension of 512 is processed by four times of nearest neighbor difference up-sampling US and MLP, the nearest neighbor difference is calculated, the down-sampled point cloud is restored to the original number, that is, the number of point clouds is increased from N / 128 to N / 64, N / 16, N / 4, N, at the same time, the feature dimension of the point cloud is sequentially reduced from 512 to 256, 128, 32, 8, and the features corresponding to the encoding-decoding layers are spliced;

[0070] The output of the decoder is given a semantic label for each point cloud through three fully connected layers FC, and finally the semantic segmentation of the point cloud scene is realized.

[0071] When the semantic segmentation of the point cloud scene is performed by using the application, the accuracy of the category corresponding to each point of the point cloud can be effectively ensured, and the verification is performed on Area5 of the S3DIS dataset, the S3DIS dataset is an indoor scene dataset divided into six parts, namely Area1, Area2, Area3, Area4, Area5 and Area6, training is performed on Area1, Area2, Area3, Area4, Area6, and the specific semantic segmentation effect of the indoor scene point cloud data is as shown in Figure 5 Figure 5 (a) is the input indoor scene point cloud data, Figure 5 (b) is the real label (the label is the category of a certain point cloud, that is, a certain point cloud belongs to a chair, a floor or a ceiling, etc., and different colors are displayed in the result visualization) of the indoor scene point cloud data, Figure 5 (c) is the effect of the indoor point cloud scene semantic segmentation realized by using the segmentation method.

[0072] The unmentioned part of the application is applicable to the prior art.​

Claims

1. A point cloud semantic segmentation method based on feature fusion and attention mechanisms, comprising a point cloud semantic segmentation network, wherein the point cloud semantic segmentation network includes a decoder and an encoder, characterized in that, Each coding layer in the encoder employs a semantic segmentation residual module. In the semantic segmentation residual module, the point cloud input features are processed by an MLP layer and then input together with the point cloud spatial coordinate information into the feature fusion module. The output of the feature fusion module is sequentially connected to a multi-layer attention pool module and an MLP layer. The point cloud feature information is then processed by another MLP layer and summed with the output of the MLP layer connected to the multi-layer attention pool module to obtain the output of the semantic segmentation residual module. The feature fusion module includes a feature encoding unit and a position encoding unit. The input of the feature encoding unit is the feature information f of the center point. i The KNN algorithm is used to collect feature information from j neighboring points and feature information f from the center point. i The feature information of the center point and the features of the neighboring points f ij The difference is then concatenated with the feature information of the center point to obtain the output of the feature encoding unit; The location encoding unit uses point cloud spatial coordinate information to calculate the spatial feature p of the center point based on the KNN algorithm. i The neighborhood point information is used to calculate the spatial features p of the center point. i Spatial features p of j neighboring points ij Relative features c in Euclidean coordinates ij and relative features a in polar coordinates ij ; The relative feature c in the Euclidean coordinates of j neighboring points ij and relative features a in polar coordinates ij After performing pooling operations separately, the features are concatenated with themselves, and finally, the updated feature C is obtained using a shared multilayer perceptron (MLP). ij A ij The updated feature C ij A ij Spatial features of the center point p i Spatial features p of j neighboring points ij relative position coordinates p i -p ij The output of the position encoding unit is obtained by concatenation; The outputs of the feature coding unit and the position coding unit are used to obtain local features H using MLP and the ReLU activation function, respectively. ij and local geometric features G ij The fused feature e is obtained by multiplying the two elements element by element. ij , fusion feature e ij This is the output of the feature fusion module; The multi-layer attention pool module includes local attention pools and cascaded attention pools. The input to the local attention pool is the fused feature e. ij The input to the cascaded attention pool is the output f' of the local attention pool. i After processing by the KNN algorithm, the spatial features p of the j neighboring points of the center point are... ij After concatenation, the vectors are processed by an MLP layer to obtain the enhanced feature vector λ. ij Then for λ ij Apply the softmax operation to calculate the attention weights s ij Then, λ ij and attention weights ij Perform element-wise multiplication to obtain the updated feature λ' ij The feature λ' will be updated. ij With λ ij After addition, the features are further enhanced through the MLP layer, and then the next level of attention weights are calculated through a softmax operation. The enhanced feature vector of the next level feature enhancement and the attention weight of the next level are multiplied by element to obtain the updated features. After summing, the output features of the cascaded attention pool are obtained. The output of the local attention pool f is obtained by using an MLP shared multilayer perceptron. ' i The output features of the cascaded attention pool are mapped to the same dimension, and then additive fusion is used to obtain the output of the multi-layer attention pool module; Semantic segmentation of point cloud scenes is performed using a point cloud semantic segmentation network.

2. The point cloud semantic segmentation method based on feature fusion and attention mechanism according to claim 1, characterized in that, The updated feature C ij A ij The expression is: C ij <MLP(c ij ||(α(c ij ))) TO ij =MLP(a ij ||(η(a ij ))) Where α is the average pooling function, η is the summation pooling function, MLP is a shared multilayer perceptron, and || is the concatenation operation; Local geometric feature G ij and local features H ij They represent: G ij =MLP(ReLU(A ij ||C ij ||p i -p ij )) A ij =MLP(ReLU((f i -f ij )||f i ))).

3. The point cloud semantic segmentation method based on feature fusion and attention mechanism according to claim 1, characterized in that, In the feature extraction process of the encoder, the number of points in the input point cloud scene is N, and the feature dimension is 8. After downsampling in four random sampling methods, the number of points in the scene point cloud decreases to N / 4, N / 16, N / 64, and N / 128 respectively, and the feature dimension increases to 32, 128, 256, and 512 respectively. After each downsampling, a semantic segmentation residual module is connected. In the decoder, the point cloud with a number of N / 128 points and a feature dimension of 512 is first subjected to four nearest neighbor differences to restore the downsampled point cloud to its original number. That is, the number of points increases from N / 128 to N / 64, N / 16, N / 4, and N, while the feature dimension of the point cloud is successively reduced from 512 to 256, 128, 32, and 8. The features of the corresponding feature layers of the encoder and decoder are then concatenated. The decoder output assigns semantic labels to each point cloud through three fully connected layers, ultimately achieving semantic segmentation of the point cloud scene.

4. The point cloud semantic segmentation method based on feature fusion and attention mechanism according to claim 1, characterized in that, The process of the local attention pool is as follows: First, for the fusion feature e ij The attention weights σ of the fused features are obtained by normalizing the fused features using the softmax function. ij Calculate the features f of the neighborhood points ij With attention weight σ ij The product of these products is then summed, specifically: Where f ' i It is the output of the local attention pool, which is the updated feature of the center point.

5. The point cloud semantic segmentation method based on feature fusion and attention mechanism according to claim 1, characterized in that, The cloud semantic segmentation method is used for semantic segmentation of indoor scene point cloud data in the S3DIS dataset.

Images