A point cloud segmentation method based on bilateral feature fusion and vector self-attention

Abstract

The application discloses a point cloud segmentation method based on bilateral feature fusion and vector self-attention, and relates to the technical field of point cloud semantic segmentation; comprising the following steps: inputting original point cloud data; using a bilateral feature fusion module and a vector self-attention module to encode the input original point cloud data; decoding the point cloud features, using a continuous FP layer to up-sample the point cloud features, and obtaining the segmentation result of the point cloud. The application proposes an efficient point cloud semantic segmentation network, which makes the semantic segmentation of the point cloud faster and more accurate, and has superior segmentation performance; a group of correction information calculation modules based on geometric semantic bilateral feature information are proposed, which adjust the edge information and alleviate the edge ambiguity problem during local region aggregation, and strengthen the aggregation effect of local information; a new offset vector self-attention module is proposed, which effectively extracts the global features of the point cloud, and on the basis of reducing the network calculation amount, better global feature extraction effect is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud semantic segmentation, and particularly relates to a point cloud segmentation method based on bilateral feature fusion and vector self-attention, which is suitable for semantic segmentation of indoor point clouds. BACKGROUND

[0002] With the deepening of research, the method of using deep learning to process 3D point clouds has achieved great success, and these methods can be generally divided into three types: projection-based methods, voxelization-based methods and point-based methods. Among them, the point-based method, i.e. using a multi-layer perceptron (MLP) to directly process a point set, has become the mainstream due to its efficiency and high performance.

[0003] In the point-based method, PointNet is a very classic network. It uses a shared multi-layer perceptron (MLP) to extract features and consistently aggregates global features through a symmetric function without considering internal order. However, Pointnet adopts a single-point sampling mode, and this method cannot effectively extract local features. Pointnet++ proposes a sampling and grouping operation based on PointNet, and this multi-level feature extraction method effectively solves the above problems. However, the grouping method of Pointnet++ for local feature extraction will cause the edge ambiguity problem of the local area. In the neighborhood construction process, it is difficult to avoid the abnormal value and overlap between neighborhoods, and in the transition area of multiple semantic classifications, this abnormality and overlap are more prominent. Moreover, the aggregation area divided by the Euclidean distance as the standard cannot well adapt to the semantic features in the local range of the semantic space, which makes PointNet++ focus on the extraction of geometric information, and the aggregation effect of local feature information is insufficient. For semantic extraction, the extraction ability is weak, and the global extraction using FPS is also insufficient. Compared with Pintnet++, PointNeXt proposed recently focuses on training skills and scale strategies to further improve the performance of Pointnet++; PointMLP achieves very high classification performance by introducing a residual MLP structure without any complex local feature extractor design. However, these methods focus on the extraction of geometric space features, and the problems of edge ambiguity and insufficient global feature extraction of PointNet++ have not been well solved. There are also some methods that focus on semantic feature extraction. DGCNN proposes an edge convolution (EdgeConv) for learning edge features, which dynamically updates the graph structure between levels by constructing a local neighborhood graph and performing EdgeConv operation on each edge. AdaptiveGraph proposes to assign learning weights to each edge to better evaluate and aggregate information. These methods also use similar grouping methods, and there is also an edge ambiguity problem similar to PointNet++. Due to the focus on semantic space feature extraction, the feature extraction part has the problem of missing geometric structure in high-dimensional semantic space. With the great success of the self-attention mechanism in natural language processing and two-dimensional image processing tasks, the self-attention mechanism is also used to process three-dimensional point clouds. The attention mechanism has strong global feature extraction capability, but has the problem of large computational complexity.

[0004] So it can be seen that, since most of the current point cloud semantic segmentation network is extracted by grouping and aggregating point cloud for local feature. The ambiguity problem of neighbor edge in grouping is difficult to solve. Moreover, the aggregation region divided by Euclidean distance as the standard cannot well adapt to the semantic features in the local range of the semantic space. Similarly, the geometric structure is missing in the high-dimensional semantic space, resulting in insufficient local feature information aggregation effect. The method of constructing multi-scale feature extraction only through downsampling will lose a lot of detail information, and cannot fully extract global features. How to solve these problems has become a technical problem that the prior art urgently needs to solve. SUMMARY

[0005] Based on the technical problems existing in the background art, the present application proposes a point cloud segmentation method based on bilateral feature fusion and vector self-attention, which improves the robustness and accuracy of point cloud scene semantic segmentation, improves the feature learning ability of the semantic segmentation network, and alleviates the edge ambiguity of local region features and the lack of global feature representation.

[0006] The technical scheme adopted by the present application is:

[0007] A point cloud segmentation method based on bilateral feature fusion and vector self-attention, the method comprising the following steps:

[0008] S1: inputting original point cloud data;

[0009] S2: using a bilateral feature fusion module and a vector self-attention module to encode the input original point cloud data;

[0010] S3: decoding the point cloud features, using a continuous FP layer to upsample the point cloud features to obtain the segmentation result of the point cloud.

[0011] Further, the input original point cloud data of step S1 is specifically:

[0012] S3DIS is used as the test indoor data set, S3DIS is a large indoor scene segmentation data set, containing 13 categories, a total of 271 rooms. Each point cloud data has 9 features, namely color information R, G, B, coordinate information x, y, z and 3 normal vectors. The 271 rooms are divided into 6 areas, and each room is divided into 1 meter x 1 meter blocks. Set the input point cloud bit F in , the dimension is [B, N, 9], where B is the batch, N is the number of points, 9 is the feature, and the total number of input features is B*N*9.

[0013] Further, the step S2 of using the bilateral feature fusion module and the vector self-attention module to encode the input original point cloud data is specifically:

[0014] The input point cloud is divided into a geometry space containing coordinate information of the point cloud, with a dimension of [B, N, 3], and a semantic space containing color information and a normal vector of the point cloud, with a dimension of [B, N, 6]. The semantic space part is subjected to an MLP operation, is converted to a semantic space, and is then respectively fed into an encoder SA. FPS on the original data is used to generate sampling points p i Corresponding feature information is represented as f i With these sampling points as the center, the ball query method is used to group the point cloud under the measurement of three-dimensional Euclidean distance. The ball query method is to set a radius r, find points within the spherical range with the FPS sampling point as the center and the radius r, and these points are called neighbor points p i of the center point p j , An SA layer and a VA layer are added to extract point cloud features. The SA layer represents a local region aggregation layer of the point cloud, is responsible for bilateral feature information fusion and local feature extraction of the point cloud, and the VA layer is an improved self-attention layer, is responsible for global feature extraction of the point cloud.

[0015] In the SA layer, the absolute position of the center point and the relative position of the neighborhood are combined as a local feature G(p i ,p j ) = [p i ; p j -p i ] in the geometry space. Similarly, in the semantic space, S(f i ,f j ) = [f i ; f j -f i ] represents the local feature in the semantic space. An MLP is used to convert the geometry information to the semantic space, and a softmax function is used to convert the geometry information into a feature mask to correct the edge feature information of the semantic space. Similarly, the same method is used to dynamically adjust the edge information of the geometry space. At the same time, the adjusted edge information and the original edge information are added to establish a residual structure to ensure the robustness of information optimization. The calculation of the corrected information can be formulated as:

[0016] f s = Softmax(mlp(G(p i ,p j )))*S e +S e (1)

[0017] p s = Softmax(mlp(S(f i ,f j )))*Ge +G e (2)

[0018] where G e (pi,pj)=[p i ;p j -p i ],S e (f i ,f j )=[f i ;f j -f i ],p i ,p j ,p s ∈R N×3 ,f i ,f j ,f s ∈R N ×d .

[0019] The obtained supplementary information is then combined with p s , f s and G(p i ,p j ), S(f i ,f j ) to obtain G'=[p i ; p j -p i ; p s ], where S'=[f i ; f j -f i ; f s ], G'∈R N×9 , S'∈R N×3×d , and then they are concatenated to obtain enhanced local feature information F c The obtained local feature information F c is sent to two layers of LBR layers for feature extraction to obtain F c ', and finally the obtained feature F c ' is sent to a max-pooling layer to complete the aggregation of local region features to obtain local aggregated features F a ; after obtaining the local aggregated features F a , the obtained local region aggregated information is sent to a VA layer for global feature extraction;

[0020] In the traditional attention mechanism, the attention mechanism is calculated in the point cloud according to the following steps: firstly, the embedding feature F a is fed into three separable convolution layers with a kernel size of 1x1, thereby generating three new feature maps F q , Fk and F v ; then, the new feature map F q will be transposed and multiplied by the new feature map F k , and an attention matrix of size n x n is generated after the softmax layer:

[0021]

[0022] In the calculation method of the traditional attention mechanism, the method of vector attention mechanism is introduced, and the attention matrix is generated by subtraction, two linear layers and softmax function, and the calculation formula is as follows

[0023] Attention(F Q ,F K ,F V )=Softmax(γ(F Q -F K ))F V (4)

[0024] Where γ is a mapping function that generates an attention vector for feature aggregation, and the obtained attention matrix F A is point multiplied with the input feature F a to obtain the attention feature F v , and a kind of offset attention mechanism is designed, which is expressed by the following formula:

[0025] F out =Relu(Batchnorm(Mlp(F a -F A )))+F a (5)

[0026] The feature information F out is obtained.

[0027] Further, the step S3 of decoding the point cloud feature is specifically:

[0028] The decoding part uses continuous FP layers to upsample the point cloud feature to achieve the effect of segmenting the point cloud; the interpolation based on distance and the hierarchical propagation strategy of spanning the skip link are adopted; in the feature propagation level, the point feature is propagated from N l x (d+C) points to N l-1 points, wherein N l-1 and N l (having N l ≤N l-1 ) are the input and output point set sizes of the lth SA layer; N l-1 points are interpolated at the coordinates of N lThe feature propagation is realized by the eigenvalue f of each point; among the numerous choices of interpolation, the inverse distance weighted average based on k nearest neighbors is used; then, the interpolated features of N l-1 The interpolated features of each point are connected with the skip link point features from the set abstraction level; then, the connected features are passed through the "Pointnet unit", and a shared fully connected and ReLU layer is applied to update the feature vector of each point; the process is repeated until the features are propagated to the original point set;

[0029]

[0030]

[0031] Finally, the segmentation result of the point cloud is obtained.

[0032] The application provides a point cloud segmentation method based on bilateral feature fusion and attention mechanism, which has at least the following beneficial effects compared with the prior art:

[0033] 1) The application proposes an efficient point cloud semantic segmentation network, which makes the semantic segmentation of the point cloud faster and more accurate, and has superior segmentation performance.

[0034] 2) The application proposes a set of correction information calculation modules based on geometric semantic bilateral feature information, which adjusts the edge information and alleviates the edge ambiguity problem during local region aggregation, and strengthens the aggregation effect of local information.

[0035] 3) The application proposes a new offset vector self-attention module, which effectively extracts the global features of the point cloud, and achieves better global feature extraction effect on the basis of reducing the network calculation amount. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 Processing schematic diagram of the bilateral feature aggregation module;

[0037] Figure 2 Processing schematic diagram of the attention module;

[0038] Figure 3 Processing schematic diagram of the point cloud semantic segmentation network based on bilateral feature fusion and attention;

[0039] Figure 4 Point cloud semantic segmentation effect diagram. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments.

[0041] Embodiment 1.

[0042] As Figures 1-4 shown, a point cloud segmentation method based on bilateral feature fusion and vector self-attention, the method comprising the following steps:

[0043] S1: input original point cloud data

[0044] Taking S3DIS as the test indoor data set, S3DIS is a large indoor scene segmentation data set, containing 13 categories, a total of 271 rooms. Each point cloud data has 9 features, namely color information R, G, B, coordinate information x, y, z and 3 normal vectors. 271 rooms are divided into 6 areas, and each room is divided into 1 meter x 1 meter blocks. Set the input point cloud bit F in , the dimension is [B, N, 9], where B is the batch, N is the number of points, and 9 is the feature. The total number of input features is B*N*9;

[0045] S2: encode the input original point cloud data using a bilateral feature fusion module and a vector self-attention module

[0046] The input point cloud is divided into a geometric space containing the coordinate information of the point cloud, with a dimension of [B, N, 3], and a semantic space containing the color information and normal vector of the point cloud, with a dimension of [B, N, 6]. Then the semantic space part is subjected to mlp operation to transform it into a semantic space and then fed into the encoder SA. First, use the FPS on the original data to generate sampling points p i The corresponding feature information is represented as f i With these sampling points as the center, the ball query method is used to group the point cloud under the measurement of three-dimensional Euclidean distance, where the ball query method is to find the points within the spherical range with the FPS sampling point as the center and r as the radius. These points are called neighbor points p i of the center point p j , In this model, 4 layers of FPS sampling are used to construct point cloud space sampling points with different scales, with N / 4, N / 16, N / 64, and N / 128 respectively. In each layer, 0.1, 0.2, 0.4, and 0.8 are used as grouping radii to group the point cloud, and SA and VA layers are added to extract the point cloud features. The SA layer represents the local area aggregation layer of the point cloud, responsible for bilateral feature information fusion and local feature extraction of the point cloud. The VA layer is an improved self-attention layer responsible for global feature extraction of the point cloud.

[0047] In the SA layer, the absolute position of the center point and the relative position of its neighborhood are combined as the geometric space local feature G(p i , p j ) = [pi ; p j -p i ] in semantic space. Similarly, in semantic space, S(f i ,f j ) = [f i ; f j -f i ] represents the local feature in semantic space. The MLP is used to convert the geometric information to the semantic space, and the geometric information is converted to a feature mask by the softmax function, which corrects the edge feature information of the semantic space. Similarly, for the geometric space, the same method is used to dynamically adjust the geometric edge information. At the same time, the adjusted edge information and the original edge information are added to establish a residual structure to ensure the robustness of information optimization. The calculation of this modified information can be formalized as:

[0048] f s = Softmax(mlp(G(p i ,p j ))) * S e + S e (1)

[0049] p s = Softmax(mlp(S(f i ,f j ))) * G e + G e (2)

[0050] where G e (p i ,p j ) = [p i ; p j -p i ], S e (f i ,f j ) = [f i ; f j -f i ], p i ,p j ,p s ∈ R N×3 ,f i ,f j ,f s ∈ R N ×d .

[0051] The resulting supplementary information is then added to p s ,f s and G(p i ,p j ), S(fi f j ) combined to get G' = [p i ; p j -p i ; p s ], where S' = [f i ; f j -f i ; f s ], G' e R N×9 , S' e R N×3×d Then concatenate them to get the enhanced local feature information Fc, and send the obtained local feature information Fc to two layers of LBR layers (Linear + Batchnorm + Relu) for feature extraction to obtain F c ', and finally send the obtained feature F c ' to the maximum pooling layer to complete the feature aggregation of the local region to obtain the local aggregated feature F a .

[0052] After obtaining the local aggregated feature F a , the obtained local region aggregated information is sent to the VA layer for global feature extraction.

[0053] Before introducing the improved attention mechanism, the following traditional attention mechanism is first introduced. In point cloud, the calculation of the attention mechanism usually needs to follow the following steps. First, the embedding feature F a is fed to three separable convolution layers with a kernel size of 1x1, thereby generating three new feature maps F q , F k and F v . Subsequently, the new feature map F q will be transposed and multiplied by the new feature map F k , and an attention matrix of size n x n is generated after the softmax layer.

[0054]

[0055] The attention module of the attention module is introduced into the calculation method of the traditional attention mechanism. The method uses subtraction and two layers of linear layers and then passes through a softmax function to generate an attention matrix. Unlike the traditional attention calculation method, the attention weight in the vector attention is a vector that can modulate a single feature channel, and through subtraction instead, the computational complexity required to calculate the attention matrix can be reduced. The calculation formula is as follows

[0056] Attention(F Q , F K , F V ) = Softmax(γ(FQ -F K ))F V (4)

[0057] where γ is a mapping function (e.g. MLP) that produces the attention matrix F A by point-wise multiplication with the input features F a The attention features F v are then typically passed through a MLP layer and added to the input features F a through a residual connection to obtain the final output features F out but to improve the weight of attention and reduce the impact of noise, an offset attention mechanism is designed, which works by replacing the attention features with the offset between the input of the self-attention module and the attention features. This can be represented by the following formula:

[0058] F a -F A )) + F a (5)

[0059] to obtain the feature information F out

[0060] S3: Decoding point cloud features

[0061] The decoding part uses consecutive FP layers to upsample the point cloud features to achieve the effect of segmenting the point cloud. In simple terms, the FP layer is a process of returning the aggregated features to the original point cloud. Another method is to propagate the features from the subsampled points to the original points.

[0062] A distance-based interpolation and hierarchical propagation strategy across skip-links is adopted. In the feature propagation stage, the point features are propagated from N l ×(d+C) points to N l-1 points, where N l-1 and N l (with N l ≤ N l-1 ) are the input and output point set sizes of the l-th SA layer. The feature propagation is achieved by interpolating the feature values f of N l-1 points at the coordinates of N l points. Among the numerous choices of interpolation, the inverse distance weighted average based on k nearest neighbors is used (as in Equation 2, p = 2, k = 3 are used by default). Then, N l-1The interpolated features at a point are concatenated with the skip-linked point features from the set abstraction level. The concatenated features are then passed through a "Pointnet unit", which is analogous to a convolution over individual points in a CNN. Some shared fully connected and ReLU layers are applied to update the feature vector for each point. This process is repeated until the features are propagated to the original point set.

[0063]

[0064]

[0065] In the present model, the point cloud is up-sampled by using four continuous FP layers, and finally the segmentation result of the point cloud is obtained, as shown in Table 1.

[0066] Table 1 Qualitative segmentation results

[0067]

[0068]

[0069] The above describes only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any skilled person in the art, according to the technical solution and the inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, should be covered within the protection scope of the present application.

Claims

1. A point cloud segmentation method based on bilateral feature fusion and vector self-attention, characterized in that, The method comprises the following steps: S1: input original point cloud data; S2: encode the input original point cloud data using a bilateral feature fusion module and a vector self-attention module, specifically: The input point cloud is divided into a geometry space containing coordinate information of the point cloud, with a dimension of [B, N, 3], and a semantic space containing color information and a normal vector of the point cloud, with a dimension of [B, N, 6]. The semantic space part is subjected to an mlp operation, is converted to a semantic space, and is then respectively fed into an encoder SA. FPS on original data is used to generate sampling points p i Corresponding feature information representation f i Ball query method is used to group the point cloud under the measurement of three-dimensional Euclidean distance, with the sampling points as the center. The ball query method is to find points within a spherical range with the FPS sampling point as the center and r as the radius. These points are called neighbor points p i of the center point p j , for all p j ∈Ni(p i ), and SA and VA layers are added to extract point cloud features. The SA layer represents a local region aggregation layer of the point cloud, is responsible for bilateral feature information fusion and local feature extraction of the point cloud, and the VA layer is an improved self-attention layer, responsible for global feature extraction of the point cloud. In the SA layer, the absolute position of the center point and the relative position of its neighborhood are combined as a geometric space local feature G(p i , p j )= [p i ;p j -p i ]; similarly, a local feature in the semantic space is represented by S(f i , f j ) = [f i ;f j -f i ]; The MLP is used to convert the geometric information to the semantic space, and the geometric information is converted to a feature mask by a softmax function to correct the edge feature information of the semantic space; similarly, the same method is used to dynamically adjust the geometric edge information; at the same time, the adjusted edge information is added to the original edge information, and a residual structure is established to ensure the robustness of information optimization; the calculation of the corrected information can be formulated as: where G e (p i ,p j ) = [p i ;p j -p i ], S e (f i ,f j ) = [f i ;f j -f i ], p i ,p j ,p s ∈R N×3 , f i ,f j ,f s ∈R N ×d ; The supplementary information obtained will then be compared with p s f s With G(p i , p j ), S(f i , f j Combining these, we get G' = [p] i ;p j -p i ;p s ], where S' = [f i ;f j -f i ;f s ] ,G' ∈R N×9 , S'∈R N×3×d Then concatenate them to obtain the enhanced local feature information F. c The obtained local feature information F c The feature is extracted by feeding it into two LBR layers to obtain F. c Finally, the feature F obtained will be... c The feature is fed into a max pooling layer to aggregate features in the local region, resulting in the local aggregated feature F. a ; after obtaining the local aggregation feature F a Then, the obtained local region aggregation information is fed into the VA layer for global feature extraction; In the calculation method of the traditional attention mechanism, the vector attention mechanism is introduced, and the attention matrix is generated by subtraction, two linear layers and a softmax function, and the calculation formula is as follows: where γ is a mapping function that produces an attention vector for feature aggregation, resulting in an attention matrix F A with input features F a dot product with the attention vector to obtain the attention feature F v An offset attention mechanism is designed, which is expressed by the following formula: Thus, the feature information F is obtained out ; S3: decode the point cloud feature, and use a continuous FP layer to upsample the point cloud feature to obtain a point cloud segmentation result.

2. The point cloud segmentation method based on bilateral feature fusion and vector self-attention according to claim 1, characterized in that, The input original point cloud data in step S1 is specifically: S3DIS is used as the test indoor dataset, which is a large indoor scene segmentation dataset containing 13 categories and a total of 271 rooms; each point cloud data has 9 features, namely color information R, G, B, coordinate information x, y, z and 3 normal vectors; 271 rooms are divided into 6 areas, and each room is divided into 1 meter x 1 meter blocks; set the input point cloud bit F in , the dimension of which is [B, N, 9], wherein B is the batch, N is the number of points, and 9 is the feature. The total number of input features is B*N*9.

3. The point cloud segmentation method based on bilateral feature fusion and vector self-attention according to claim 1, characterized in that, The decoding of the point cloud feature in step S3 is specifically: The decoding part uses a continuous FP layer to upsample the point cloud feature to achieve the effect of segmenting the point cloud; a distance-based interpolation and a hierarchical propagation strategy across a skip link are used; In the feature propagation stage, the point features are propagated from N l × (d + C) points to N l-1 points, where N l-1 and N l are the input and output point set size of the l-th SA layer, where N l ≤ N l-1 ; the feature propagation is achieved by interpolating the feature values f of N l-1 points at the coordinates of N l points; among the numerous choices of interpolation, the inverse distance weighted average based on k nearest neighbors is used; then, the interpolated features on N l-1 points are concatenated with the skip-linked point features from the set abstraction stage, a shared fully connected and ReLU layer is applied to update the feature vector of each point; the process is repeated until the features are propagated to the original point set; Finally, a point cloud segmentation result is obtained.

Images