Three-dimensional point cloud semantic segmentation method based on attention soft pooling

Abstract

The invention provides a three-dimensional point cloud semantic segmentation method based on attention soft pooling. Firstly, original three-dimensional point cloud data are sent to an encoder for feature extraction; the local feature aggregation module constructs a multi-scale neighborhood, and learns point cloud features on different scales by using graph convolution; the point cloud edge attention pooling module encodes structural features through an edge convolution attention structure, adopts a space attention strategy to weight local features, fuses global shape features, and pools the fused local features; then, the outputs of the two feature mapping modules are connected, multi-level feature representation is constructed, and multi-level features are globally pooled; an asymmetric decoder structure is adopted, and sparse point cloud data is restored into high-resolution point cloud features by using an up-sampling module; and finally, connecting the up-sampled features with corresponding encoder feature points through residual connection, and finally forming high-precision semantic information of each point cloud.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional point cloud segmentation and discloses a three-dimensional point cloud semantic segmentation method based on attention soft pooling. Background Art

[0002] Point cloud data, with its real-world geometric form and rich semantic information, is widely used in many fields such as autonomous driving, intelligent robots, virtual reality, and urban planning. In this context, efficient classification and segmentation technology of point clouds has become the key to realizing intelligent 3D scene processing. However, the irregularity and disorder of point clouds pose great challenges to point cloud feature learning. At present, most point cloud segmentation methods use the symmetric function of maximum pooling to aggregate local features. However, this makes local point clouds insensitive to global features.

[0003] Currently, under the patents for 3D point cloud segmentation, there is a 3D point cloud segmentation method based on graph convolution that fused direction and distance (application number: CN202110923542.6); an improved LCCP point cloud segmentation method based on Gaussian curvature and local convexity (application number: CN202311431572.0); a 3D point cloud segmentation method based on PointNet++ neural network (application number: CN202111232095.6), etc. Related papers include: (1) AVPhan, M.Le Nguyen, YLHNguyen, and LTBui, "Dgcnn: Aconvolutional neural network over large-scale labeled graphs," Neural Networks, vol. 108, pp. 533-543, 2018. The graph relationship between point cloud neighborhoods was constructed; (2) H.Zhao, L.Jiang, J.Jia, PHTorr, and V.Koltun, "Point transformer," in Proceedings of the IEEE / CVF international conference on computer vision, pp. 16259-16268, 2021. The self-attention mechanism is used to allow the network to dynamically pay attention to the relationship between different points when learning point cloud features;

[0004] Most of the above studies use symmetric functions to process disordered point cloud data when aggregating local features. This makes the model insensitive to the order of input points. Most symmetric functions use the maximum pooling function. However, this simple maximum pooling function selects the maximum value from each local area to generate the feature representation of the area. But this rigid pooling method is limited to using the feature information of a small number of points in the local area. This makes the point cloud feature aggregation insufficient, resulting in poor prediction of fine-grained point cloud labels by point cloud. Therefore, it is urgent to propose an efficient pooling method for 3D point cloud semantic segmentation to understand 3D point cloud scenes. Summary of the invention

[0005] The present invention proposes a soft pooling feature encoding module to construct a point cloud segmentation network. The network is mainly composed of two key modules: an attention pooling module based on edge convolution and a local feature aggregation module based on graph convolution. The attention pooling module based on edge convolution combines the edge convolution attention structure to effectively encode the structural features of each input. In order to promote the integration of global features, the present invention adopts a spatial attention strategy to fuse input information. In addition, the present invention introduces a residual module to alleviate the problems of information similarity and redundancy. The local feature aggregation module combines graph convolution to group points and aggregate local features in the point cloud. Finally, the semantic information of each point cloud is constructed through multi-level feature extraction and asymmetric decoding modules to achieve high-precision point cloud semantic segmentation effect.

[0006] The present invention discloses a 3D point cloud semantic segmentation method based on attention soft pooling, and the 3D point cloud semantic segmentation method based on attention soft pooling comprises the following steps:

[0007] Step 1: Obtain the original three-dimensional point cloud data, map the point cloud data to a high-dimensional feature space through a multi-layer perceptron for feature extraction, and obtain the feature vector of the high-dimensional feature space;

[0008] Step 2: Use a soft pooling feature encoder to perform multi-level feature extraction on the point cloud features for the feature vector of the high-dimensional feature space;

[0009] Step 3: The features output by the soft pooling feature encoder are asymmetrically decoded through the point cloud upsampling module and the multi-layer perceptron module, and the multi-layer perceptron module is used to further map the point cloud features to the preset dimension and then predict the semantic category of each point through the Softmax function.

[0010] Furthermore, in step 2, the following steps are also included:

[0011] Step 2.1, in the soft pooling feature encoder, a sampling point set of the point cloud is obtained by sampling the farthest point, and each point in the sampled set is taken as the center point, and a multi-scale neighborhood of each sampling point is constructed by ball query grouping;

[0012] Step 2.2: Construct a multi-scale neighborhood and transform each sampling point x in each scale i Neighborhood point x j The coordinate information p j and feature information f j Perform context fusion and extract local features of point cloud based on fusion features;

[0013] Step 2.3: Generate aggregated local features through the point cloud edge attention pooling module;

[0014] Step 2.4: Concatenate the aggregated local features at different scales as the output of the soft pooling feature encoder.

[0015] Furthermore, in step 2.1, the n sampling points of the point cloud are obtained by sampling the farthest point, denoted as S = {x1, x2, ..., x n}; For each point x in the sampled S set i is the center point, where i = (0, 1, ..., n), and a multi-scale local neighborhood of each sampling point is constructed by ball query grouping.

[0016] Furthermore, in step 2.2, based on each sampling point x in the multi-scale local neighborhood of each sampling point i Neighborhood point x j The coordinate information p j and feature information f j After context fusion, the features after fusion are:

[0017]

[0018] Among them, f j Used to represent the semantic information of local neighborhood point clouds, Used to characterize the collective distribution of local neighborhood point clouds in space; and establish sampling points x i With neighboring point x j The relationship is defined as:

[0019]

[0020] in, For the sampling point x i Its adjacent point x j Semantic relations, spatial relations, and fusion features Perform aggregation.

[0021] Furthermore, in step 2.3, graph convolution is used to extract local features of the point cloud:

[0022]

[0023] Among them, Conv(·) is a 1×1 convolution operation, and EAP(·) is a point cloud edge attention pooling module operation.

[0024] Furthermore, in step 2.3, the point cloud edge attention pooling module also includes an edge convolution attention structure, a spatial attention module and a residual module;

[0025] The edge convolution attention structure combines the grouped point features with the learnable parameters θ f Compare and finally use maximum pooling to aggregate the results:

[0026]

[0027] Where f and f′ represent the results of edge convolution on input x and x′ respectively, N(·; k) is the k nearest neighbor grouping of the point cloud, ψ(·; θ f ) is to apply edge attention to convolution, MaxPool(·) is the maximum pooling;

[0028] Then, the query and key are matrix-multiplied to generate the attention feature map. The attention feature matrix Y=[y1,y2,…,y n ] T ,y i It is expressed as:

[0029]

[0030] Among them, softmax(·) represents a softmax function, Q(·) represents the matrix of attention Q, K(·) represents the matrix of attention K, and V(·) represents the matrix of attention V.

[0031] Furthermore, in step 2.3, the spatial attention module performs a pooling operation on the input features in the channel dimension to compress the features. The output of the pooling operation is the result of linear mapping after the maximum pooling and average pooling are connected:

[0032] G=β(g([MaxPool(F);AvgPool(F)]))

[0033] Where g(·) is a (2,1) linear mapping, β is the sigmoid function, and the spatial attention module optimizes the input feature F through the obtained spatial attention weight mask G.

[0034] Furthermore, in step 2.3, the pooled input feature x is fused with the complete attention feature using a residual structure, and the output of the residual module is:

[0035]

[0036] Among them, γ(·) is a function containing linear layers, norm layers, and activation layers, and ρ(·) represents a similarity function.

[0037] Furthermore, in step 2.4, the feature aggregation representation f of i local point clouds is i ′, and F′=(f1′,f2′,…,f s ′) as a local point aggregation feature set at a single scale, the output of the soft pooling feature encoding module is the concatenation of F′ in the feature dimension at scale m:

[0038] F out =concat(F′1,F′2,…,F′ m )

[0039] Among them, F out Represents the output of the soft pooling feature aggregation module.

[0040] The beneficial effects achieved by the present invention are:

[0041] This paper proposes an edge convolution attention pooling module, which weights input features through a spatial attention strategy to improve the information loss problem caused by point cloud pooling.

[0042] Based on edge convolution attention pooling, this paper proposes a multi-scale local feature aggregation module based on graph convolution to comprehensively capture point cloud information.

[0043] The present invention proposes a network for efficient point cloud segmentation through the above two key feature aggregation modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is an overall block diagram of a three-dimensional point cloud semantic segmentation based on attention soft pooling according to the present invention.

[0045] Figure 2 This is a block diagram of the local feature aggregation module based on graph convolution in the present invention.

[0046] Figure 3 This is a block diagram of the edge convolution-based attention pooling module of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are exemplary only and do not constitute any limitation to the scope of the present invention. It should be understood by those skilled in the art that the details and forms of the technical solution of the present invention may be modified or replaced without departing from the spirit and scope of the present invention, but these modifications and replacements all fall within the scope of protection of the present invention.

[0048] like Figure 1 As shown, this embodiment provides a 3D point cloud semantic segmentation method based on attention soft pooling, and its main steps are described as follows:

[0049] Step 1: Get the original 3D point cloud. Suppose there are N original 3D point cloud data P = {p1, p2, ..., p N}, where p k Represents the three-dimensional coordinates of the kth point, k = {0,1,…,N}. The point cloud data is mapped to a high-dimensional feature space through a multi-layer perceptron to obtain the feature representation F = {f1,f2,…,f N}, where f k It's point p k The corresponding high-dimensional feature space feature vector.

[0050] Step 2: After the feature vector of the high-dimensional feature space is sent as input to the feature mapping module, the soft pooling feature encoder is used to perform multi-level feature extraction on the point cloud features. Each soft pooling feature encoder consists of a multi-scale local feature aggregation module based on graph convolution and a point cloud edge attention pooling module. Among them, the local feature aggregation module constructs a multi-scale neighborhood and combines graph convolution to learn features of point cloud data at different scales. The point cloud edge attention pooling module weights local features through a spatial attention strategy, integrates global shape features, and performs a pooling operation on the fused local features.

[0051] In this embodiment, two soft pooling feature encoders are used to perform a pooling operation on the fused local features to extract attention soft pooling aggregation features.

[0052] Step 2.1: Figure 2 As shown in the figure, in the soft pooling feature encoder, in order to efficiently aggregate local features to sampling points, the present invention adopts a multi-scale local feature aggregation module based on graph convolution. In order to ensure the diversity of the receptive field, the farthest point sampling (FPS) is used to obtain n sampling points of the point cloud, denoted as S = {x1, x2, ..., x n}. For each point x in the sampled S set iis the center point, where i = (0, 1, ..., n), and a multi-scale local neighborhood of each sampling point is constructed by ball query grouping.

[0053] Step 2.2: After constructing the multi-scale neighborhood, each sampling point x i Neighborhood point x j The coordinate information p j and feature information f j Perform context fusion. The fused features are recorded as:

[0054]

[0055] where f j Used to represent the semantic information of local neighborhood point clouds, It is used to characterize the collective distribution of local neighborhood point clouds in space. For better feature encoding, the present invention establishes the sampling point x i With neighboring point x j The relationship is defined as:

[0056]

[0057] in, For the sampling point x i Its adjacent point x j Semantic relations, spatial relations, and fusion features Perform aggregation.

[0058] Step 2.3, after local point cloud feature extraction, such as Figure 3 As shown, the present invention proposes an edge attention pooling module to aggregate local information.

[0059] Soft pooling feature encoder is based on fusion feature SN ij , after extracting local features of the point cloud using graph convolution, the local point cloud is aggregated through the point cloud edge attention pooling module, and finally the feature aggregation representation f of i local point clouds is obtained i ′, can be expressed as:

[0060]

[0061] Among them, Conv(·) is a graph convolution operation on a point cloud, and EAP(·) is a point cloud edge attention pooling module to be described in the next step of the present invention, so as to better aggregate local information to the corresponding sampling points.

[0062] This module mainly consists of three key parts: edge convolution attention structure, spatial attention module and residual module.

[0063] (1) Edge convolution attention structure:

[0064] 3D point cloud data has inherent permutation invariance, so for input local features Without position encoding, the edge convolution feature mapping is directly performed on the query, key and value. The input feature is integrated into the global information of the point cloud through spatial attention to enhance the feature representation of each point, which is denoted as And construct the matrix Key and Value. For the construction of the Query matrix, the input features are directly pooled in the group point cloud and then aggregated to generate features, which is recorded as The constructed Query, Key, and Value matrices can be expressed as:

[0065]

[0066] Among them, W q ,W k ,W v is a learnable weight matrix. In the edge convolutional attention pool module proposed in the present invention, edge convolution is used to replace the traditional linear mapping of query, key and value. The k nearest neighbor (kNN) grouping of the point cloud is used, denoted as N(·; k), and edge attention is applied to the convolution, denoted as ψ(·; θ f ) to solve the problem of local feature loss when using the traditional multi-layer perceptron (MLP) in the feature mapping process. The grouped point features are compared with the learnable parameters θ f The results are compared and finally aggregated using maximum pooling.

[0067]

[0068] Among them, f and f′ represent the results of edge convolution of input x and x′ respectively. Then the query and key are matrix-multiplied to generate the attention feature map, and the attention feature matrix Y=[y1,y2,…,y n ] T , so y i It can be expressed as:

[0069]

[0070] Among them, softmax(·) represents a softmax function, Q(·) represents the matrix of attention Q, K(·) represents the matrix of attention K, and V(·) represents the matrix of attention V.

[0071] The point cloud edge convolution self-attention pooling module can not only encode the structural features of each input through edge convolution operations, but also be used to represent the local geometric structure of the input and represent the global structure through attention operations.

[0072] (2) Spatial Attention Module:

[0073] In order to associate global features, the present invention proposes a spatial attention optimization strategy, which allows the model to automatically learn the key features in the point cloud data, so that the model can better represent the point cloud data. First, the input features are pooled in the channel dimension to compress the features. The output of the pooling operation is the result of linear mapping after the maximum pooling and average pooling are connected, which can be expressed as:

[0074] G=β(g([MaxPool(F);AvgPool(F)])) (7)

[0075] Where g(·) is a (2,1) linear mapping. β is the sigmoid function. The spatial attention module optimizes the input feature F through the obtained spatial attention weight mask G.

[0076] (3) Residual module:

[0077] The present invention uses an improved residual structure to fuse the pooled input feature x with the complete attention feature. This improved residual structure not only maintains the original channel dimension, but also avoids similar / redundant features. The output of the residual module, that is, the attention pooling module EAP, can be expressed as:

[0078]

[0079] where γ(·) is a function consisting of a linear layer, a norm layer, and an activation layer, and ρ(·) represents a similarity function.

[0080] Step 2.4 Finally, for the feature aggregation representation f of i local point clouds i ′, and F′=(f1′,f2′,…,f s ′) as a local point aggregation feature set at a single scale, the output of the soft pooling feature encoding module is the concatenation of F′ in the feature dimension at scale m, which can be expressed as:

[0081] F out =concat(F′1,F′2,…,′ m ) (9)

[0082] Among them, F out Represents the output of the soft pooling feature aggregation module.

[0083] Step 3: Figure 1As shown in the figure, the outputs of multiple soft pooling feature aggregation modules are connected to construct multi-level features, and the multi-level features are further globally pooled to aggregate the point cloud features into a (1×1024)-dimensional vector representation. The point cloud features are asymmetrically decoded to (2048×128) dimensions by a decoder composed of a point cloud upsampling module and a multi-layer perceptron module, and then the multi-layer perceptron module is used to further map the point cloud to (2048×D seg ) feature dimension and then use the Softmax function to predict the semantic category of each point, where D seg is the number of semantic categories.

[0084] The above are only specific steps of the present invention and do not constitute any limitation to the protection scope of the present invention; any technical solutions formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention; some parts not elaborated in detail in the present invention belong to the known technologies of those skilled in the art.

Claims

1. A 3D point cloud semantic segmentation method based on attention soft pooling, characterized in that: The three-dimensional point cloud semantic segmentation method based on attention soft pooling comprises the following steps: Step 1: Obtain the original three-dimensional point cloud data, map the point cloud data to a high-dimensional feature space through a multi-layer perceptron for feature extraction, and obtain the feature vector of the high-dimensional feature space; Step 2: Use a soft pooling feature encoder to perform multi-level feature extraction on the point cloud features for the feature vector of the high-dimensional feature space; Step 3: The features output by the soft pooling feature encoder are asymmetrically decoded through the point cloud upsampling module and the multi-layer perceptron module, and the multi-layer perceptron module is used to further map the point cloud features to the preset dimensions and then predict the semantic category of each point through the Softmax function.

2. According to claim 1, the three-dimensional point cloud semantic segmentation method based on attention soft pooling is characterized in that: In step 2, the following steps are also included: Step 2.1, in the soft pooling feature encoder, a sampling point set of the point cloud is obtained by sampling the farthest point, and each point in the sampled set is taken as the center point, and a multi-scale neighborhood of each sampling point is constructed by ball query grouping; Step 2.2: Construct a multi-scale neighborhood and transform each sampling point x in each scale i Neighborhood point x j The coordinate information p j and feature information f j Perform context fusion and extract local features of point cloud based on fusion features; Step 2.3: Generate aggregated local features through the point cloud edge attention pooling module; Step 2.4: Concatenate the aggregated local features at different scales as the output of the soft pooling feature encoder.

3. The 3D point cloud semantic segmentation method based on attention soft pooling according to claim 2, characterized in that: In step 2.1, the n sampling points of the point cloud are obtained by sampling the farthest point, denoted as S = {x1, x2, ..., x n }; For each point x in the sampled S set i is the center point, where i = (0, 1, ..., n), and a multi-scale local neighborhood of each sampling point is constructed by ball query grouping.

4. The 3D point cloud semantic segmentation method based on attention soft pooling according to claim 3, characterized in that: In step 2.2, for each sampling point x in the multi-scale local neighborhood of each sampling point i Neighborhood point x j The coordinate information p j and feature information f j After context fusion, the features after fusion are: Among them, f j Used to represent the semantic information of local neighborhood point clouds, Used to characterize the collective distribution of local neighborhood point clouds in space; and establish sampling points x i With neighboring point x j The relationship is defined as: in, For the sampling point x i Its adjacent point x j Semantic relations, spatial relations, and fusion features Perform aggregation.

5. The 3D point cloud semantic segmentation method based on attention soft pooling according to claim 4, characterized in that: In step 2.3, graph convolution is used to extract local features of the point cloud: Among them, Conv(·) is a 1×1 convolution operation, and EAP(·) is a point cloud edge attention pooling module operation.

6. The 3D point cloud semantic segmentation method based on attention soft pooling according to claim 5, characterized in that: In step 2.3, the point cloud edge attention pooling module also includes an edge convolution attention structure, a spatial attention module and a residual module; The edge convolution attention structure combines the grouped point features with the learnable parameters θ f Compare and finally use maximum pooling to aggregate the results: Where f and f′ represent the results of edge convolution on input x and x′ respectively, N(·; k) is the k nearest neighbor grouping of the point cloud, ψ(·; θ f ) is to apply edge attention to convolution, MaxPool(·) is the maximum pooling; Then, the query and key are matrix-multiplied to generate the attention feature map. The attention feature matrix Y=[y1,y2,…,y n ] T ,y i It is expressed as: Among them, softmax(·) represents a softmax function, Q(·) represents the matrix of attention Q, K(·) represents the matrix of attention K, and V(·) represents the matrix of attention V.

7. The 3D point cloud semantic segmentation method based on attention soft pooling according to claim 6, characterized in that: In step 2.3, the spatial attention module performs a pooling operation on the input features in the channel dimension to compress the features. The output of the pooling operation is the result of linear mapping after the maximum pooling and average pooling are connected: G=β(g([MaxPool(F);AvgPool(F)])) Where g(·) is a (2,1) linear mapping, β is the sigmoid function, and the spatial attention module optimizes the input feature F through the obtained spatial attention weight mask G.

8. The 3D point cloud semantic segmentation method based on attention soft pooling according to claim 7, characterized in that: In step 2.3, the pooled input feature x is fused with the complete attention feature using the residual structure, and the output of the residual module is: Among them, γ(·) is a function containing linear layers, norm layers, and activation layers, and ρ(·) represents a similarity function.

9. The 3D point cloud semantic segmentation method based on attention soft pooling according to claim 5, characterized in that: In step 2.4, the feature aggregation representation f of i local point clouds is i ′, and F′=(f1′,f2′,…,f s ′) as a local point aggregation feature set at a single scale, the output of the soft pooling feature encoding module is the concatenation of F′ in the feature dimension at scale m: F out =concat(F′1,F′2,…,F′ m ) Among them, F out Represents the output of the soft pooling feature aggregation module.

Images