Three-dimensional point cloud semantic segmentation method for reconstructing 3D sequence Transform

Abstract

The invention provides a three-dimensional point cloud semantic segmentation method for reconstructing a 3D sequence Transform, and the method comprises the steps: 1, inputting original three-dimensional point cloud data, carrying out the voxelization of the original three-dimensional point cloud data in a three-dimensional space, and dividing the voxelized three-dimensional point cloud space into densely-arranged and non-overlapping 3D windows; step 2, carrying out 3D adaptive average pool operation on voxels in each 3D local window by using three different outputs, and connecting the three pooling outputs in series to serve as input of a multi-head attention mechanism to construct K and V to carry out attention calculation so as to obtain voxel features; step 3, performing point-by-point self-attention feature extraction on the input point cloud data to obtain a point-by-point level feature of a single point; and step 4, adding the point-by-point features and the voxel features to obtain final output. According to the method, the calculation complexity of the model is reduced by compressing the sequence length, the multi-scale feature representation is fused, and the point-level features with finer granularity are further combined, so that the model can efficiently understand the point cloud scene.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional point cloud segmentation and discloses a three-dimensional point cloud semantic segmentation method for reconstructing a 3D sequence Transformer. Background Art

[0002] Point cloud analysis and processing is an important part of the field of computer vision. It enables machines to better understand and interact with their surroundings by representing objects or scenes in three-dimensional space as a large amount of point cloud data. However, unlike regularly arranged two-dimensional images, point cloud data is disordered and irregular. How to efficiently process point cloud data is an important challenge in the current point cloud field.

[0003] Application No. 202111232095.6, invention name "3D point cloud segmentation method based on PointNet++ neural network" discloses a 3D point cloud segmentation method based on PointNet++ neural network, the method comprising: (1) inputting 3D point cloud data to be segmented, wherein represents the coordinate dimension and represents other dimensional features; (2) using the granularity ball k-medoids algorithm in the point cloud to obtain the best cluster center set; (3) traversing each cluster to obtain the representative point cloud of all clusters; using the PointNet neural network to extract the feature information of each cluster to obtain the downsampled point cloud; (4) repeating steps 2 to 3 four times, the network structure is obtained, and downsampled layer by layer; (5) using the interpolation and backtracking method to upsample and restore the original point cloud scale; using the multi-layer perceptron and softmax to judge the probability of belonging to the target or background point by point. This method uses the granularity ball k-medoids algorithm and the process of traversing each cluster seriously increases the model inference speed, which is not suitable for large-scene point cloud segmentation methods. Summary of the invention

[0004] The present invention discloses a 3D point cloud semantic segmentation method for reconstructing a 3D sequence Transformer, and the 3D point cloud semantic segmentation method for reconstructing a 3D sequence Transformer comprises the following steps:

[0005] Step 1, inputting original 3D point cloud data, voxelizing the original 3D point cloud data in 3D space, and dividing the voxelized 3D point cloud space into densely arranged and non-overlapping 3D windows;

[0006] Step 2: Use multiple 3D adaptive average pooling operations with different outputs for each voxel in each 3D local window, and concatenate multiple pooling outputs as the input of the multi-head attention mechanism to construct K and V for attention calculation to obtain the final voxel feature F V ;

[0007] Step 3: Extract point-by-point self-attention features of the input point cloud data to obtain point-by-point features of a single point;

[0008] Step 4: Add the point-by-point features to the voxel features to get the final output.

[0009] Furthermore, in step 1, the following steps are also included:

[0010] Step 1.1, given the input point cloud {(p i ,f i )}, where p i =(x i ,y i ,z i ) represents the coordinates of the point, f i Represents the corresponding eigenvector, and the point cloud is converted into a voxel grid {V u,v,w};

[0011] Step 1.2: Divide the voxel grid after voxelization into a series of local windows of equal size and uniform arrangement in three-dimensional space.

[0012] Furthermore, in step 1.1, the point cloud is converted into a voxel grid {V u,v,w}for:

[0013]

[0014] N u,v,w is the normalization factor, represents the number of points that fall into the voxel grid (u, v, w), I[condition] is a binary indicator function indicating whether the condition is true, r is the voxel resolution, floor(x) means returning the largest integer not greater than x, floor(y) means returning the largest integer not greater than y, floor(z) means returning the largest integer not greater than z, n is the number of points in the point cloud, and f i,c Represented as p i The corresponding average eigenvalue of the Cth channel.

[0015] Furthermore, in step 1.2, the process of creating local windows of equal size is:

[0016] W x,y,z ={(u,v,w)|x≤u <x+M,y≤v<y+M,z≤w<z+M}

[0017] Among them, the local 3D window W x,y,z It consists of a subset of the voxel grid, where the voxel position (u,v,w) satisfies the horizontal position [x|x+M], the vertical position [y|y+M], and the depth position [z|z+M], where M is the window side length.

[0018] Furthermore, in step 2, the following steps are also included:

[0019] Step 2.1: Convert the voxelized features F in the 3D local grid v After normalization, the contextual relationship is captured by 3D adaptive pooling multi-head attention to obtain the output of the multi-head attention. The output of the multi-head attention is added to the input voxel feature and normalized again to obtain the final voxel feature F V Model;

[0020] Step 2.2: In the 3D adaptive pooling multi-head attention capture context relationship, the input voxel feature F with shape N×C is transformed into v Reconstructed into the three-dimensional space of C×D×H×W, the reconstructed feature F v Apply 3D adaptive average pooling operation multiple times to obtain features of different output sizes;

[0021] Step 2.3: Concatenate feature maps of different output sizes and use them as the input of the multi-head attention mechanism to construct K and V for attention calculation.

[0022] Furthermore, in step 2.1, the final voxel feature F V The model is:

[0023] F V = devoxel(MLP(LayerNorm(F att )))

[0024] Among them, F V is the final voxel feature, 3DAPAtten(·) represents the multi-head attention module of 3D adaptive pooling, LayerNorm(·) and MLP(·) represent the feature normalization and feature encoding modules respectively, devoxel(·) is the decomposition voxel module, and F att To generate voxel features.

[0025] Furthermore, in step 2.2, the features of the different output sizes are represented as:

[0026] P1=3DAvgPool1(F v )

[0027] P2=3DAvgPool2(F v )

[0028] …

[0029] P s =3DAvgPool s (F v )

[0030] Among them, 3DAvgPool s (·) represents the 3D adaptive pooling function, P1,P2,…,P s Represents the pooled feature map output by 3D adaptive average pooling, and n represents the number of 3D adaptive pooling layers;

[0031] In step 2.3, features of different output sizes are concatenated to obtain:

[0032] P=LayerNorm(Concat(P1,P2,…,P s ))

[0033] Among them, P represents the sequence length after the features of different output sizes are concatenated, LayerNorm(·) is the normalization layer, and Concat(·) is used to concatenate multiple adaptive pooling features.

[0034] Furthermore, in step 2.3, for the input feature F v And its feature sequence after multi-scale pooling in multiple dimensions, the constructed Q (Query), K (Key), V (Valur) can be expressed as:

[0035] (Q,K,V)=(F v W q ,PW k ,PW v )

[0036] Among them, W q , W k , W v The weight matrices in the self-attention mechanism of Q, K, and V are used to convert the input data into Q, K, and V vectors, and perform self-attention calculations to generate voxel features F. att for:

[0037]

[0038] Among them, Softmax(·) means applying the Softmax function to each row in the similarity matrix to convert the similarity score into an attention weight.

[0039] Furthermore, in step 3, the point-wise features are expressed as:

[0040] F P =MLP(Softmax(Q T K)V)+F

[0041] in, Represents the global features aggregated from the entire point cloud context, Q, K, V is generated from the input point features F.

[0042] Going one step further, in step 4, the final output is:

[0043] F′=F V +F P

[0044] in,

[0045] The beneficial effects achieved by the present invention are:

[0046] This paper proposes a 3D point cloud semantic segmentation method that reconstructs a 3D sequence Transformer. This method greatly reduces the sequence length in multi-head self-attention calculation through 3D adaptive pooling attention, and establishes richer semantic information through multi-scale feature fusion. It speeds up the network reasoning speed and improves the efficiency of point cloud semantic segmentation.

[0047] The present invention combines point-based features with voxel-based features to extract point cloud features, taking into full account the advantages and disadvantages of the two methods, so that the two methods complement each other to achieve efficient and accurate point cloud processing. ;

[0048] The present invention proposes a network for efficient point cloud classification and segmentation by combining a 3D adaptive pooling Transformer module with a point attention module. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is an overall block diagram of a three-dimensional point cloud semantic segmentation method for reconstructing a 3D sequence Transformer according to the present invention.

[0050] Figure 2 Schematic diagram of the 3D adaptive pooling process of the present invention.

[0051] Figure 3 This is a diagram of the experimental effect of the present invention on the S3DIS dataset. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are exemplary only and do not constitute any limitation to the scope of the present invention. It should be understood by those skilled in the art that the details and forms of the technical solution of the present invention may be modified or replaced without departing from the spirit and scope of the present invention, but these modifications and replacements all fall within the scope of protection of the present invention.

[0053] like Figure 1As shown, this embodiment provides a 3D point cloud semantic segmentation for reconstructing a 3D sequence Transformer.

[0054] The main steps are described as follows:

[0055] Step 1: Input the original 3D point cloud data, voxelize the original 3D point cloud data in the 3D space, and divide the voxelized 3D point cloud space into densely arranged and non-overlapping 3D windows to obtain a series of 3D local windows.

[0056] The specific steps include:

[0057] Step 1.1: Figure 1 As shown in the 3D adaptive Transformer module in, given an input point cloud {)p i ,f i )}, where p i =(x i ,y i ,z i ) represents the coordinates of the point, f i Represents the corresponding eigenvector. The present invention discretizes the coordinates and converts them into voxel grids {V u,v,w}.

[0058] The conversion process is defined as:

[0059]

[0060] Among them, N u,v,w is the normalization factor, represents the number of points that fall into the voxel grid (u, v, w), I[condition] is a binary indicator function indicating whether the condition is true, r is the voxel resolution, floor(x) means returning the largest integer not greater than x, floor(y) means returning the largest integer not greater than y, floor(z) means returning the largest integer not greater than z, n is the number of points in the point cloud, and f i,c Represented as p i The corresponding average eigenvalue of the Cth channel.

[0061] Step 1.2: Divide the voxel grid after voxelization into a series of local windows of equal size and uniform arrangement in three-dimensional space. The present invention divides the voxel grid evenly in each dimension to create local windows of equal size. The process can be defined as:

[0062] W x,y,z ={(u,v,w)|x≤u <x+M,y≤v<y+M,z≤w<z+M} (2)

[0063] Among them, the local 3D window W x,y,zIt consists of a subset of voxel grids, where the voxel position (u, v, w) satisfies the horizontal position [x|x+M], the vertical position [y|y+M], and the depth position [z|z+M]. By adjusting the value of M, a series of local 3D windows of equal size and uniform arrangement can be obtained. In the present invention, the value of M is set to 12.

[0064] Step 2: The present invention uses multiple 3D adaptive average pooling operations with different outputs on the voxels in each 3D local window, and then connects multiple pooling outputs in series as the input of the multi-head attention mechanism to construct K, V for attention calculation, thereby obtaining the final voxel feature F V .

[0065] The specific steps include:

[0066] Step 2.1

[0067] like Figure 1 As shown in , the voxelized features F in the 3D local grid are v After normalization, the contextual relationship is captured by multi-head attention with 3D adaptive pooling to obtain the output of multi-head attention. The output of multi-head attention is added to the input voxelized features and normalized again, and finally feature projection is performed, denoted as F V The above process can be expressed as:

[0068] F V = devoxel(MLP(LayerNorm(F att ))) (3)

[0069] Among them, 3DAPAtten(·) represents the multi-head attention module of 3D adaptive pooling. LayerNorm(·) and MLP(·) represent feature normalization and feature encoding modules respectively, and devoxel(·) is the de-voxel module.

[0070] Step 2.2: Figure 2 As shown in , for 3D adaptive pooling attention, the present invention first transforms the input voxel feature F of shape N×C into v Reconstruct into a three-dimensional space of C×D×H×W. Then, after the reconstruction, the feature F v The 3D adaptive average pooling operation is applied multiple times to obtain features of different output sizes, which can be expressed as:

[0071]

[0072] Among them, 3DAvgPool s (·) represents the 3D adaptive pooling function, P1,P2,…,P srepresents the pooled feature map output by 3D adaptive average pooling, and s represents the number of 3D adaptive pooling layers.

[0073] Step 2.3, connect these pooled feature maps in series and use them as the input of the multi-head attention mechanism to construct K, V for attention calculation, thereby obtaining voxel features;

[0074] After connecting these pooled feature maps in series, we get:

[0075] P=LayerNorm(Concat(P1,P2,…,P s )) (5)

[0076] Where P represents the sequence length of the sum of features obtained after all 3D adaptive average pooling. For voxels, the obtained P is the input voxel F v The feature sequence after multi-scale pooling in multiple dimensions will be smaller than F v The length of the sequence.

[0077] For the input feature F v , the constructed Q (Query), K (Key), V (Valur) can be expressed as:

[0078] (Q,K,V)=(F v W q ,PW k ,PW v ) (6)

[0079] Among them, W q , W k , W v are the weight matrices in the self-attention mechanism of Q, K, and V, respectively, which are used to convert the input data into Q, K, and V vectors so as to perform self-attention calculation to generate voxel features, which can be expressed as:

[0080]

[0081] Where Softmax(·) means applying the Softmax function to each row in the similarity matrix to convert the similarity scores into attention weights. The channel size of K is taken as d K ,and then As an approximate normalization to make the model easier to train.

[0082] Step 3: Perform point-by-point self-attention feature extraction on the input point cloud data to obtain the point-level features of a single point.

[0083] Since the feature learning network of voxels is based on fusing coarse-grained neighborhood information in a local window, this will cause the model to lack a global receptive field. In order to capture long-range dependencies in the low-resolution voxel branch, the input point cloud data is subjected to point-by-point self-attention feature extraction to obtain the point-level features of a single point. Figure 1 As shown in the midpoint attention module, the present invention uses self-attention to perform global context aggregation on the entire point cloud, that is, to calculate the attention between each point and other points, so as to extract the point-level features of a single point. The point-level self-attention formula can be expressed as:

[0084] F P =MLP(Softmax(Q T K)V)+F (8)

[0085] in, Represents the global feature of the entire point cloud context aggregation. Q, K, V are generated by the input point feature F.

[0086] Step 4: Add the point-by-point features and voxel features to get the final output. This can fully utilize the advantages of both methods, allowing the model to efficiently and fully understand the point cloud scene.

[0087] like Figure 1 As shown in , the point-based and voxel-based feature aggregation branches are added together to obtain the final output F′, which can be expressed as:

[0088] F′=F V +F P (9)

[0089] in High-resolution individual point information is used to supplement the coarse-grained voxel-based information for feature representation. This can fully utilize the advantages of both methods, allowing the model to fully understand the point cloud scene.

[0090] Figure 3 A visualized semantic segmentation map of the present invention in a challenging indoor scene in S3DIS dataset region 5 is given in FIG. The point cloud segmentation model of the present invention is closer to the true label and has a faster model inference speed than most other models.

[0091] The above are only specific steps of the present invention and do not constitute any limitation to the protection scope of the present invention; any technical solutions formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention; some parts not elaborated in detail in the present invention belong to the known technologies of those skilled in the art.

Claims

1. A 3D point cloud semantic segmentation method based on a reconstructed 3D sequence Transformer, characterized in that: The 3D point cloud semantic segmentation method of reconstructing the 3D sequence Transformer comprises the following steps: Step 1, inputting original 3D point cloud data, voxelizing the original 3D point cloud data in 3D space, and dividing the voxelized 3D point cloud space into densely arranged and non-overlapping 3D windows; Step 2: Use multiple 3D adaptive average pooling operations with different outputs for each voxel in each 3D local window, and concatenate multiple pooling outputs as the input of the multi-head attention mechanism to construct K and V for attention calculation to obtain the final voxel feature F V ; Step 3: Extract point-by-point self-attention features of the input point cloud data to obtain point-by-point features of a single point; Step 4: Add the point-by-point features to the voxel features to get the final output.

2. The 3D point cloud semantic segmentation method of reconstructing the 3D sequence Transformer according to claim 1, characterized in that: In step 1, the following steps are also included: Step 1.1, given the input point cloud {(p i ,f i )}, where p i =(x i ,y i ,z i ) represents the coordinates of the point, f i Represents the corresponding eigenvector, and the point cloud is converted into a voxel grid {V u,v,w }; Step 1.2: Divide the voxel grid after voxelization into a series of local windows of equal size and uniform arrangement in three-dimensional space.

3. The 3D point cloud semantic segmentation method of reconstructing the 3D sequence Transformer according to claim 2, characterized in that: In step 1.1, the point cloud is converted into a voxel grid {V u,v,w }for: N u,v,w is the normalization factor, represents the number of points that fall into the voxel grid (u, v, w), I[condition] is a binary indicator function indicating whether the condition is true, r is the voxel resolution, floor(x) means returning the largest integer not greater than x, floor(y) means returning the largest integer not greater than y, floor(z) means returning the largest integer not greater than z, n is the number of points in the point cloud, and f i,c Represented as p i The corresponding average eigenvalue of the Cth channel.

4. The 3D point cloud semantic segmentation method of reconstructing the 3D sequence Transformer according to claim 2, characterized in that: In step 1.2, the process of creating local windows of equal size is: W x,y,2 ={(u,v,w)|x≤u<x+M,y≤v<y+M,z≤w<z+M} Among them, the local 3D window W x,y,z It consists of a subset of the voxel grid, where the voxel position (u,v,w) satisfies the horizontal position [x|x+M], the vertical position [y|y+M], and the depth position [z|z+M], where M is the window side length.

5. The 3D point cloud semantic segmentation method of reconstructing the 3D sequence Transformer according to claim 1, characterized in that: In step 2, the following steps are also included: Step 2.1: Convert the voxelized features F in the 3D local grid v After normalization, the contextual relationship is captured by 3D adaptive pooling multi-head attention to obtain the output of the multi-head attention. The output of the multi-head attention is added to the input voxel feature and normalized again to obtain the final voxel feature F V Model; Step 2.2: In the 3D adaptive pooling multi-head attention capture context relationship, the input voxel feature F with shape N×C is transformed into v Reconstructed into the three-dimensional space of C×D×H×W, the reconstructed feature F v Apply 3D adaptive average pooling operation multiple times to obtain features of different output sizes; Step 2.3: Concatenate feature maps of different output sizes and use them as the input of the multi-head attention mechanism to construct K and V for attention calculation.

6. The 3D point cloud semantic segmentation method of reconstructing the 3D sequence Transformer according to claim 5, characterized in that: In step 2.1, the final voxel feature F V The model is: F V =devoxel(MLP(LayerNorm(F att ))) Among them, F V is the final voxel feature, 3DAPAtten(·) represents the multi-head attention module of 3D adaptive pooling, LayerNorm(·) and MLP(·) represent the feature normalization and feature encoding modules respectively, devoxel(·) is the decomposition voxel module, and F att To generate voxel features.

7. The 3D point cloud semantic segmentation method of reconstructing a 3D sequence Transformer according to claim 6, characterized in that: In step 2.2, the features of different output sizes are represented as: P1=3DAvgPool1(F v ) P2=3DAvgPool2(F v ) … P s =3DAvgPool s (F v ) Among them, 3DAvgPool s (·) represents the 3D adaptive pooling function, P1,P2,…,P s Represents the pooled feature map output by 3D adaptive average pooling, and n represents the number of 3D adaptive pooling layers; In step 2.3, features of different output sizes are concatenated to obtain: P=LayerNorm(Concat(P1,P2,…,P s )) Among them, P represents the sequence length after the features of different output sizes are concatenated, LayerNorm(·) is the normalization layer, and Concat(·) is used to concatenate multiple adaptive pooling features.

8. The 3D point cloud semantic segmentation method of reconstructing a 3D sequence Transformer according to claim 7, characterized in that: In step 2.3, for the input feature F v And its feature sequence after multi-scale pooling in multiple dimensions, the constructed Q (Query), K (Key), V (Valur) can be expressed as: (Q,K,V)=(F v W q ,PW k ,PW v ) Among them, W q , W k , W v The weight matrices in the self-attention mechanism of Q, K, and V are used to convert the input data into Q, K, and V vectors, and perform self-attention calculations to generate voxel features F. att for: Among them, Softmax(·) means applying the Softmax function to each row in the similarity matrix to convert the similarity score into an attention weight.

9. The 3D point cloud semantic segmentation method of reconstructing a 3D sequence Transformer according to claim 1, characterized in that: In step 3, the point-wise features are represented as: F P =MLP(Softmax(Q T K)V)+F in, Represents the global features aggregated from the entire point cloud context, Q, K, V is generated from the input point features F.

10. The 3D point cloud semantic segmentation method of reconstructing a 3D sequence Transformer according to claim 8, characterized in that: In step 4, the final output is: F ′ =F V +F P in,

Images