Large-scale point cloud semantic segmentation method based on double-branch enhancement and multi-scale fusion

Abstract

The invention discloses a large-scale point cloud semantic segmentation method based on double-branch feature enhancement and multi-scale fusion, and the method comprises the steps: designing a double-branch enhancement aggregation module which focuses on the coding of the attribute information and semantic features of a point cloud in a neighborhood, generates offset features according to bilateral features, enables the offset features to be embedded into corresponding original features, and achieves the segmentation of the semantic features of the point cloud in the neighborhood; therefore, the local sensing capability of the model is improved. Meanwhile, in order to effectively weaken semantic gaps among features under different scales, a multi-scale feature fusion module is additionally designed, global features including all coding layer outputs are obtained by fusing adjacent features of different scales, the global context sensing ability of the model is improved, and upper and bottom coding outputs are fused to improve the feature recognition degree.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional point cloud semantic segmentation, and specifically relates to a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion. Background Art

[0002] In recent years, with the continuous development and maturity of 3D laser scanning technology, 3D perception of outdoor scenes based on discrete point cloud representation has become an important research direction in the field of computer vision. Efficient semantic segmentation of large-scale point cloud scenes has shown broad application prospects in fields such as autonomous driving, virtual reality and smart cities. In smart cities, it helps urban planning and management, infrastructure monitoring and maintenance, traffic flow analysis and management, and environmental protection and resource management; in the field of virtual reality, it can enhance the realism and immersion of virtual environments, achieve efficient scene reconstruction and editing, and enhance the naturalness and accuracy of human-computer interaction; in the field of autonomous driving, it can improve environmental perception capabilities, enhance the reliability of decision-making and planning, and improve the robustness and adaptability of the system to different environments.

[0003] Point cloud semantic segmentation refers to the classification of points in point cloud data through algorithms. Semantic categories are assigned to each point based on object features. In recent years, due to the improvement of computer hardware performance and the emergence of large-scale data sets, deep learning has gradually become a research hotspot in the field of point cloud semantic segmentation. RandLA-Net is a deep learning point cloud semantic segmentation method based on U-Net. This method first uses a random point sampling method to reduce the size of the point cloud, significantly reducing computational and memory overhead. Local spatial encoding is designed to encode the relative position of each point and its neighboring points, and the geometric information of the point is converted into a feature vector through MLP to help the network understand the local geometric structure. However, the color information of the point cloud is ignored in the encoding process, and there is a lack of connection between the point cloud attribute information and its semantic features. In addition, due to the wide variety of objects in large-scale scenes and the large scale differences between objects of different categories, it is difficult to complete the semantic segmentation of large-scale scenes by simply relying on resolution to perform simple jump connections to fuse features. Summary of the invention

[0004] The purpose of the present invention is to provide a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion, which solves the problem of feature loss caused by downsampling operations and the problem of excessive differences between multi-scale features.

[0005] The technical solution adopted by the present invention is a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion, which is specifically implemented in the following steps:

[0006] Step 1, data preprocessing: input point cloud data, downsample the data using voxel grid filtering, and then construct a KD tree;

[0007] Step 2, initial feature extraction: send the preprocessed data into the multi-layer perceptron to obtain the initial feature matrix F;

[0008] Step 3, dual-branch feature encoding: encode the semantic feature F and the point cloud attribute feature I by K-nearest neighbor information, and cross-enhance the relative features in the semantic feature encoding and attribute encoding to obtain the offset features F and I, respectively. offset and I offset ;

[0009] Step 4, feature enhancement: Send the feature matrices in the two feature branches into the multi-layer perceptron to obtain the enhanced feature F E ,I E ;

[0010] Step 5, feature aggregation: calculate the adaptive weight W, then use the softmax function to normalize the feature dimension, and finally sum the weighted features in the feature dimension to obtain the aggregated feature F agg ;

[0011] Step 6, build the encoder: repeat steps 3-5, use a skip connection between the two output features, use the above content as a dilated residual module, use random sampling after each dilated residual block, and repeat multiple times to obtain an encoder containing features of different resolutions;

[0012] Step 7, multi-scale feature fusion: Combine the output features of adjacent layers in the encoder to fuse the semantic information in the upper layer features with the details in the lower layer features;

[0013] Step 8: Build a classifier to complete the semantic segmentation of the point cloud.

[0014] The present invention is also characterized in that:

[0015] Step 1 is as follows:

[0016] Step 1.1, input 3D point cloud data P = {p1, p2, ..., p n}∈R F , F represents the dimension information, and each point is represented as

[0017] Step 1.2, process the point cloud data P using a voxel filtering algorithm and save the result;

[0018] Step 1.3, construct a KD tree for the filtered data and save the KD tree and projection file;

[0019] Step 1.4, encapsulate the data obtained in steps 1.2 and 1.3 into a complete data set to complete the preprocessing.

[0020] Step 3 is as follows:

[0021] Step 3.1: Use the k-nearest neighbor algorithm to obtain the nearest neighbor feature I K ∈R k×6 , the center point attribute information, neighbor attribute information and relative attribute information are integrated, as shown in formula (1),

[0022]

[0023] In formula (1), I T is the center point attribute information, I T -I K is the relative attribute information, Represents feature splicing;

[0024] Step 3.2: For semantic features, the same encoding method as attribute features is used to obtain the semantic features F of neighboring points using the k-nearest neighbor algorithm. K ∈R k×c And integrate three encoding features, as shown in formula (2):

[0025]

[0026] In formula (2), F T is the center point attribute information, F T -F K is a relative semantic feature, Represents feature splicing;

[0027] Step 3.3, calculate the offset feature to enhance the association between attribute information and semantic features. The specific calculation of the offset feature is shown in formula (3) and (4).

[0028] I offset =(I T -I K )+M(F T -F K ) (3)

[0029] F offset =(F T -F K )+M(I T -I K ) (4)

[0030] In formula (3), M represents MLP.

[0031] Step 4 is as follows:

[0032] Step 4.1, using the offset feature I offset 、F offset For local features I C 、F CPerform cross enhancement and complete feature fusion, as shown in formula (5) and (6).

[0033]

[0034] Step 4.2, fuse the two enhanced features to obtain the fused feature, as shown in formula (7):

[0035]

[0036] Step 5 is as follows:

[0037] Step 5.1, calculate the adaptive weight, as shown in formula (8):

[0038] Weight = δ(M(F fusion )) (8)

[0039] In formula (8), δ is the softmax function;

[0040] Step 5.2: Use the softmax function to normalize the feature dimension, and finally sum the weighted features in the feature dimension to obtain the aggregate feature F. agg , as shown in formula (9),

[0041]

[0042] In formula (9), ⊙ is the matrix dot product.

[0043] Step 6 is as follows:

[0044] Step 6.1: Name the structure formed by steps 3-5 as a dual-branch enhanced aggregation (DEA) module. In each encoder, the point cloud attribute information and semantic features are fed into the above module to aggregate the point cloud features F in a smaller range. S , as shown in formula (10),

[0045] F S =DEA(I,F) (10)

[0046] In formula (10), DEA represents the double-branched enhanced aggregation module;

[0047] Step 6.2, F S After the second dual-branch enhanced aggregation module, the features F in a larger neighborhood are aggregated. L To balance the receptive field and local details, as shown in formula (11),

[0048] F L =DEA(I,F S ) (11)

[0049] Step 6.3 fuses the features of the two different receptive fields. The specific calculation is shown in formula (12):

[0050]

[0051] Step 7 is as follows:

[0052] Step 7.1: Take the second-layer feature E2 and the fourth-layer feature E4 in the encoder as the fusion starting point, combine the upper and lower encoder features, and use MLP to fuse to obtain F1 H 、F1 L , as shown in formula (13) and (14),

[0053]

[0054] In formula (13), DS and US represent downsampling and upsampling operations respectively, and Conv() represents the convolution operation;

[0055] Step 7.2: Combine the encoder intermediate layer features E3 and F1 H and F1 L Fusion, to obtain the global fusion feature F2 that combines features of different scales G In addition, the encoder top-level feature E1 and bottom-level feature E5 are respectively compared with F1 H 、F1 L Fusion To ensure the distinction between features of different scales, as shown in formulas (15), (16), and (17),

[0056]

[0057]

[0058] Step 7.3, in the final stage of fusion, the three fusion features are processed symmetrically, and F2 G Decompose and The specific integration is shown in the following formulas (18) and (19):

[0059]

[0060] In the decoding stage, the fused features are decomposed and aligned with the decoder feature dimensions. In addition, long skip connections are used to integrate the encoder and decoder features to ensure the stability of the model.

[0061] The final decoded features are passed through two multi-layer perceptrons and a Dropout layer, and the final classification result is obtained through Softmax.

[0062] The beneficial effects of the present invention are:

[0063] The present invention is a large-scale point cloud semantic segmentation method based on dual-branch feature enhancement and multi-scale fusion. A dual-branch enhancement aggregation module is designed. The module focuses on the encoding of point cloud attribute information and semantic features in the neighborhood, generates offset features based on bilateral features, and embeds the offset features into the corresponding original features, thereby improving the local perception ability of the model. At the same time, in order to effectively reduce the semantic gap between features at different scales, a multi-scale feature fusion module is also designed. By fusing adjacent features of different scales, a global feature containing the output of all coding layers is obtained, thereby improving the global context perception ability of the model and fusing the upper and lower layer coding outputs to improve feature recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is the overall network structure diagram of the large-scale point cloud semantic segmentation method based on dual-branch feature enhancement and multi-scale fusion;

[0065] Figure 2 It is a structural diagram of a dual-branch enhanced aggregation module;

[0066] Figure 3 It is a structural diagram of the multi-scale feature fusion module;

[0067] Figure 4 This is a visualization of the large-scale point cloud semantic segmentation method based on dual-branch feature enhancement and multi-scale fusion on the SensatUrban dataset. DETAILED DESCRIPTION

[0068] The present invention is described in detail below in conjunction with specific implementation modes.

[0069] Example 1

[0070] This embodiment provides a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion. Figure 1 As shown, the specific steps are as follows:

[0071] Step 1, data preprocessing: input point cloud data, use voxel grid filtering to downsample the data, and then build a KD tree to facilitate subsequent access to point cloud data;

[0072] Step 1.1, input 3D point cloud data P = {p1, p2, ..., p n}∈R F , F represents the dimension information, and each point is represented as

[0073] Step 1.2, process the point cloud data P using a voxel filtering algorithm and save the result;

[0074] Step 1.3, construct a KD tree for the filtered data and save the KD tree and projection file;

[0075] Step 1.4, encapsulate the data obtained in steps 1.2 and 1.3 into a complete data set to complete preprocessing;

[0076] Step 2, initial feature extraction: send the preprocessed data into the multi-layer perceptron to obtain the initial feature matrix F;

[0077] Step 3, dual-branch feature encoding: encode the semantic feature F and the point cloud attribute feature I by K-nearest neighbor information, and cross-enhance the relative features in the semantic feature encoding and attribute encoding to obtain the offset features F and I, respectively. offset and I offset ;

[0078] Step 4, feature enhancement: Send the feature matrices in the two feature branches into the multi-layer perceptron to obtain the enhanced feature F E ,I E ;

[0079] Step 5, feature aggregation: Use the multi-layer perceptron to calculate the adaptive weight W, then use the softmax function to normalize the feature dimension, and finally sum the weighted features in the feature dimension to obtain the aggregated feature F agg ;

[0080] Step 6, build the encoder: repeat steps 3-5, use a skip connection between the two output features, use the above content as a dilated residual module, use random sampling after each dilated residual block, repeat 5 times to obtain an encoder containing features of different resolutions;

[0081] Step 7, multi-scale feature fusion: Combine the output features of adjacent layers in the encoder to fuse the semantic information in the upper layer features with the details in the lower layer features;

[0082] Step 8: Build a classifier: pass the final decoded features through two multi-layer perceptrons and a Dropout layer, and obtain the final classification result through Softmax to complete the semantic segmentation of the point cloud.

[0083] Example 2

[0084] This embodiment provides a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion. On the basis of Embodiment 1, Figure 2 As shown, step 3 is specifically as follows:

[0085] Step 3.1: Use the k-nearest neighbor algorithm to obtain the nearest neighbor feature I K ∈R k×6 , the center point attribute information, neighbor attribute information and relative attribute information are integrated, as shown in formula (1),

[0086]

[0087] In formula (1), I T is the center point attribute information, I T -I K is the relative attribute information, Represents feature splicing;

[0088] Step 3.2: For semantic features, the same encoding method as attribute features is used to obtain the semantic features F of neighboring points using the k-nearest neighbor algorithm. K ∈R k×c And integrate three encoding features, as shown in formula (2):

[0089]

[0090] In formula (2), F T is the center point attribute information, F T -F K is a relative semantic feature, Represents feature splicing;

[0091] Step 3.3, calculate the offset feature, enhance the association between attribute information and semantic features, and enrich the local context. The specific calculation of the offset feature is shown in formula (3) (4):

[0092] I offset =(I T -I K )+M(F T -F K ) (twenty two)

[0093] F offset =(F T -F K )+M(I T -I K ) (twenty three)

[0094] In formula (3), M represents MLP.

[0095] Example 3

[0096] This embodiment provides a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion. On the basis of Embodiment 2, step 4 is specifically as follows:

[0097] Step 4.1, using the offset feature I offset 、F offset For local features I C 、F C Perform cross enhancement and complete feature fusion, as shown in formula (5) and (6).

[0098]

[0099] Step 4.2, fuse the two enhanced features to obtain the fused feature, as shown in formula (7):

[0100]

[0101] Example 4

[0102] This embodiment provides a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion. On the basis of Embodiment 3, step 5 is specifically as follows:

[0103] Step 5.1, calculate the adaptive weight, as shown in formula (8):

[0104] Weight = δ(M(F fusion )) (27)

[0105] In formula (8), δ is the softmax function;

[0106] Step 5.2: Use the softmax function to normalize the feature dimension, and finally sum the weighted features in the feature dimension to obtain the aggregate feature F. agg , as shown in formula (9),

[0107]

[0108] In formula (9), ⊙ is the matrix dot product.

[0109] Example 5

[0110] This embodiment provides a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion. On the basis of Embodiment 4, step 6 is specifically as follows:

[0111] Step 6.1: Name the structure formed by steps 3-5 as a dual-branch enhanced aggregation (DEA) module. In each encoder, the point cloud attribute information and semantic features are fed into the above module to aggregate the point cloud features F in a smaller range. S , as shown in formula (10),

[0112] F S =DEA(I,F) (29)

[0113] In formula (10), DEA represents the double-branched enhanced aggregation module;

[0114] Step 6.2, F S After the second dual-branch enhanced aggregation module, the features F in a larger neighborhood are aggregated. L To balance the receptive field and local details, as shown in formula (11),

[0115] F L =DEA(I,F S ) (30)

[0116] Step 6.3 fuses the features of the two different receptive fields. The specific calculation is shown in formula (12):

[0117]

[0118] Example 6

[0119] This embodiment provides a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion. On the basis of Embodiment 1, Figure 3 As shown, step 7 is as follows:

[0120] Step 7.1: Take the second-layer feature E2 and the fourth-layer feature E4 in the encoder as the fusion starting point, combine the upper and lower encoder features, and use MLP to fuse to obtain F1 H 、F1 L , as shown in formula (13) and (14),

[0121]

[0122] In formula (13), DS and US represent downsampling and upsampling operations respectively, and Conv() represents the convolution operation;

[0123] Step 7.2: Combine the encoder intermediate layer features E3 and F1 H and F1 L Fusion, to obtain the global fusion feature F2 that combines features of different scales G In addition, the encoder top-level feature E1 and bottom-level feature E5 are respectively compared with F1 H 、F1 L Fusion To ensure the distinction between features of different scales, as shown in formulas (15), (16), and (17),

[0124]

[0125] Step 7.3, in the final stage of fusion, similar operations are used to symmetrically process the three fused features, and F2 G Decompose and The specific integration is shown in the following formulas (18) and (19):

[0126]

[0127] In the decoding stage, the fused features are decomposed and aligned with the decoder feature dimensions. In addition, long skip connections are used to integrate the encoder and decoder features to ensure the stability of the model.

[0128] Table 1 shows the quantitative results of this embodiment on the SensatUrban dataset. Figure 4 The visualization effect of this embodiment on the SensatUrban dataset is shown. This embodiment is based on the large-scale point cloud semantic segmentation method of dual-branch enhancement and multi-scale fusion. A large number of experimental verifications and analyses are carried out on the SensatUrban scene dataset. The results show that the average intersection-over-union ratio of the proposed method reaches 61.6%.

[0129] Table 1

[0130]

[0131] From the above content, it can be seen that the present invention is based on a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion. On the basis of the RandLA-Net method, by introducing a dual-branch enhancement aggregation module and a multi-scale feature fusion module, a large-scale point cloud semantic segmentation method based on dual-branch enhancement and multi-scale fusion is proposed, which solves the problem of feature loss caused by the downsampling operation used in the existing method and the problem of excessive difference between multi-scale features, thereby improving the model's perception of details and local features and improving the accuracy of semantic segmentation.

Claims

1. A large-scale point cloud semantic segmentation method based on dual-branch feature enhancement and multi-scale fusion, characterized in that: The specific steps are as follows: Step 1, data preprocessing: input point cloud data, downsample the data using voxel grid filtering, and then construct a KD tree; Step 2, initial feature extraction: send the preprocessed data into the multi-layer perceptron to obtain the initial feature matrix F; Step 3, dual-branch feature encoding: encode the semantic feature F and the point cloud attribute feature I by K-nearest neighbor information, and cross-enhance the relative features in the semantic feature encoding and attribute encoding to obtain the offset features F and I, respectively. offset and I offset ; Step 4, feature enhancement: Send the feature matrices in the two feature branches into the multi-layer perceptron to obtain the enhanced feature F E ,I E ; Step 5, feature aggregation: calculate the adaptive weight W, then use the softmax function to normalize the feature dimension, and finally sum the weighted features in the feature dimension to obtain the aggregated feature F agg ; Step 6, build the encoder: repeat steps 3-5, use a skip connection between the two output features, use the above content as a dilated residual module, use random sampling after each dilated residual block, and repeat multiple times to obtain an encoder containing features of different resolutions; Step 7, multi-scale feature fusion: Combine the output features of adjacent layers in the encoder to fuse the semantic information in the upper layer features with the details in the lower layer features; Step 8: Build a classifier to complete the semantic segmentation of the point cloud.

2. The large-scale point cloud semantic segmentation method based on dual-branch feature enhancement and multi-scale fusion according to claim 1 is characterized in that: The step 1 is specifically as follows: Step 1.1, input 3D point cloud data P = {p1, p2, ..., p n }∈R F , F represents the dimension information, and each point is represented as Step 1.2, process the point cloud data P using a voxel filtering algorithm and save the result; Step 1.3, construct a KD tree for the filtered data and save the KD tree and projection file; Step 1.4, encapsulate the data obtained in steps 1.2 and 1.3 into a complete data set to complete the preprocessing.

3. The repetitive single object extraction method based on center point detection and clustering according to claim 2 is characterized in that: The step 3 is specifically as follows: Step 3.1: Use the k-nearest neighbor algorithm to obtain the nearest neighbor feature I K ∈R k×6 , the center point attribute information, neighbor attribute information and relative attribute information are integrated, as shown in formula (1), In formula (1), I T is the center point attribute information, I T -I K is the relative attribute information, Represents feature splicing; Step 3.2: For semantic features, the same encoding method as attribute features is used to obtain the semantic features F of neighboring points using the k-nearest neighbor algorithm. K ∈R k×c And integrate three encoding features, as shown in formula (2): In formula (2), F T is the center point attribute information, F T -F K is a relative semantic feature, Represents feature splicing; Step 3.3, calculate the offset feature to enhance the association between attribute information and semantic features. The specific calculation of the offset feature is shown in formula (3) and (4). I offset =(I T -I K )+M(F T -F K ) (3) F offset =(F T -F K )+M(I T -I K ) (4) In formula (3), M represents MLP.

4. The repetitive single object extraction method based on center point detection and clustering according to claim 3 is characterized in that: The step 4 is specifically as follows: Step 4.1, using the offset feature I offset 、F offset For local features I C 、F C Perform cross enhancement and complete feature fusion, as shown in formula (5) and (6). Step 4.2, fuse the two enhanced features to obtain the fused feature, as shown in formula (7):

5. The repetitive single object extraction method based on center point detection and clustering according to claim 4 is characterized in that: The step 5 is specifically as follows: Step 5.1, calculate the adaptive weight, as shown in formula (8): Weight=δ(M(F fusion )) (8) In formula (8), δ is the softmax function; Step 5.2: Use the softmax function to normalize the feature dimension, and finally sum the weighted features in the feature dimension to obtain the aggregate feature F. agg , as shown in formula (9), In formula (9), ⊙ is the matrix dot product.

6. The repetitive single object extraction method based on center point detection and clustering according to claim 5, characterized in that: The step 6 is specifically as follows: Step 6.1: Name the structure formed by steps 3-5 as a dual-branch enhanced aggregation module. In each encoder, the point cloud attribute information and semantic features are fed into the above module to aggregate the point cloud features F in a smaller range. S , as shown in formula (10), F S =DEA(I,F) (10) In formula (10), DEA represents the double-branched enhanced aggregation module; Step 6.2, F S After the second dual-branch enhanced aggregation module, the features F in a larger neighborhood are aggregated. L To balance the receptive field and local details, as shown in formula (11), F L =DEA(I,F S ) (11) Step 6.3 fuses the features of the two different receptive fields. The specific calculation is shown in formula (12):

7. The repetitive single object extraction method based on center point detection and clustering according to claim 6, characterized in that: The step 7 is specifically as follows: Step 7.1: Take the second-layer feature E2 and the fourth-layer feature E4 in the encoder as the fusion starting point, combine the upper and lower encoder features, and use MLP to fuse to obtain F1 H 、F1 L , as shown in formula (13) and (14), In formula (13), DS and US represent downsampling and upsampling operations respectively, and Conv() represents the convolution operation; Step 7.2: Combine the encoder intermediate layer features E3 and F1 H and F1 L Fusion, to obtain the global fusion feature F2 that combines features of different scales G In addition, the encoder top-level feature E1 and bottom-level feature E5 are respectively compared with F1 H 、F1 L Fusion to obtain F2 S F2 D , to ensure the distinction between features of different scales, as shown in formulas (15), (16), and (17). Step 7.3, in the final stage of fusion, the three fusion features are processed symmetrically, and F2 G Decompose and combine with F2 S 、F2 D The specific integration is shown in the following formulas (18) and (19): In the decoding stage, the fused features are decomposed and aligned with the decoder feature dimensions. In addition, long skip connections are used to integrate the encoder and decoder features to ensure the stability of the model.

8. The repetitive single object extraction method based on center point detection and clustering according to claim 7, characterized in that: Specifically, step 8 is to pass the final decoded features through two multi-layer perceptrons and a Dropout layer, and obtain the final classification result through Softmax.

Images