Three-dimensional point cloud semantic segmentation method based on multi-scale feature jump fusion

Abstract

The application provides a three-dimensional point cloud semantic segmentation method based on multi-scale feature jump fusion, which is used for improving the accuracy of three-dimensional point cloud semantic segmentation. In the method, a multi-scale learning space context feature network is constructed, which mainly comprises a learning space context feature module, an improved feature jump fusion module and a multi-scale connection module. The learning space context feature module can enhance the connection between local and global context features, thereby improving the segmentation performance. The feature jump fusion module adopts an attention mechanism, which can eliminate the semantic gap between the encoding layer and the decoding layer, and can extract low-level features from high-level features, thereby reducing the computational complexity of the network. The multi-scale connection module fully utilizes the relevant information in the encoding layer, so that the network can obtain available features of different scales, thereby capturing the dependency relationship between different scale features and realizing the global consistency of the network.

Description

Technical Field

[0001] This invention belongs to the field of 3D point cloud processing, specifically involving a 3D point cloud semantic segmentation method based on multi-scale feature jump fusion. Background Technology

[0002] 3D point cloud semantic segmentation methods based on deep learning technology can be categorized into indirect point cloud-based methods (including projection-based, voxel-based, and multi-view-based methods) and direct point cloud-based methods, based on the data representation. Indirect methods transform irregular point clouds into regular structures before inputting them into the network for segmentation. This type of method effectively compensates for the limitation of Convolutional Neural Networks (CNNs) in applying them to point clouds, but inevitably loses some important information features during the transformation process. Direct methods, on the other hand, do not require transformation; they directly input the point cloud data into the network for segmentation. This type of method fully utilizes the characteristics of point cloud data, reduces network computational complexity, and improves the accuracy of semantic segmentation.

[0003] The most representative methods based directly on point clouds are PointNet[1] and PointNet++[2] proposed by Qi of Stanford University. PointNet is mainly composed of three modules: spatial transformation network module, recurrent neural network module, and symmetric function module. The spatial transformation network module can process all data into a normal form, the recurrent neural network module can be trained in the form of continuous signals, and the symmetric function module can summarize the information of all points in the point cloud. Through the processing of these three modules, the point cloud has rotation invariance and sequence invariance, which successfully solves the disorder of the point cloud. However, since the features of each point in PointNet are learned independently, it is impossible to capture the local features between points to extract fine-grained features. In order to solve this problem, based on PointNet, the PointNet++ method learns the features of the point cloud with hierarchical structure features. By deepening the network layers, the local features are more accurate. However, due to its complex architecture, the computational complexity is too high, and the overall accuracy needs to be further improved.

[0004] The above references are:

[0005] [1]Qi CR,Su H,Mo K,et al.PointNet:Deep learning on point sets for 3Dclassification and segmentation[C]. / / 30th IEEE Conference on ComputerVisionand Pattern Recognition,2017:77-85.

[0006] [2]Qi CR,Yi L,Su H,et al.PointNet++:Deep hierarchical featurelearning on point sets in a metric space[C]. / / 31st Annual Conference onNeuralInformation Processing Systems,2017:5100-5109. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a 3D point cloud semantic segmentation method to improve the accuracy of 3D point cloud semantic segmentation. The technical solution adopted in this invention is as follows:

[0008] This invention provides a 3D point cloud semantic segmentation method based on multi-scale feature skip fusion, characterized by the following steps: Step S1, constructing a multi-scale learning spatial context feature network, which has a learning spatial context feature module, an information feature skip fusion module, a multi-scale connection module, an encoding layer, a decoding layer, and a fully connected layer; Step S2, inputting the 3D point cloud data to be tested into the multi-scale learning spatial context feature network, generating the corresponding first point cloud feature (N, 8) based on the fully connected layer, thus serving as the input to the learning spatial context feature module; Step S3, the learning spatial context feature module learns the corresponding local information features and global context information based on the first point cloud feature to obtain the second point cloud feature (N, 16); Step S4, performing a random sampling on the second point cloud feature to generate a third point cloud feature (N / 4, 16). 6); Step S5, the encoding layer performs four rounds of context feature extraction and random sampling on the third point cloud features, i.e., repeats steps S3-S4, to generate the first point cloud feature set; Step S6, the decoding layer upsamples the first point cloud feature set using nearest neighbor interpolation to generate the second point cloud feature set; Step S7, the information feature skip fusion module performs feature mapping between the first point cloud feature set and the second point cloud feature set based on a shared perceptron, thereby eliminating the semantic gap between the encoding layer and the encoding layer; Step S8, the multi-scale connection module connects the point cloud features output from the first layer of the encoding layer with the second point cloud feature set to obtain new point cloud features; Step S9, the new point cloud features are passed through three fully connected layers and one Dropout layer to obtain the semantic label of the point cloud data to be tested, and the semantic label is used as the final output of the multi-scale learning space context feature network.

[0009] The 3D point cloud semantic segmentation method with multi-scale feature jump fusion provided by this invention can also have the following technical features: the learning spatial context feature module is mainly composed of three sub-modules: local polar coordinate representation block, dual-distance attention pooling block, and global context feature block. The local polar coordinate representation block is used to construct a rotation-invariant local spatial representation of each 3D point cloud with respect to the Z-axis. The dual-distance attention pooling block uses the geometric distance and feature distance between points to learn more distinctive local information features. The global context feature block learns the global context information of each 3D point cloud based on the ratio of the spatial position of the point to the volume of the global point cloud occupied by the neighboring points.

[0010] The 3D point cloud semantic segmentation method with multi-scale feature skip fusion provided by this invention may also have the following technical features: the information feature skip fusion module adopts an attention mechanism to connect the low-level features output by the encoder, namely the first point cloud feature set, and the high-level features output by the decoder, namely the second point cloud feature set, so that the network pays more attention to global context information during the learning process, thereby improving the global semantic consistency.

[0011] The multi-scale feature skip fusion method for 3D point cloud semantic segmentation provided by this invention may also have the following technical features, wherein step S7 includes the following sub-steps: Step S7-1, passing the low-level input features through F L Transformed into feature F by MLP A The process can be expressed by the following formula:

[0012] F A =MLP(F L );

[0013] Step S7-2, transfer feature F A and low-level input features F H The feature F is obtained by extrusion processing. SA and feature F SB The process can be expressed by the following formula:

[0014] F SA =Squeeze(F A )

[0015] F SB =Squeeze(F H );

[0016] Step S7-3, obtain the feature F SB After the Softmax operation, the attention score of the high-level features is calculated, and the score is multiplied by the feature F. SA The low-level attention feature F is derived. SC :

[0017] F SC =Soft max(F SB )⊙F SA ;

[0018] Step S7-4, transfer the high-level features F SB With low-level attention features F SC By combining them, we obtain a new feature F. SD :

[0019] F SD =Cat(F SB ,F SC )

[0020] In the formula, Cat represents the splicing operation;

[0021] Step S7-5: After the expansion operation and MLP, the final output F of the information feature skip fusion module is obtained. D :

[0022] F D=MLP(Expand(F SD )).

[0023] The 3D point cloud semantic segmentation method with multi-scale feature skip fusion provided by this invention may also have the following technical features: the first point cloud feature set generated by the encoding layer is (N / 16, 64), (N / 64, 128), (N / 256, 256), (N / 512, 512); the second point cloud feature set generated by the decoding layer based on the first point cloud feature set is (N / 512, 512), (N / 256, 256), (N / 64, 128), (N / 16, 64), (N / 4, 16); the multi-scale connection module upsamples the features in the decoding layer and connects them with the point cloud features output by the first layer of the encoding layer to obtain five new point cloud features; the relevant features are evaluated by MLP to obtain five new 16-channel features; and these five new 16-channel features are concatenated into a 90-channel feature, which is used as the output of the multi-scale connection module.

[0024] The 3D point cloud semantic segmentation method with multi-scale feature skip fusion provided by this invention may also have the following technical features, wherein step S8 includes the following sub-steps: Step S8-1, upsampling the point cloud features in the second point cloud feature set and transforming them into features (N, 512), (N, 256), (N, 128), (N, 64), and (N, 16) respectively, which is represented as follows:

[0025]

[0026] In the formula, US(*) represents the feature of the i-th layer in the decoding layer, and US(*) represents upsampling the feature *. represent The features obtained after upsampling are i = 1, 2, 3, 4, 5;

[0027] Step S8-2: The five features obtained in step S8-1 are concatenated with the point cloud features output from the first layer of the coding layer to obtain five new features, namely (N, 528), (N, 272), (N, 144), (N, 80), and (N, 32). This process is represented as follows:

[0028]

[0029] In the formula, F E This represents the output feature of the first layer in the coding layer, and Cat represents the concatenation operation. Represents the features obtained after the splicing operation, i = 1, 2, 3, 4, 5;

[0030] Step S8-3, obtain the features The relevant features were evaluated using MLP, resulting in five 16-channel features (N, 16):

[0031]

[0032] In the formula, The features obtained after MLP;

[0033] Step S8-4 involves concatenating the five 16-channel features (N, 16) into a single (N, 90) feature representation, which is then used as the final output of the multi-scale connection module. This process is illustrated below:

[0034]

[0035] Invention Function and Effect

[0036] This invention discloses a multi-scale feature skip fusion method for 3D point cloud semantic segmentation, which integrates multi-scale feature skipping with a deep learning network to improve the accuracy of 3D point cloud semantic segmentation. The method constructs a multi-scale learning spatial context feature network, which mainly includes a learning spatial context feature module, an improved feature skip fusion module, and a multi-scale connection module. The learning spatial context feature module enhances the connection between local and global context features, thereby improving segmentation performance. The proposed feature skip fusion module, employing an attention mechanism, eliminates the semantic gap between the encoding and decoding layers while extracting low-level features from high-level features, thus reducing the computational complexity of the network. Furthermore, because this module connects low-level and high-level features, the network pays more attention to global context information during the learning process, thereby improving global semantic consistency. The multi-scale connection module fully utilizes relevant information in the encoding layer, enabling the multi-scale learning spatial context feature network to obtain available features at different scales, thereby capturing the dependencies between features at different scales and achieving global consistency of the network. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the structure of the Multi-Scale Learning Spatial Context Feature Network (MSSCF) in an embodiment of the present invention;

[0038] Figure 2 This is a flowchart illustrating the three-dimensional point cloud semantic segmentation method using multi-scale feature skip fusion in an embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram of the spatial context feature module in an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the information feature skip fusion module in an embodiment of the present invention;

[0041] Figure 5 This is a structural schematic diagram of the multi-scale connection module in an embodiment of the present invention;

[0042] Figure 6 This is a visualization diagram of the S3DIS dataset used in the experiments of this invention embodiment;

[0043] Figure 7 This is a visualization diagram of five indoor scenes of MSSCF in region 6 of the S3DIS dataset in an embodiment of the present invention;

[0044] Figure 8 This is a visualization diagram of ten office scenarios in region 6 of the S3DIS dataset using MSSCF in an embodiment of the present invention; and

[0045] Figure 9 This is a schematic diagram illustrating the partial segmentation results of applying the MSSCF-based 3D point cloud semantic segmentation method to a 3D point cloud model dataset in an embodiment of the present invention. Detailed Implementation

[0046] Figure 1 This is a schematic diagram of the structure of the Multi-Scale Learning Spatial Context Feature Network (MSSCF) in an embodiment of the present invention.

[0047] The overall structure of the Multi-Scale Spatial Context Features Network (MSSCF) for point cloud semantic segmentation is as follows: Figure 1 As shown, it mainly consists of a Spatial Context Features (SCF) module, an Information Feature Skip Fusion (IFSF) module, a Multi-Scale Concatenate (MSC) module, as well as a fully connected layer (FC), a RS random sampling layer, an MLP multilayer perceptron, a US upsampling layer, and a Dropout layer.

[0048] Depend on Figure 1The process of 3D point cloud semantic segmentation based on the MSSCF network is as follows: For the input point cloud data, it first passes through a fully connected layer to output point cloud features (N, 8); then it is used as the input to the learning space context feature module, and after passing through SCF, it outputs point cloud features (N, 16); next, random sampling is performed, outputting point cloud features (N / 4, 16); then SCF and RS are used four times, and the output point cloud features are (N / 16, 64), (N / 64, 128), (N / 256, 256), and (N / 512, 512), respectively. This process is the encoding layer of MSSCF. After the above encoding layer, four decoding layers are used. For each decoding layer, the point cloud feature set is first upsampled using nearest neighbor interpolation. Then, the upsampled feature map is connected to the intermediate feature map generated by the encoding layer through an improved feature skip fusion module, and MLP is applied to the connected feature map. The point cloud features at each layer in the decoding process correspond to (N / 512, 512), (N / 256, 256), (N / 64, 128), (N / 16, 64), and (N / 4, 16), respectively. After the decoding layers, a multi-scale connection module concatenates the point cloud features (N, 16) from the encoding layer with each point cloud feature from the decoding layer, resulting in a new point cloud feature (N, 128). This new feature is then passed through three fully connected layers and one Dropout layer to obtain the final output semantic label. The point cloud features at each layer in this process correspond to (N, 128), (N, 64), and (N, 32), respectively.

[0049] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following describes the multi-scale feature jump fusion three-dimensional point cloud semantic segmentation method of this invention in detail with reference to embodiments and accompanying drawings.

[0050] <Example>

[0051] Figure 2 This is a flowchart illustrating the three-dimensional point cloud semantic segmentation method using multi-scale feature skip fusion in an embodiment of the present invention.

[0052] like Figure 2 As shown, the 3D point cloud semantic segmentation method based on multi-scale feature skip fusion includes the following steps:

[0053] Step S1: Construct a multi-scale learning spatial context feature network.

[0054] like Figure 1 As shown, the multi-scale learning space context feature network in this embodiment has a learning space context feature module, an information feature jump fusion module, a multi-scale connection module, an encoding layer, a decoding layer, and a fully connected layer.

[0055] Step S2: Input the 3D point cloud data to be tested into the multi-scale learning space context feature network, and generate the corresponding point cloud features (N, 8) based on the fully connected layer, which are then used as the input of the learning space context feature module.

[0056] Step S3: The spatial context feature learning module learns the corresponding local information features and global context information based on the first point cloud features to obtain point cloud features (N, 16).

[0057] For large-scale 3D point cloud semantic segmentation tasks, in order to further learn the spatial context information of the point cloud and thus improve segmentation performance, it is necessary to correctly represent the local context information of the point cloud and correctly learn the information features of local and global context. Therefore, this embodiment introduces a plug-and-play Spatial Context Feature Module (SCF) to learn the information features of spatial context.

[0058] Figure 3 This is a schematic diagram of the spatial context feature module in an embodiment of the present invention.

[0059] like Figure 3 As shown, the learning space context feature module mainly consists of three sub-modules: Local Polar Representation (LPR), Dual-Distance Attentive Pooling (DDAP), and Global Contextual Feature (GCF).

[0060] Among them, the local polar coordinate representation block is used to construct a rotation-invariant local spatial representation of each 3D point cloud with respect to the Z-axis, the dual-distance attention pooling block uses the geometric distance and feature distance between points to learn more distinctive local information features, and the global context feature block learns the global context information of each 3D point cloud based on the ratio of the spatial position of the point to the volume of the global point cloud occupied by the neighboring points.

[0061] Step S4: Perform a random sampling on the point cloud feature (N, 16) to generate point cloud feature (N / 4, 16).

[0062] Step S5: The encoding layer performs four context feature extractions and random samplings on the point cloud features (N / 4, 16), that is, repeats steps S3-S4 to generate the first point cloud feature set (N / 16, 64), (N / 64, 128), (N / 256, 256), (N / 512, 512).

[0063] Step S6: The decoding layer upsamples the first point cloud feature set using nearest neighbor interpolation to generate the second point cloud feature set (N / 512, 512), (N / 256, 256), (N / 64, 128), (N / 16, 64), (N / 4, 16).

[0064] Step S7: The information feature skip fusion module performs feature mapping between the first point cloud feature set and the second point cloud feature set based on the shared perceptron, thereby eliminating the semantic gap between the coding layer and the encoding layer.

[0065] In 2D semantic segmentation tasks, common encoder and decoder networks primarily use feature information at different scales to recover object boundaries in the decoding path. They can also leverage high-level feature information with rich categories to extract low-level feature information, thereby selecting more accurate resolution details. Furthermore, to refine resolution details using high-level feature information, attention mechanisms have become an important component for capturing contextual information. Based on this idea, this embodiment proposes an information feature skipping fusion module employing an attention mechanism.

[0066] Figure 4 A schematic diagram of the information feature skipping fusion module in this embodiment of the invention.

[0067] like Figure 4 As shown, the Information Feature Skip Fusion (IFSF) module has two inputs. The first is low-level input features from the encoder; the second is high-level input features from the decoder. The specific process of step S7 is as follows:

[0068] First, F with low-level input features L Transformed into feature F by MLP A The process can be expressed by the following formula:

[0069] F A =MLP(F L )

[0070] Next, feature F A and low-level input features F H The feature F is obtained by extrusion processing. SA and feature F SB The process can be expressed by the following formula:

[0071] F SA =Squeeze(F A )

[0072] F SB =Squeeze(F H )

[0073] Secondly, the obtained feature F SB After the Softmax operation, the attention score of the high-level features is calculated, and the score is multiplied by the feature F. SA The low-level attention feature F is derived. SC The process can be expressed by the following formula:

[0074] F SC =Soft max(F SB )⊙F SA

[0075] Then, the high-level features F SB With low-level attention features F SC By combining them, we obtain a new feature F. SD The process can be expressed by the following formula:

[0076] F SD =Cat(F SB ,F SC )

[0077] Here, Cat represents the splicing operation.

[0078] Finally, after the expansion operation and MLP, the final output F of the module is obtained. D The process can be expressed by the following formula:

[0079] F D =MLP(Expand(F SD ))

[0080] As mentioned above, the IFS module employs an attention mechanism, which, while bridging the semantic gap between the encoder and decoder layers, can extract low-level features from high-level features, and its computational complexity is lower than other methods. Furthermore, the IFS module can connect low-level and high-level features together, allowing the network to pay more attention to global contextual information during the learning process, thereby improving global semantic consistency. Therefore, this information feature skipping fusion module has the characteristic of balancing the encoder and decoder.

[0081] In step S8, the multi-scale connection module connects the point cloud features output from the first layer of the coding layer with the second point cloud feature set to obtain new point cloud features.

[0082] In the field of computer vision, multi-scale modules have become an indispensable part of improving the performance of some representative methods due to their ability to aggregate multi-scale contextual information. This embodiment proposes a multi-scale connectivity module for point cloud semantic segmentation.

[0083] Figure 5 This is a structural schematic diagram of the multi-scale connection module in an embodiment of the present invention.

[0084] Depend on Figure 5 As can be seen, the Multi-Scale Connectivity Module (MSC) takes the output of the first layer of the coding layer and the features of the five layers of the decoding layer as inputs to the module. Then, the features in the decoding layer are upsampled and concatenated with the output of the first layer of the coding layer to obtain five new features. Next, the relevant features are evaluated using an Multi-Scale Connectivity Module (MLP) to obtain five new 16-channel features. Finally, these five 16-channel features are concatenated to form a 90-channel feature set, which is the output of the module. The detailed operation is as follows:

[0085] First, the point cloud features in the second point cloud feature set are upsampled and transformed into features (N, 512), (N, 256), (N, 128), (N, 64), and (N, 16), respectively. This process is represented as follows:

[0086]

[0087] In the formula, US(*) represents the feature of the i-th layer in the decoding layer, and US(*) represents upsampling the feature *. represent The features obtained after upsampling are i = 1, 2, 3, 4, 5.

[0088] Next, the five features obtained in the previous step are concatenated with the point cloud features output from the first layer of the encoding layer to obtain five new features, namely (N, 528), (N, 272), (N, 144), (N, 80), and (N, 32). This process is represented as follows:

[0089]

[0090] In the formula, F E This represents the output feature of the first layer in the coding layer, and Cat represents the concatenation operation. Represents the features obtained after the splicing operation, i = 1, 2, 3, 4, 5;

[0091] Then, the obtained features The relevant features were evaluated using MLP, resulting in five 16-channel features (N, 16):

[0092]

[0093] In the formula, The features obtained after MLP;

[0094] Finally, the five 16-channel features (N, 16) are concatenated into a single (N, 90) feature representation, which is then used as the final output of the multi-scale connection module. This process is represented as follows:

[0095]

[0096] As shown above, the MSC module makes full use of the relevant information in the coding layer, enabling the MSSCF network to obtain available features at different scales, thereby capturing the dependencies between features at different scales and achieving global consistency of the network.

[0097] Step S9: The new point cloud features are passed through three fully connected layers and one Dropout layer to obtain the semantic label of the point cloud data to be tested. The semantic label is used as the final output of the multi-scale learning spatial context feature network.

[0098] To verify the effectiveness of the proposed network model, experiments were conducted on a large-scale indoor scene using a selected experimental dataset. The experimental environment consisted of a computer running Ubuntu 20.04, equipped with two NVIDIA Titan Xp GPUs and 64GB of memory. The algorithm used in the experiments was Python 3.6, TensorFlow 1.14, CUDA 10.1, and CUDA 7.6.5. The software and hardware configurations used in this experiment are shown in Table 1 below.

[0099]

[0100] Table 1

[0101] The experiment used four evaluation criteria to assess the segmentation performance of the model, including overall accuracy (oAcc), mean accuracy (mAcc), intersection over union ratio (Iou) for each class, and mean intersection over union ratio (mIou), which were defined as follows:

[0102] oAcc: refers to the ratio of the semantic annotation result of each random sample to the actual data annotation type. It is a relatively simple metric, which can be expressed by the formula:

[0103]

[0104] mAcc: refers to the average precision of all classes, which can be expressed by the formula:

[0105]

[0106] Iou: refers to the overlap rate between the actual and predicted annotations, which can be expressed by the formula:

[0107]

[0108] mIou: This refers to the average of the Iou calculated for each category. It is one of the most effective metrics for evaluating segmentation accuracy and can be expressed by the formula:

[0109]

[0110] Where i represents the true value, j represents the predicted value, Pij means predicting i as j, and N represents the number of categories.

[0111] To verify the superior performance of the proposed method, this experiment was conducted on the large-scale indoor scene S3DIS dataset. The visualization of the S3DIS dataset is shown below. Figure 6 As shown.

[0112] The S3DIS dataset is a semantic segmentation dataset for large-scale indoor scenes. The dataset covers approximately 6,000 square meters and contains over 70,000 RGB images with more than 200 million pixels. It consists of three buildings primarily used for education and offices, divided into six different large indoor areas with a total of 271 rooms. These rooms are categorized into 13 semantic categories: ceiling, floor, wall, beam, column, window, door, chair, table, bookcase, sofa, board, and clutter. The total number of categories for each area is shown in Table 2.

[0113]

[0114] Table 2

[0115] Since the S3DIS dataset contains six distinct regions, this embodiment also employs the same evaluation criterion used in PointNet, namely, performing 6x cross-validation on all regions. Specifically, five regions are selected as the training set in each training iteration, and the remaining region is used as the test set.

[0116] To verify the effectiveness of the MSSCF network structure, this experiment compares the point cloud semantic segmentation results of MSSCF with those of PointNet, RSNet, 3P-RNN, SPG, PointCNN, PointWeb, ShellNet, and KpConv. The performance comparison is shown in Table 3 and Table 4 below.

[0117]

[0118] Table 3. Semantic segmentation accuracy of different methods on the S3DIS dataset.

[0119] As shown in Table 3 above, MSSCF achieves semantic segmentation oAcc, mAcc, and mIou of 88.3%, 82.0%, and 71.1% respectively on the S3DIS dataset, outperforming other methods. Specifically, compared to PointNet, MSSCF improves oAcc, mAcc, and mIou performance by 9.7%, 15.8%, and 23.5% respectively; compared to RSNet, MSSCF improves mAcc and mIou performance by 15.5% and 14.6% respectively; compared to the 3P-RNN algorithm, MSSCF improves oAcc and mIou performance by 1.4% and 14.8% respectively; compared to the SPG architecture, MSSCF improves oAcc, mAcc, and mIou performance by 1.9%, 9.0%, and 9.0% respectively; and compared to Poi... While MSSCF only improved oAcc performance by 0.2% in the ntCNN network, it improved mAcc and mIou performance by 6.4% and 5.7%, respectively. Compared to the PointWeb method, MSSCF improved oAcc, mAcc, and mIou performance by 1.0%, 5.8%, and 4.4%, respectively. Compared to the ShellNet framework, MSSCF improved oAcc and mIou performance by 1.8% and 4.3%, respectively. Compared to the KpConv model, MSSCF improved mAcc and mIou performance by 1.7% and 0.5%, respectively.

[0120]

[0121]

[0122] Table 4 compares the Iou values ​​of different methods for each semantic class in the S3DIS dataset.

[0123] As shown in Table 4, the MSSCF proposed in this invention achieves good performance metrics for each semantic class Iou on the S3DIS dataset. Specifically, the Iou values ​​are as follows: ceiling (94.4%), floor (96.2%), wall (79.8%), beam (62.7%), column (47.3%), window (69.6%), door (72.6%), table (71.2%), chair (79.1%), sofa (60.2%), bookcase (64.7%), wooden board (66.2%), and miscellaneous items (60.6%). Compared to PointNet, RSNet, and 3P-RNN, MSSCF improved the IOU of all semantic classes. Compared to SPG, MSSCF had slightly lower IOU values ​​for beams, tables, and sofas, but outperformed in the IOU of the remaining 10 semantic classes, especially showing a significant improvement in the IOU value of planks. Compared to PointCNN and PointWeb, MSSCF had the biggest lead in the IOU values ​​of doors and chairs, respectively. Compared to ShellNet and KpConv, MSSCF achieved 10 and 7 class results, respectively.

[0124] To verify the effectiveness of the MSSCF network and the IFSF and MSC modules proposed in this invention, an ablation experiment was added to explore the effectiveness of the IFSF and MSC modules in MSSCF. The ablation experiments are shown in Table 5 below, where [-] indicates the removal of the module.

[0125]

[0126] Table 5. Impact of each module in MSSCF on mIou

[0127] As shown in Table 5 above, when MSSCF removes both the IFSF and MSC modules, the mean Iou decreases by 3.6%. Compared to removing both IFSF and MSC modules simultaneously, when MSSCF removes the IFSF module but retains the MSC module, the mean Iou increases by 1.7%; when MSSCF removes the MSC module but retains the IFSF module, the mean Iou increases by 2.1%. When MSSCF retains both the FAA and FSF modules, the mean Iou reaches 71.1%. This demonstrates the effectiveness of the IFSF and MSC modules and their ability to improve the mean Iou. The ablation experiments show that the MSSCF network model with IFSF and MSC modules proposed in this embodiment achieves superior point cloud semantic segmentation performance on the S3DIS dataset.

[0128] The MSSCF proposed in this embodiment is used to visualize five indoor scenes in region 6 of the S3DIS dataset, such as... Figure 7 As shown, the diagram includes three corridor scenes (rows 1, 2, and 5), one printing room scene (row 3), and one lounge scene (row 4). The first column corresponds to the RGB color input point cloud, the second column corresponds to the semantic segmentation prediction obtained based on the MSSCF method, and the third column corresponds to the ground truth.

[0129] In addition, this experiment also provides ten office scene visualization examples of MSSCF in region 6 of the S3DIS dataset, such as... Figure 8 As shown (different items are distinguished by shades of gray). Figure 8 The first column represents the RGB color input point cloud, the second column represents the semantic segmentation prediction obtained by MSSCF in this embodiment, and the third column represents the ground reality.

[0130] Depend on Figure 7 and Figure 8 It is known that semantic segmentation of some indoor scenes (such as printing rooms and lounges) is quite difficult. This may be because some categories in the dataset have limited data, resulting in generally poor category classification. Furthermore, some categories, while finely divided, may contain noise, distortion, or similarity to other categories, making misclassification by the network inevitable. However, in indoor scenes with good point cloud quality, the semantic segmentation performance of the MSSCF proposed in this invention is excellent.

[0131] Furthermore, the 3D point cloud semantic segmentation method of this embodiment can also be applied to the partial segmentation task of 3D point cloud models. The 3D point cloud model used is ShapeNet Part. This dataset contains 16 classes and 50 parts of object models, totaling 16,881 3D point cloud model samples. The dataset includes 12,137 training samples, 1,870 validation samples, and 2,874 test samples. Each point cloud model contains 2-6 parts. The 3D point cloud semantic segmentation method based on the MSSCF network is applied to the partial segmentation of the ShapeNet Part 3D point cloud model dataset. The results of partial segmentation of the point cloud models are shown in [the figure]. Figure 9 The comparison results of semantic segmentation accuracy of different methods on the ShapeNet Part dataset are shown in Table 6 below:

[0132]

[0133] Table 6

[0134] The AGConv method, with 200 epochs, takes 30.4 hours to train when using dual NVIDIA RTX 2080 graphics cards. The MSSCF method of this invention, with 136 epochs, takes only 20.3 hours to train and achieves an average crossover ratio mIou = 88.3%.

[0135] Functions and effects of the embodiments

[0136] This embodiment provides a multi-scale feature skip fusion method for 3D point cloud semantic segmentation, designed to improve the accuracy of 3D point cloud semantic segmentation. The method constructs a multi-scale learning spatial context feature network, which mainly includes a learning spatial context feature module, an improved feature skip fusion module, and a multi-scale connection module. The learning spatial context feature module enhances the connection between local and global context features, thereby improving segmentation performance. The proposed feature skip fusion module, employing an attention mechanism, eliminates the semantic gap between the encoding and decoding layers while extracting low-level features from high-level features, thus reducing the computational complexity of the network. Furthermore, by connecting low-level and high-level features, the module allows the network to focus more on global context information during the learning process, improving global semantic consistency. The multi-scale connection module fully utilizes relevant information in the encoding layer, enabling the multi-scale learning spatial context feature network to obtain usable features at different scales, thereby capturing the dependencies between features at different scales and achieving global consistency of the network.

[0137] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.

Claims

1. A three-dimensional point cloud semantic segmentation method based on multi-scale feature jump fusion, characterized in that, Includes the following steps: Step S1: Construct a multi-scale learning space context feature network. This network has a learning space context feature module, an information feature skip fusion module, a multi-scale connection module, an encoding layer, a decoding layer, and a fully connected layer. Step S2: Input the 3D point cloud data to be tested into the multi-scale learning space context feature network, and generate the corresponding first point cloud feature (N, 8) based on the fully connected layer, so as to serve as the input of the learning space context feature module; Step S3: The learning space context feature module learns the corresponding local information features and global context information based on the first point cloud features to obtain the second point cloud features (N, 16). Step S4: Perform a random sampling on the second point cloud feature to generate a third point cloud feature (N / 4, 16); Step S5: The encoding layer performs four context feature extractions and random samplings on the third point cloud features, that is, repeats steps S3-S4 to generate the first point cloud feature set. Step S6: The decoding layer upsamples the first point cloud feature set using nearest neighbor interpolation to generate a second point cloud feature set. Step S7: The information feature skip fusion module performs feature mapping between the first point cloud feature set and the second point cloud feature set based on the shared perceptron, thereby eliminating the semantic gap between the coding layer and the encoding layer. Step S8: The multi-scale connection module connects the point cloud features output from the first layer of the coding layer with the second point cloud feature set to obtain new point cloud features. Step S9: The new point cloud features are passed through three fully connected layers and one Dropout layer to obtain the semantic label of the point cloud data to be tested, and the semantic label is used as the final output of the multi-scale learning spatial context feature network.

2. The 3D point cloud semantic segmentation method based on multi-scale feature skip fusion according to claim 1, characterized in that: in, The learning space context feature module mainly consists of three sub-modules: a local polar coordinate representation block, a dual-distance attention pooling block, and a global context feature block. The local polar coordinate representation block is used to construct a rotation-invariant local spatial representation of each 3D point cloud with respect to the Z-axis. The dual-distance attention pooling block utilizes the geometric distance and feature distance between points to learn more discriminative local information features. The global context feature block learns the global context information of each 3D point cloud based on the ratio of the spatial location of the point to the volume of the global point cloud occupied by its neighboring points.

3. The 3D point cloud semantic segmentation method based on multi-scale feature skip fusion according to claim 1, characterized in that: in, The information feature skip fusion module adopts an attention mechanism to connect the low-level features output by the encoder, namely the first point cloud feature set, and the high-level features output by the decoder, namely the second point cloud feature set, so that the network pays more attention to global context information during the learning process, thereby improving global semantic consistency.

4. The 3D point cloud semantic segmentation method based on multi-scale feature skip fusion according to claim 1, characterized in that: in, Step S7 includes the following sub-steps: Step S7-1, pass the low-level input features through F L Transformed into feature F by MLP A The process can be expressed by the following formula: F A =MLP(F L ); Step S7-2, transfer feature F A and low-level input features F H Feature F is obtained by extrusion processing. SA and feature F SB The process can be expressed by the following formula: F SA =Squeeze(F A ) F SB =Squeeze(F H ); Step S7-3, obtain the feature F SB After the Softmax operation, the attention score of the high-level feature is calculated, and the score is multiplied by the feature F. SA The low-level attention feature F is derived. SC : Step S7-4, transfer the high-level features F SB With low-level attention features F SC By combining them, we obtain a new feature F. SD : F SD =Cat(F SB ,F SC ) In the formula, Cat represents the splicing operation; Step S7-5: After the expansion operation and MLP, the final output F of the information feature skip fusion module is obtained. D : F D =MLP(Expand(F SD ))。 5. The 3D point cloud semantic segmentation method based on multi-scale feature skip fusion according to claim 4, characterized in that: in, The first point cloud feature set generated by the coding layer is: (N / 16, 64), (N / 64, 128), (N / 256, 256), (N / 512, 512). The second point cloud feature set generated by the decoding layer based on the first point cloud feature set is: (N / 512, 512), (N / 256, 256), (N / 64, 128), (N / 16, 64), (N / 4, 16). The multi-scale connection module upsamples the features in the decoding layer and connects them with the point cloud features output from the first layer of the coding layer to obtain five new point cloud features. The relevant features are evaluated by MLP to obtain five new 16-channel features. These five new 16-channel features are then concatenated into a 90-channel feature, which is used as the output of the multi-scale connection module.

6. The 3D point cloud semantic segmentation method based on multi-scale feature skip fusion according to claim 5, Its features are: in, Step S8 includes the following sub-steps: Step S8-1: Upsample the point cloud features in the second point cloud feature set and transform them into features (N, 512), (N, 256), (N, 128), (N, 64), and (N, 16), respectively. This process is represented as follows: In the formula, US(*) represents the feature of the i-th layer in the decoding layer, and US(*) represents upsampling the feature *. represent The features obtained after upsampling are i = 1, 2, 3, 4, 5; Step S8-2: The five features obtained in step S8-1 are concatenated with the point cloud features output from the first layer of the coding layer to obtain five new features, namely (N, 528), (N, 272), (N, 144), (N, 80), and (N, 32). This process is represented as follows: In the formula, F E This represents the output feature of the first layer in the coding layer, and Cat represents the concatenation operation. Represents the features obtained after the splicing operation, i = 1, 2, 3, 4, 5; Step S8-3, obtain the features The relevant features were evaluated using MLP, resulting in five 16-channel features (N, 16): In the formula, The features obtained after MLP; Step S8-4 involves concatenating the five 16-channel features (N, 16) into a single (N, 90) feature representation, which is then used as the final output of the multi-scale connection module. This process is illustrated below:

Images