A point cloud semantic segmentation method based on point hole direction convolution

Abstract

A point cloud semantic segmentation method based on dilated directional convolution is applicable to the field of computer vision. It is a hierarchical encoder-decoder network that alternately uses dilated directional convolution modules, edge-preserving pooling modules, and edge-preserving non-pooling modules. The dilated directional encoding units can perform equivalent sparse sampling of neighboring points by changing the dilation rate, while considering the directional and distance information of local neighbors. This allows for the arbitrary expansion of the receptive field while encoding eight directional features, thus capturing local neighborhood information more comprehensively. Multiple dilated directional encoding units are then stacked together to form a dilated directional convolution module, which has scale awareness and portability. Edge-preserving pooling and edge-preserving non-pooling modules are used to preserve edge features, recover high-dimensional features of the point cloud, and improve the accuracy of point cloud semantic segmentation. This method includes local neighborhood selection and feature extraction of the point cloud to achieve better semantic segmentation performance.

Description

Technical Field

[0001] This invention is applicable to point cloud semantic segmentation technology in the field of computer vision. Background Technology

[0002] 3D point clouds, sampled from the surfaces of real-world objects using a 3D scanner, have wide applications in computer vision, such as autonomous driving, medical imaging, and intelligent robot navigation. Segmenting 3D point clouds into different semantic parts is a fundamental task in 3D modeling and scene understanding. In recent years, various point cloud semantic segmentation methods have been proposed, which can be broadly categorized into three types: voxel-based methods, projection-based methods, and point-based methods.

[0003] (1) Voxel-based method

[0004] Early researchers used voxel representation to organize and regularize the disordered and irregular structure of point clouds, and then used traditional convolutional neural networks to learn features for semantic segmentation of point clouds. VoxNet represents point clouds as occupied grid voxels and then uses 3D convolutional neural networks for supervised learning. However, as the spatial resolution of voxel grids increases, problems such as a significant increase in memory consumption and loss of local geometric information arise. To reduce the computational cost of these voxel-based methods, OctNet uses a hybrid grid octree structure to hierarchically partition the point cloud. This effective spatial partitioning structure reduces memory consumption, thereby enabling high-resolution deep learning and solving the problem of voxel data sparsity. However, since voxels are discrete spatial representations, memory consumption remains high when deeper convolutional neural networks are involved. To fully capture the local geometric information of point clouds, PointGrid integrates point and grid representations and samples a constant number of points in each embedded voxel grid cell, thereby using 3D convolution for feature learning. While voxelization can regularize the structure of point clouds, it is prone to information loss and cannot better balance the relationship between resolution and memory.

[0005] (2) Projection-based methods

[0006] Convolutional neural networks (CNNs) have been widely and maturely applied to 2D images. Therefore, many researchers have attempted to project 3D point data onto various specific image planes to form 2D images, and then use traditional 2D convolution for semantic segmentation of 3D point clouds. SnapNet uses multiple camera positions to generate multiple RGB and depth maps of the point cloud, then uses a 2D fully convolutional network to perform pixel-level labeling on these maps, followed by pixel-labeled backprojection to fuse the prediction scores of the RGB and depth images. Multi-view convolutional neural networks use 2D rendered images of 3D objects from multiple perspectives as training data, and based on CNNs, they recognize 3D objects, achieving good results in point cloud semantic segmentation tasks. However, since max pooling only retains the maximum element from a specific perspective, it cannot fully utilize the local geometric information of each view, easily leading to information loss. While projection-based methods can reduce model complexity and improve the speed and accuracy of semantic segmentation, they ignore the inherent geometric relationships of 3D points and are susceptible to the influence of projection angle and occlusion factors, easily losing crucial detailed structural information.

[0007] (3) Point-based methods

[0008] To fully utilize the inherent characteristics of point cloud data, some scholars have proposed several effective architectures for directly processing point clouds. Point-based networks can directly operate on disordered and irregular point clouds. PointNet was the first network model to perform semantic segmentation directly on disordered point clouds. It can directly input disordered point clouds for processing and extract features using a multilayer perceptron. However, because the network directly and brute-forces max pooling of all points into a single global feature, the relationships between local points are not learned by the network. PointNet++ designed a hierarchical structure, using multiple smaller PointNets to combine local feature information. It can extract local features at different scales and obtain deep features through a multilayer network structure. To effectively utilize local contextual information, PointWeb tightly connects each pair of points in a local region and adaptively adjusts its features, enhancing the feature's ability to describe the local neighborhood. However, these point cloud semantic segmentation networks still suffer from problems such as coarse semantic segmentation edges, insufficient mining of local geometric and contextual information, poor feature learning ability, and high computational cost and low efficiency.

[0009] Due to the sparsity, disorder, and irregularity of 3D point clouds, the positional and semantic relationships between points are difficult to capture. Therefore, existing point cloud segmentation methods have three main drawbacks: (a) they cannot learn multi-scale point features; (b) they cannot fully exploit the local geometric and contextual information of the point cloud; and (c) they cannot learn accurate edge features. Key solutions to these problems include designing sophisticated local neighborhood selection methods and robust feature description methods. To address this, a point-dip oriented convolutional network for 3D point cloud semantic segmentation is proposed. It alternates between point-dip oriented convolutional modules and edge-preserving pooling / edge-preserving non-pooling modules, making it a deep encoder-decoder network that fully utilizes multi-scale, multi-directional local neighborhood feature information of the point cloud. The most crucial component is the point-dip oriented encoding unit. By introducing point-dip oriented convolution into the learning of local neighborhood features in the point cloud, local neighborhood information can be captured more comprehensively, enhancing point-by-point feature representation capabilities and effectively improving the semantic segmentation performance of the point cloud. Summary of the Invention

[0010] This invention proposes a point cloud semantic segmentation method based on point-diffused directional convolution. The method proposes a point-diffused directional convolutional network for learning multi-scale, multi-directional features of point clouds. It is a hierarchical encoder-decoder network that alternately uses point-diffused directional convolutional modules, edge-preserving pooling modules, and edge-preserving non-pooling modules. The point-diffused directional encoding units can perform equivalent sparse sampling of neighboring points by changing the dilation rate, while considering the directional and distance information of local neighbors. It can arbitrarily expand its receptive field while encoding eight directional features, thereby capturing local neighborhood information more comprehensively. Then, multiple point-diffused directional encoding units are stacked together to form a point-diffused directional convolutional module, which has scale awareness and portability. Furthermore, edge-preserving pooling and edge-preserving non-pooling modules are used to preserve edge features and recover high-dimensional features of the point cloud, further improving the semantic segmentation accuracy. The main technical problems addressed by this method are local neighborhood selection and feature extraction of point clouds to achieve better semantic segmentation performance.

[0011] The point cloud semantic segmentation method proposed in this invention based on point-diffuse directional convolution can effectively avoid the shortcomings of traditional semantic segmentation methods. By introducing point-diffuse directional convolution operation, it can effectively capture local neighborhood feature information from multiple directions and scales. Multiple point-diffuse directional encoding units are stacked to form a point-diffuse directional convolution module, which makes the module portable and scale-aware. The combination of edge-preserving pooling module and edge-preserving non-pooling module forms an encoding and decoding architecture, ensuring hierarchical utilization of multi-scale local geometric details and gradual recovery of high-dimensional point feature space information. Figure 2 The overall structure diagram of the point cloud semantic segmentation network based on point-hole directional convolution is given.

[0012] This invention is achieved through the following technical solution:

[0013] The input is raw 3D point cloud data. First, the point cloud data in the dataset is processed for model training. Point hole orientation encoding units are designed to build a deep network model based on local neighborhood feature extraction. Then, the processed point cloud data is fed into the network to train the network parameters. The trained model is tested, and finally, the point cloud semantic segmentation label results are output. The point cloud semantic segmentation flowchart is as follows: Figure 1 As shown, the specific steps are as follows:

[0014] Step 1: Process the point cloud data in the dataset.

[0015] The network proposed in this invention can directly input unordered point cloud data, taking 8192 points and their features (3D features of ScanNet and 9D features of S3DIS) as input to the network.

[0016] ScanNet is a 3D reconstruction annotation dataset for indoor scenes, containing 1513 scanned scenes across 21 classes. Following the experimental setup in PointNet++, 1201 scenes were used for training, and 312 scenes were used for testing. Without RGB information, 8192 points were randomly selected from each scene as training or testing samples.

[0017] The large-scale Stanford 3D Indoor Semantic Dataset (S3DIS) consists of six folders of RGB-D point cloud data from three different buildings (containing 271 rooms). Each point is annotated with labels from 13 categories. To ensure fair comparison, the training set was prepared according to PointNet requirements, and the points were divided into 1m×1m blocks by room and sample room. 4096 points were randomly selected from each block for training, and all points were used for testing.

[0018] Step 2: Establish a deep network model based on local neighborhood feature extraction

[0019] 1. Design of point void direction coding unit

[0020] 3D point clouds are disordered and unstructured. However, more information can be obtained by using ordering operators. The ordering caused by the three coordinate axes (X, Y, and Z) is the natural ordering of point clouds. Although point-dilated convolution can capture multi-scale features of point clouds along the Euclidean distance, it may lose information from different directions. Therefore, a point-dilated orientation encoding unit is proposed.

[0021] A point hole orientation coding unit involves two hyperparameters: a sampling rate r and a neighborhood size K, such that 8|K and K≤8r. For a given centroid point p, the point hole orientation coding unit searches for the first... The nearest neighbor points are used to select multi-directional neighborhood points of p, and the centroid features are updated by the following equation (1):

[0022]

[0023] Where q1,...,q K x is a selected neighborhood point of point p. p ,x qK Let p and q represent points respectively. K Pointwise features. h θ This is the edge kernel implemented by a shared multilayer perceptron operation, cat(x,y) is the concatenation operation of two vectors, and g() is the max pooling function.

[0024] Next, we will explain the neighborhood selection strategy of the point hole orientation coding unit. When r=1 and K=8, the point hole orientation coding unit selects the nearest neighbor point in each quadrant; when r=2 and K=8, it selects the second nearest neighbor point in each quadrant; when r=2 and K=16, it selects the second and fourth nearest neighbors in each quadrant. In this way, the point hole orientation coding unit can arbitrarily expand its receptive field while encoding multi-directional feature information without increasing the additional computational cost, which is beneficial for generating better global point features and thus improving semantic segmentation accuracy.

[0025] Let's examine the advantages of the point hole orientation coding unit compared to existing neighborhood selection strategies. Figure 3 This illustrates different neighborhood selection strategies for K-Nearest Neighbors (KNN) search (used in PointNet++), S8N search (used in PointSIFT), and point hole orientation encoding units. For example... Figure 3 As shown, given the centroid as the origin, KNN selects K = 8 nearest neighbors (represented by lines), 6 of which are located in the (-,-,-) quadrant. S8N selects the nearest neighbor from each of the eight quadrants (represented by circles), and these 8 neighbors are likely to be close to each other. In contrast, the point-diffuse orientation encoding unit convolves the features of the centroid in all eight quadrants and uses a flexible dilation rate (r = 2) to select neighboring points (represented by triangles), which effectively expands its receptive field without increasing the number of parameters.

[0026] 2. Constructing a point-diffused direction convolution module by stacking point-diffused direction encoding units.

[0027] Inspired by dilated convolution algorithms and the SIFT operator, a point-dilated directional convolution module is proposed, which is constructed by stacking N² point-dilated directional coding units with different dilation rates. Compared with traditional point-dilated convolution, the point-dilated directional convolution module can capture multi-scale and multi-directional point cloud contextual information by arbitrarily expanding the receptive field, thus facilitating more refined point cloud segmentation.

[0028] use Figure 2 The lower left sub-image illustrates the multi-scale perception capability of the point-dilated directional convolutional module. For each point-dilated directional convolutional module, the input features are processed through a series of point-dilated directional encoding units, transforming them into features of different scales, and then concatenated together through skip connections. Then, point-wise convolution transforms the cascaded features into d-dimensional multi-scale features. Based on this architecture, the point-dilated directional convolutional module can selectively focus on appropriate scales. Figure 2 Taking the lower left sub-image as an example, the input is a 3D point cloud, where all points are considered centroids. For each centroid, by setting K=8, the point-orifice orientation convolutional module updates the features of all points once after they pass through a point-orifice orientation encoding unit, feeding the updated point features with different dilation rates to the next point-orifice orientation encoding unit. After passing through all three point-orifice orientation encoding units, a feature block is obtained. Then, the features output by the first point-orifice orientation encoding unit, the features output by the second point-orifice orientation encoding unit, and the features after passing through three point-orifice orientation encoding units are concatenated and fed into a pointwise convolution operation to obtain the output features. Therefore, higher-level point-orifice orientation encoding units have a larger receptive field than lower-level point-orifice orientation encoding units, and they are jointly optimized to adapt to different scales. In summary, the point-orifice orientation convolutional network has multi-scale perception capabilities and can effectively extract multi-scale local edge features of point clouds.

[0029] 3. Edge-preserving pooling / edge-preserving non-pooling modules

[0030] 1) Edge-preserving pooling module

[0031] This module extracts edge features by connecting the features of the centroid and its neighboring points. This module effectively captures local geometric details while preserving the individual features of each point. The edge-preserving pooling module can be described as follows:

[0032]

[0033] Where q1,...,q K x is a selected neighborhood of point p. p ,x qK Let p and q represent points respectively. KThe point-by-point features are obtained. During the downsampling process, the centroid is first selected using the farthest point sampling algorithm. Then, edge-preserving pooling is used to explicitly propagate the features of each centroid and its local neighborhood features, thereby gradually expanding the receptive field by reducing the feature density of the three-dimensional points.

[0034] 2) Edge-preserving non-pooling modules

[0035] During upsampling, to recover the spatial information of 3D points, skip connections are used to connect the point features of layer L in the decoder to the point features of layer L-1 in the encoder. These points are then used as the centroids of layer L in the decoder, and their neighborhoods are selected using inverse distance weighted interpolation (IDWI) to update the centroid features. Unlike PointNet++, the EU module does not need to consider the d-dimensional coordinates associated with each point. The edge-preserving non-pooling module can be described as follows:

[0036]

[0037] Where p is the centroid, and p′ is the output point of the corresponding PASIFT module connected by jumps from point p. q1,...,q K Let p be the K nearest neighbors of the point in the previous PASIFT module, and g′() be the IDWI operation. The edge-preserving non-pooling operation gradually recovers the feature density and spatial information of the edge features of the high-dimensional points by considering the spatial position of the 3D points.

[0038] 4. Design a deep hierarchical encoding and decoding semantic segmentation network architecture

[0039] For a given input point cloud, the point-dilated orthogonal convolutional network employs a deep hierarchical encoder-decoder architecture to learn its multi-scale features, following an encoder-decoder architecture and alternately using point-dilated orthogonal convolutional modules. Figure 1 (bottom left box) and edge-preserving pooling / edge-preserving non-pooling modules ( Figure 1 (The bottom right box). The overall network structure is as follows: Figure 1 As shown.

[0040] In the encoder section (i.e.) Figure 1 The downsampling process (corresponding to the left side) alternates between point-dilated directional convolutional modules and edge-preserving pooling modules N1 times according to different downsampling rates. The point-dilated directional convolutional module consists of multiple point-dilated directional encoding units with different dilation rates. Then, edge-preserving pooling is used to retain key edge features, thereby hierarchically utilizing and preserving multi-scale local geometric details. Three skip connections are added to the network, allowing features from corresponding layers in the encoding process to be directly passed to the decoding layer. In the decoder part (i.e.... Figure 1The upsampling process (corresponding to the right side) alternates between point-dip orientation convolutional modules and edge-preserving non-pooling modules N1 times according to different downsampling rates. The point-dip orientation convolutional modules consist of point-dip orientation encoding units with different dilation rates. Then, edge-preserving non-pooling operations are used to consider the 3D point feature positions, gradually recovering the spatial information of the sparse high-dimensional point features. Finally, the point features of the last upsampling layer are used for point-by-point semantic label prediction through a fully connected layer.

[0041] Step 3: Adjust network parameters and train and test the neural network.

[0042] 1. Model Details

[0043] In the encoding process of the point-dilated directional convolutional network, three point-dilated directional convolutional modules and three edge-preserving pooling modules are used alternately. Each point-dilated directional convolutional module has a dilation rate of 1, 2, and 4, respectively. Each edge-preserving pooling module has 64, 128, and 512 feature channels and 1024, 256, and 64 output points, respectively. In the decoding process of the point-dilated directional convolutional network, three point-dilated directional convolutional modules and three edge-preserving non-pooling modules are used alternately. Each point-dilated directional convolutional module has a dilation rate of 4, 2, and 1, respectively. Each edge-preserving non-pooling module has 512, 256, and 128 feature channels and 256, 1024, and 8192 output points, respectively. The number of skip connection operators from the encoder to the decoder is N1 = 3. The output matrix size of the fully connected layer is 8192 × C, where C represents the number of semantic categories. In each point-diffusing direction convolutional module, the number of neighboring points searched by each point-diffusing direction coding unit is set to K=8. In each point-diffusing direction convolutional module of the encoder, the number of stacked point-diffusing direction coding units is set to N²=3. In each point-diffusing direction convolutional module of the decoder, the same number of stacked point-diffusing direction coding units is set.

[0044] 2. Parameter Settings

[0045] Implemented on a single NVIDIA TITAN X PGPU using the TensorFlow architecture. The Adam optimizer was used with an initial learning rate of 0.001. For the ScanNet dataset, the maximum number of iterations was set to 1000, the number of decay steps to 20k, the decay rate to 0.7, and the batch size to 8. For the S3DIS dataset, the maximum number of iterations was set to 200, the momentum to 0.9, the number of decay steps to 30k, the decay rate to 0.5, and the batch size to 8. Attached Figure Description

[0046] Figure 1 This is a flowchart of point cloud semantic segmentation.

[0047] Figure 2 This is a diagram of the overall network structure of a point-drilled convolutional network.

[0048] Figure 3 This is a comparison of k-nearest neighbors with S8N and point hole orientation coding units.

[0049] Figure 4 Visualization on the S3DIS dataset

[0050] Figure 5 Three close-up example images from the S3DIS dataset. Detailed Implementation

[0051] This invention experimentally validates the above method, evaluating its performance on point cloud semantic segmentation tasks using both the S3DIS and ScanNet datasets. Furthermore, some semantic segmentation results are visualized. The furthest-point sampling algorithm is applied to sub-sampling points, following the same training and testing settings as in PointNet.

[0052] 1. Qualitative assessment

[0053] Table 1 shows the quantitative results on ScanNet. Compared with other competitive methods, the Point-Oscillating Oriented Convolutional Network achieves an overall semantic segmentation accuracy of 86.8% on ScanNet, improving upon other methods by at least 0.6%. Table 2 shows the quantitative results on S3DIS. It can be seen that the Point-Oscillating Oriented Convolutional Network significantly improves the semantic segmentation results on this dataset, especially achieving a mean intersection-over-union (mIOU) of 68.3%. Although the overall accuracy of the Point-Oscillating Oriented Convolutional Network is not the best, considering that SPG and 3P-RNN gain an unfair advantage by applying specific preprocessing or post-processing techniques (such as superplots or recurrent neural networks), the results of the Point-Oscillating Oriented Convolutional Network without additional processing are still acceptable.

[0054] Table 3 shows the Cross-Intersection Over-Union (IoU) results for each category on S3DIS. It can be seen that the Point-Dip Oriented Convolutional Network (PONCNN) performs best in terms of mIOU, achieving the best semantic segmentation results in the four categories of windows, doors, tables, and clutter. While PointCNN achieves the best results in the other four categories, its results are lower in the bookshelf category (IOU = 39.1%). In contrast, the Point-Dip Oriented Convolutional Network (PONCNN) achieves satisfactory results in all categories and good results in some difficult categories, such as clutter (IOU = 66.8%) and boards (IOU = 59.6%).

[0055] Table 1 Overall accuracy on the ScanNet dataset

[0056] Method Accuracy (%) 3DCNN 73.0 PointNet 73.9 PointNet++ 84.5 3P-RNN 76.5 PointSIFT 86.2 Ours 86.8

[0057] Table 2 Overall accuracy and mIOU on the S3DIS dataset

[0058] Method Accuracy (%) mIOU (%) PointNet 78.6 47.7 3P-RNN 86.9 56.3 SPG 85.5 62.1 ASIS 86.2 59.3 PAG 88.1 65.9 MP-Net 86.8 61.3 Ours 86.5 68.3

[0059] Table 3. IOU results for each category on the S3DIS dataset.

[0060]

[0061] 2. Subjective effect

[0062] The semantic segmentation results of point clouds on the S3DIS dataset were visualized using Tensorboard. For example... Figure 4 As shown, each row displays two samples, where odd-numbered columns represent the results of the point-oriented convolutional network (POC), and even-numbered columns represent the ground truth. The POC generally captures the overall layout of an interior room correctly, achieving satisfactory point cloud semantic segmentation results. However, in some categories, the semantic segmentation effect remains unsatisfactory, such as clutter, windows, and pillars. For example, in the examples on the left of the first row, a window is classified as a door or wall, and a chair is identified as clutter. In the examples on the right of the first row, a bookshelf cannot be correctly identified. In the examples on the right of the second row, a pillar is incorrectly identified. In the examples on the left of the third row, a pillar is incorrectly identified as a ceiling. In summary, the POC achieves correct semantic segmentation results in most cases, but often fails to distinguish between a few very similar semantic parts (e.g., windows and walls).

[0063] Figure 5Three close-up examples of the results of the Point-Oriented Convolutional Network (PON) on S3DIS are presented. The left sub-image shows the PON result for the entire scene, while the right sub-image shows close-up results for two details (from top to bottom: PointNet's result, the PON result, and the ground truth). In the first example, PointNet incorrectly identifies a table in the corner of the conference room as clutter, fails to recognize a window on the back wall, and has unclear chair leg boundaries. In contrast, the PON successfully identifies the outlines of the table and some windows and essentially segments the entire chair. In the second example, PointNet identifies several points near the corner of the wall (top left frame) as clutter and a large number of points near the chair (bottom right frame), with most of the misclassified points located on the back of the chair. In contrast, the PON misclassifies a small number of points as clutter, demonstrating that the PON is less affected by surrounding objects and can selectively focus on appropriate scales to achieve optimal semantic segmentation results. In the example of the third corridor, PointNet confused doors with clutter and identified windows as doors. In contrast, PointNet's point-dipping orienting convolutional network achieved better performance in these cases, better distinguishing objects with similar shapes but different color information.

Claims

1. A point cloud semantic segmentation method based on point-drilled direction convolution, characterized in that: The input is raw 3D point cloud data. First, the point cloud data in the dataset is processed for model training. Point hole orientation encoding units are designed to establish a deep network model based on local neighborhood feature extraction. Then, the processed point cloud data is fed into the network to train the parameters of the established network. The trained model is tested, and finally, the point cloud semantic segmentation label results are output. The specific steps are as follows: Step 1: Process the point cloud data in the dataset. Directly input unordered point cloud data, and use the unordered points and their features, namely the 3D features of ScanNet and the 9D features of S3DIS, as the input to the network. A training set was prepared, and the point-based rooms and sample rooms were divided into... The blocks; several points are randomly selected from each block for training, and all points are used for testing; Step 2: Establish a deep network model based on local neighborhood feature extraction 1) Design point void direction coding unit A point hole orientation coding unit involves two hyperparameters: sampling rate and neighborhood size , making and For a given centroid point The point void direction encoding unit searches within each quadrant. The Choose from the nearest neighbor points The multi-directional neighborhood points are updated, and the centroid features are updated using the following equation (1): in It is a point Selected neighborhood points, Representing points respectively Point-by-point features; For edge kernels implemented by shared multilayer perceptron operations, This is the concatenation operation between two vectors. It is a max pooling function; Next, we will explain the neighborhood selection strategy for the hole orientation coding unit; when When, the hole orientation encoding unit selects the nearest neighbor point in each quadrant; when When, it selects the second nearest neighbor point in each quadrant; when At that time, it selects the second nearest neighbor and the fourth nearest neighbor in each quadrant; 2) Stacked point-diffusing direction encoding units construct point-diffusing direction convolutional modules A point-drilled directional convolution module is proposed, which combines convolutions with different dilation rates. It is constructed by stacking together point-hole orientation coding units; For each point-dilated directional convolutional module, the input features are processed through a series of point-dilated directional encoding units, transforming them into features of different scales. These features are then concatenated together via skip connections. Finally, point-wise convolutions transform the cascaded features into... Multidimensional scale features; The input is a 3D point cloud, where all points are considered centroids; for each centroid, settings are configured... The point-directed dilated convolution module updates the features of all points once after they pass through a point-directed dilated coding unit, and feeds the updated point features into the next point-directed dilated coding unit with different dilation rates. After it passes through all three point-directed dilated coding units, a feature block is obtained. Then, the features output by the first point-directed dilated coding unit, the features output by the second point-directed dilated coding unit, and the features after passing through three point-directed dilated coding units are concatenated and fed into a point-wise convolution operation to obtain the output features. 3) Edge-preserving pooling / edge-preserving non-pooling modules 3.1 Edge-preserving pooling module This module extracts edge features by connecting the features of the centroid and its neighboring points. The edge-preserving pooling module is described as follows: in It is a point Selected neighborhood points, Representing points respectively The point-by-point features; during the downsampling process, the centroid is first selected using the farthest point sampling algorithm, and then edge-preserving pooling is used to explicitly propagate the features of each centroid and its local neighborhood features, thereby gradually expanding the receptive field by reducing the feature density of the three-dimensional points; 3.2 Edge-preserving non-pooling module During upsampling, the first step in the decoder is achieved through skip connections. Point features of layer and the first layer in the encoder The point features of the layer are connected; then, these points are used as the first layer in the decoder. The centroids of the layers are determined, and their features are updated by selecting their neighborhoods using inverse distance weighted interpolation (IDWI) operations; the edge-preserving non-pooling module is described as follows: in It is the center of mass. These are the points in the corresponding PASIFT module. Output points connected via jumps; It is a point In the previous PASIFT module The nearest neighbor, It is an IDWI operation; 4) Design a deep hierarchical encoding and decoding semantic segmentation network architecture For a given input point cloud, the point-dilation orientation convolutional network employs a deep hierarchical encoder-decoder architecture to learn its multi-scale features. Following the encoder-decoder architecture, the point-dipping direction convolution module and the edge-preserving pooling / edge-preserving non-pooling module are used alternately; In the encoder section, point-drilled convolutional modules and edge-preserving pooling modules are used alternately according to different downsampling rates. The point-diffuse direction convolution module is composed of multiple point-diffuse direction encoding units with different dilation rates. Then, edge-preserving pooling is used to preserve key edge features, thereby hierarchically utilizing and preserving multi-scale local geometric details. Three skip connection operations are added to the network to directly pass the corresponding layer features in the encoding process to the decoding layer through skip connections. In the decoder section, point-dip orientation convolutional modules and edge-preserving non-pooling modules are used alternately according to different downsampling rates. Next, the point-drilled direction convolutional module is composed of point-drilled direction encoding units with different dilation rates; then, edge-preserving non-pooling operation is used to consider the three-dimensional point feature positions, gradually recovering the spatial information of sparse high-dimensional point features; finally, the point features of the last upsampling layer are used to predict point-by-point semantic labels through a fully connected layer. Step 3: Adjust network parameters and train and test the neural network. In the encoding process of the point-dilated directional convolutional network, three point-dilated directional convolutional modules and three edge-preserving pooling modules are used alternately. Each point-dilated directional convolutional module has a dilation rate of 1, 2, and 4, respectively. Each edge-preserving pooling module has 64, 128, and 512 feature channels and 1024, 256, and 64 output points, respectively. In the decoding process of the point-dilated directional convolutional network, three point-dilated directional convolutional modules and three edge-preserving non-pooling modules are used alternately. Each point-dilated directional convolutional module has a dilation rate of 4, 2, and 1, respectively. Each edge-preserving non-pooling module has 512, 256, and 128 feature channels and 256, 1024, and 8192 output points, respectively. The number of jump connection operators from the encoder to the decoder is... The size of the output matrix of the fully connected layer is ,in This represents the number of semantic categories; in each point-dilated orientation convolutional module, the number of neighborhood points searched by each point-dilated orientation encoding unit is set to... In each point-drilled direction convolutional module of the encoder, the number of stacked point-drilled direction coding units is set to... In each point-dilated direction convolutional module of the decoder, the same number of stacked point-dilated direction coding units are set. The implementation uses the TensorFlow architecture and is run on a single NVIDIA TITAN XP GPU. The Adam optimizer is used with an initial learning rate of 0.

001. For the ScanNet dataset, the maximum number of iterations is set to 1000, the number of decay steps is 20k, the decay rate is 0.7, and the batch size is 8. For the S3DIS dataset, the maximum number of iterations is set to 200, the momentum is 0.9, the number of decay steps is 30K, the decay rate is 0.5, and the batch size is 8.

Images