Large-scale point cloud semantic segmentation method based on deep learning

Abstract

The invention discloses a large-scale point cloud semantic segmentation method based on deep learning. The method comprises the following steps: 1) preprocessing large-scale point cloud data; the preprocessing comprises grid division processing according to a set density; 2) randomly selecting a point from each grid for reservation, removing other points, and obtaining preprocessed point cloud data; and 3) inputting the preprocessed point cloud data into a deep learning network for semantic segmentation to realize final point cloud semantic segmentation. Through a point cloud grid sparse strategy and a feature extraction function of model optimization, the method is suitable for large-scale point cloud semantic segmentation, and meanwhile, the point cloud segmentation performance is optimized.

Description

Technical Field

[0001] The present invention relates to point cloud semantic segmentation technology, and in particular to a large-scale point cloud semantic segmentation method based on deep learning. Background Art

[0002] Current 3D point cloud semantic segmentation technologies can be divided into traditional methods and methods based on deep learning strategies. In deep learning-based strategies, methods can be further classified into projection-based, voxel-based, and point-based methods. Among them, the projection-based method converts 3D point clouds into 2D views in order to apply mature 2D convolutional neural networks. Voxel-based technology converts point cloud data into voxel grids and uses 3D convolution for feature extraction.

[0003] In the field of point cloud semantic segmentation with deep learning, projection-based methods face challenges such as occlusion and overlap; voxel-based methods often lead to data redundancy and increased computational requirements, making them less suitable for large-scale data sets; grid-based random thinning technology improves the efficiency of point cloud preprocessing, but due to its fixed grid size, it fails to solve the problem of uneven distribution of object categories in large-scale point clouds.

[0004] Given that large-scale point clouds usually include hundreds of millions of points, it is impractical to process these data directly. Therefore, it is particularly important to select a representative subset of points to reduce processing complexity during the data preprocessing stage before model training. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a large-scale point cloud semantic segmentation method based on deep learning in view of the defects in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a large-scale point cloud semantic segmentation method based on deep learning, comprising the following steps:

[0007] 1) Preprocessing the large-scale point cloud data; the preprocessing includes meshing according to a set density; the details are as follows:

[0008] Determine the maximum and minimum values ​​of the three-dimensional coordinates of each point in the point cloud data;

[0009] Based on the extreme values ​​of the three-dimensional coordinates of each point in the point cloud data, the grid size is set, and the total number of grids in the x, y, and z axes is calculated;

[0010] Assign an index label to each grid based on the coordinate depth information;

[0011] The number of points in each grid is evaluated. If the number of points in a grid exceeds the set threshold, the grid is divided into two sub-grids until the number of points in all grids does not exceed the threshold.

[0012] 2) Randomly select a point in each grid and keep it, and remove the rest of the points to obtain the preprocessed point cloud data;

[0013] 3) Input the preprocessed point cloud data into a deep learning network for semantic segmentation to achieve the final point cloud semantic segmentation;

[0014] The network sequentially includes a fully connected layer, four encoding layers, a multi-layer perceptron MLP, four decoding layers corresponding to the four encoding layers, and a classifier;

[0015] Among them, there are four coding layers, each of which consists of a RS module and a LAA module;

[0016] Each of the four decoding layers has an upsampling (US) module and an attention residual (AR) module connected to the corresponding encoding layer.

[0017] According to the above scheme, in step 3), the feature dimension of the network is [Batch, Number, 6] when it is initialized. The point cloud passes through a fully connected layer, and its feature dimension is adjusted to the size [B, N, 8]. The point cloud data will undergo a step-by-step downsampling process when passing through 4 RS modules, and the number of feature channels is adjusted accordingly: from [B, N, 8] to [B, N / 4, 32], [B, N / 16, 128], [B, N / 64, 256], [B, N / 256, 512]. ([B, N, 6] → [B, N / 4, 32] → [B, N / 16, 128] → [B, N / 64, 256] → [B, N / 256, 512]).

[0018] According to the above scheme, in step 3), the point cloud data is randomly downsampled by the RS module and then input into the LAA module; the processing process of the LAA module is as follows:

[0019] First, the K nearest neighbor point cloud blocks P = {p1, p2, ..., p k};

[0020] Next, the coordinate features of these points are encoded and the block center point p is calculated. i Its neighboring points The relative position information between rel，k and absolute distance dist k ;

[0021] P rel，k =P coor,k-P coor,i ;

[0022] Among them, p coor，i is the center point coordinate, p coor，k For neighboring points The coordinates of

[0023] dist k =||p coor，k -p coor，i ||;

[0024] The relative position information and absolute distance data are combined with the center point coordinates p coor，i and its neighboring point coordinates p coor，k Combined and processed through a convolutional layer to extract the relative position vector f pos，k ;

[0025]

[0026] f pos，k =Conv(r k );

[0027] in, represents a concatenation operation, ||·|| represents the absolute value distance calculation, and Conv represents a convolution operation.

[0028] Afterwards, the resulting relative position vector is merged with the corresponding point features and sum pooling is performed.

[0029] According to the above scheme, a multi-layer perceptron MLP is used to process the output of the encoding layer to generate new neighboring features, thereby accurately encoding the local geometric structure of the point cloud.

[0030] According to the above scheme, in the decoding layer, the function of the four US modules is to restore the feature size of the point cloud and adjust its channel size from [B, N / 256, 512] [B, N, 8] to [B, N / 64, 256], [B, N / 16, 128], [B, N /

[0031] 4,32],[B,N,8];

[0032] The AR module uses the output features of each previous layer and the relative point position vector fp generated by the LAA module os， k is taken as input, and the module processes as follows:

[0033] First, the output of the previous layer is upsampled to generate a new upsampled point cloud, which is used as the query for attention calculation;

[0034] Query = US × W q ;

[0035] At the same time, the relative point position vector f from the LAA module is obtained through a 1×1 convolutional layer. pos，k Get the Key and Value used for attention calculation;

[0036] Key = f posk ×W k ;

[0037] Value = f pos，k ×W v ;

[0038] Among them, Key is the attention key, Value is the attention value; w represents the weight matrix;

[0039] Perform attention calculation, merge the output of the attention mechanism with the upsampled point cloud, and further obtain the attention residual features through a shared multi-layer perceptron; where:

[0040]

[0041] Where d represents the number of feature channels.

[0042] According to the above scheme, in the classifier, the decoded point cloud features are further processed, and the final point cloud semantic segmentation is achieved through a three-layer fully connected network; in this process, the channel size of the point cloud features is adjusted to ([B, N, 8] → [B, N, 64] → [B, N.32] → [B, N, n_class]) to match the number of categories for classification;

[0043] n_class is the number of categories that need to be classified.

[0044] The beneficial effects produced by the present invention are:

[0045] The present invention is suitable for large-scale point cloud semantic segmentation through the point cloud grid sparsification strategy and model optimized feature extraction function, while optimizing the point cloud segmentation performance, balancing the category distribution and retaining key geometric details. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0047] Figure 1 is a method flow chart of an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of the deep learning network structure of an embodiment of the present invention;

[0049] Figure 3 is a working principle diagram of the LAA module of an embodiment of the present invention;

[0050] Figure 4 is a working principle diagram of the attention residual module of an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of test results on the Toronto3D dataset according to an embodiment of the present invention;

[0052] Figure 6 It is a schematic diagram of test results on the Semantic3D dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] like Figure 1 As shown, a large-scale point cloud semantic segmentation method based on deep learning includes the following steps:

[0055] 1) Preprocessing the large-scale point cloud data; the preprocessing includes meshing according to a set density; the details are as follows:

[0056] Determine the maximum and minimum values ​​of the three-dimensional coordinates of each point in the point cloud data;

[0057] Maximum value {x max ,y max , z max} and the minimum value {x min ,y min , z min};

[0058] x max , x min =max(x1,x2,x3...x n ), nm(x1, x2, x3...x n )

[0059] y max ,y min =max(y1,y2,y3...y n ), min(y1, y2, y3...y n )

[0060] z max , z min =max(z1,z2,z3...z n ), min(z1, z2, z3...z n )

[0061] Based on the extreme values ​​of the three-dimensional coordinates of each point in the point cloud data, the grid size r is set, and the total number of grids in the x, y, and z directions is calculated. x , D y , D z};

[0062] D x =floor(((x max -x min ) / r))

[0063] D y =floor(((y max -y min ) / r))

[0064] D z =floor(((z max -z min ) / r))

[0065] Assign an index label to each grid based on the coordinate depth information;

[0066]

[0067]

[0068]

[0069]

[0070] Among them, floor() is the rounding function, D x is the total number of grids in the x-axis direction, D y is the total number of grids in the y-axis direction;

[0071] The number of points in each grid is evaluated. If the number of points in a grid exceeds the set threshold, the grid is divided into two sub-grids until the number of points in all grids does not exceed the threshold.

[0072] 2) Randomly select a point in each grid and keep it, and remove the rest of the points to obtain the preprocessed point cloud data;

[0073] 3) Input the preprocessed point cloud data into a deep learning network for semantic segmentation to achieve the final point cloud semantic segmentation;

[0074] The network sequentially includes a fully connected layer, four encoding layers, a multi-layer perceptron MLP, four decoding layers corresponding to the four encoding layers, and a classifier; the structure is as follows Figure 2 ;

[0075] Among them, there are four coding layers, each of which consists of a RS module and a LAA module;

[0076] Each of the four decoding layers has an upsampling (US) module and an attention residual (AR) module connected to the corresponding encoding layer.

[0077] In step 3), the feature dimension of the network is initialized to [Batch, Number, 6]. The point cloud passes through a fully connected layer, and its feature dimension is adjusted to [B, N, 8]. The point cloud data undergoes a gradual downsampling process when passing through four RS modules, and the number of feature channels is adjusted accordingly: from [B, N, 8] to [B, N / 4, 32], [B, N / 16, 128], [B, N / 64, 256], [B, N / 256, 512] in sequence.

[0078] The point cloud data is randomly downsampled by the RS module and then input into the LAA module; the processing process of the LAA module is as follows: Figure 3 :

[0079] First, the K nearest neighbor point cloud blocks P = {p1, p2, ..., p k};

[0080] Next, the coordinate features of these points are encoded and the block center point p is calculated. i Its neighboring points The relative position information between rel，k and absolute distance dist k ;

[0081] P rel，k =p coor,k -P coor,i ;

[0082] Among them, p coor，i The center point p i Coordinate, p coor，k The center point p i Neighboring points The coordinates of

[0083] dist k =||p coor，k -p coor，i ||;

[0084] The relative position information and absolute distance data are combined with the center point coordinates p coor，i and its neighboring point coordinates p coor，k Combined and processed through a convolutional layer to extract the relative position vector f pos，k ;

[0085]

[0086] fpos，k =Conv(r k );

[0087] in, represents concatenation operation, ||·|| represents absolute value distance calculation, and Conv represents convolution operation;

[0088] Afterwards, the resulting relative position vector is merged with the corresponding point features and sum pooling is performed.

[0089] The multi-layer perceptron (MLP) is used to process the output of the encoding layer to generate new neighboring features, thereby accurately encoding the local geometric structure of the point cloud. The multi-layer perceptron is used to perform in-depth processing of point cloud features. The key to this link is that despite the multiple rounds of transformation and processing of the features, their dimensions remain unchanged, ensuring the integrity of the original data structure and information. The features of each point are carefully optimized to maximize the amount of information contained in them. This process is not just a data conversion, but also an intelligent feature extraction and reshaping, with the aim of providing a richer and more effective data representation for subsequent computing tasks.

[0090] In the decoding layer, the function of the four US modules is to restore the feature size of the point cloud and adjust its channel size from [B, N / 256, 512] [B, N, 8] to [B, N / 64, 256], [B, N / 16, 128], [B, N / 4, 32], [B, N, 8] in sequence;

[0091] The AR module uses the output features of each previous layer and the relative point position vector fp generated by the LAA module os， k as input, the module processing process is as follows, Figure 4 :

[0092] First, the output of the previous layer is upsampled to generate a new upsampled point cloud, which is used as the query for attention calculation;

[0093] Query = US × W q ;

[0094] At the same time, the relative point position vector fp from the LAA module is obtained through a 1×1 convolutional layer os， Get the Key and Value used for attention calculation from k;

[0095] Key = f pos，k ×W k ;

[0096] Value = f pos，k ×W v ;

[0097] Among them, Key is the attention key, Value is the attention value; W k and W v Represents the corresponding weight matrix;

[0098] Perform attention calculation, merge the output of the attention mechanism with the upsampled point cloud, and further obtain the attention residual features through a shared multi-layer perceptron; where:

[0099]

[0100] Where d represents the number of feature channels.

[0101] In the classifier, the decoded point cloud features are further processed and the final point cloud semantic segmentation is achieved through a three-layer fully connected network. In this process, the channel size of the point cloud features is adjusted to ([B, N, 8] → [B, N.64] → [B, N.32] → [B, N, n_class]) to match the number of categories to be classified; n_class is the number of categories to be classified. This step not only completes the mapping of point cloud features to semantic categories, but also optimizes the accuracy and reliability of the classification results.

[0102] Data-based effect verification:

[0103] The performance of the method of the present invention is demonstrated by a specific experimental setup and its Figure 5 and Figure 6 The actual test results in the paper are verified.

[0104] The training of this experiment involves processing 6 groups of data per batch, each containing 40,960 data points, and lasting for 100 training cycles. The initial learning rate is set to 0.01, and after each training cycle, the learning rate will be reduced to 95% of its previous cycle. The system uses the Adam optimizer configured with default parameters for model training. In addition, the preprocessing grid size used in the experiment is 0.1. K nearest neighbor search is implemented by using the KD-Tree algorithm in the Scikit-learn package, where the value of K is set to 16. In order to overcome the problem of class imbalance, a weighted cross entropy loss function is used, and the weight of each class is assigned according to its inverse frequency in the training set. All model training and testing are performed on an Ubuntu 22.04 system, equipped with PyTorch 1.13.1 and CUDA 12.4, using an NVIDIA GeForce RTX 3090 24G GPU.

[0105] The test results on the Toronto3D dataset are as follows Figure 5As shown, mIoU and IoU values ​​are calculated for each category. The highest accuracy result reported in a study published in 2021 using the RandLA-Net model was selected, and the proposed method showed significant performance improvement compared with it, with mIoU reaching 86.79%.

[0106] The proposed method significantly improves the performance of point cloud segmentation, especially in identifying road markings (81.22% accuracy), electric poles (90.45% accuracy) and fences (70.29% accuracy). These results are attributed to the model's excellent feature extraction capabilities and a density-based point cloud grid sparsification strategy, which balances the class distribution and preserves key geometric details.

[0107] The test results on the Semantic3D dataset are as follows Figure 6 As shown, the mIoU and IoU values ​​of each category are shown. This technique outperforms other methods, with an overall mIoU of 79.88%, and shows excellent accuracy in the high vegetation, hardscape, and car categories, reaching 90.35%, 60.48%, and 67.55%, respectively. These results highlight the powerful segmentation capabilities and wide applicability of this technology model.

[0108] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A large-scale point cloud semantic segmentation method based on deep learning, characterized in that: The following steps are involved: 1) Preprocessing large-scale point cloud data; the preprocessing includes grid division according to a set density; 2) Randomly select a point in each grid and keep it, and remove the rest of the points to obtain the preprocessed point cloud data; 3) Input the preprocessed point cloud data into a deep learning network for semantic segmentation to achieve the final point cloud semantic segmentation; The network sequentially includes a fully connected layer, four encoding layers, a multi-layer perceptron MLP, four decoding layers corresponding to the four encoding layers, and a classifier; Among them, there are four coding layers, each of which consists of an RS module and an LAA module; Each of the four decoding layers has an upsampling module and an attention residual module connected to the corresponding encoding layer.

2. The large-scale point cloud semantic segmentation method based on deep learning according to claim 1, characterized in that: In the step 1), the grid division process is performed according to the set density as follows: Determine the maximum and minimum values ​​of the three-dimensional coordinates of each point in the point cloud data; Based on the extreme values ​​of the three-dimensional coordinates of each point in the point cloud data, the grid size is set, and the total number of grids in the x, y, and z axes is calculated; Assign an index label to each grid based on the coordinate depth information; The number of points in each grid is evaluated. If the number of points in a grid exceeds the set threshold, the grid is divided into two sub-grids until the number of points in all grids does not exceed the threshold.

3. The large-scale point cloud semantic segmentation method based on deep learning according to claim 1, characterized in that: In the step 3), the feature dimension of the network is [Batch, Number, 6] when it is initialized. The point cloud passes through a fully connected layer, and its feature dimension is adjusted to the size [B, N, 8]. The point cloud data undergoes a gradual downsampling process when passing through four RS modules, and the number of feature channels is adjusted accordingly: from [B, N, 8] to [B, N / 4, 32], [B, N / 16, 128], [B, N / 64, 256], [B, N / 256, 512] in sequence.

4. The large-scale point cloud semantic segmentation method based on deep learning according to claim 1, characterized in that: In step 3), the point cloud data is randomly downsampled by the RS module and then input into the LAA module; the processing process of the LAA module is as follows: First, the KNN algorithm is used to construct the K nearest neighbor point cloud blocks P = {p1, p2..., p k }; Next, the coordinate features of these points are encoded and the center point p of the nearest neighbor point cloud block is calculated. i Points close to the center The relative position information between rel,k and absolute distance dist k ; p rel,k =p coor,k -p coor,i ; Among them, p coor,i is the center point coordinate, p coor,k For neighboring points The coordinates of dist k =||p coor，k -p coor，i ||; The relative position information and absolute distance data are combined with the center point coordinates p coor,i and its neighboring point coordinates p coor,k Combined and processed through a convolutional layer to extract the relative position vector f posk ; f pos，k =Conv(r k ); in, represents concatenation operation, ||·|| represents absolute value distance calculation, and Conv represents convolution operation; Afterwards, the resulting relative position vector is merged with the corresponding point features and sum pooling is performed.

5. The large-scale point cloud semantic segmentation method based on deep learning according to claim 1, characterized in that: The multi-layer perceptron MLP is used to process the output of the encoding layer to generate new neighboring features, thereby accurately encoding the local geometric structure of the point cloud.

6. The large-scale point cloud semantic segmentation method based on deep learning according to claim 3, characterized in that: In the decoding layer, the function of the four US modules is to restore the feature size of the point cloud and adjust its channel size from [B, N / 256, 512] [B, N, 8] to [B, N / 64, 256], [B, N / 16, 128], [B, N / 4, 32], [B, N, 8] in sequence; The AR module uses the output features of each previous layer and the relative point position vector f generated by the LAA module pos,k As input, the module processes as follows: First, the output of the previous layer is upsampled to generate a new upsampled point cloud, which is used as the query for attention calculation; Query=US×W q ; At the same time, the relative point position vector f from the LAA module is obtained through a 1×1 convolutional layer. pos,k Get the Key and Value used for attention calculation; Key=f pos，k ×W k ; Value=f pos，k ×W v ; Among them, Key is the attention key, Value is the attention value; W k and W v represents the corresponding weight matrix; Perform attention calculation, merge the output of the attention mechanism with the upsampled point cloud, and further obtain the attention residual features through a shared multi-layer perceptron; where: Where d represents the number of feature channels.

7. The large-scale point cloud semantic segmentation method based on deep learning according to claim 1, characterized in that: In the classifier, the decoded point cloud features are further processed, and the final point cloud semantic segmentation is achieved through a three-layer fully connected network; in this process, the channel size of the point cloud features is adjusted to ([B, N, 8] → [B, N, 64] → [B, N, 32] → [B, N, n_class]) to match the number of categories to be classified; n_class is the number of categories that need to be classified.

8. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images