Large-scale point cloud semantic segmentation method based on convolutional network and key point modeling

Abstract

The invention discloses a large-scale point cloud semantic segmentation method based on a convolutional network and key point modeling. The method comprises the following steps: firstly, dynamically extracting and enhancing local geometric features of a point cloud through a local position convolution module; secondly, capturing the long-term dependency relationship of the point cloud through a global key point module and a key point sampling method, and remarkably reducing the calculation complexity; and finally, adaptively aggregating the local and global features through a fusion module, and generating optimized point features for semantic segmentation. According to the method, the accuracy and efficiency of point cloud semantic segmentation are improved by enhancing local geometric feature representation, capturing a global long-term dependency relationship and adaptively fusing multi-scale features.

Description

Technical Field

[0001] This invention belongs to the field of point cloud semantic segmentation, specifically involving a large-scale point cloud semantic segmentation method based on convolutional networks and key point modeling. Background Technology

[0002] Point clouds, as a primary source of 3D geographic information, play a crucial role in applications such as autonomous driving, augmented reality, and intelligent robotics. Essentially, point clouds are collections of discrete sampled points on the surface of an object, exhibiting irregularity, disorder, and non-uniform density, which presents unique challenges for point cloud semantic segmentation. Unlike regularly structured 2D images, point cloud data is difficult to directly apply traditional convolutional neural networks because convolution operations rely on the order and locality of the input data. Points in point clouds are unordered and their adjacency relationships are not fixed, leading to difficulties in feature extraction.

[0003] With the development of deep learning, existing point cloud semantic segmentation methods are mainly divided into voxel-based and point-based methods. Voxel-based methods convert point clouds into regular 3D meshes and process them using 3D convolutional neural networks. Although these methods improve the regularity of processing, they lead to the loss of geometric information and excessive memory consumption, especially in large-scale scenes where voxel resolution is limited, making it difficult to balance computational efficiency and accuracy. VVNet accelerates convolution operations by encoding local geometry within voxels through variational autoencoders, but it still cannot avoid the information loss caused by voxelization. Supervoxel-CNN uses supervoxel clustering to achieve 2D-3D joint learning, improving semantic segmentation accuracy, but its preprocessing is complex and its applicability is limited.

[0004] Point-based methods directly process raw point clouds, avoiding the drawbacks of data transformation. PointNet uses a multilayer perceptron (MLP) and max pooling to extract global features, but ignores local structural information. PointNet++ captures multi-scale local features through hierarchical feature extraction and nearest neighbor search, improving the understanding of complex scenes. However, it struggles to effectively model long-term dependencies when processing large-scale point clouds and is susceptible to noise and occlusion. Subsequent work, PointMLP, improves 3D relationship capture through a geometric transformation module, but still has limitations in large-scale urban scenes.

[0005] For large-scale point cloud processing, SPGraph represents the point cloud as a supergraph to encode contextual relationships, avoiding the loss of details, but requires complex preprocessing. TangentConv improves efficiency by projecting local point clouds through tangent convolution, but fails to capture global semantics. RandLA-Net uses random sampling to reduce computational complexity and introduces local spatial encoding and attention pooling, but random sampling may lead to the loss of geometric information, and the model does not pay enough attention to global context. The Transformer model is introduced into point cloud processing to capture long-term dependencies, but the computational complexity of global self-attention is high. However, it cannot be directly applied to massive point clouds. The Fast Point Transformer reduces complexity through downsampling, but voxel processing introduces feature loss, resulting in unsatisfactory performance in noisy outdoor scenes. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention proposes a large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling (Local Position Conv with Global Key-Point Transformer, LPGPFormer). This method aims to improve the accuracy and efficiency of point cloud semantic segmentation by enhancing local geometric feature representation, capturing global long-term dependencies, and adaptively fusing multi-scale features. First, the method dynamically extracts and enhances the local geometric features of the point cloud through a local position convolutional module. Then, it captures the long-term dependencies of the point cloud through a global keypoint module and keypoint sampling methods, significantly reducing computational complexity. Finally, a fusion module adaptively aggregates local and global features to generate optimized point features for semantic segmentation.

[0007] The technical solution adopted in this invention is: A large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling includes the following steps: Step 1: Construct neighborhood features; Step 2, Local position weight convolution feature extraction: Local geometric features of the point cloud are dynamically learned through Local position weight convolution (LPWC) to enhance the representation ability of local point cloud features; Step 3, Global Key-Point Attention Modeling: The Global Key-Point Transformer (GKPT) is used to capture long-term dependencies in the point cloud, and the Key-Point Sample (KPS) method is used to reduce computation and simulate the global context. Step 4: Multi-scale feature adaptive fusion: By combining local and global features through the fusion module (Fusion), weights are adaptively allocated to achieve feature refinement; Step 5, Encoder Structure Design: The execution steps from Step 2 to Step 4 are denoted as LPGPFormer. LPGPFormer is executed repeatedly in sequence. Each execution is recorded as one layer of encoding module. Before each layer of encoding module, the number of points in the point cloud is downsampled using the farthest point sampling algorithm, and the neighborhood features are reconstructed from the downsampled point cloud in step one. Step 6: Network Decoding and Semantic Segmentation Result Calculation: In the decoding stage, encoder features are fused through layer-by-layer upsampling and skip connections to recover the number of point clouds and enhance semantic information. Finally, the class probability distribution of each point is output through MLP and Softmax activation functions, thereby outputting the final semantic segmentation result.

[0008] Furthermore, the process of step one is as follows; Step 1.1) Define the point cloud dataset Point cloud data consists of a dense set of three-dimensional points, where each point has a similar representational meaning to its neighboring points. Given a point cloud dataset containing N points... Feature set ,in The number of channels for the input feature. Let i be the i-th point in the point cloud dataset. The features of the i-th point; Step 1.2) Obtain neighborhood features Feature extraction of local point clouds is performed using the KNN neighborhood search algorithm from machine learning. As the i-th sampling point, it has a total of K neighboring points. Therefore, the j-th neighboring point of the i-th sampling point can be represented as... Its characteristics can be expressed as .

[0009] Furthermore, the process of step two is as follows; Step 2.1) Obtain position weights The points obtained in step 1) Neighboring points As input, the weights between local point clouds are learned using their geometric prior knowledge. Specifically, the sampling points coordinates, neighboring points The coordinates of the two points, the relative position vector between them, and the three-dimensional Euclidean distance are concatenated, and then the features are mapped through a multilayer perceptron, thereby more flexibly describing the spatial geometry of point cloud objects and enhancing geometric feature representation: ; Where ⊕ represents vector concatenation, Represents the Euclidean norm, i.e., the sampling point. and neighboring points The three-dimensional Euclidean distance between them.

[0010] Step 2.2) Perform local position weight convolution To explicitly reflect the correlation of local point clouds, local location-weighted convolution is used. First, the sampling points are calculated. To nearby points The difference of coordinate vectors and through a scalar function Process it and combine it with the sampling point-based data. and neighboring points Position weight term and scalar functions Transformed sampling point features Perform the product operation, and finally sum the interaction results of all sampling point ranges N and neighboring point ranges K, as shown in the following formula: ; in, , These represent the coordinates of the sampling point and its neighboring points, respectively. Indicates the characteristics of the sampling points, and These are two scalar functions approximated by MLP; Step 2.3) Local Feature Acquisition To avoid excessive resource consumption and computational burden, the feature vector difference between local points and their corresponding query points is utilized. This is used to obtain weights, and points with high similarity values ​​are assigned higher weights. Based on the given LPWC module, the calculated local location weight convolution result is compared with the feature vector difference between the local point and its corresponding query point. Add them together to obtain the final local features. The formula is as follows: .

[0011] Furthermore, the process of step three is as follows; Step 3.1) Key point sampling: KPS selects points that are more representative of global semantic modeling by fusing semantic features and spatial geometric information while maintaining semantic integrity; Step 3.2) Key points in the use of Transformer; Step 3.3) Global Feature Calculation: Multiply the attention weights by the value, reducing the computational complexity from... Down to To achieve efficient long-term dependency modeling and obtain global features : .

[0012] Preferably, the process of step 3.1) is as follows: Step 3.1.1) Spatial distribution modeling Farthest Point Sampling (FPS) was used to sample the original point cloud using the classic sampling method. Select a subset , as the initial sampling point. For each sampling point The KNN algorithm is used to find its local neighborhood points and generate an index mapping: ; in, Point The set of indices of the K nearest neighbors, the neighborhood set is defined as: ; in This indicates that the region containing the K nearest points is obtained by querying different indexes i; Step 3.1.2) Semantic kernel function design To explicitly capture the spatial distribution geometry of point clouds, a radial basis function (RBF) kernel is introduced for spatial encoding, and the point cloud is calculated. to every point in its neighborhood The sum of the squares of the Euclidean distances is multiplied by a negative scaling factor for scaling and inversion. An exponential function is then applied to obtain the local response at that point. Finally, the local responses of all points are aggregated into a global metric, i.e., the spatial kernel function. , is represented as: ; in This indicates adjusting the value of the receiving space's receiving domain. If A larger value indicates a smaller spatial receptive field. Indicates exponentiation; Step 3.1.3) Keypoint Generation Use learnable matrices Point features With spatial kernel function Fusion to generate a semantic weight matrix : ; in It is the transpose of the characteristic matrix; Then semantic weights Point cloud coordinates Multiplication, from the point cloud Calculate the key points , where n represents the number of key points. Represented as: .

[0013] The process of step 3.2) is as follows: Step 3.2.1) Feature Projection Original point features and key points Features Projected as query (Q), key (K), and value (V) respectively: ; in , , It is a learnable weight matrix; Step 3.2.2) Self-attention calculation To utilize keypoint generation and To refer to global point information, first calculate the query matrix. Key point key matrix The product of the transposes is used to measure the correlation, and then the result is divided by the dimension. The square root of the value is used for scaling adjustment, and finally, the softmax function is applied for normalization to calculate the attention weights. , is represented as: ; in It is a scaling factor to prevent the inner product from becoming too large. express The transpose of .

[0014] The process of step four is as follows: The local features obtained in step 2) And the global features obtained in step 3) As input, and by passing local features as input to step 3), we obtain the features. Ultimately, the network adaptively assigns different weights to the features. This can be described as follows: .

[0015] in, For learnable parameters, The representations are generated based on point-based attention and local point attention. Finally, MLP is used for batch normalization and residual connections to extract high-level point-by-point representations.

[0016] The process of step five is as follows: The execution steps from step two to step four are denoted as LPGPFormer. LPGPFormer consists of three main modules: LPWC, GKPT, and Fusion. LPGPFormer is executed repeatedly in this order. Each execution is recorded as one layer of encoding module. Before each layer of encoding module, the number of points in the point cloud is downsampled using the farthest point sampling algorithm to ensure the uniformity of the samples. Then, the neighborhood features of the downsampled point cloud are reconstructed in step one to ensure the accuracy and effectiveness of subsequent processing. Farthest point sampling is a commonly used sampling algorithm that is widely used in the field of point cloud data processing.

[0017] The process of step six is ​​as follows: The decoding module employs a layer-by-layer upsampling strategy, aiming to gradually restore sparse point cloud features to their original resolution. This process is achieved using the feature upsampling strategy in 3D point cloud processing and an inverse distance weighted interpolation algorithm.

[0018] Preferably, in step six, the Euclidean distance between two adjacent point cloud layers is first calculated to establish a spatial correspondence, and each sampling point of the layer to be recovered is selected from the previous layer. The target point is then identified by finding its nearest neighbors. Subsequently, a weighted aggregation based on the inverse distance is performed on these neighbors, multiplying the neighborhood feature by the inverse of its distance, and summing the results after normalization to calculate the semantic features of the target point, thus completing the feature transfer from sparse to dense. Building upon this, a skip connection mechanism is introduced to concatenate the interpolated features with features of the same scale from the encoder along the channel dimension, fusing deep semantics with shallow details. This process is repeated iteratively. The process continues until the full resolution is restored. The resulting high-level semantic features are then processed by MLP dimensionality reduction and Softmax activation function to output a semantic label for each point, thus completing the segmentation task.

[0019] The beneficial effects of this invention are mainly reflected in the following aspects: 1. This invention introduces a local position convolution module, which dynamically learns the position weights between point clouds by utilizing the geometric prior knowledge between local point clouds. This enables a more flexible description of the spatial geometric structure of point cloud objects, thereby significantly enhancing the representation ability of local geometric features. This module avoids the incompatibility of traditional convolution kernels on point clouds and effectively captures the detailed information of sparsely distributed point clouds through weight learning, improving the model's perception accuracy of local context in complex urban scenes. 2. This invention designs a global keypoint module and proposes a semantic space keypoint sampling method. By fusing semantic features and spatial coordinates to adaptively filter keypoints, the computational complexity of the global attention of the Transformer is reduced from... Down to While significantly reducing computational resource consumption, this invention effectively simulates long-term dependencies in point clouds, enhances the ability to capture global context features, and significantly suppresses noise and occlusion interference. 3. This invention dynamically combines local and global features through a fusion module and introduces learnable parameters to adaptively allocate hybrid weights, achieving efficient complementarity between low-level geometric features and high-level semantic features. This improves the overall effect and efficiency of point cloud feature refinement, enabling the model to maintain high-precision segmentation in large-scale scenes. Attached Figure Description

[0020] Figure 1 This is a flowchart of the network structure.

[0021] Figure 2 This is a schematic diagram of key point downsampling.

[0022] Figure 3 It is the point cloud semantic segmentation result of the method of this invention and other methods. Detailed Implementation

[0023] The present invention will now be further described with reference to the accompanying drawings.

[0024] Reference Figures 1-3 A large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling includes the following steps: Step 1: Construct neighborhood features, the process is as follows: Step 1.1) Define the point cloud dataset Point cloud data consists of a dense set of three-dimensional points, where each point has a similar representational meaning to its neighboring points. Given a point cloud dataset containing N points... Feature set ,in The number of channels for the input feature. Let i be the i-th point in the point cloud dataset. The features of the i-th point; Step 1.2) Obtain neighborhood features Feature extraction of local point clouds is performed using the KNN neighborhood search algorithm from machine learning. As the i-th sampling point, it has a total of K neighboring points; then, the j-th neighboring point of the i-th sampling point can be represented as Its characteristics can be expressed as .

[0025] Step 2: Local convolutional feature extraction, the process is as follows: Step 2.1) Obtain position weights The points obtained in step 1) Neighboring points As input, the weights between local point clouds are learned using their geometric prior knowledge. Specifically, sampling points coordinates, neighboring points The coordinates of the two points, the relative position vector between them, and the three-dimensional Euclidean distance are concatenated, and then the features are mapped through a multilayer perceptron, thereby more flexibly describing the spatial geometry of point cloud objects and enhancing geometric feature representation: ; Where ⊕ represents vector concatenation, Represents the Euclidean norm, i.e., the sampling point. and neighboring points The three-dimensional Euclidean distance between them.

[0026] Step 2.2) Perform local position weight convolution To explicitly reflect the correlation of local point clouds, this embodiment proposes local location weighted convolution, first calculating the sampling points... To nearby points The difference of coordinate vectors and through a scalar function Process it and combine it with the sampling point-based data. and neighboring points Position weight term and scalar functions Transformed sampling point features Perform the product operation, and finally sum the interaction results of all sampling point ranges N and neighboring point ranges K, as shown in the following formula: ; in, , These represent the coordinates of the sampling point and its neighboring points, respectively. Indicates the characteristics of the sampling points, and These are two scalar functions approximated by MLP; Step 2.3) Local Feature Acquisition To avoid excessive resource consumption and computational burden, this embodiment utilizes the feature vector difference between local points and their corresponding query points. This is used to obtain weights, and points with high similarity values ​​are assigned higher weights. Based on the given LPWC module, the calculated local location weight convolution result is compared with the feature vector difference between the local point and its corresponding query point. Add them together to obtain the final local features. The formula is as follows: Step 3: Global keypoint attention modeling, the process is as follows: Step 3.1) Key point sampling like Figure 2 KPS, by fusing semantic features and spatial geometric information, selects points that are more representative of global semantic modeling while maintaining semantic integrity. The process is as follows: Step 3.1.1) Spatial distribution modeling Farthest Point Sampling (FPS) was used to sample the original point cloud using the classic sampling method. Select a subset As the initial sampling point, for each sampling point The KNN algorithm is used to find its local neighborhood points and generate an index mapping: ; in, Point The set of indices of the K nearest neighbors, the neighborhood set is defined as: ; in, This indicates that the region containing the K nearest points is obtained by querying different indexes i; Step 3.1.2) Semantic kernel function design To explicitly capture the spatial distribution geometry of point clouds, a radial basis function (RBF) kernel is introduced for spatial encoding, and the point cloud is calculated. to every point in its neighborhood The sum of the squares of the Euclidean distances is multiplied by a negative scaling factor for scaling and inversion. An exponential function is then applied to obtain the local response at that point. Finally, the local responses of all points are aggregated into a global metric, i.e., the spatial kernel function. , is represented as: ; in, This indicates adjusting the value of the receiving space's receiving domain. If A larger value indicates a smaller spatial receptive field. Indicates exponentiation; Step 3.1.3) Keypoint Generation Use learnable matrices Point features With spatial kernel function Fusion to generate a semantic weight matrix : ; in, It is the transpose of the characteristic matrix; Then semantic weights Point cloud coordinates Multiplication, from the point cloud Calculate the key points , where n represents the number of key points. Represented as: ; Step 3.2) The key point is the use of Transformer, the process is as follows: Step 3.2.1) Feature Projection Original point features and key points Features Projected as query (Q), key (K), and value (V) respectively: ; in, , , It is a learnable weight matrix.

[0027] Step 3.2.2) Self-attention calculation To utilize keypoint generation and To refer to global point information, first calculate the query matrix. Key point key matrix The product of the transposes is used to measure the correlation, and then the result is divided by the dimension. The square root of the value is used for scaling adjustment, and finally, the softmax function is applied for normalization to calculate the attention weights. , is represented as: ; in, It is a scaling factor to prevent the inner product from becoming too large. express The transpose of .

[0028] Step 3.3) Global Feature Calculation Multiplying the attention weights by the values ​​reduces the computational complexity from Down to To achieve efficient long-term dependency modeling and obtain global features : .

[0029] Step 4: Adaptive fusion of multi-scale features, the process is as follows: The local features obtained in step 2) And the global features obtained in step 3) As input, and by passing local features as input to step 3), we obtain the features. Finally, the network adaptively assigns different weights to the features, as described below: ; in, For learnable parameters, The representations are generated based on point-based attention and local point attention. Finally, MLP is used for batch normalization and residual connections to extract high-level point-by-point representations.

[0030] Step 5: Encoder structure design, the process is as follows: like Figure 1 The input point cloud is converted into a feature vector that can be processed by a neural network through point embedding before entering the encoder module; point embedding refers to the process of encoding the position information of points into a high-dimensional vector through trigonometric functions and other methods. The execution steps from step two to step four are denoted as LPGPFormer. LPGPFormer consists of three main modules: LPWC, GKPT, and Fusion. LPGPFormer is executed repeatedly in this order. Each execution is recorded as one encoding module. Before each encoding module, the number of points in the point cloud is downsampled using the farthest point sampling algorithm to ensure the uniformity of the samples. Then, the neighborhood features of the downsampled point cloud are reconstructed in step one to ensure the accuracy and effectiveness of subsequent processing. Farthest point sampling is a commonly used sampling algorithm that is widely used in the field of point cloud data processing.

[0031] Step 6: Calculation of network decoding and semantic segmentation results, the process is as follows: The decoding module employs a layer-by-layer upsampling strategy to progressively restore sparse point cloud features to their original resolution. This process utilizes the feature upsampling strategy found in 3D point cloud processing, employing an inverse distance-weighted interpolation algorithm. First, the Euclidean distance between adjacent point cloud layers is calculated to establish a spatial correspondence. Then, for each sampling point in the layer to be restored, samples are selected from the previous layer. The target point is then identified by finding its nearest neighbors. Subsequently, a weighted aggregation based on the inverse distance is performed on these neighbors, multiplying the neighborhood feature by the inverse of its distance, and summing the results after normalization to calculate the semantic features of the target point, thus completing the feature transfer from sparse to dense. Building upon this, a skip connection mechanism is introduced to concatenate the interpolated features with features of the same scale from the encoder along the channel dimension, fusing deep semantics with shallow details. This process is repeated iteratively. The process continues until the full resolution is restored. The resulting high-level semantic features are then processed by MLP dimensionality reduction and Softmax activation function to output a semantic label for each point, thus completing the segmentation task.

[0032] Reference Figure 3 The method of this invention is compared and analyzed with other point cloud semantic segmentation methods, including the following steps: Step 1: Define 6 point cloud semantic segmentation methods SegCloud combines voxelization and point-level optimization. It first performs coarse voxel-level prediction using a 3D fully convolutional network, and then uses trilinear interpolation and optimized point-level prediction to achieve fine segmentation. Voxelization converts irregular point clouds into regular 3D meshes; trilinear interpolation is a common upsampling method; and fully connected conditional random fields are a post-processing technique that considers the global relationships between points to optimize the initial segmentation results and make the boundaries clearer.

[0033] SPGraph: SPGraph abstracts point clouds into supergraphs. It first uses unsupervised segmentation to extract geometric superpoints as graph nodes, then uses a graph convolutional network to learn the relationships between superpoints, utilizing contextual information for segmentation. Superpoints are sets of points with similar geometric and semantic attributes; a supergraph is a graph model constructed with superpoints as nodes and the spatial relationships between them as edges.

[0034] RandLA-Net: RandLA-Net uses random sampling to reduce computational overhead and retains key geometric features through a local feature aggregation module, achieving a balance between accuracy and efficiency.

[0035] Kpconv: Kpconv generates different offsets at each convolution position, thereby adjusting the shape of the convolution kernel for different point cloud regions.

[0036] CFSA-Net: CFSA-Net dynamically fuses the geometric and semantic information of points through a cross-fusion self-attention mechanism, thereby more comprehensively describing local features, enhancing local feature description, expanding the receptive field, and effectively alleviating the information loss problem of random sampling.

[0037] LPGPFormer: This invention proposes a large-scale point cloud semantic segmentation method based on local positional convolution and global keypoint Transformer.

[0038] Step 2: Point Cloud Semantic Segmentation Dataset The Semantic3D dataset is widely recognized as one of the most challenging large-scale datasets for outdoor urban scene point cloud datasets. It contains over four billion manually labeled points, covering various outdoor scenes including churches, streets, railway tracks, and squares. Due to severe occlusion by different objects, the point cloud density varies greatly in this dataset. The dataset consists of eight classes: man-made terrain, natural terrain, high vegetation, low vegetation, buildings, hard landscape, scanning artifacts, and cars. This benchmark dataset comprises 15 training point clouds and 15 online test cloud points. Each point cloud has a spatial extent of approximately 160×240×30m and covers more than 108 points.

[0039] Step 3: Define evaluation indicators The model's performance is validated by calculating the mean intersection-union ratio (mIoU) and overall pointwise accuracy (OA) on the union set, as shown in the following formulas: ; ; Among them, TP, FP, TN, and FN represent true positive, false positive, true negative, and false negative, respectively. It represents the number of categories in the dataset.

[0040] Among them, the larger the values ​​of mIou and OA, the better the model performance.

[0041] Step 4: Evaluate and compare the results Figure 3 Quantitative evaluation experimental results of the method of this invention and other comparative models on the reduced-8 dataset of Semantic3D are presented. Figure 3The results show that, compared to previous work, the method proposed in this invention achieves optimal performance in both mIoU and OA. Compared to the current state-of-the-art method RandLA-Net, the model proposed in this invention improves the mIoU and OA metrics by 1.1% and 0.5%, respectively, verifying the effectiveness of the proposed model in outdoor point cloud semantic segmentation tasks. Furthermore, LPGPFormer exhibits excellent capture performance for 3D planar geometry, achieving optimal segmentation results on objects such as artificial terrain, low vegetation, hardscapes, and vehicles.

[0042] This embodiment uses specific point cloud data analysis as an example to further illustrate the effectiveness of the network module of the present invention, as follows: Step 1: Introduction to the Network Module LPWC is used to dynamically learn the uneven distribution of local point cloud features in large-scale scenes, while GKPT is used to explore global context relationships. In addition, this invention also designs a Fusion module to dynamically aggregate the features of the two.

[0043] Step 2: Introduction to Point Cloud Datasets The Jiliang-3D dataset is a large-scale urban outdoor point cloud dataset obtained through UAV oblique photogrammetry and 3D reconstruction, covering an area of ​​approximately 320,000 square meters. The data is primarily used to test the performance of point cloud semantic segmentation algorithms for large-scale urban scenes, including various complex scenes such as parking lots, small squares, lawns, dormitories, and laboratory buildings, with significant variations in point cloud density. The dataset is divided into 10 regions, containing manually labeled categories for 9 categories: bicycles, buildings, cars, debris, grass, poles, roads, vegetation, and water bodies.

[0044] Step 3: Implementation Design and Process Table 1 presents the quantitative ablation analysis results of the proposed method on the Jiliang-3D dataset for different modules. First, the proposed method removes all modules, essentially using a neighbor attention operation similar to PointNet++. The proposed method uses model(a) as the baseline model and sequentially adds LPWC, GKPT, and Fusion modules to this model to construct four comparative models. Model(b) uses LPWC to obtain local point cloud feature distribution, model(c) uses GKPT to obtain global point cloud feature distribution, model(d) integrates model(a) and model(b) to obtain both local and global feature distributions, and model(e) adds a fusion model to model(d) to aggregate features. The quantitative analysis results are shown in Table 1. Based on model(a), model(b) shows a significant performance improvement of 3.6% due to the enhanced robustness of LPWC for local feature extraction. However, using only GKPT results in a smaller performance improvement, mainly due to the significant semantic gap between local and global representations and the large amount of noise in point clouds in large-scale scenes. Therefore, based on the model(d) model, the integration of LPWC and GKPT improved performance by 5.4%. Building on this, the present invention uses the Fusion module to effectively combine the features of both, achieving another 2.5% improvement. Finally, combining all the above modules achieves optimal performance. This demonstrates the effectiveness of the proposed components and illustrates their synergistic impact on achieving state-of-the-art performance.

[0045] Table 1 shows the ablation quantitative analysis results of different modules using the method of the present invention on the Jiliang-3D dataset; The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling, characterized in that, The method includes the following steps: Step 1: Construct neighborhood features; Step 2, Local Location Convolution Feature Extraction: Local geometric features of the point cloud are dynamically learned through Local Location Weighted Convolution (LPWC) to enhance the representation ability of local point cloud features; Step 3, Global Keypoint Attention Modeling: The Global Keypoint Module (GKPT) is used to capture long-term dependencies in the point cloud, and the Keypoint Sampling KPS method is used to reduce computation and simulate the global context. Step 4: Multi-scale feature adaptive fusion: By combining local and global features through the fusion module, weights are adaptively allocated to achieve feature refinement; Step 5, Encoder Structure Design: The execution steps from Step 2 to Step 4 are denoted as LPGPFormer. LPGPFormer is executed repeatedly in sequence. Each execution is recorded as one layer of encoding module. Before each layer of encoding module, the number of points in the point cloud is downsampled using the farthest point sampling algorithm, and the neighborhood features are reconstructed from the downsampled point cloud in step one. Step 6: Network Decoding and Semantic Segmentation Result Calculation: In the decoding stage, encoder features are fused through layer-by-layer upsampling and skip connections to recover the number of point clouds and enhance semantic information. Finally, the class probability distribution of each point is output through MLP and Softmax activation functions, thereby outputting the final semantic segmentation result.

2. The large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling as described in claim 1, characterized in that, The process of step one is as follows; Step 1.1) Define the point cloud dataset Point cloud data consists of a dense set of three-dimensional points, where each point has a similar representational meaning to its neighboring points. Given a point cloud dataset containing N points... Feature set ,in The number of channels for the input feature. Let i be the i-th point in the point cloud dataset. The features of the i-th point; Step 1.2) Obtain neighborhood features Feature extraction of local point clouds is performed using the KNN neighborhood search algorithm from machine learning. As the i-th sampling point, it has a total of K neighboring points. The j-th neighboring point of the i-th sampling point is denoted as... Its characteristics are represented as .

3. The large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling as described in claim 1 or 2, characterized in that, The process of step two is as follows; Step 2.1) Obtain position weights The points obtained in step 1) Neighboring points As input, the weights between local point clouds are learned using their geometric prior knowledge. , sampling points coordinates, neighboring points The coordinates of the two points, the relative position vector between the two points, and the three-dimensional Euclidean distance are used to concatenate the features, and a multilayer perceptron is used for feature mapping, so as to more flexibly describe the spatial geometric structure of the point cloud object and enhance the geometric features. Step 2.2) Perform local position weight convolution First, calculate the sampling points. To nearby points The difference of coordinate vectors and through a scalar function Process it and combine it with the sampling point-based data. and neighboring points Position weight term and scalar functions Transformed sampling point features Perform the product operation, and finally sum the interaction results of all sampling point ranges N and neighboring point ranges K. Step 2.3) Local Feature Acquisition Utilizing the feature vector difference between a local point and its corresponding query point To obtain weights, points with high similarity values ​​are assigned higher weights. Based on the given LPWC module, the calculated local location weight convolution result is compared with the feature vector difference between the local point and its corresponding query point. Add them together to obtain the final local features. .

4. The large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling as described in claim 1 or 2, characterized in that, The process of step three is as follows; Step 3.1) Key point sampling: KPS selects points that are more representative of global semantic modeling by fusing semantic features and spatial geometric information while maintaining semantic integrity; Step 3.2) Key points in the use of Transformer; Step 3.3) Global Feature Calculation: Multiply the attention weights by the value, reducing the computational complexity from... Down to To achieve efficient long-term dependency modeling and obtain global features : 。 5. The large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling as described in claim 4, characterized in that, The process of step 3.1) is as follows: Step 3.1.1) Spatial distribution modeling Using the farthest point sampling FPS from the original point cloud Select a subset As the initial sampling point, for each sampling point The KNN algorithm is used to find its local neighborhood points, generate an index mapping, and define a set of neighborhood points; Step 3.1.2) Semantic kernel function design To explicitly capture the spatial distribution geometry of point clouds, a radial basis function (RBF) kernel is introduced for spatial encoding, and the point cloud is calculated. to every point in its neighborhood The sum of the squares of the Euclidean distances is multiplied by a negative scaling factor for scaling and inversion. An exponential function is then applied to obtain the local response at that point. Finally, the local responses of all points are aggregated into a global metric, i.e., the spatial kernel function. ; Step 3.1.3) Keypoint Generation Use learnable matrices Point features With spatial kernel function Fusion to generate a semantic weight matrix Then semantic weights Point cloud coordinates Multiplication, from the point cloud Calculate the key points , where n represents the number of key points.

6. The large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling as described in claim 4, characterized in that, The process of step 3.2) is as follows: Step 3.2.1) Feature Projection Original point features and key points Features Project them as query, key, and value, respectively; Step 3.2.2) Self-attention calculation To utilize keypoint generation and To refer to global point information, first calculate the query matrix. Key point key matrix The product of the transposes is used to measure the correlation, and then the result is divided by the dimension. The square root of the value is used for scaling adjustment, and finally, the softmax function is applied for normalization to calculate the attention weights. .

7. The large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling as described in claim 1 or 2, characterized in that, The process of step four is as follows: The local features obtained in step 2) And the global features obtained in step 3) As input, and by passing local features as input to step 3), we obtain the features. Finally, the network adaptively assigns different weights to the features, and then uses MLP for batch normalization and residual connections to extract pointwise high-level representations.

8. The large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling as described in claim 1 or 2, characterized in that, The process of step five is as follows: The execution steps from step two to step four are denoted as LPGPFormer. LPGPFormer consists of three main modules: LPWC, GKPT, and Fusion. LPGPFormer is executed repeatedly in this order. Each execution is recorded as one layer of encoding module. Before each layer of encoding module, the number of points in the point cloud is downsampled using the farthest point sampling algorithm to ensure the uniformity of the samples. Then, the neighborhood features of the downsampled point cloud are reconstructed in step one to ensure the accuracy and effectiveness of subsequent processing. Farthest point sampling is a commonly used sampling algorithm that is widely used in the field of point cloud data processing.

9. The large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling as described in claim 1 or 2, characterized in that, The process of step six is ​​as follows: The decoding module adopts a layer-by-layer upsampling strategy to gradually restore the sparse point cloud features to the original resolution. This process is achieved by using the feature upsampling strategy in 3D point cloud processing and employing an inverse distance weighted interpolation algorithm.

10. The large-scale point cloud semantic segmentation method based on convolutional networks and keypoint modeling as described in claim 9, characterized in that, In step six, the Euclidean distance between two adjacent point cloud layers is first calculated to establish a spatial correspondence, and then each sampling point of the layer to be recovered is selected from the previous layer. The nearest neighbor; Subsequently, a weighted aggregation based on the inverse distance is performed on these neighborhood points. This involves multiplying the neighborhood features by the inverse of their distances, normalizing and summing the results, and then calculating the semantic features of the target point, thus completing the feature transfer from sparse to dense. Building upon this, a skip connection mechanism is introduced to concatenate the interpolated features with features of the same scale from the encoder along the channel dimension, thereby fusing deep semantics with shallow details. This process is repeated iteratively. The process continues until the full resolution is restored. The resulting high-level semantic features are then processed by MLP dimensionality reduction and the Softmax activation function to output the semantic category label for each point, thus completing the segmentation task.

Images