A point cloud classification and segmentation method based on dual-path feature fusion
By using a dual-path feature fusion method, the accuracy of point cloud classification and segmentation is improved, solving the problem that local geometric features are not fully utilized in existing technologies, and achieving efficient classification and segmentation in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LIAONING UNIVERSITY
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-05
Smart Images
Figure CN122156780A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of point cloud processing technology, and provides a point cloud classification and segmentation method based on dual-path feature fusion. Background Technology The inherent disorder, irregularity, and sparsity of point cloud data determine that local geometry is the core element representing the essential features of 3D targets, and its extraction quality directly determines the performance ceiling of point cloud classification and segmentation tasks. In point cloud classification tasks, local shape features are the key basis for distinguishing targets with similar global contours but different fine-grained structures. For example, the global contour height of point clouds of different vehicle models is similar, but the differences in local geometric features such as wheel distribution and window contours are the core support for achieving accurate category differentiation. In point cloud segmentation tasks, subtle differences in the geometric shape of local regions directly determine the positioning accuracy of target component boundaries, which is the core prerequisite for improving the completeness and consistency of segmentation results. Most existing point cloud classification and segmentation methods directly encode features or perform simple neighborhood aggregation based on the 3D coordinates of the point cloud, focusing only on the shallow fusion of features of neighboring points. They fail to fully explore the geometric differences between the center point and neighboring points and ignore the contextual dependencies between local points, resulting in insufficient local feature representation capabilities and difficulty in adapting to the classification and segmentation needs of complex scenarios.
[0002] To address the shortcomings of existing methods in local geometric feature extraction, this chapter proposes a dual-path feature fusion method for point cloud classification and segmentation. The overall architecture of this method comprises three core modules: First, an input mapping module transforms the 3D coordinates of the original point cloud into high-dimensional feature vectors through dimensionality enhancement and nonlinear transformation, providing effective input for subsequent feature extraction. Second, a difference-aware neighborhood aggregation module incorporates the feature differences between the center point and its neighbors, strengthening the representation of local structural differences and overcoming the insensitivity of traditional neighborhood aggregation to subtle geometric differences. Third, a dual-path feature extraction module focuses on learning geometric details, employing a multilayer perceptron and hybrid pooling to learn local features; the other path uses a local cross-attention mechanism to accurately model the feature dependencies between the center point and its neighbors, mining local contextual information. Finally, the output features of the two paths are fused. This method effectively improves the accuracy of point cloud classification and segmentation, fully extracts key feature information, achieves accurate diagnosis in point cloud classification and segmentation tasks, and is applicable to point cloud processing tasks in various complex scenarios. Summary of the Invention
[0003] This invention proposes a point cloud classification and segmentation method based on dual-path feature fusion, which addresses the problem of low recognition accuracy caused by existing methods failing to pay attention to local geometric features. At the same time, it provides an innovative technical approach and implementation scheme for efficient feature learning and fine segmentation and recognition of point clouds.
[0004] To achieve the above objectives, the technical solution adopted in this invention is: a point cloud classification and segmentation method based on dual-path feature fusion, the specific steps of which are as follows:
[0005] 1) Obtain the point cloud classification and segmentation dataset, and preprocess the dataset to form a .txt file dataset.
[0006] 2) Construct a network model, which consists of two branches;
[0007] 2.1) For the obtained point cloud classification and segmentation dataset, an input mapping module is first designed to encode the low-dimensional geometric features of the point cloud into high-dimensional feature vectors, providing a more adaptable input form for subsequent feature extraction. This module consists of a 1×1 convolutional layer, a batch normalization (BN) layer, a LeakyReLU activation layer, and residual connections.
[0008] 2.2) A difference-aware neighborhood aggregation module is constructed to enhance the expressive power of local features, explore the dependencies between center points and neighboring points, and alleviate the interference of the actual sparse point cloud region on local structure extraction. First, the original coordinate set P is downsampled using FPS to obtain the center point coordinate set. (N2 is the number of center points after downsampling), and simultaneously extract the feature set corresponding to the center points. For each center point The KNN method is used to select the k nearest points in the coordinate set P' as the neighborhood point set. At the same time, the original features corresponding to these neighborhood points are extracted. For each center point , center point features By concatenating the differences between the features of the center point and the features of the neighboring points, a set of local difference features of the center point is obtained, as shown in formula (1).
[0009] (1)
[0010] in, Represents the distance from the center point in the original coordinate set P. The set of the k nearest points.
[0011] By integrating the local difference feature sets of all center points, we obtain the global local feature set. This set contains both the basic features of the center point and the differential structure of the neighborhood, providing a more robust local feature input for the subsequent dual-path feature extraction module.
[0012] 2.3) Branch 1 is constructed by using shared weight MLP combined with gated hybrid pooling to learn local geometric features, so as to achieve robust extraction of local geometric features, as shown in formulas (2)-(6).
[0013] (2)
[0014] (3)
[0015] (4)
[0016] (5)
[0017] (6)
[0018] Where: mlp is a multilayer perceptron, F g This is the set of local difference features obtained by the difference-aware aggregation module. The features are after dimensionality upscaling via a multilayer perceptron; MaxPool is the maximum pooling method; F... max The features are after max pooling, AvgPool is for average pooling, and F is for average pooling. avg F represents the features after average pooling. MLP Let σ be the feature after adaptive mixing weights, w be the learnable parameter, and α be the learnable mixing weight vector.
[0019] 2.4) Constructing a branch two local cross-attention mechanism: Using the center point features as the query and the neighbor point features as the key and value, the mechanism adaptively assigns contribution weights to neighboring points, and the output is denoted as F. CA Specifically, the feature set is first processed using an MLP. Perform dimensional transformation and use the reshape function to transform the shape of the features to obtain feature set F2. Combine feature set F2 with... The input is fed into three different linear functions to obtain three feature matrices query, key, and value. The calculation method is shown in formula (7).
[0020] (7)
[0021] The similarity between query and key is calculated by scaling the dot product, and the attention weights of neighboring points are obtained by softmax normalization. The weight matrix W is calculated as shown in formula (8).
[0022] (8)
[0023] in, This is the transpose of the feature matrix key, where C represents the dimension of the input features. After obtaining the weight matrix W, it is multiplied by the feature matrix value, and the reshape function is used to transform the feature shape to obtain the local features F of the center point. CA The calculation method is shown in formula (9).
[0024] (9)
[0025] 3) Construct a dual-path feature fusion network, train the model using the training set, and achieve effective fusion of global and local features:
[0026] 3.1) This model is based on The original point cloud data is used as input. First, the feature embedding module performs dimensionality enhancement and nonlinear transformation on the original 3D coordinates to obtain... The initial high-dimensional features;
[0027] 3.2) Enter the feature extraction stage, downsampling to FPS. For each point, the difference-aware neighborhood aggregation module performs neighborhood difference encoding on the initial features of the sampled center point;
[0028] 3.3) The dual-path feature extraction module then performs geometric detail and inter-point dependency relationship fusion encoding on the feature to output enhanced features;
[0029] 3.4) Following this, the process of differential-aware neighborhood aggregation and dual-path feature extraction is repeated for the enhanced feature to obtain the final high-level semantic feature. This feature is directly input into the classification branch to complete the category determination of the point cloud; the segmentation branch uses the feature upsampling module to gradually restore the number of points to the desired level. The original number of points is N. At the same time, the initial features obtained by the feature embedding module are introduced through SkipConnection to supplement the original geometric details. Finally, the point-by-point segmentation features are obtained to complete the point-by-point category prediction of the point cloud.
[0030] 3.5) Train the model with an initial learning rate of 0.001, an exponential decay learning rate adjustment strategy, and the Adam optimizer.
[0031] 4) Load the trained network parameters, perform end-to-end inference on the test set, and output the point cloud classification and point-by-point segmentation results.
[0032] The beneficial effects of this invention are as follows: This invention accurately mines local geometric difference features of point clouds through the difference-aware neighborhood aggregation module, effectively overcomes the interference of sparse point clouds, and significantly improves the robustness of local feature representation; the dual-path branches focus on geometric detail extraction and inter-point dependency modeling respectively, realizing complementary feature learning and breaking the accuracy bottleneck of single feature extraction; with the help of jump-connected feature fusion and adaptive weight mechanism, it takes into account both global semantics and local details, significantly improving the accuracy of point cloud classification and segmentation precision; the model is adaptable to point cloud data in various scenarios, has high end-to-end inference efficiency, and has strong generalization ability and engineering application value. Attached Figure Description
[0033] Figure 1 This is a diagram of a point cloud network model for dual-path feature fusion.
[0034] Figure 2 This is a diagram of the dual-path feature extraction module.
[0035] Figure 3 This is a local cross-attention map.
[0036] Figure 4 This is a diagram showing the results of component segmentation in the dataset.
[0037] Figure 5 The diagram shows the impact of the number of K-nearest neighbors on model accuracy. Detailed Implementation
[0038] This invention provides a point cloud classification and segmentation method based on dual-path feature fusion, specifically including the following steps:
[0039] 1) Obtain the point cloud classification and segmentation dataset, and preprocess the dataset to form a .txt file dataset.
[0040] 2) Construct a network model, which consists of two branches; such as Figure 1 As shown in the diagram, this is the overall model of the network, which mainly consists of three modules: an input mapping module, a difference-aware aggregation module, and a dual-path feature extraction module. Figure 2 As shown in the diagram, this diagram illustrates the design of two branch paths in the dual-path feature extraction module. One path uses a multilayer perceptron and hybrid pooling strategy to extract basic local features; the other path constructs a local cross-attention mechanism, dynamically learning attention weights to obtain refined local features rich in neighborhood dependencies. By integrating the outputs of the two paths, the model can obtain a local representation that combines basic geometric information with refined neighborhood relationships. Figure 3 As shown in the figure, this diagram represents the local cross-attention module of the second path in the dual-path feature extraction module.
[0041] 2.1) For the obtained point cloud classification and segmentation dataset, an input mapping module is first designed to encode the low-dimensional geometric features of the point cloud into high-dimensional feature vectors, providing a more adaptable input form for subsequent feature extraction. This module consists of a 1×1 convolutional layer, a batch normalization (BN) layer, a LeakyReLU activation layer, and residual connections.
[0042] 2.2) A difference-aware neighborhood aggregation module is constructed to enhance the expressive power of local features, explore the dependencies between center points and neighboring points, and alleviate the interference of the actual sparse point cloud region on local structure extraction. First, the original coordinate set P is downsampled using FPS to obtain the center point coordinate set. (N2 is the number of center points after downsampling), and simultaneously extract the feature set corresponding to the center points. For each center point The KNN method is used to select the k nearest points in the coordinate set P' as the neighborhood point set. At the same time, the original features corresponding to these neighborhood points are extracted. For each center point , center point features By concatenating the differences between the features of the center point and the features of the neighboring points, a set of local difference features of the center point is obtained, as shown in formula (1).
[0043] (1)
[0044] in, Represents the distance from the center point in the original coordinate set P. The set of the k nearest points.
[0045] By integrating the local difference feature sets of all center points, we obtain the global local feature set. This set contains both the basic features of the center point and the differential structure of the neighborhood, providing a more robust local feature input for the subsequent dual-path feature extraction module.
[0046] 2.3) Branch 1 is constructed by using shared weight MLP combined with gated hybrid pooling to learn local geometric features, so as to achieve robust extraction of local geometric features, as shown in formulas (2)-(6).
[0047] (2)
[0048] (3)
[0049] (4)
[0050] (5)
[0051] (6)
[0052] Where: mlp is a multilayer perceptron, F g This is the set of local difference features obtained by the difference-aware aggregation module. The features are after dimensionality upscaling via a multilayer perceptron; MaxPool is the maximum pooling method; F... max The features are after max pooling, AvgPool is for average pooling, and F is for average pooling. avg F represents the features after average pooling. MLP Let σ be the feature after adaptive mixing weights, w be the learnable parameter, and α be the learnable mixing weight vector.
[0053] 2.4) Constructing a branch two local cross-attention mechanism: Using the center point features as the query and the neighbor point features as the key and value, the mechanism adaptively assigns contribution weights to neighboring points, and the output is denoted as F. CA Specifically, the feature set is first processed using an MLP. Perform dimensional transformation and use the reshape function to transform the shape of the features to obtain feature set F2. Combine feature set F2 with... The input is fed into three different linear functions to obtain three feature matrices query, key, and value. The calculation method is shown in formula (7).
[0054] (7)
[0055] The similarity between query and key is calculated by scaling the dot product, and the attention weights of neighboring points are obtained by softmax normalization. The weight matrix W is calculated as shown in formula (8).
[0056] (8)
[0057] in, This is the transpose of the feature matrix key, where C represents the dimension of the input features. After obtaining the weight matrix W, it is multiplied by the feature matrix value, and the reshape function is used to transform the feature shape to obtain the local features F of the center point. CA The calculation method is shown in formula (9).
[0058] (9)
[0059] 3) Construct a dual-path feature fusion network, train the model using the training set, and achieve effective fusion of global and local features:
[0060] 3.1) This model is based on The original point cloud data is used as input. First, the feature embedding module performs dimensionality enhancement and nonlinear transformation on the original 3D coordinates to obtain... The initial high-dimensional features;
[0061] 3.2) Enter the feature extraction stage, downsampling to FPS. For each point, the difference-aware neighborhood aggregation module performs neighborhood difference encoding on the initial features of the sampled center point;
[0062] 3.3) The dual-path feature extraction module then performs geometric detail and inter-point dependency relationship fusion encoding on the feature to output enhanced features;
[0063] 3.4) Following this, the process of differential-aware neighborhood aggregation and dual-path feature extraction is repeated for the enhanced feature to obtain the final high-level semantic feature. This feature is directly input into the classification branch to complete the category determination of the point cloud; the segmentation branch uses the feature upsampling module to gradually restore the number of points to the desired level. The original number of points is N. At the same time, the initial features obtained by the feature embedding module are introduced through SkipConnection to supplement the original geometric details. Finally, the point-by-point segmentation features are obtained to complete the point-by-point category prediction of the point cloud.
[0064] 3.5) Train the model with an initial learning rate of 0.001, an exponential decay learning rate adjustment strategy, and the Adam optimizer.
[0065] 4) Load the trained network parameters, perform end-to-end inference on the test set, and output point cloud classification and point-by-point segmentation results. For example... Figure 4 The image shows a schematic diagram of the network's component segmentation results, displaying the visualization results of the 10 category predictions and their respective ground truth labels. The first and third rows are visualizations based on the ground truth labels, while the second and fourth rows are visualizations based on the predictions. The visualization results demonstrate that the model in this chapter can predict the component category of each point, exhibiting clear segmentation effects in the boundary regions of different components and effectively distinguishing points from adjacent components. Figure 5 As shown, the model's classification accuracy first increases and then decreases with the increase in the number of neighborhood points. The model achieves its best average class accuracy and overall accuracy when the number of neighborhood points reaches 20. After this point, the model's classification accuracy begins to decrease with further increases in the number of neighborhood points.
[0066] Example 1:
[0067] The dataset was trained and tested using the Princeton University ModelNet40 classification dataset and the ShapeNetPart dataset, respectively.
[0068] Step 1: The Princeton University ModelNet40 dataset was divided into 9843 training samples and 2468 test samples according to the official standard. The original samples in the dataset were 3D models in .obj format. Preprocessing was required in the experiment, converting them into point clouds through surface information extraction and uniform sampling (1024 points were uniformly sampled in this paper). The ShapeNetPart large-scale part segmentation benchmark dataset was divided into a training set of 12137 samples, a validation set of 1870 samples, and a test set of 2874 samples according to the official standard. This was converted into a point cloud with a fixed number of points (2048 points were uniformly sampled in this paper), allowing it to be adapted to the segmentation model input without additional complex preprocessing.
[0069] Step 2: Set up the model's runtime environment and hyperparameter variables
[0070] In terms of the experimental environment, all experiments were conducted on a Windows server with an Intel(R) Xeon(R) Silver 4110 processor, 24GB of memory, and an NVIDIA RTX A5000 graphics card to accelerate the network training and inference process. Python version 3.8 was used, and the network model was built based on the PyTorch deep learning framework. The specific training parameters for the classification and segmentation datasets are shown below:
[0071] Table 1 Experimental Parameter Settings
[0072]
[0073] Step 3: Construct a point cloud classification and segmentation method model based on dual-path feature fusion.
[0074] The main modules include: an input mapping module, a difference-aware neighborhood aggregation module, and a dual-path feature extraction module. One path focuses on learning geometric details, using a multilayer perceptron and hybrid pooling to learn local features; the other path designs a local cross-attention mechanism to accurately model the feature dependencies between the center point and its neighbors, and to mine local contextual information.
[0075] Step 4: Using this model, conduct experiments. The results of the comparison between classification and segmentation experiments are shown in Tables 2 and 3.
[0076] Table 2. Classification results on the ModelNet40 dataset.
[0077]
[0078] surface Part segmentation results table on the ShapeNetPart dataset
[0079]
[0080] As shown in Tables 2 and 3, the method of this invention achieved optimal evaluation results on standard point cloud datasets, and the core indicators fully demonstrate the technical superiority of the solution. Among them, the ModelNet40 dataset serves as an authoritative benchmark for point cloud classification. mAcc (mean class accuracy) reflects the model's balanced recognition ability across various target classes, while OA (overall accuracy) reflects the overall classification accuracy. This invention achieves mAcc of 91.1% and OA of 93.5%, both surpassing mainstream classic methods such as PointNet, PointNet++, and DGCNN, proving that the dual-path feature fusion strategy can effectively capture local geometric features and significantly improve the accuracy of 3D model classification. The ShapeNetPart dataset focuses on fine-grained part segmentation. Cls.mIoU (class average intersection-over-union ratio) and Inst.mIoU (instance average intersection-over-union ratio) are the core metrics for measuring segmentation quality. This invention achieves 83.1% Cls.mIoU and 85.5% Inst.mIoU. For class-by-class segmentation accuracy of various 3D parts such as bags, cars, chairs, and motorcycles, this invention also generally outperforms existing solutions, especially for structurally complex targets with sparse local features, where the segmentation improvement is particularly significant. This invention proposes a dual-path feature fusion point cloud classification and segmentation method based on deep learning. This method can fully mine key feature information, effectively improving classification and segmentation accuracy. It is suitable for point cloud processing tasks in various complex scenarios and possesses high accuracy, stability, and practical value.
Claims
1. A point cloud classification and segmentation method based on dual-path feature fusion, characterized in that, The steps are as follows: Step 1) Obtain the point cloud classification and segmentation dataset, and preprocess the dataset to form a .txt file dataset; Step 2) Construct a network model consisting of two branches: Construct a dual-branch feature extraction structure. Branch 1 uses a multilayer perceptron and hybrid pooling to extract global structural features, while branch 2 mines local neighborhood geometric correlation features of point clouds through a local cross-attention mechanism. Step 3) Construct a dual-path feature fusion network, train the model using the training set, and achieve effective fusion of global and local features: Step 4) Load the trained network parameters, perform end-to-end inference on the test set, and output the point cloud classification and point-by-point segmentation results.
2. The point cloud classification and segmentation method based on dual-path feature fusion according to claim 1, characterized in that, The specific method in step 2) is as follows: Step 2.1) For the obtained point cloud classification and segmentation dataset, firstly design an input mapping module to encode the low-dimensional geometric features of the point cloud into high-dimensional feature vectors, providing an adaptive input form for subsequent feature extraction; this module consists of a 1×1 convolutional layer, a batch normalization layer (BN), a LeakyReLU activation layer, and residual connections. Step 2.2) Construct a difference-aware neighborhood aggregation module to enhance the expressive power of local features, explore the dependency relationship between the center point and the neighborhood points, and at the same time alleviate the interference of the actual sparse point cloud region on the extraction of local structure. First, the original coordinate set P is downsampled using FPS to obtain the center point coordinate set. Where: N2 is the number of center points after downsampling, and the feature set corresponding to the center points is extracted simultaneously. For each center point The KNN method is used to select the k nearest points in the coordinate set P' as the neighborhood point set. At the same time, the original features corresponding to these neighborhood points are extracted. For each center point , center point features By concatenating the differences between the center point features and the neighborhood point features, a set of local difference features for the center point is obtained, as shown in formula (1): (1) in, Represents the distance from the center point in the original coordinate set P. The set of the k nearest points; By integrating the local difference feature sets of all center points, we obtain the global local feature set. This set contains both the basic features of the center point and the differential structure of the neighborhood, providing local feature input for the subsequent dual-path feature extraction module; Step 2.3) Constructing Branch 1: Using shared-weight MLP combined with gated hybrid pooling to learn local geometric features, robust extraction of local geometric features is achieved, as shown in formulas (2)-(6): (2) (3) (4) (5) (6) Where: mlp is a multilayer perceptron, F g This is the set of local difference features obtained by the difference-aware aggregation module. The features are after dimensionality upscaling via a multilayer perceptron; MaxPool is the maximum pooling method; F... max The features are after max pooling, AvgPool is for average pooling, and F is for average pooling. avg F represents the features after average pooling. MLP σ represents the features after adaptive mixing weights, w represents the learnable parameter, and α represents the learnable mixing weight vector. Step 2.4) Construct a branch two local cross-attention mechanism using the center point features as the query and the neighbor point features as the key and value. Adaptively allocate the contribution weights of the neighboring points, and denote the output as F. CA First, the feature set is processed using MLP. Perform dimensional transformation and use the reshape function to transform the shape of the features to obtain feature set F2. Combine feature set F2 with... The input is fed into three different linear functions to obtain three feature matrices: query, key, and value. The calculation method is shown in formula (7). (7) The similarity between query and key is calculated by scaling the dot product, and the attention weights of neighboring points are obtained by softmax normalization. The weight matrix W is calculated as shown in formula (8): (8) in, It is the transpose of the feature matrix key, where C represents the dimension of the input features; after obtaining the weight matrix W, it is multiplied by the feature matrix value and the reshape function is used to transform the feature shape to obtain the local features F of the center point. CA , (9) The calculation method is shown in formula (9).
3. The point cloud classification and segmentation method based on dual-path feature fusion according to claim 1, characterized in that, The specific method in step 3) is as follows: Step 3.1) The model uses The original point cloud data is used as input. First, the feature embedding module performs dimensionality enhancement and nonlinear transformation on the original 3D coordinates to obtain... The initial high-dimensional features; Step 3.2) Proceed to the feature extraction stage, downsampling to FPS. For each point, the difference-aware neighborhood aggregation module performs neighborhood difference encoding on the initial features of the sampled center point; Step 3.3) The dual-path feature extraction module then performs geometric detail and inter-point dependency relationship fusion encoding on the feature to output the enhanced feature; After step 3.4), the process of difference-aware neighborhood aggregation and dual-path feature extraction is repeated for the enhanced feature to obtain the final high-level semantic feature. This feature is directly input into the classification branch to complete the category determination of the point cloud; the segmentation branch uses the feature upsampling module to gradually restore the number of points to the desired level. The original number of points is N. At the same time, the initial features obtained by the feature embedding module are introduced through jump connection to supplement the original geometric details. Finally, the point-by-point segmentation features are obtained to complete the point-by-point category prediction of the point cloud. Step 3.5) Train the model with an initial learning rate of 0.001, an exponential decay learning rate adjustment strategy, and the Adam optimizer.