A 3D point cloud classification segmentation method fusing a dense connection mechanism and PointNet++
By introducing a dense connection mechanism and a skip connection method into the PointNet++ model, and fusing the low-level and high-level global features of point clouds, the problems of illumination variation and computational resource consumption in point cloud classification and segmentation are solved, and higher classification and segmentation accuracy is achieved.
Patent Information
- Application Number
- CN202211190932.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-28
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2042-09-28
AI Technical Summary
Existing point cloud classification and segmentation methods suffer from unstable feature information under varying illumination. Traditional convolutional networks cannot directly extract point cloud features, resulting in excessive computational resource and time consumption. PointNet++ only utilizes high-level global point cloud information while ignoring low-level information, leading to insufficient classification and segmentation accuracy.
The PointNet++ model introduces a dense connection mechanism, which extracts low-level and high-level global and local features of the point cloud through a dense set sampling layer module, and fuses the features through skip connections and inverse interpolation methods, and uses a fully connected network for classification and segmentation.
It improves the accuracy of point cloud classification and segmentation, makes full use of the feature information of multilayer perceptron, is applicable to other deep learning models, and enhances the performance of the model in different tasks.
Smart Images
Figure CN115527032B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application is mainly based on the PointNet++ model, and the dense connection mechanism in the dense convolutional network (DenseNet) is referred to in the feature extraction layer, and the global features and local features of the point cloud at the bottom and high levels are used to improve the accuracy of the classification and segmentation results of the model. BACKGROUND
[0002] With the rapid development of computer science, technologies related to neural networks have gradually become the focus of research and have been widely used in target detection, autonomous driving and other tasks, which are of great significance for subsequent intelligent and unmanned technologies.
[0003] At present, the related technologies based on deep learning can be mainly divided into two categories: images and point clouds. After 2013, the deep learning technologies related to images represented by convolutional neural network (CNN) and You Only Look Once (YOLO) algorithm have developed rapidly. These methods mainly establish a model for extracting rich color information of images to achieve classification, segmentation, splicing and other tasks. Compared with the traditional method which is complex and inefficient, this method is more rapid and efficient. However, since it mainly relies on the color information of the picture, the feature information obtained by the neural network model will change under different light conditions, which will affect the result.
[0004] The deep learning method related to point cloud can overcome the shortcomings of image. Point cloud is composed of a group of points containing (x, y, z) 3D coordinate information, and these points are not affected by the transformation of light conditions. Due to its robustness to light, point cloud has become the focus of recent research. However, point cloud has no order, which leads to the fact that the traditional convolutional network cannot directly extract its feature information. In the early research, researchers chose to project the point cloud to multiple 2D planes or convert the points in the point cloud into voxels, grids and other forms, and then use the convolutional network for processing. Due to the additional processing steps, these methods consume more computer memory and time.
[0005] Until 2017, the PointNet network which can directly extract the feature of point cloud appeared. Since it does not need to process the point cloud additionally, PointNet is more rapid and efficient, providing a new idea for subsequent algorithms. Based on PointNet, PointNet++ uses the local features between points. The results show that compared with PointNet, PointNet++ achieves higher accuracy in classification and segmentation tasks. SUMMARY
[0006] The application provides a PointNet++ optimization algorithm.
[0007] The technical scheme of the application is a 3D point cloud classification and segmentation method fusing a dense connection mechanism and PointNet++, comprising the following steps.
[0008] Step 1: two designed dense set abstraction layer modules are used to extract global features and local features of bottom layers and high layers of input point clouds, wherein the dense set abstraction layer module mainly comprises a sampling layer, a grouping layer and a feature extraction layer;
[0009] Step 2: global features obtained by the dense set abstraction layer module and a full connection network are used to obtain a prediction score of each class, and the class of the input point cloud is determined according to the highest prediction score;
[0010] Step 3: a reverse interpolation method is used to propagate the features of the sampling points in the last layer to all points of the original point cloud, the propagated features are fused with the global features and the local features obtained in step 1 in a skip connection mode, and a full connection network is used to obtain a prediction score of each point in the point cloud, so that the segmentation task is realized.
[0011] Further, the step 1 specifically comprises:
[0012] Step 1.1: set the input point cloud set containing N points as Each point contains (x, y, z) three-dimensional coordinates;
[0013] Step 1.2: use the first dense set abstraction layer module to obtain global features and local features of the bottom layers of the point cloud;
[0014] Step 1.3: use the second dense set abstraction layer module to obtain global features and local features of the high layers of the point cloud.
[0015] Further, the specific process of step 1.2 is as follows:
[0016] Step 1.2.1: use the farthest distance sampling method in the sampling layer to sample 512 points from the original point cloud as sampling points;
[0017] Step 1.2.2: in the grouping layer, the sampling points obtained in step 1.2.1 are used as center points, and the nearest 32 points to each center point within a radius of 0.2 are used as neighbor points, and when the number of neighbor points is less than 32, the first point coordinates are repeated;
[0018] Step 1.2.3, the local feature information of the point cloud is extracted using a dense PointNet layer as a feature extraction layer, and the input of the multi-layer perceptron in the dense PointNet layer is the feature splicing form of all the output layers before, and the local feature of the point cloud in the region divided in step 1.2.2 is extracted;
[0019] Step 1.2.4, a dense PointNet layer is used to extract the global feature of the point cloud. The dense PointNet layer directly extracts the features of all the points sampled in step 1.2.1 as the global feature of the point cloud.
[0020] Further, the specific process of step 1.3 is as follows:
[0021] Step 1.3.1, 128 points are sampled from 512 points in the first dense set sampling layer module using the farthest distance sampling method;
[0022] Step 1.3.2, the sampling points obtained in step 1.3.1 are used as center points in the grouping layer, and the nearest 64 points to each center point within a radius of 0.4 are used as neighbor points, and when the number of neighbor points is less than 64, the first point coordinate is repeated;
[0023] Step 1.3.3, the local feature of the point cloud is extracted using a dense PointNet layer, and the input of the multi-layer perceptron in the dense PointNet layer is the feature splicing form of all the output layers before, and the local feature of the point cloud in the region divided in step 1.3.2 is extracted;
[0024] Step 1.3.4, a dense PointNet layer is used to extract the global feature of the point cloud, and the dense PointNet layer directly extracts the features of all the points sampled in step 1.3.1 as the global feature of the point cloud.
[0025] Further, the specific process of step 2 is as follows:
[0026] Step 2.1, the point cloud bottom layer features obtained in steps 1.2.4 and 1.3.4 are spliced with the high layer features of the point cloud;
[0027] Step 2.2, the global feature of the point cloud obtained by splicing in step 2.1 is input into a fully connected network to obtain the prediction score of k classes, and the class with the highest prediction score is the class of the input point cloud. In order to prevent overfitting, a dropout layer with a dropout rate of 0.4 is added in the fully connected network.
[0028] Further, the specific process of step 3 is as follows:
[0029] Step 3.1, the global features obtained in the second dense set sampling layer module are up-sampled point feature propagation by using the reverse interpolation method, and the local features obtained in step 1.3.3 are spliced by using the way of jump connection;
[0030] Step 3.2, the point cloud features obtained are updated by using a convolution kernel of 1x1 convolution;
[0031] Step 3.3, the point cloud features obtained in step 3.2 are up-sampled point feature propagation by using the reverse interpolation method, and the global features and local obtained in step 1.2.3 and step 1.2.4 are spliced by using the way of jump connection;
[0032] Step 3.4, the point cloud features obtained are updated by using a convolution kernel of 1x1 convolution;
[0033] Step 3.5, the point cloud features obtained in step 3.4 are up-sampled point feature propagation by using the reverse interpolation method, and the features corresponding to all points in the original point cloud are obtained;
[0034] Step 3.6, the point cloud features obtained are updated by using a convolution kernel of 1x1 convolution;
[0035] Step 3.7, the prediction score of each point in the point cloud is obtained by using the full connection network, and the category of each point is determined according to the prediction score, and the segmentation task is realized.
[0036] The beneficial effects of the application are: the application improves the PointNet++ model to improve the classification and segmentation performance of the model. In the feature extraction layer of the improved model, the dense connection mechanism in the DenseNet model is referenced, and the input of each layer of the multilayer perceptron is the splicing form of all previous output features, so as to fully utilize the feature information obtained by each layer of the multilayer perceptron. And this method has universality and can also be used in other deep learning models. In addition, the unimproved PointNet++ only uses the highest layer point cloud global information of the set abstraction levels module in the classification task, and also ignores the bottom layer point cloud global information of the set abstraction levels module in the segmentation task. The improved network model aims at this point, and the global features of the bottom layer and the high layer are fused in the classification task to improve the accuracy of the model classification. In the segmentation task, the improved model connects the bottom layer global information and local information of the point cloud in parallel with the features obtained by feature propagation in the way of jump connection to improve the accuracy of segmentation. BRIEF DESCRIPTION OF DRAWINGS
[0037] Figure 1 It is the structure diagram of the PointNet++ model fused with the dense connection mechanism.
[0038] Figure 2 A dense PointNet layer for extracting point cloud features is provided for the present application. DETAILED DESCRIPTION
[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.
[0040] Figure 1 The PointNet++ model structure diagram fusing a dense connection mechanism is shown in FIG. 1. The specific process of using the model to implement a classification and segmentation task is as follows:
[0041] Step 1: Based on the input point cloud data, a first dense set sampling layer module is used to extract point cloud bottom layer features.
[0042] 1) Sampling layer: The farthest distance sampling method is used to down-sample the original point cloud: a null set S = *+ is set to store the sampled points. In the point cloud set A point x is randomly selected i is stored in the set S, and the distances of other points in the point cloud set to the point x are calculated. The point x i farthest to the point x is selected from the point cloud set j and stored in the set S, so that S = *x i ,x j +. The distances of the remaining points to all points in the set S are calculated, and the point x k farthest to the set S is selected and stored in the set S. In this way, a set of points with a set number is selected.
[0043] 2) Grouping layer: The points obtained by the sampling layer are used as center points, and the neighbor points of each center point are obtained in a spherical query manner, so as to construct a local region.
[0044] 3) Dense PointNet layer: The points in the local region are converted into a local coordinate system equivalent to the center point, and the local features of the point cloud are extracted by the dense PointNet. The features extracted from all points in the sampling layer are used as the global features of the point cloud.
[0045] Step 2: Based on the bottom layer features of the point cloud, a second dense set sampling layer module is used to extract high layer features of the point cloud in the same way.
[0046] Step 3: In the classification task, the bottom layer features and the high layer features of the point cloud obtained by the two dense set sampling layer modules are spliced, and then a fully connected network is used to obtain the prediction score of each class. The higher the prediction score is, the more likely the input point cloud belongs to this class.
[0047] Step 4: In the segmentation task, the feature of the upper sampling point is obtained by using the reverse interpolation method: the feature of the l-th layer containing N l points is propagated to the l-1-th layer containing N l-1 points (N l <N l-1 ). At each point of the l-1-th layer, find its k nearest points in the l-th layer in the original point cloud coordinate space, and obtain the feature of the point in the l-1-th layer by weighted sum of the corresponding features. Wherein, the size of the weight is inversely related to the distance of the point. After propagating the feature of the upper sampling point, the global feature and the local feature corresponding to the point cloud are spliced in a skip connection manner, and then a convolution kernel of 1*1 is used to update the point cloud feature. Finally, the prediction score of each point in the original point cloud is obtained, so as to realize the segmentation task.
[0048] Figure 2 Fig. 1 is a schematic diagram of a dense PointNet layer for extracting point cloud feature information.
[0049] Suppose the multi-layer perceptron in the feature extraction layer is d mlp1 ,d mlp2 ,d mlp3 -, the number of point clouds is N, the original feature dimension is d in , and K is the number of neighbor points in each local region. In addition to the original feature, the three-dimensional coordinates of the point cloud are also important features, so the dimension of the point cloud input into the network is 3+d in .
[0050] In the PointNet layer of the PointNet++ model, due to the effect of the multi-layer perceptron, the feature dimension of the point cloud changes as follows: 3+d in →d mlp1 →d mlp2 →d mlp3 .
[0051] The DenseNet model realizes the propagation of information flow by using the dense connection mechanism, and splices the features obtained by all previous layers in the dimension as the input of the next layer, so as to more fully utilize the feature information of the network.
[0052] Inspired by the DenseNet model, the feature is extracted in the form of dense connection in the feature extraction layer. The input point cloud feature dimension is 3+d in , and the dimension of the point cloud is improved to d mlp1 after the first feature extraction of the multi-layer perceptron. The feature of the original point cloud is spliced with the feature obtained by the first layer of the multi-layer perceptron as the input of the second layer of the multi-layer perceptron, and the dimension of the input point cloud feature becomes 3+d in +d mlp1Similarly, after feature extraction of the first layer of the multi-layer perceptron, the feature dimension of the point cloud is transformed into d mlp2 The features of the input point cloud, the features obtained by the first and second layers of the multi-layer perceptron are spliced as the input of the last layer of the multi-layer perceptron, and the feature dimension of the input point cloud is 3+d in +d mlp1 +d mlp2 The feature dimension of the output point cloud is d mlp3 After the maximum pooling operation, the feature information of the point cloud is obtained.
[0053] We respectively carried out classification and segmentation experiments on the ModelNet40 and ShapeNet data sets, and compared with PointNet and PointNet++, and the experimental results are shown in Table 1 and Table 2. The experimental results show that compared with PointNet and PointNet++, the method proposed in the application has improved classification accuracy and segmentation accuracy.
[0054] Table 1: Classification results on ModelNet40 dataset
[0055]
[0056] Table 2: Segmentation results on ShapeNet dataset
[0057]
[0058]
[0059] In summary, the application improves the original PointNet++ model to extract the features of the point cloud in the form of dense connection. And in the classification and segmentation tasks, the global and local features of the bottom and high layer point clouds are used to improve the accuracy of the results.
[0060] The above is only the preferred embodiment of the application, and does not limit the application in any other form. Those skilled in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and purposes of the application, and the scope of the application is defined by the claims and their equivalents.
Claims
1. A 3D point cloud classification segmentation method fusing a dense connection mechanism and PointNet++, characterized in that, The method comprises the following steps: Step 1, extracting global features and local features of the input point cloud at the bottom layer and the high layer by using two designed dense set sampling layer modules, wherein the dense set sampling layer module comprises a sampling layer, a grouping layer and a feature extraction layer; Step 2, obtaining the prediction score of each class by using the global features obtained by the dense set sampling layer module and the full connection network, and determining the class of the input point cloud according to the highest prediction score; Step 3, propagating the features of the last layer of sampling points to all points of the original point cloud by using a reverse interpolation method, fusing the propagated features with the global features and the local features obtained in step 1 in a skip connection manner, and obtaining the prediction score of each point in the point cloud by using a full connection network to realize the segmentation task; The step 1 specifically comprises: Step 1.1, set the input point cloud set containing N points as Each point contains (x, y, z) three-dimensional coordinates; Step 1.2, obtaining the global features and the local features of the point cloud at the bottom layer by using the first dense set sampling layer module; Step 1.3, obtaining the global features and the local features of the point cloud at the high layer by using the second dense set sampling layer module; The specific process of step 1.2 is as follows: Step 1.2.1, using the farthest distance sampling method in the sampling layer to sample 512 points from the original point cloud as sampling points; Step 1.2.2, taking the sampling points obtained in step 1.2.1 as center points in the grouping layer, and taking the nearest 32 points to each center point within a radius of 0.2 as neighbor points, and when the number of neighbor points is less than 32, repeating the first point coordinate; Step 1.2.3, using a dense PointNet layer as a feature extraction layer to extract local feature information of the point cloud, wherein the input of the multi-layer perceptron in the dense PointNet layer is the feature splicing form of all previous output layers, and the local features of the point cloud in the region divided in step 1.2.2 are extracted; Step 1.2.4, using a dense PointNet layer to extract global features of the point cloud, wherein the dense PointNet layer directly extracts the features of all points sampled in step 1.2.1 as the global features of the point cloud; The specific process of step 1.3 is as follows: Step 1.3.1, using the farthest distance sampling method to sample 128 points from the 512 points in the first dense set sampling layer module; Step 1.3.2, taking the sampling points obtained in step 1.3.1 as center points in the grouping layer, and taking the nearest 64 points to each center point within a radius of 0.4 as neighbor points, and when the number of neighbor points is less than 64, repeating the first point coordinate; Step 1.3.3, using a dense PointNet layer to extract local features of the point cloud, wherein the input of the multi-layer perceptron in the dense PointNet layer is the feature splicing form of all previous output layers, and the dense PointNet layer extracts the local features of the point cloud in the region divided in step 1.3.2; Step 1.3.4, using a dense PointNet layer to extract global features of the point cloud, wherein the dense PointNet layer directly extracts the features of all points sampled in step 1.3.1 as the global features of the point cloud; The specific process of step 3 is as follows: Step 3.1, the global features obtained in the second dense set sampling layer module are up-sampled point feature propagation using the reverse interpolation method, and the local features obtained in step 1.3.3 are spliced using the method of jump connection; Step 3.2, a 1x1 convolution kernel is used to update the obtained point cloud features; Step 3.3, the point cloud features obtained in step 3.2 are up-sampled point feature propagation using the reverse interpolation method, and the global features obtained in step 1.2.3 and step 1.2.4 are spliced with the local using the method of jump connection; Step 3.4, a 1x1 convolution kernel is used to update the obtained point cloud features; Step 3.5, the point cloud features obtained in step 3.4 are up-sampled point feature propagation using the reverse interpolation method, and the features corresponding to all points in the original point cloud are obtained; Step 3.6, a 1x1 convolution kernel is used to update the obtained point cloud features; Step 3.7, a fully connected network is used to obtain the prediction score of each point in the point cloud, and the class of each point is determined according to the prediction score, realizing the segmentation task.
2. The 3D point cloud classification and segmentation method of fusing the dense connection mechanism and PointNet++, according to claim 1, characterized in that: The specific process of step 2 is as follows: Step 2.1, the point cloud bottom layer features obtained in step 1.2.4 and step 1.3.4 are spliced with the point cloud high layer features; Step 2.2, the point cloud global features obtained in step 2.1 are input into a fully connected network to obtain the prediction score of k classes, and the class with the highest prediction score is the class of the input point cloud. In order to prevent overfitting, a dropout layer with a dropout rate of 0.4 is added in the fully connected network.