Point cloud classification segmentation method based on adaptive convolution and self-attention

Abstract

The invention discloses a point cloud classification and segmentation method, and particularly relates to a point cloud classification and segmentation method based on adaptive convolution and self-attention, and the method comprises the following steps: S1, reading three-dimensional point cloud data, carrying out the sampling and preprocessing of the point cloud data, and inputting the preprocessed point cloud data into a model; s2, inputting the three-dimensional point cloud data into an adaptive residual convolution module, and performing feature extraction; s3, combining the plurality of local features obtained in the step S2, then passing through a multi-layer perceptron, and finally passing through a self-attention module to obtain advanced features; and S4, carrying out maximum pooling operation and average pooling operation on the advanced features, splicing the advanced features to obtain high-dimensional features, inputting the high-dimensional features to a full connection layer for dimension reduction, and outputting classification and segmentation results of the three-dimensional point cloud data.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a point cloud classification and segmentation method based on adaptive convolution and self-attention. Background Art

[0002] With the rapid development of 3D sensor technology, 3D point cloud data has been widely used in many fields such as autonomous driving, robot navigation, virtual reality, and building information modeling. 3D point cloud data can intuitively capture the shape and spatial relationship of objects, and is an important basis for environmental perception, scene reconstruction, and object recognition. However, the unstructured, disordered, and sparse nature of point cloud data poses a huge challenge to its effective processing. How to achieve efficient and accurate processing in point cloud classification and segmentation tasks has become a key research topic.

[0003] Existing point cloud processing methods mainly include voxel-based, multi-view-based and point-based methods. Voxel-based methods such as VoxNet extract features by voxelizing point clouds and applying 3D convolutional networks. Although this method is helpful for feature extraction, the voxelization process inevitably leads to loss of spatial resolution and significantly increases computing and storage requirements. Multi-view-based methods such as MVCNN project point clouds into multiple 2D views and then process them using 2D convolutional networks. Although this method reduces the reliance on 3D structures, it cannot avoid the loss of 3D information, especially in complex scenes. Point-based methods directly use the original point cloud data input to avoid information loss in preprocessing. PointNet, as a pioneering work of this type of method, successfully retains the original structure of the point cloud by processing each point independently and extracting global features through global pooling operations. However, PointNet only focuses on global features and fails to fully capture local geometric structure information, which limits its performance in complex scenes. PointNet++ enhances the ability to extract local features by introducing a hierarchical feature learning mechanism, but it still has certain limitations when processing neighborhood point relationships. PointASNL and So-Net further optimized the relationship modeling between points and neighboring points on this PointNet++, but local feature extraction is still insufficient. DGCNN aggregates the features of each point and its neighborhood points through the EdgeConv module, significantly improving the ability to capture local geometric structures and enhancing the classification and segmentation performance of the model. However, even DGCNN still faces challenges in computational efficiency and global feature capture when processing complex and large-scale point cloud data. PointCNN can flexibly process irregular point cloud data and achieve more accurate feature learning by designing a new type of X-Conv operation; SpiderCNN uses polynomial functions to define convolution kernels to extract neighborhood features; PCNN further improves the ability to capture local features through spatial coordinate convolution kernel design; 3D-GCN proposes the concept of deformable kernels, and the shape and weight of each convolution kernel are learnable during training; AdaptConv effectively captures the relationship between points in different semantic parts through adaptive kernels; KPConv uses deformable kernel convolution mechanism to effectively capture local features. However, these methods still have limitations in the feature extraction process, discarding many features in the neighborhood and failing to fully capture the global information in the point cloud. Summary of the invention

[0004] In order to solve the above-mentioned problems in the prior art, the present invention provides a point cloud classification and segmentation method based on adaptive convolution and self-attention to solve the current technical problems.

[0005] The technical solution adopted by the present invention to solve its technical problem is: The present invention provides a point cloud classification and segmentation method based on adaptive convolution and self-attention, comprising the following steps: Step S1, reading three-dimensional point cloud data, sampling and preprocessing the point cloud data, and inputting the preprocessed point cloud data into the model; Step S2, inputting the 3D point cloud data into the adaptive residual convolution module, after the classification task is subjected to feature extraction by four layers of adaptive residual convolution modules with different dimensions, four local features are obtained respectively, and after the segmentation task is subjected to feature extraction by five layers of adaptive residual convolution modules with different dimensions, five local features are obtained respectively; Step S3: stitch together the multiple local features obtained in step S2, merge the features, and form a global fine-scale feature. X feature , then pass through the multi-layer perceptron, and finally pass through the self-attention module to obtain high-level features F h ; Step S4: high-level features F h The maximum pooling operation and the average pooling operation are performed respectively and then concatenated to obtain high-dimensional features, which are then input into the fully connected layer for dimensionality reduction and output the classification and segmentation results of the three-dimensional point cloud data.

[0006] Preferably, the step S1 can be divided into the following sub-steps: Step S11: read the 3D point cloud data; the ModelNet40 dataset is used for the classification task, and the ShapeNet dataset is used for the segmentation task for training and testing; Step S12: downsampling the three-dimensional coordinates of the original three-dimensional point cloud data; Step S13: preprocessing the data, using a data enhancement method of random rotation, movement and scaling; Step S14: input the preprocessed point cloud data into the model.

[0007] Preferably, step S2 can be divided into the following sub-steps: Step S21: Enter a size N *3 3D point cloud data X in To the adaptive residual convolution module AdaptiveResConv {64}, extract the features and get N *64 feature vector, record the first layer of local features X feature1 ; ; Step S22: The previous layer X feature1Input the adaptive residual convolution module AdaptiveResConv{64} to extract features and obtain N *64 feature vector, record the second layer of local features X feature2 ; ; Step S23: The second layer local features obtained in step S22 X feature2 Input the adaptive residual convolution module AdaptiveResConv {128} to extract features and get N *128 feature vector, record the third layer local features X feature3 ; ; Step S24: The third layer local features obtained in step S23 X feature3 Input the adaptive residual convolution module AdaptiveResConv {256} to extract features and get N *256 feature vector, record the fourth layer local features X feature4 ; ; Step S25: If it is a segmentation task, it needs to go through one more layer of adaptive residual convolution module AdaptiveResConv {512} than the classification task to obtain N *512 feature vector, record the fifth layer local features X feature5 ; .

[0008] Preferably, step S3 can be divided into the following sub-steps: Step S31: If it is a classification task, X feature1 , X feature2 , X feature3 , X feature4 Four layers of local features are combined to form global fine-density features X feature If it is a segmentation task, then S2 X feature1 , X feature2 , X feature3 , Xfeature4 , X feature5 To carry out splicing and merging; ; Step S32: X feature Send it to the multi-layer perceptron network MLP{1024}, and get N *1024-dimensional features, denoted as F 1024 ; Step S33: Then, the X 1024 The features are fed into the SelfAttention module to obtain high-level features F h ; .

[0009] Preferably, the step S4 can be divided into the following sub-steps: Step S41: High-level features F h Perform maximum pooling and average pooling operations respectively and then concatenate them to obtain high-dimensional features; Step S42: The high-dimensional features obtained in step S41 are passed through a fully connected layer, and finally the classification result of the point cloud is output; Step S43: Connect the high-dimensional features obtained in step S41 with the 1024 features of each point to obtain the global features F global At the same time, the object label is one-hot encoded and converted into feature representation, which is consistent with the global feature F global Connect, pass through the fully connected layer, and finally output the segmentation result of the point cloud.

[0010] The beneficial effects of the present invention are: The adaptive residual convolution module designed in the present invention is designed based on the ideas of adaptive convolution and residual connection, and effectively integrates local neighborhood information and global context information by fusing main features with detailed features.

[0011] The self-attention module designed in the present invention weakens redundant features and enhances the sensitivity of the model to key features by dynamically adjusting the attention weights.

[0012] The present invention is a point cloud classification and segmentation method based on adaptive convolution and self-attention, which can show excellent point cloud data processing capabilities when processing complex point cloud data, fully extract deep semantic information of point clouds, and accurately represent the feature correspondence between 3D points. The method proposed by the present invention improves the accuracy of point cloud classification and segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 This is a flowchart of the point cloud classification and segmentation method based on adaptive convolution and self-attention of the present invention.

[0014] Figure 2 This is a flowchart of step S1 of the point cloud classification and segmentation method based on adaptive convolution and self-attention of the present invention.

[0015] Figure 3 This is a flowchart of step S2 of the point cloud classification and segmentation method based on adaptive convolution and self-attention of the present invention.

[0016] Figure 4 This is a structural diagram of the adaptive residual convolution module of the point cloud classification and segmentation method based on adaptive convolution and self-attention of the present invention.

[0017] Figure 5 This is a flowchart of step S3 of the point cloud classification and segmentation method based on adaptive convolution and self-attention of the present invention.

[0018] Figure 6 This is a structural diagram of the self-attention module of the point cloud classification and segmentation method based on adaptive convolution and self-attention of the present invention.

[0019] Figure 7 This is a flowchart of step S4 of the point cloud classification and segmentation method based on adaptive convolution and self-attention of the present invention.

[0020] Figure 8 This is a method model diagram of the point cloud classification and segmentation method based on adaptive convolution and self-attention of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] A point cloud classification and segmentation method based on adaptive convolution and self-attention provided in an embodiment of the present invention includes four steps.

[0023] Step S1: read the three-dimensional point cloud data, sample and preprocess the point cloud data, and input the preprocessed point cloud data into the model.

[0024] like Figure 2 As shown, specifically including: Step S11: Read 3D point cloud data. The ModelNet40 dataset is used for classification tasks, and the ShapeNet dataset is used for segmentation tasks for training and testing; Step S12: downsample the three-dimensional coordinates of the original point cloud and select the number of sampling points. For the classification task, 1024 sampling points are selected, and for the segmentation task, 2048 sampling points are selected; Step S13: preprocessing the data, using a data enhancement method of random rotation, movement and scaling; Step S14: input the preprocessed point cloud data into the model.

[0025] Step S2: The model accepts a size of N *3 Input data, N The data is 1024 or 2048, and the data is input into the adaptive residual convolution module. The classification task needs to go through four layers of adaptive residual convolution modules with different dimensions for feature extraction, and four local features are obtained respectively; the segmentation task has one more layer, and five local features are obtained. Figure 3 As shown, specifically including: Step S21: Enter a size N *3 Point cloud data X in To the adaptive residual convolution module AdaptiveResConv {64}, extract the features and get N *64 feature vector, record the first layer of local features X feature1 ; (1); Step S22: The first layer of local features obtained from the previous layer X feature1 Input into the adaptive residual convolution module AdaptiveResConv {64}, extract features, and get N *64 feature vector, record the second layer of local features X feature2 ; (2); Step S23: The second layer local features obtained in step S22X feature2 Input the adaptive residual convolution module AdaptiveResConv {128} to extract features and get N *128 feature vector, record the third layer local features X feature3 ; (3); Step S24: The third layer local features obtained in step S23 X feature3 Input the adaptive residual convolution module AdaptiveResConv {256} to extract features and get N *256 feature vector, record the fourth layer local features X feature4 ; (4); Step S25: If it is a segmentation task, it needs to go through one more layer of adaptive residual convolution module AdaptiveResConv{512} to obtain N *512 feature vector, record the fifth layer local features X feature5； (5); The structure of the adaptive residual convolution module AdaptiveResConv is as follows Figure 4 As shown: The design ideas of the AdaptiveResConv module include the ideas of adaptive convolution and residual connection.

[0026] The AdaptiveResConv module consists of two core parts: Primary Feature Extraction and Fine-density Feature Extraction.

[0027] In the Primary Feature Extraction section, record X ={ x i ︱ i =1,2,…, N}∈ℝ N *3 is the original point cloud, F in is the input of the module. First, F inThe global context information feature F1 is obtained through the get graph feature function and GraphConv. The main task of the get graph feature function is to aggregate the neighborhood features of each node to the node to generate a richer feature representation; while GraphConv further extracts and learns these feature representations to capture the deep features in the graph structure.

[0028] (6).

[0029] In addition, the original point cloud X is constructed through the K-nearest neighbor algorithm (KNN) and the get graph feature function to construct the spatial relationship between each point and its K nearest neighbor points, forming a K-nearest neighbor graph (K-Nearest Neighbor Graph), and then generating a graph feature (Graph Feature).

[0030] (7).

[0031] The definition of the constructed graph structure Graph Feature is: (8).

[0032] in V express N A local set of vertices, E Represents a collection of edges between vertices. Graph Feature helps capture local geometric features in point clouds, thereby improving feature expression capabilities.

[0033] (9); (10).

[0034] Then, by using the global context information feature F 1 Perform matrix multiplication with graph feature to generate adaptive features F adapt . Adaptive Features F adapt Not only does it retain the global context information, but it also provides a more detailed description of the structure of the point cloud data.

[0035] (11); Next, F adapt with by F inEdge features obtained by EdgeConv F edge Perform feature fusion.

[0036] (12); (13); Among them, the EdgeConv module aims to make full use of the features of the K nearest neighbor points of each point and aggregate these features to generate edge features, where the edge features are: (14); This feature aggregation not only combines the global shape information ( x i ) and local neighborhood information ( x j - x i ), and also use the maximum pooling operation to highlight important features and reduce the impact of redundant information on the final feature representation.

[0037] (15); (16).

[0038] In addition, the self-attention mechanism is applied to model the relationship between features, dynamically allocate attention weights between different points, remove redundant features, and finally generate the main features. F Primary .

[0039] (17).

[0040] For the Fine-density Feature Extraction part, the LFA (Local Feature Aggregation) convolution layer is used, which passes a 1* k The convolution kernel of the local area k The feature weights of neighbor points are calculated, and the weighted sum of the features of these neighbor points is added. Then, the self-attention module is used to further improve the feature distinguishing ability, ensure that the detailed features are fully expressed, and thus generate detailed local context features. F detail .

[0041] (18); (19).

[0042] Finally, the features obtained by Primary Feature Extraction and Fine-density Feature Extraction are merged to obtain the output of the module. F out This multi-level feature fusion strategy helps to capture rich information of point cloud data at different scales, thereby improving the accuracy and robustness of point cloud feature representation.

[0043] (20).

[0044] The function of the adaptive residual convolution module AdaptiveResConv is to enhance the representation ability of point cloud data by extracting and fusing features at different levels, and finally generate a new global feature representation through feature integration.

[0045] Step S3: Splice together the multiple local features obtained in step S2 and merge the features to form a global fine-scale feature. X feature , and then through the multi-layer perceptron MLP{1024}, the output dimension is N *1024 features, and finally pass through the self-attention module to obtain high-level features F h .like Figure 5 As shown, specifically including: Step S31: X feature1 , X feature2 , X feature3 , X feature4 Four layers of local features are combined to form global fine-density features X feature The segmentation task requires X feature5 To carry out splicing and merging; (twenty one); Step S32: Obtaining the global fine-density features after splicing X feature After that, X feature Send it to the multi-layer perceptron network MLP{1024}, and get N *1024-dimensional features, denoted as F 1024 ; Step S33: Then, the X 1024The features are sent to the SelfAttention module, which dynamically adjusts the attention weights to weaken redundant features, enhance the model's sensitivity to key features, and obtain high-level features. F h .

[0046] (twenty two).

[0047] Among them, the structure of the self-attention module is as follows Figure 6 shown.

[0048] The self-attention mechanism has been widely used in many fields in recent years, such as image processing and natural language processing. Its core advantage is that it can dynamically adjust the correlation between different features, thereby enhancing the model's ability to focus on key features. The structure of the self-attention module introduced in this paper is shown in the figure. The input data is mapped to the query matrix through linear transformation. Q , Keyword Matrix K and the numerical matrix V , these matrices are all passed through the learnable weight matrix W q , W k and W v get.

[0049] (twenty three).

[0050] The key to the self-attention mechanism is to calculate the query matrix Q and keyword matrix K Specifically, we first calculate the dot product between the two through matrix multiplication, and then normalize the result to obtain the attention matrix A . Attention Matrix A It represents the relative importance between each pair of features, that is, the weight that should be given to each feature when considering other features comprehensively.

[0051] (twenty four); (25).

[0052] Next, the attention matrix A With numerical matrices V Weighted multiplication to generate new feature representation O This ensures that the final generated features not only contain the original information of the input, but also enhance the correlation with the key features through weighted sum operations, thereby improving the quality of feature representation.O Sum it with the input matrix N to get the output of the module.

[0053] (26); (27).

[0054] This mechanism enhances the model's sensitivity to key features by assigning different weights to different features, while effectively suppressing the interference of minor features, thereby improving the performance of the overall model. Figure 6 middle' multiple ' is matrix multiplication,' sum ' is matrix addition.

[0055] Step S4: High-level features F h After performing the maximum pooling operation and the average pooling operation respectively, the high-dimensional features are obtained, which are then input into the fully connected layer for dimensionality reduction, and finally the classification and segmentation results of the point cloud are output. Figure 7 As shown, specifically including: Step S41: High-level features F h Perform the maximum pooling operation and the average pooling operation respectively and then concatenate them to obtain high-dimensional features; Step S42: The high-dimensional features are passed through the fully connected layer MLP {512, 256, c}, and finally output the classification results of the point cloud. c is the number of categories for classification; Step S43: Connect the high-dimensional features with the 1024 features of each point to obtain the global features F global At the same time, the object label is one-hot encoded and converted into feature representation, which is consistent with the global feature F global Connect, through the fully connected layer MLP {512, 256, s}, and finally output the segmentation result of the point cloud. s is the number of categories for classification.

[0056] A point cloud classification and segmentation method model based on adaptive convolution and self-attention used in an embodiment of the present invention is as follows: Figure 8 shown.

[0057] The present invention proposes a point cloud classification and segmentation method based on adaptive convolution and self-attention. In the feature extraction process, the local neighborhood information and global context information are integrated, so that the model can more flexibly handle the relationship between points, thereby improving the model's ability to understand complex point clouds. The method proposed by the present invention improves the accuracy of point cloud classification and segmentation tasks.

[0058] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A point cloud classification and segmentation method based on adaptive convolution and self-attention, characterized in that: The following steps are involved: Step S1: read the three-dimensional point cloud data, sample and preprocess the point cloud data, and input the preprocessed point cloud data into the model; Step S2: Input the 3D point cloud data into the adaptive residual convolution module. After the classification task is subjected to feature extraction by four layers of adaptive residual convolution modules with different dimensions, four local features are obtained respectively. After the segmentation task is subjected to feature extraction by five layers of adaptive residual convolution modules with different dimensions, five local features are obtained respectively. Step S3: Splice together the multiple local features obtained in step S2 and merge the features to form a global fine-scale feature. X feature , then pass through the multi-layer perceptron, and finally pass through the self-attention module to obtain high-level features F h ; Step S4: High-level features F h The maximum pooling operation and the average pooling operation are performed respectively and then concatenated to obtain high-dimensional features, which are then input into the fully connected layer for dimensionality reduction and output the classification and segmentation results of the three-dimensional point cloud data.

2. According to the point cloud classification and segmentation method based on adaptive convolution and self-attention according to claim 1, the step S1 can be divided into the following sub-steps: Step S11: read the 3D point cloud data; the ModelNet40 dataset is used for the classification task, and the ShapeNet dataset is used for the segmentation task for training and testing; Step S12: downsampling the three-dimensional coordinates of the original three-dimensional point cloud data; Step S13: preprocessing the data, using a data enhancement method of random rotation, movement and scaling; Step S14: input the preprocessed point cloud data into the model.

3. According to the point cloud classification and segmentation method based on adaptive convolution and self-attention according to claim 1, the step S2 can be divided into the following sub-steps: Step S21: Enter a size N *3 3D point cloud data X in To the adaptive residual convolution module AdaptiveResConv {64}, extract the features and get N *64 feature vector, record the first layer of local features X feature1 ; ； Step S22: The previous layer X feature1 Input into the adaptive residual convolution module AdaptiveResConv {64}, extract features, and get N *64 feature vector, record the second layer of local features X feature2 ; ； Step S23: The second layer local features obtained in step S22 X feature2 Input the adaptive residual convolution module AdaptiveResConv {128} to extract features and get N *128 feature vector, record the third layer local features X feature3 ; ； Step S24: The third layer local features obtained in step S23 X feature3 Input the adaptive residual convolution module AdaptiveResConv {256} to extract features and get N *256 feature vector, record the fourth layer local features X feature4 ; ； Step S25: If it is a segmentation task, it needs to go through one more layer of adaptive residual convolution module AdaptiveResConv {512} than the classification task to obtain N *512 feature vector, record the fifth layer local features X feature5 ; 。 4. According to the point cloud classification and segmentation method based on adaptive convolution and self-attention according to claim 1, the step S3 can be divided into the following sub-steps: Step S31: If it is a classification task, X feature1 , X feature2 , X feature3 , X feature4 Four layers of local features are combined to form global fine-density features X feature If it is a segmentation task, then S2 X feature1 , X feature2 , X feature3 , X feature4 , X feature5 To carry out splicing and merging; ； Step S32: X feature Send it to the multi-layer perceptron network MLP{1024}, and get N* 1024-dimensional features, denoted as F 1024 ; Step S33: Then, the X 1024 The features are fed into the SelfAttention module to obtain high-level features F h ; 。 5. According to the point cloud classification and segmentation method based on adaptive convolution and self-attention according to claim 1, the step S4 can be divided into the following sub-steps: Step S41: High-level features F h Perform maximum pooling and average pooling operations respectively and then concatenate them to obtain high-dimensional features; Step S42: The high-dimensional features obtained in step S41 are passed through a fully connected layer, and finally the classification result of the point cloud is output; Step S43: Connect the high-dimensional features obtained in step S41 with the 1024 features of each point to obtain the global features F global At the same time, the object label is one-hot encoded and converted into feature representation, which is consistent with the global feature F global Connect, pass through the fully connected layer, and finally output the segmentation result of the point cloud.

Images