A point cloud semantic segmentation method and system based on local neighborhood attention
By constructing a local neighborhood attention module that integrates local and global positional encoding, the problems of information loss and difficulty in extracting global geometric structure in point cloud learning are solved, achieving higher accuracy and robustness in point cloud semantic segmentation.
Patent Information
- Application Number
- CN202411977023.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-10-28
- Estimated Expiration
- 2044-12-31
AI Technical Summary
Existing voxel-based and multi-view-based methods suffer from significant information loss during voxelization and projection, making it difficult for the model to extract the fine geometric structure of the point cloud. Previous point cloud learning methods based on attention mechanisms mostly only consider the relative positional relationships of points within the neighborhood, while ignoring the spatial positional relationships between neighborhoods, making it difficult for the model to learn the global geometric structure information of the point cloud.
A point cloud semantic segmentation method based on local neighborhood attention is adopted. By constructing a local neighborhood attention module that integrates local and global positional encoding, vector attention is used to calculate point cloud features. Local and global positional encoding is used to provide the spatial positional relationships between points in the neighborhood and between neighborhoods. Non-downsampling and downsampling attention modules are designed for feature extraction, and a deep learning network based on local neighborhood attention is constructed.
It improves the robustness and semantic segmentation accuracy of the model, effectively learns the local structure and global spatial geometric structure information of point clouds, and improves the accuracy of point cloud semantic segmentation.
Smart Images
Figure CN119785032B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of 3D point cloud processing and deep learning technology, and particularly relates to a point cloud semantic segmentation method and system based on local neighborhood attention. Background Technology
[0002] 3D point clouds are one of the most important ways to represent 3D data, facilitating environmental perception. Currently, deep learning is developing rapidly, and point cloud deep learning is becoming increasingly important in the field of computer vision. Point cloud semantic segmentation has wide applications, helping robots recognize their surroundings, aiding scene understanding and interaction in augmented / virtual reality, and even digitizing cultural relics in cultural heritage preservation. However, designing efficient point cloud semantic segmentation networks is quite challenging.
[0003] Because point cloud data is irregular and disordered, early methods involved converting the point cloud into a regular voxel mesh and processing it using 3D convolution, or projecting it onto a 2D planar view for convolution to extract features. However, this approach easily loses fine local geometric information during voxelization and projection, making it difficult to improve model accuracy. Furthermore, this method is primarily suitable for point cloud classification and not easily extended to segmentation tasks. To address this issue, this invention abandons the approach of regularizing the point cloud before using convolution to learn features, and instead processes the point cloud directly to prevent information loss.
[0004] In recent years, Transformers, with their ability to model long-range dependencies, have been frequently used in point cloud deep learning. Their core attention mechanism adapts to the characteristics of point clouds, improving model performance by focusing on different parts of the data. However, current methods in this field only use local position encoding when calculating local attention, considering only the positional relationships of points within a neighborhood and neglecting the spatial relationships between neighborhoods, resulting in a weak global shape perception. This invention constructs a local neighborhood attention module that integrates local and global position encoding for point cloud learning. This attention mechanism uses vector attention and employs two different position encodings to provide the positional relationships of points within a neighborhood and the spatial relationships between neighborhoods, respectively. This allows the model to adapt to the local structure of point cloud data while also adapting to the spatial geometry of neighborhoods globally, enriching the features extracted by the attention mechanism.
[0005] Based on the above analysis, the problems and defects of the existing technology are as follows: existing voxel-based and multi-view-based methods will cause a lot of information loss during voxelization and projection, resulting in the model being unable to extract the fine geometric structure of the point cloud; most previous point cloud learning methods based on attention mechanisms only consider the relative positional relationship of points in the neighborhood, while ignoring the spatial positional relationship between neighborhoods, making it difficult for the model to learn the global geometric structure information of the point cloud.
[0006] Based on this, the present invention designs a point cloud semantic segmentation method and system based on local neighborhood attention to solve the above problems. Summary of the Invention
[0007] The purpose of this invention is to address the problem that existing voxel-based and multi-view-based methods suffer from significant information loss during voxelization and projection, resulting in the model's inability to extract the fine geometric structure of point clouds. Furthermore, previous point cloud learning methods based on attention mechanisms mostly only consider the relative positional relationships of points within a neighborhood, neglecting the spatial positional relationships between neighborhoods, which makes it difficult for the model to learn the global geometric structure information of point clouds. Therefore, this invention proposes a point cloud semantic segmentation method and system based on local neighborhood attention.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] A point cloud semantic segmentation method and system based on local neighborhood attention includes the following steps:
[0010] S101, divide the 3D point cloud data into training, validation, and test sets and preprocess them;
[0011] S102, Construct a deep learning network based on local neighborhood attention, train it using the training set, and obtain the optimal model using the validation set;
[0012] S103, input the test set into the optimal model to obtain the semantic segmentation result.
[0013] As a further description of the above technical solution:
[0014] The point cloud semantic segmentation method based on local neighborhood attention includes the following steps:
[0015] Step 1: Divide the 3D point cloud data into training set, validation set, and test set, and preprocess the point cloud dataset.
[0016] Step 2: Construct a local neighborhood attention module that integrates local and global location encoding, and build a deep learning network based on local neighborhood attention;
[0017] Step 3: Train the network using the training set, validate the network using the validation set during training, and optimize the model.
[0018] Step four: Input the test set into the obtained optimal deep network model to obtain the semantic segmentation results of the 3D point cloud.
[0019] As a further description of the above technical solution:
[0020] The preprocessing of the point cloud dataset in step one includes:
[0021] The coordinates of points in each point cloud file in the point cloud dataset are normalized to the interval [0,1] in order to divide the neighborhood range and perform standardization.
[0022] As a further description of the above technical solution:
[0023] Step two involves constructing a local neighborhood attention module that integrates local and global location encoding, and building a deep learning network based on local neighborhood attention, including:
[0024] (1) Construct a local neighborhood attention module that integrates local positional encoding and global positional encoding:
[0025] The local neighborhood attention module, which integrates local and global location encoding, uses vector attention to compute point cloud features and aggregates them to the center point. It also uses two types of location encoding to provide the model with spatial relationships between points within and between neighborhoods, respectively.
[0026] For a point cloud P containing n points, its features and coordinates are represented as follows: and Sample m points from the point cloud P as center points, and their features and coordinates are represented as follows: and Directly use the center point feature F center After linear mapping, the query vector Q is obtained, then:
[0027]
[0028] In the formula, α represents a linear layer, which reduces the number of feature channels from C. in Mapped to C out .
[0029] To obtain the key vector and value vector, the input point cloud features F are first processed. in After two different linear mappings, and then with the center point X center Centered on a point, a ball query is used to divide the neighborhood. Based on this, the two features obtained from the linear mapping are divided into m subsets, each containing k points, resulting in a key vector K and a value vector V, and the corresponding coordinates X of the neighboring points. neighbor In order to calculate the position code later:
[0030]
[0031] In the formula, β and γ are linear layers, which reduce the number of feature channels from C. in Mapped to C out group(a,(b,c)) means dividing the neighborhood of feature a into groups based on b as the center and c as the center; k represents the number of points in each neighborhood.
[0032] Subtract the Q vector from the K vector, and after passing through a multi-layer perceptron (MLP) and normalization, the attention weights can be obtained. However, to help the model learn the structural relationships between neighborhoods, global position encoding is added. In addition, to prevent the network from overfitting, random dropout is also applied to the attention weights:
[0033]
[0034] In the formula, represents the MLP composed of a convolutional layer, batch normalization, and Swish activation function, η represents the softmax normalization function, and drop represents the random dropout function.
[0035] In addition to the structural relationships between neighborhoods, the positional relationships of points within neighborhoods are also crucial for point cloud learning. Therefore, a local position encoding is added to the V vector. After that, the attention weights W are used for weighted summation to aggregate the features of points within each neighborhood.
[0036] To adapt to feature extraction in different situations, two attention modules are designed. The first attention module does not perform downsampling (m = n), and it is called the non-downsampling attention module. It divides the neighborhood with each point as the center and calculates the attention:
[0037] <
[0043] In the formula, λ and θ are both MLPs composed of convolutional layers, batch normalization, and Swish activation functions.
[0044] Considering global location coding POS global It is obtained from the coordinates of the neighborhood center point. Among Q, K, and V, only Q is obtained from the features of the neighborhood center point. Therefore, global position encoding (POS) is added when calculating the attention weights. global This provides spatial location information between neighborhoods for learning feature weights; local location encoding (POS) local It is then added after the V vector to provide local location information for points in the neighborhood.
[0045] (2) Construct a deep learning network based on local neighborhood attention:
[0046] A deep learning network based on local neighborhood attention is used for point cloud segmentation and classification. It consists of four modules: pre-embedding, downsampling feature extraction, residual block, and upsampling feature extraction. Its main feature aggregation operators are the non-downsampling attention module and the downsampling attention module.
[0047] Pre-embedding. Before extracting deep features, we first use a pre-embedding module to expand the point cloud feature channels and extract shallow features from the point cloud. The pre-embedding consists of two consecutive non-downsampling attention modules.
[0048] Downsampling feature extraction. The encoder part of the network downsamples the point cloud layer by layer, expanding the number of feature channels while reducing the number of points, thus learning the deep features of the point cloud. Downsampling feature extraction is the main component of the encoder, consisting of a non-downsampling attention module and a downsampling attention module. The non-downsampling attention module is used to extract fine structural information, while the downsampling attention module is used to downsample the point cloud while aggregating neighborhood features.
[0049] Residual blocks. For segmentation tasks, deeper structural features are needed to learn the geometry of the point cloud. Therefore, a residual block is added after each layer of downsampled feature extraction in the segmentation task. It contains a non-downsampled attention module that connects its input residuals to the output, which is then passed through an activation function. In practice, the number of residual blocks added after downsampled feature extraction can be adjusted according to different tasks or datasets to obtain better results.
[0050] Upsampling Feature Extraction. For segmentation tasks, the encoder downsamples the point cloud. To obtain the classification score for each point, the decoder needs to upsample the point cloud. The point cloud features from the previous layer are upsampled using trilinear interpolation and then concatenated with the output features of the corresponding layer in the encoder. Finally, a non-downsampling attention module extracts the fused point cloud features and feeds them into the next layer.
[0051] This network architecture is used for point cloud segmentation and classification. For segmentation tasks, the encoder part of the network consists of pre-embedding, downsampling feature extraction, and residual blocks. The decoder part uses upsampling feature extraction to upsample the features obtained from the previous layer and connects them to the output of the corresponding layer of the encoder. The last layer of the decoder obtains the features of each point, and finally uses an MLP to map them to the number of segmentation classes to obtain the score for each point. For classification tasks, the encoder part removes the residual blocks compared to the segmentation task because the deep feature extraction of the residual blocks would cause the network to overlearn the details of the point cloud, leading to model overfitting. After the last downsampling feature extraction, two non-downsampling attention modules further extract the point cloud features. The decoder part performs max pooling on the obtained features to obtain the global features of the point cloud, and finally uses an MLP to obtain the classification score.
[0052] As a further description of the above technical solution:
[0053] Step three, which involves training the network using the training set, validating the network using the validation set during training, and optimizing the model, includes:
[0054] (1) Train the network using the training set, input the training set into the network, calculate the cross-entropy loss, and update the network parameters;
[0055] (2) Test using the validation set and select the optimal network model;
[0056] The validation set is input into the model, and data processing includes normalization and standardization to obtain predicted labels. For segmentation tasks, the mean intersection-over-union ratio (mIoU) is used as the evaluation metric, mIoU = TP / (TP + FP + FN), where TP represents true positives, FP represents false positives, and FN represents false negatives. For classification tasks, the overall accuracy is used as the evaluation metric. The validation set accuracy is calculated by comparing the predicted labels with the actual labels, and the optimal model on the validation set during training is saved.
[0057] As a further description of the above technical solution:
[0058] The semantic segmentation results of the 3D point cloud obtained by inputting the test set into the optimal deep network model in step four include:
[0059] (1) Input the test set into the model selected in step three. Data processing includes normalization and standardization, and then the predicted labels are obtained.
[0060] (2) Calculate the accuracy of the test set.
[0061] As a further description of the above technical solution:
[0062] A point cloud semantic segmentation system based on local neighborhood attention includes:
[0063] The 3D point cloud data preprocessing module is used to divide the dataset into training set, validation set and test set according to a certain ratio, and to preprocess the dataset and normalize and standardize the point cloud data.
[0064] The point cloud semantic segmentation network construction module is used to build a local neighborhood attention module that integrates local and global positional encoding, and to build a deep learning network based on local neighborhood attention.
[0065] The point cloud semantic segmentation network training module is used to train the network using the training set and to validate the network and select the network model during the training process using the validation set.
[0066] The point cloud semantic segmentation module includes point cloud classification and point cloud segmentation. It is used to input the test set into the selected optimal classification or segmentation model to obtain the classification or segmentation results of the point cloud.
[0067] As a further description of the above technical solution:
[0068] A point cloud semantic segmentation system based on local neighborhood attention includes: a computer device having a program for a point cloud semantic segmentation system based on local neighborhood attention stored and executable, and a processor capable of executing a point cloud semantic segmentation method based on local neighborhood attention.
[0069] As a further description of the above technical solution:
[0070] A point cloud semantic segmentation system based on local neighborhood attention includes: a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform a point cloud semantic segmentation method based on local neighborhood attention.
[0071] As a further description of the above technical solution:
[0072] A point cloud semantic segmentation system based on local neighborhood attention includes an information data processing terminal, which is used to implement the system.
[0073] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0074] First, addressing the technical problems existing in the prior art and the difficulty of solving them, this paper closely analyzes, in conjunction with the technical solution to be protected by this invention and the results and data obtained during the research and development process, how the technical solution of this invention solves the technical problems, and the inventive technical effects brought about by solving these problems. The specific description is as follows:
[0075] This invention employs a point cloud deep learning network based on local neighborhood attention for the classification and segmentation of 3D point clouds. This network learns the local structural information and global spatial geometric structure information of the point cloud by fusing local and global positional encoding through a local neighborhood attention module, thereby improving the robustness of the model and the accuracy of semantic segmentation. This invention yields a well-fitting 3D point cloud semantic segmentation model, effectively addressing the weakness of existing point cloud deep learning models in extracting global spatial geometric structure information, and significantly improving the accuracy of point cloud semantic segmentation.
[0076] Second, considering the technical solution as a whole or from a product perspective, the technical effects and advantages of the technical solution to be protected by this invention are specifically described as follows:
[0077] The point cloud semantic segmentation method based on local neighborhood attention provided by this invention can effectively improve the problem that previous point cloud learning methods based on attention mechanisms only consider the relative positions of points within the neighborhood. This enables the model to adapt to the local structure of point cloud data while also adapting to the spatial geometric structure of the neighborhood in the global range, thereby improving the robustness of the model and the accuracy of point cloud semantic segmentation.
[0078] Third, as supporting evidence of the inventiveness of this invention, it is also reflected in the following important aspects:
[0079] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:
[0080] This invention aims to improve upon the problem that attention-based point cloud learning methods only consider the relative positions of points within the neighborhood. It will bring many expected benefits and commercial value, improve the accuracy and reliability of point cloud semantic segmentation, enable robots to understand their surroundings more quickly, improve the robot's reaction speed, understand the surrounding scene faster, improve the robot's action efficiency, provide better robot services, and promote innovation and development in the field of robotics.
[0081] (2) The technical solution of the present invention solves a technical problem that people have long wanted to solve but have never been able to solve successfully:
[0082] This invention improves upon the problem that attention-based point cloud learning methods only consider the relative position information of points within the neighborhood, resulting in weak global structural information extraction capabilities. This invention enables the model to learn point clouds more comprehensively from both local and global spatial geometric information, making the classification and segmentation of point clouds more accurate. Attached Figure Description
[0083] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0084] Figure 1 This is a flowchart of the point cloud semantic segmentation method based on local neighborhood attention provided in the embodiments of the present invention;
[0085] Figure 2 This is a schematic diagram of the point cloud semantic segmentation method based on local neighborhood attention provided in the embodiments of the present invention.
[0086] Figure 3 This is a schematic diagram of a local neighborhood attention module structure that integrates local position encoding and global position encoding, provided in an embodiment of the present invention.
[0087] Figure 4 This is a schematic diagram of a point cloud semantic segmentation network structure based on local neighborhood attention provided in an embodiment of the present invention. Detailed Implementation
[0088] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0089] Please see the appendix Figure 1 -Appendix Figure 4 This invention provides a technical solution: a point cloud semantic segmentation method and system based on local neighborhood attention, comprising the following steps:
[0090] S101 divides the 3D point cloud data into training set, validation set, and test set, and preprocesses the point cloud dataset.
[0091] S102, Construct a deep learning network based on local neighborhood attention, train it using the training set, and obtain the optimal deep learning network model using the validation set.
[0092] S103 takes the test set as input and outputs the semantic segmentation results of the 3D point cloud into the optimal deep network model.
[0093] As a preferred embodiment, such as Figure 2 As shown, the point cloud semantic segmentation method based on local neighborhood attention provided in this embodiment of the invention specifically includes the following steps:
[0094] (1) Divide the 3D point cloud data into training set, validation set, and test set, and preprocess the point cloud dataset:
[0095] The points in each point cloud file of the point cloud dataset are downsampled, and the coordinates of the downsampled point cloud are normalized to the interval [0,1] in order to divide the neighborhood range and perform standardization.
[0096] (2) Construct a local neighborhood attention module that integrates local and global positional encoding (see...). Figure 3 And construct a deep learning network based on local neighborhood attention (see...). Figure 4 Specifically, it includes:
[0097] (2.1) Construct a local neighborhood attention module that integrates local and global positional encoding (see...). Figure 3 ):
[0098] (2.1.1) The local neighborhood attention module, which integrates local and global location encoding, uses vector attention to calculate point cloud features and aggregates them to the center point. It also uses two types of location encoding to provide the model with the spatial positional relationships of points within and between neighborhoods.
[0099] (2.1.2) For a point cloud P containing n points, its features and coordinates are represented as follows: and Sample m points from the point cloud P as center points, and their features and coordinates are represented as follows: and Directly use the center point feature F center After linear mapping, the query vector Q is obtained, then:
[0100]
[0101] In the formula, α represents a linear layer, which reduces the number of feature channels from C. in Mapped to C out .
[0102] To obtain the key vector and value vector, the input point cloud features F are first processed. in After two different linear mappings, and then with the center point X centerCentered around, use ball query to divide the neighborhood, and accordingly divide the two features obtained by linear mapping into m subsets, each subset contains k points, obtain the key vector K and the value vector V respectively, and obtain the corresponding neighbor point coordinates X neighbor For subsequent calculation of position encoding:
[0103]
[0104] In the formula, β and γ are linear layers, which map the number of feature channels from C in to C out ; group(a, (b, c)) means dividing the neighborhood on c with b as the center, thereby grouping the feature a; k represents the number of points in each neighborhood.
[0105] Subtract the Q vector from the K vector, and the attention weight can be obtained after passing through a multi-layer perceptron (MLP) and normalization. However, in order to help the model learn the structural relationship between neighborhoods, a global position encoding is added In addition, in order to prevent network overfitting, random dropout is also performed on the attention weight:
[0106]
[0107] In the formula, represents an MLP composed of a convolutional layer, batch normalization, and Swish activation function, η represents the softmax normalization function, and drop represents the random dropout function.
[0108] In addition to the structural relationship between neighborhoods, the positional relationship of points within a neighborhood is also crucial for point cloud learning. Therefore, a local position encoding is added to the V vector and then weighted summation is performed using the attention weight W to aggregate the features of points within each neighborhood.
[0109] To adapt to feature extraction in different situations, two attention modules are designed. The first attention module does not perform downsampling (m = n), and it is called the non-downsampling attention module. It divides the neighborhood with each point as the center and calculates the attention:
[0110]
[0111] In the formula, represents element-wise multiplication in sequence, and the sum function adds the weighted point features within the neighborhood. The second attention module performs downsampling (m < n), and it is called the downsampling attention module. It divides the neighborhood with the downsampled points as the center and calculates the attention. In addition, in order to improve the stability of the model, a residual connection is added, and the sampled point features are mapped to the same feature channels through a convolution and then added to the output:
[0112]
[0113] In the formula, τ represents one-dimensional convolution, and act is the activation function.
[0114] (2.1.3) Learnable, parametric global position coding (POS) global (Provides spatial location information between neighborhoods) and local location coding (POS) local (Provides the positional relationships of points within the neighborhood) respectively, based on the coordinates of the center point. The relative coordinates X of the neighboring points neighbor -X center After MLP, we obtain:
[0115]
[0116] In the formula, λ and θ are both MLPs composed of convolutional layers, batch normalization, and Swish activation functions.
[0117] Considering global location coding POS global It is obtained from the coordinates of the neighborhood center point. Among Q, K, and V, only Q is obtained from the features of the neighborhood center point. Therefore, global position encoding (POS) is added when calculating the attention weights. global This provides spatial location information between neighborhoods for learning feature weights; local location encoding (POS) local It is then added after the V vector to provide local location information for points in the neighborhood.
[0118] (2.2) Construct a deep learning network based on local neighborhood attention (see...) Figure 4 ):
[0119] (2.2.1) A deep learning network based on local neighborhood attention is used for point cloud segmentation and point cloud classification. It consists of four modules: pre-embedding, downsampling feature extraction, residual block, and upsampling feature extraction. Its main feature aggregation operators are the non-downsampling attention module and the downsampling attention module.
[0120] (2.2.2) Pre-embedding. Before extracting deep features, we first use a pre-embedding module to expand the feature channels of the point cloud and extract the shallow features of the point cloud. The pre-embedding consists of two consecutive non-downsampling attention modules.
[0121] (2.2.3) Downsampling Feature Extraction. The encoder part of the network downsamples the point cloud layer by layer, expanding the number of feature channels while reducing the number of points, and learning the deep features of the point cloud. Downsampling feature extraction is the main component of the encoder. It consists of a non-downsampling attention module and a downsampling attention module. The non-downsampling attention module is used to extract fine structural information, while the downsampling attention module is used to downsample the point cloud and aggregate neighborhood features.
[0122] (2.2.4) Residual Block. For segmentation tasks, deeper structural features are needed to learn the geometry of the point cloud. Therefore, a residual block is added after each layer of downsampled feature extraction in the segmentation task. It contains a non-downsampled attention module that connects its input residuals to the output, which is then passed through an activation function. In practice, the number of residual blocks added after downsampled feature extraction can be adjusted according to different tasks or datasets to obtain better results.
[0123] (2.2.5) Upsampling Feature Extraction. For the segmentation task, the encoder downsamples the point cloud. In order to obtain the classification score of each point, the decoder needs to upsample the point cloud. The point cloud features from the previous layer are upsampled using trilinear interpolation and then concatenated with the output features of the corresponding layer in the encoder. Then, a non-downsampling attention module is used to extract the fused point cloud features and feed them into the next layer.
[0124] (2.2.6) Network architecture for point cloud segmentation and classification. For segmentation tasks, the encoder part of the network consists of pre-embedding, downsampling feature extraction, and residual blocks. The decoder part uses upsampling feature extraction to upsample the features obtained from the previous layer and connects them to the output of the corresponding layer of the encoder. The last layer of the decoder obtains the features of each point, and finally uses MLP to map them to the number of segmentation categories to obtain the score for each point. For classification tasks, the encoder part removes the residual blocks compared to the segmentation task because the deep feature extraction of the residual blocks will cause the network to overlearn the details of the point cloud, leading to model overfitting. After the last downsampling feature extraction, the network further extracts point cloud features through two non-downsampling attention modules. The decoder part performs max pooling on the obtained features to obtain the global features of the point cloud, and finally uses MLP to obtain the classification score.
[0125] (3) Use the training set to train the network, use the validation set to validate the network during the training process, and optimize the model;
[0126] (3.1) Train the network using the training set, input the training set into the network, calculate the cross-entropy loss, and update the network parameters;
[0127] (3.2) Test using the validation set and select the optimal network model:
[0128] (3.2.1) Input the validation set into the model. Data processing includes normalization and standardization, and then the predicted labels are obtained. For segmentation tasks, the mean intersection-over-union ratio (mIoU) is used as the evaluation metric, mIoU = TP / (TP + FP + FN), where TP represents true positives, FP represents false positives, and FN represents false negatives. For classification tasks, the overall accuracy is used as the evaluation metric. The validation set accuracy is calculated by comparing the predicted labels with the actual labels, and the optimal model on the validation set during training is saved.
[0129] (4) Input the test set into the obtained optimal depth network model to obtain the semantic segmentation result of the three-dimensional point cloud.
[0130] (4.1) Input the test set into the model selected in step (3), and the data processing includes normalization and standardization, and then obtain the predicted label.
[0131] (4.2) Calculate the accuracy of the test set.
[0132] This invention provides a point cloud semantic segmentation system based on local neighborhood attention, comprising:
[0133] The 3D point cloud data preprocessing module is used to divide the dataset into training set, validation set and test set according to a certain ratio, and to preprocess the dataset and normalize and standardize the point cloud data.
[0134] The point cloud semantic segmentation network construction module is used to build a local neighborhood attention module that integrates local and global positional encoding, and to build a deep learning network based on local neighborhood attention.
[0135] The point cloud semantic segmentation network training module is used to train the network using the training set and to validate the network and select the network model during the training process using the validation set.
[0136] The point cloud semantic segmentation module includes point cloud classification and point cloud segmentation. It is used to input the test set into the selected optimal classification or segmentation model to obtain the classification or segmentation results of the point cloud.
[0137] To demonstrate the inventiveness and technical value of the present invention, this section provides specific product or related technology application examples of the technical solution.
[0138] As a preferred embodiment, such as Figure 2 As shown, the point cloud semantic segmentation method based on local neighborhood attention provided in this embodiment of the invention specifically includes the following steps:
[0139] (1) Divide the 3D point cloud data into training set, validation set and test set, and preprocess the point cloud dataset;
[0140] In a preferred embodiment, the present invention obtains two publicly available point cloud datasets for point cloud classification and point cloud segmentation, respectively. The dataset for point cloud classification contains 12,311 point cloud files, divided into 40 different categories. 9,843 of these point cloud files are used as the training set, and 2,468 are used as the test set. The validation set is set to be consistent with the test set. The dataset for point cloud component segmentation contains 16,880 point cloud files with 50 different component category labels. 14,006 of these point cloud files are used as the training set, and 2,874 are used as the test set. The validation set is set to be consistent with the test set.
[0141] The point clouds are then preprocessed. For point cloud classification tasks, 1024 points are randomly sampled from each point cloud file as model input, while for point cloud component tasks, 2048 points are sampled. The coordinates of all points in the point cloud dataset are normalized to the [0,1] interval to divide the neighborhood range and standardize the data.
[0142] (2) Construct a local neighborhood attention module that integrates local and global location encoding, and build a deep learning network based on local neighborhood attention, specifically including:
[0143] (2.1) Construct a local neighborhood attention module that integrates local and global positional encoding, specifically including:
[0144] (2.1.1) The local neighborhood attention module, which integrates local and global location encoding, uses vector attention to calculate point cloud features and aggregates them to the center point. It also uses two types of location encoding to provide the model with the spatial positional relationships of points within and between neighborhoods.
[0145] (2.1.2) The local neighborhood attention module, which integrates local and global location encoding, is described as follows: For a point cloud P containing n points, its features and coordinates are represented as follows: and Sample m points from the point cloud P as center points, and their features and coordinates are represented as follows: and Directly use the center point feature F center After linear mapping, the query vector Q is obtained, then:
[0146]
[0147] In the formula, α represents a linear layer, which reduces the number of feature channels from C. in Mapped to C out .
[0148] To obtain the key vector and value vector, the input point cloud features F are first processed. in After two different linear mappings, and then with the center point X center Centered on a point, a ball query is used to divide the neighborhood. Based on this, the two features obtained from the linear mapping are divided into m subsets, each containing k points, resulting in a key vector K and a value vector V, and the corresponding coordinates X of the neighboring points. neighbor In order to calculate the position code later:
[0149]
[0150] In the formula, β and γ are linear layers, which reduce the number of feature channels from C. in Mapped to C out group(a,(b,c)) means dividing the neighborhood of feature a into groups based on b as the center and c as the center; k represents the number of points in each neighborhood.
[0151] Subtracting the Q vector from the K vector, and then passing it through a multilayer perceptron (MLP) and normalizing, yields the attention weights. However, to help the model learn the structural relationships between neighborhoods, global position encoding is added. In addition, to prevent network overfitting, the attention weights were randomly dropped:
[0152]
[0153] In the formula, This represents an MLP consisting of convolutional layers, batch normalization, and the Swish activation function, where η represents the softmax normalization function and drop represents the random drop function.
[0154] Besides the structural relationships between neighborhoods, the positional relationships of points within the neighborhood are also crucial for point cloud learning. Therefore, a local positional encoding is added to the V vector. Then, the attention weight W is used to perform a weighted summation to aggregate the features of each point in the neighborhood.
[0155] To adapt to feature extraction in different situations, two attention modules were designed. The first attention module does not perform downsampling (m=n), and is called the non-downsampling attention module. It divides the neighborhood around each point and calculates the attention:
[0156]
[0157] In the formula, denotes element - by - element multiplication in sequence, and the sum function adds the weighted point features within the neighborhood. The second attention module performs downsampling (m < n), which is called the downsampling attention module. It divides the neighborhood centered on the downsampled points, calculates the attention. In addition, to improve the stability of the model, a residual connection is added. The sampled point features are mapped to the same feature channels through a convolution and then added to the output:
[0158]
[0159] In the formula, τ represents one - dimensional convolution, and act is the activation function.
[0160] As a preferred embodiment, when the present invention uses the downsampling attention module, the number of downsampled point clouds is 1 / 4 of that before sampling, that is, m = n / 4. The number of points k in the subset divided by ball query is 32, and act is the Swish activation function.
[0161] (2.1.3) Learnable, parametric global position encoding POS global (providing spatial position information between neighborhoods) and local position encoding POS local (providing the positional relationship of points within the neighborhood) are respectively obtained from the center point coordinates and the relative coordinates X of neighbor points neighbor -X center through MLP:
[0162]
[0163] In the formula, both λ and θ are MLP composed of convolutional layers, batch normalization, and Swish activation functions.
[0164] Considering that the global position encoding POS global is obtained from the center point coordinates of the neighborhood, and only Q among Q, K, and V is obtained from the center point features of the neighborhood. Therefore, when calculating the attention weight, the global position encoding POS global is added to provide spatial position information between neighborhoods for the learning of feature weights; the local position encoding POS local is added after the V vector to provide local position information for the points within the neighborhood.
[0165] (2.2) Construct a deep learning network based on local neighborhood attention, specifically including:
[0166] (2.2.1) The deep learning network based on local neighborhood attention is used for point cloud segmentation and point cloud classification. It consists of four modules: pre - embedding, downsampling feature extraction, residual block, and upsampling feature extraction. Its main feature aggregation operators are non - downsampling attention module and downsampling attention module.
[0167] (2.2.2) Pre-embedding. Before extracting deep features, we first use a pre-embedding module to expand the feature channels of the point cloud and extract the shallow features of the point cloud. The pre-embedding consists of two consecutive non-downsampling attention modules.
[0168] (2.2.3) Downsampling Feature Extraction. The encoder part of the network downsamples the point cloud layer by layer, expanding the number of feature channels while reducing the number of points, and learning the deep features of the point cloud. Downsampling feature extraction is the main component of the encoder. It consists of a non-downsampling attention module and a downsampling attention module. The non-downsampling attention module is used to extract fine structural information, while the downsampling attention module is used to downsample the point cloud and aggregate neighborhood features.
[0169] (2.2.4) Residual Block. For segmentation tasks, deeper structural features are needed to learn the geometry of the point cloud. Therefore, a residual block is added after each layer of downsampled feature extraction in the segmentation task. It contains a non-downsampled attention module that connects its input residuals to the output, which is then passed through an activation function. In practice, the number of residual blocks added after downsampled feature extraction can be adjusted according to different tasks or datasets to obtain better results.
[0170] (2.2.5) Upsampling Feature Extraction. For the segmentation task, the encoder downsamples the point cloud. In order to obtain the classification score of each point, the decoder needs to upsample the point cloud. The point cloud features from the previous layer are upsampled using trilinear interpolation and then concatenated with the output features of the corresponding layer in the encoder. Then, a non-downsampling attention module is used to extract the fused point cloud features and feed them into the next layer.
[0171] (2.2.6) Network architecture for point cloud segmentation and classification. For segmentation tasks, the encoder part of the network consists of pre-embedding, downsampling feature extraction, and residual blocks. The decoder part uses upsampling feature extraction to upsample the features obtained from the previous layer and connects them to the output of the corresponding layer of the encoder. The last layer of the decoder obtains the features of each point, and finally uses MLP to map them to the number of segmentation categories to obtain the score for each point. For classification tasks, the encoder part removes the residual blocks compared to the segmentation task because the deep feature extraction of the residual blocks will cause the network to overlearn the details of the point cloud, leading to model overfitting. After the last downsampling feature extraction, the network further extracts point cloud features through two non-downsampling attention modules. The decoder part performs max pooling on the obtained features to obtain the global features of the point cloud, and finally uses MLP to obtain the classification score.
[0172] In a preferred embodiment, the present invention uses four downsampling feature extraction modules in the classification model; in the segmentation model, four downsampling feature extraction modules are used in the encoder section, with a residual block after each downsampling feature extraction module; and four upsampling feature extraction modules are used in the decoder section. The point cloud file of the input model contains the coordinate information x, y, z of each point, as well as the normal vector information n. x n y n z The pre-embedded module extracts features with a dimension of 32. After each downsampling feature extraction, the number of point cloud feature channels doubles; while after each upsampling feature extraction, the number of point cloud feature channels is halved.
[0173] (3) Train the network using the training set, validate the network using the validation set during training, and optimize the model. Specifically, this includes:
[0174] (3.1) Train the network using the training set, input the training set into the network, calculate the cross-entropy loss, and update the network parameters;
[0175] In a preferred embodiment, during training, the present invention sets the batch size to 13 for point cloud classification, performs 400 training rounds, and sets the initial learning rate to 0.0001. The Adam optimizer is used to adjust the model parameters, and the cosine annealing algorithm is used to adjust the learning rate. For point cloud component segmentation, the batch size is set to 4, and the training is performed for 120 rounds. The initial learning rate is 0.0001, and the learning rate is adjusted to 0.2 times at rounds 55, 77, and 110. The Adam optimizer is used to adjust the model parameters.
[0176] (3.2) Test using a validation set and select the optimal network model, specifically including:
[0177] (3.2.1) Input the validation set into the model. Data processing includes normalization and standardization, and then the predicted labels are obtained. For segmentation tasks, the mean intersection-over-union ratio (mIoU) is used as the evaluation metric, mIoU = TP / (TP + FP + FN), where TP represents true positives, FP represents false positives, and FN represents false negatives. For classification tasks, the overall accuracy is used as the evaluation metric. The validation set accuracy is calculated by comparing the predicted labels with the actual labels, and the optimal model on the validation set during training is saved.
[0178] (4) Input the test set into the obtained optimal deep network model to obtain the semantic cut results of the 3D point cloud, specifically including:
[0179] (4.1) Input the test set into the model selected in step (3), and the data processing includes normalization and standardization, and then obtain the predicted label;
[0180] (4.2) Calculate the accuracy of the test set.
[0181] The evaluation criterion for the point cloud semantic segmentation method based on local neighborhood attention provided in this invention is the accuracy of the test set. For point cloud classification, the overall accuracy is used as the evaluation metric, and for point cloud segmentation, the average intersection-over-union ratio (AUC) is used. In the point cloud classification task, this invention is compared with three methods: PointNet++, PointCNN, and Point-PlaneNet; in the point cloud segmentation task, this invention is compared with three...
[0182] Table 1 Comparison of experimental results
[0183] The method is compared with PointNet, SO-Net, and SPLATNet 3D. As shown in Table 1, the experimental results show that, compared with previous methods, the point cloud semantic segmentation method based on local neighborhood attention provided in this embodiment of the invention achieves better experimental results on the test set of public point cloud datasets, indicating that the point cloud semantic segmentation method of this invention has good generalization ability and robustness.
[0184] Embodiments of the present invention can be implemented using hardware, software, or a combination of both. Hardware components are constructed using dedicated logic circuits; the software portion can be stored in a storage device and operated by suitable hardware such as a microprocessor or specially designed hardware. Those skilled in the art will understand that the above-described devices and methods are implemented using computer-executable instructions or processor control code, which are stored on storage media such as disks, CDs, or DVD-ROMs, or provided in data media such as read-only memory, programmable memory, optical or electronic signal carriers. The devices and modules of the present invention are implemented using semiconductor circuits such as very large-scale integrated circuits, gate arrays, logic chips, and transistors, or by software executed by various types of processors, or by a combination of the aforementioned hardware circuits and software.
[0185] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A point cloud semantic segmentation method based on local neighborhood attention, characterized in that, This method helps robots understand their surroundings and includes the following steps: Step 1: Divide the 3D point cloud data into training set, validation set, and test set, and preprocess the point cloud dataset; Step 2: Construct a local neighborhood attention module that integrates local and global location encoding, and build a deep learning network based on local neighborhood attention, including: (1) Construct a local neighborhood attention module that integrates local positional encoding and global positional encoding; The local neighborhood attention module, which integrates local and global location encoding, uses vector attention to calculate point cloud features and aggregates them to the center point. It also uses two types of location encoding to provide the model with the spatial positional relationships of points within and between neighborhoods, respectively. For a point cloud P containing n points, its features and coordinates are represented as follows: and Sample m points from the point cloud P as center points, and their features and coordinates are represented as follows: and Directly use the center point feature F center After linear mapping, the query vector Q is obtained, then: In the formula, α represents a linear layer, which reduces the number of feature channels from C. in Mapped to C out ; To obtain the key vector and value vector, the input point cloud features F are first processed. in After two different linear mappings, and then with the center point X center Centered on a point, a ball query is used to divide the neighborhood. Based on this, the two features obtained from the linear mapping are divided into m subsets, each containing k points, resulting in a key vector K and a value vector V, and the corresponding coordinates X of the neighboring points. neighbor In order to calculate the position code later; In the formula, β and γ are linear layers, which reduce the number of feature channels from C. in Mapped to C out group(a,(b,c)) means dividing the neighborhood of feature a into groups based on b as the center and c as the center; k represents the number of points in each neighborhood. Subtracting the Q vector from the K vector, and then passing it through a multilayer perceptron (MLP) and normalizing it, yields the attention weights. However, to help the model learn the structural relationships between neighborhoods, global position encoding is added. In addition, to prevent network overfitting, the attention weights were randomly dropped. In the formula, This represents an MLP consisting of convolutional layers, batch normalization, and the Swish activation function, where η represents the softmax normalization function and drop represents the random drop function. Besides the structural relationships between neighborhoods, the positional relationships of points within the neighborhood are also crucial for point cloud learning. Therefore, a local positional encoding is added to the V vector. Then, the attention weight W is used to perform a weighted summation to aggregate the features of each point in the neighborhood; To adapt to feature extraction in different situations, two attention modules were designed. The first attention module does not perform downsampling when m=n, and is called the non-downsampling attention module. It divides the neighborhood with each point as the center and calculates the attention. Wherein, represents element-wise multiplication in sequence. The sum function adds the point features within the weighted neighborhood. The second attention module performs downsampling, where m < n, and it is called the downsampling attention module. It divides the neighborhood centered on the points obtained by downsampling, calculates the attention. In addition, to improve the stability of the model, a residual connection is also added. The sampled point features are mapped to the same feature channels through a convolution and then added to the output. In the formula, τ represents one-dimensional convolution, and act is the activation function; Learnable, parametric global position coding (POS) global It provides spatial location information between neighborhoods and local location coding (POS). local It provides the positional relationships of points within the neighborhood, respectively, based on the coordinates of the center point. The relative coordinates X of the neighboring points neighbor -X center Obtained via MLP; In the formula, λ and θ are both MLPs composed of convolutional layers, batch normalization, and Swish activation functions; Considering global location coding POS global It is obtained from the coordinates of the neighborhood center point. Among Q, K, and V, only Q is obtained from the features of the neighborhood center point. Therefore, global position encoding (POS) is added when calculating the attention weights. global This provides spatial location information between neighborhoods for learning feature weights; local location encoding (POS) local It is then added after the V vector to provide local location information for points in the neighborhood; (2) Construct a deep learning network based on local neighborhood attention; A deep learning network based on local neighborhood attention is used for point cloud segmentation and point cloud classification. It consists of four modules: pre-embedding, downsampling feature extraction, residual block, and upsampling feature extraction. Its main feature aggregation operators are the non-downsampling attention module and the downsampling attention module. Pre-embedding: Before extracting deep features, we first use a pre-embedding module to expand the point cloud feature channels and extract shallow features of the point cloud. The pre-embedding consists of two consecutive non-downsampling attention modules. Downsampling feature extraction: The encoder part of the network downsamples the point cloud layer by layer, expanding the number of feature channels while reducing the number of points, and learning the deep features of the point cloud. Downsampling feature extraction is the main component of the encoder. It consists of a non-downsampling attention module and a downsampling attention module. The non-downsampling attention module is used to extract fine structure information, while the downsampling attention module is used to downsample the point cloud and aggregate neighborhood features. Residual Block: For segmentation tasks, deeper structural features are needed to learn the geometric structure of point clouds. Therefore, a residual block is added after each downsampled feature extraction layer in the segmentation task. It contains a non-downsampled attention module that connects its input residual to the output and then passes it through an activation function. In fact, the number of residual blocks added after downsampled feature extraction is adjusted according to different tasks or datasets to obtain better results. Upsampling feature extraction: For the segmentation task, the encoder downsamples the point cloud. In order to obtain the classification score of each point, the decoder needs to upsample the point cloud. The point cloud features from the previous layer are upsampled using trilinear interpolation and then concatenated with the output features of the corresponding layer in the encoder. Then, a non-downsampling attention module is used to extract the fused point cloud features and feed them into the next layer. Network architecture for point cloud segmentation and classification: For segmentation tasks, the encoder part of the network consists of pre-embedding, downsampling feature extraction, and residual blocks. The decoder part uses upsampling feature extraction to upsample the features obtained from the previous layer and connects them to the output of the corresponding layer of the encoder. The last layer of the decoder obtains the features of each point and finally uses MLP to map them to the number of segmentation categories to obtain the score of each point. For classification tasks, the encoder part removes the residual blocks compared to the segmentation task because the deep feature extraction of the residual blocks will cause the network to overlearn the details of the point cloud, resulting in model overfitting. After the last downsampling feature extraction, the point cloud features are further extracted by two non-downsampling attention modules. The decoder part performs max pooling on the obtained features to obtain the global features of the point cloud. Finally, MLP is used to obtain the classification score. Step 3: Train the network using the training set, validate the network using the validation set during training, and optimize the model. Step 4: Input the test set into the obtained optimal deep network model to obtain the semantic segmentation results of the 3D point cloud.
2. The point cloud semantic segmentation method based on local neighborhood attention according to claim 1, characterized in that, The preprocessing of the point cloud dataset in step one includes: The coordinates of points in each point cloud file in the point cloud dataset are normalized to the interval [0,1] in order to divide the neighborhood range and perform standardization.
3. The point cloud semantic segmentation method based on local neighborhood attention according to claim 1, characterized in that, Step three, which involves training the network using the training set, validating the network using the validation set during training, and optimizing the model, includes: (1) Train the network using the training set, input the training set into the network, calculate the cross-entropy loss, and update the network parameters; (2) Test using the validation set and select the optimal network model; The validation set is input into the model, and data processing includes normalization and standardization to obtain predicted labels. For segmentation tasks, the mean intersection-union ratio (mIoU) is used as the evaluation metric, where mIoU = TP / (TP + FP + FN), and TP represents true positives, FP represents false positives, and FN represents false negatives. For classification tasks, the overall accuracy is used as the evaluation metric. The accuracy of the validation set is calculated by comparing the predicted labels with the actual labels, and the optimal model on the validation set during training is saved.
4. The point cloud semantic segmentation method based on local neighborhood attention according to claim 1, characterized in that, The semantic segmentation results of the 3D point cloud obtained by inputting the test set into the optimal deep network model in step four include: (1) Input the test set into the model selected in step three. Data processing includes normalization and standardization, and then the predicted labels are obtained. (2) Calculate the accuracy of the test set.
5. A point cloud semantic segmentation system based on local neighborhood attention, as described in any one of claims 1 to 4, characterized in that, include: The 3D point cloud data preprocessing module is used to divide the dataset into training set, validation set and test set according to a certain ratio, and to preprocess the dataset and normalize and standardize the point cloud data. The point cloud semantic segmentation network construction module is used to build a local neighborhood attention module that integrates local and global positional encoding, and to build a deep learning network based on local neighborhood attention. The point cloud semantic segmentation network training module is used to train the network using the training set and to validate the network and select the network model during the training process using the validation set. The point cloud semantic segmentation module includes point cloud classification and point cloud segmentation. It is used to input the test set into the selected optimal classification or segmentation model to obtain the classification or segmentation results of the point cloud.
6. A point cloud semantic segmentation system based on local neighborhood attention, characterized in that, The device includes a computer device having a program for storing and running a point cloud semantic segmentation system based on local neighborhood attention, and a processor capable of executing any one of the point cloud semantic segmentation methods based on local neighborhood attention as claimed in claims 1 to 4.
7. A point cloud semantic segmentation system based on local neighborhood attention, characterized in that, It includes a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform any one of the point cloud semantic segmentation methods based on local neighborhood attention as claimed in claims 1 to 4.
8. A point cloud semantic segmentation system based on local neighborhood attention, characterized in that, It includes an information data processing terminal, which is used to implement any one of the point cloud semantic segmentation methods based on local neighborhood attention as claimed in claims 1 to 4.