Point cloud semantic segmentation method and system based on local neighborhood attention

Abstract

The invention discloses a point cloud semantic segmentation method and system based on local neighborhood attention, and belongs to the technical field of three-dimensional point cloud processing and deep learning, and the method comprises the following steps: S101, dividing three-dimensional point cloud data into training, verification and test sets, and carrying out the preprocessing; s102, constructing a deep learning network based on local neighborhood attention, training by using the training set, and obtaining an optimal model by using the verification set; and S103, inputting the test set into the optimal model to obtain a semantic segmentation result. According to the method, the semantic segmentation model is constructed by fusing local neighborhood attention of global position coding and local position coding, so that the problem of poor space geometric structure information extraction due to the fact that a conventional model only pays attention to the relative position of points in a neighborhood is effectively solved, and the feature extraction capability of an attention mechanism is enhanced; and the robustness, generalization and segmentation accuracy of the three-dimensional point cloud semantic segmentation model are improved.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional point cloud processing and deep learning technology, and in particular relates to a point cloud semantic segmentation method and system based on local neighborhood attention. Background Art

[0002] 3D point cloud is one of the most important ways to express 3D data, which is helpful for perceiving the environment. At present, deep learning is developing rapidly, and point cloud deep learning is becoming more and more important in the field of computer vision. Point cloud semantic segmentation tasks are widely used, which can help robots recognize the surroundings, help scene understanding and interaction in augmented / virtual reality, and can also be used for the digitization of cultural relics in cultural heritage protection. However, designing an efficient point cloud semantic segmentation network is quite challenging.

[0003] Because point cloud data is irregular and disordered, in the early days, point clouds were first converted into regular voxel grids and processed using three-dimensional convolution, or projected onto two-dimensional plane views for convolution to extract features. However, this method is prone to losing fine local geometric information during voxelization and projection, making it difficult to improve the accuracy of the model, and it is mostly applicable to point cloud classification and difficult to expand to segmentation tasks. To address this problem, the present invention abandons the method of regularizing point clouds and then using convolution to learn features, and instead directly processes point clouds to prevent information loss.

[0004] In recent years, Transformer, which has the ability to model long-range dependencies, is often used in deep learning of point clouds. Its core attention mechanism can adapt to the characteristics of point clouds and improve model performance by paying attention to different parts of the data. However, current methods of this type only use local position encoding to calculate local attention, and only consider the positional relationship of points within the neighborhood, ignoring the spatial relationship between neighbors, resulting in weak global shape perception of the model. The present invention constructs a local neighborhood attention module that integrates local position encoding and global position encoding for point cloud learning. The attention mechanism uses vector attention and adopts two different position encodings to provide the positional relationship of points within the neighborhood and the spatial relationship between neighbors, respectively, so that the model can adapt to the local structure of point cloud data while adapting to the spatial geometric structure between neighbors in the global range, enriching the extraction features of the attention mechanism.

[0005] Through the above analysis, the problems and defects of the existing technology are as follows: the existing voxel-based and multi-view-based methods will cause a lot of information loss in the voxelization and projection process, resulting in the model being unable to extract the fine geometric structure of the point cloud; most of the previous attention-based point cloud learning methods only consider the relative position relationship of points within the neighborhood, while ignoring the spatial position relationship between neighbors, resulting in the model being difficult to learn the global geometric structure information of the point cloud.

[0006] Based on this, the present invention designs a point cloud semantic segmentation method and system based on local neighborhood attention to solve the above problems. Summary of the invention

[0007] The purpose of the present invention is to solve the problem that the existing voxel-based and multi-view-based methods will cause a large amount of information loss in the voxelization and projection process, resulting in the model being unable to extract the fine geometric structure of the point cloud; the previous point cloud learning methods based on the attention mechanism mostly only consider the relative position relationship of points within the neighborhood, but ignore the spatial position relationship between the neighborhoods, which makes it difficult for the model to learn the global geometric structure information of the point cloud. A point cloud semantic segmentation method and system based on local neighborhood attention is proposed.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions: A point cloud semantic segmentation method and system based on local neighborhood attention, comprising the following steps: S101, dividing the three-dimensional point cloud data into training, verification, and test sets and preprocessing; S102, build a deep learning network based on local neighborhood attention, and use the training set and the validation set to obtain the optimal model; S103, input the test set into the optimal model to obtain the semantic segmentation result.

[0009] As a further description of the above technical solution: The point cloud semantic segmentation method based on local neighborhood attention comprises the following steps: Step 1: divide the 3D point cloud data into a training set, a validation set, and a test set, and preprocess the point cloud data set; Step 2: construct a local neighborhood attention module that integrates local position encoding and global position encoding, and build a deep learning network based on local neighborhood attention; Step 3: Use the training set to train the network, use the validation set to validate the network during the training process, and optimize the model; Step 4: Input the test set into the optimal deep network model to obtain the semantic segmentation result of the three-dimensional point cloud.

[0010] As a further description of the above technical solution: The preprocessing of the point cloud data set in step 1 includes: The coordinates of the points in each point cloud file in the point cloud dataset are normalized to the interval [0, 1] in order to divide the neighborhood range and perform standardization.

[0011] As a further description of the above technical solution: The step 2 constructs a local neighborhood attention module that integrates local position encoding and global position encoding, and constructs a deep learning network based on local neighborhood attention, including: (1) Construct a local neighborhood attention module that combines local position encoding and global position encoding: The local neighborhood attention module, which combines local position encoding and global position encoding, uses vector attention to calculate point cloud features and aggregates them to the center point. Two types of position encoding are used to provide the model with spatial position relationships between points within the neighborhood and between neighborhoods.

[0012] For a point cloud P containing n points, its features and coordinates are expressed as and . Sample m points from the point cloud P as the center point, and their features and coordinates are expressed as and . Directly move the center point feature After linear mapping, we get the query vector Q, then: ; In the formula, is a linear layer, which reduces the number of feature channels from Mapping .

[0013] In order to obtain the key vector and value vector, first input the point cloud feature After two different linear mappings, the center point The ball query is used to divide the neighborhood into m subsets based on the center. Each subset contains k points. The key vector K and value vector V are obtained respectively, and the corresponding neighbor point coordinates are obtained. So that the position encoding can be calculated later: ; In the formula, is a linear layer, which reduces the number of feature channels from Mapping ; It means taking b as the center and dividing the neighborhood on c to group feature a; k represents the number of points in each neighborhood.

[0014] Subtract the Q vector from the K vector, pass it through a multi-layer perceptron (MLP) and normalize it to get the attention weight. However, in order to help the model learn the structural relationship between neighborhoods, a global position encoding is added In addition, in order to prevent the network from overfitting, the attention weights are randomly lost: ; In the formula, represents an MLP consisting of a convolutional layer, batch normalization, and a Swish activation function, represents the softmax normalization function, and drop represents the random loss function.

[0015] In addition to the structural relationship between neighborhoods, the positional relationship of points within a neighborhood is also crucial for point cloud learning. Therefore, a local positional encoding is added to the V vector. After that, the attention weight W is used for weighted summation to aggregate the features of points within each neighborhood.

[0016] To adapt to feature extraction in different situations, two attention modules are designed. The first attention module does not perform downsampling (m = n), and it is called the non-downsampling attention module. It divides the neighborhood centered on each point and calculates the attention: ; In the formula, represents element-wise multiplication in sequence, and the sum function adds the weighted point features within the neighborhood. The second attention module performs downsampling (m < n), and it is called the downsampling attention module. It divides the neighborhood centered on the downsampled points and calculates the attention. In addition, to improve the stability of the model, a residual connection is added. The sampled point features are mapped to the same feature channels through a convolution and then added to the output: ; In the formula, represents one-dimensional convolution, and act is the activation function.

[0017] Learnable, parametric global positional encoding (providing spatial positional information between neighborhoods) and local positional encoding (providing the positional relationship of points within a neighborhood) are respectively obtained from the central point coordinates and the relative coordinates of neighbor points through MLP: ; In the formula, and are both MLP composed of convolutional layers, batch normalization, and Swish activation functions.

[0018] Considering that the global positional encoding is obtained from the central point coordinates of the neighborhood, only Q among Q, K, and V is obtained from the central point features of the neighborhood. Therefore, the global positional encoding is added when calculating the attention weight to provide spatial positional information between neighborhoods for the learning of feature weights; the local positional encoding is added after the V vector to provide local positional information for the points within the neighborhood.

[0019] (2) Construct a deep learning network based on local neighborhood attention: The deep learning network based on local neighborhood attention can be used for point cloud segmentation and point cloud classification. It consists of four modules: pre-embedding, downsampling feature extraction, residual block, and upsampling feature extraction. Its main feature aggregation operators are non-downsampling attention module and downsampling attention module.

[0020] Pre-embedding. Before extracting deep features, we first use a pre-embedding module to expand the point cloud feature channel and extract shallow features of the point cloud. Pre-embedding consists of two consecutive non-subsampling attention modules.

[0021] Downsampling feature extraction. The encoder part of the network downsamples the point cloud layer by layer, expanding the number of feature channels while reducing the number of points, and learning the deep features of the point cloud. Downsampling feature extraction is the main component of the encoder, which consists of a non-downsampling attention module and a downsampling attention module. The non-downsampling attention module is used to extract fine structure information, and the downsampling attention module is used to aggregate neighborhood features while downsampling the point cloud.

[0022] Residual block. For the segmentation task, it requires deeper structural features for learning the geometric structure of the point cloud, so a residual block is added after each layer of downsampled feature extraction in the segmentation task. It contains a non-downsampled attention module, which connects its input residual to the output and then passes through an activation function. In fact, according to different tasks or datasets, the number of residual blocks added after the downsampled feature extraction can be adjusted to obtain better results.

[0023] Upsampling feature extraction. For the segmentation task, the encoder part downsamples the point cloud. In order to finally get the classification score of each point, the decoder needs to upsample the point cloud. The point cloud features from the previous layer are upsampled using trilinear interpolation, concatenated with the output features of the corresponding layer in the encoder, and then a non-downsampled attention module is used to extract the fused point cloud features and send them to the next layer.

[0024] Network architecture for point cloud segmentation and classification. For segmentation tasks, the encoder part of the network consists of pre-embedding, downsampling feature extraction and residual blocks. The decoder part uses upsampling feature extraction to upsample the features obtained in the previous layer and connects the output of the corresponding layer of the encoder. The last layer of the decoder will obtain the features of each point, and finally use MLP to map it to the number of segmentation categories to obtain the score of each point. For classification tasks, compared with segmentation tasks, the encoder part removes the residual block because the deep feature extraction of the residual block will cause the network to over-learn the details of the point cloud, resulting in model overfitting. After the last downsampling feature extraction, two non-downsampling attention modules are used to further extract point cloud features. The decoder part performs maximum pooling on the obtained features to obtain the global features of the point cloud, and finally uses MLP to obtain the classification score.

[0025] As a further description of the above technical solution: The step 3 of training the network using the training set, validating the network using the validation set during the training process, and optimizing the model includes: (1) Use the training set to train the network, input the training set into the network, calculate the cross entropy loss, and update the network parameters; (2) Use the validation set to test and select the optimal network model; The validation set is input into the model, and the data processing includes normalization and standardization, and then the predicted label is obtained. For segmentation tasks, the mean intersection over union (mIoU) is used as the evaluation indicator, mIoU=TP / (TP+FP+FN), where TP represents true positive examples, FP represents false positive examples, and FN represents false negative examples. For classification tasks, the overall accuracy is used as the evaluation indicator. The predicted label is compared with the actual label to calculate the validation set accuracy, and the optimal model on the validation set during the training process is saved.

[0026] As a further description of the above technical solution: In step 4, the test set is input into the obtained optimal deep network model to obtain the semantic segmentation result of the three-dimensional point cloud, including: (1) Input the test set into the model selected in step 3, perform data processing including normalization and standardization, and then obtain the predicted label; (2) Calculate the accuracy of the test set.

[0027] As a further description of the above technical solution: A point cloud semantic segmentation system based on local neighborhood attention includes: The 3D point cloud data preprocessing module is used to divide the data set into training set, validation set and test set according to a certain ratio, preprocess the data set and normalize and standardize the point cloud data; Point cloud semantic segmentation network construction module, which is used to build a local neighborhood attention module that integrates local position encoding and global position encoding, and to build a deep learning network based on local neighborhood attention; Point cloud semantic segmentation network training module, used to train the network using the training set, verify the network using the verification set during the training process and select the network model; The point cloud semantic segmentation module includes point cloud classification and point cloud segmentation, which is used to input the test set into the selected optimal classification or segmentation model to obtain the classification or segmentation results of the point cloud.

[0028] As a further description of the above technical solution: A point cloud semantic segmentation system based on local neighborhood attention comprises: a computer device having a stored executable point cloud semantic segmentation system program based on local neighborhood attention and a processor capable of executing any one of the point cloud semantic segmentation methods based on local neighborhood attention as claimed in claims 1 to 6.

[0029] As a further description of the above technical solution: A point cloud semantic segmentation system based on local neighborhood attention comprises: a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor executes any one of the point cloud semantic segmentation methods based on local neighborhood attention as claimed in claims 1 to 6.

[0030] As a further description of the above technical solution: A point cloud semantic segmentation system based on local neighborhood attention includes an information data processing terminal, wherein the information data processing terminal is used to implement the system.

[0031] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows: The present invention uses a point cloud deep learning network based on local neighborhood attention to classify and segment three-dimensional point clouds. The network learns the local structure information and global spatial geometric structure information of the point cloud by integrating the local neighborhood attention module of local position encoding and global position encoding, thereby improving the robustness of the model and the accuracy of semantic segmentation. The present invention obtains a well-fitting three-dimensional point cloud semantic segmentation model, effectively solving the problem that the existing point cloud deep learning model has a weak ability to extract the global spatial geometric structure information of the point cloud, and effectively improving the accuracy of point cloud semantic segmentation.

[0032] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows: The point cloud semantic segmentation method based on local neighborhood attention provided by the present invention can effectively improve the problem that the previous point cloud learning method based on the attention mechanism only considers the relative positions of points in the neighborhood, so that the model can adapt to the local structure of the point cloud data while adapting to the spatial geometric structure within the neighborhood in the global range, and improve the robustness of the model and the accuracy of point cloud semantic segmentation.

[0033] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects: (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are: The present invention aims to improve the problem that the point cloud learning method based on the attention mechanism only considers the relative positions of points in the neighborhood. It will bring many expected benefits and commercial value, improve the accuracy and reliability of point cloud semantic segmentation, and robots can understand the surrounding environment more quickly, improve the robot's reaction speed, understand the surrounding scenes faster, improve the robot's action efficiency, provide better robot services, and promote innovation and development in the field of robotics.

[0034] (2) The technical solution of the present invention solves a technical problem that people have been eager to solve but have never been able to solve successfully: The present invention improves the problem that the point cloud learning method based on the attention mechanism only considers the relative position information of points in the neighborhood, resulting in weak ability to extract global structural information. The present invention enables the model to learn point clouds more comprehensively from the aspects of local structure and global spatial geometric structure information, making the classification and segmentation of point clouds more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 is a flow chart of a point cloud semantic segmentation method based on local neighborhood attention provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a point cloud semantic segmentation method based on local neighborhood attention provided by an embodiment of the present invention; Figure 3 It is a schematic diagram of the structure of a local neighborhood attention module that integrates local position coding and global position coding provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the point cloud semantic segmentation network structure based on local neighborhood attention provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Please see attached Figure 1 -Attached Figure 4 The present invention provides a technical solution: a point cloud semantic segmentation method and system based on local neighborhood attention, comprising the following steps: S101, dividing the three-dimensional point cloud data into a training set, a validation set, and a test set, and preprocessing the point cloud data set; S102, constructing a deep learning network based on local neighborhood attention, and training it using the training set, and obtaining the optimal deep learning network model using the validation set; S103, input the test set into the obtained optimal deep network model, and output the semantic segmentation result of the three-dimensional point cloud.

[0039] As a preferred embodiment, Figure 2 As shown, the point cloud semantic segmentation method based on local neighborhood attention provided by the embodiment of the present invention specifically includes the following steps: (1) Divide the 3D point cloud data into training set, validation set, and test set, and preprocess the point cloud data set: The points in each point cloud file in the point cloud dataset are downsampled, and then the downsampled point cloud coordinates are normalized to the interval [0, 1] in order to divide the neighborhood range and perform standardization.

[0040] (2) Construct a local neighborhood attention module that combines local position encoding and global position encoding (see Figure 3), and construct a deep learning network based on local neighborhood attention (see Figure 4 ), specifically including; (2.1) Construct a local neighborhood attention module that combines local position encoding and global position encoding (see Figure 3 ): (2.1.1) The local neighborhood attention module, which combines local position encoding and global position encoding, uses vector attention to calculate point cloud features and aggregates them to the center point. Two types of position encoding are used to provide the model with spatial position relationships between points within the neighborhood and between neighborhoods.

[0041] (2.1.2) For a point cloud P containing n points, its features and coordinates are expressed as and . Sample m points from the point cloud P as the center point, and their features and coordinates are expressed as and . Directly move the center point feature After linear mapping, we get the query vector Q, then: ; In the formula, is a linear layer, which reduces the number of feature channels from Mapping .

[0042] In order to obtain the key vector and value vector, first input the point cloud feature After two different linear mappings, the center point The ball query is used to divide the neighborhood into m subsets, each of which contains k points. The key vector K and value vector V are obtained respectively, and the corresponding neighbor point coordinates are obtained. So that the position encoding can be calculated later: ; In the formula, is a linear layer, which reduces the number of feature channels from Mapping ; It means taking b as the center and dividing the neighborhood on c to group feature a; k represents the number of points in each neighborhood.

[0043] Subtract the Q vector from the K vector, pass it through a multi-layer perceptron (MLP) and normalize it to get the attention weight. However, in order to help the model learn the structural relationship between neighborhoods, a global position encoding is added In addition, in order to prevent the network from overfitting, the attention weights are randomly lost: ; In the formula, represents an MLP composed of a convolutional layer, batch normalization, and a Swish activation function, represents the softmax normalization function, and drop represents the dropout function.

[0044] In addition to the structural relationship between neighborhoods, the positional relationship of points within a neighborhood is also crucial for point cloud learning. Therefore, a local positional encoding is added to the V vector and then the attention weight W is used for weighted summation to aggregate the features of points within each neighborhood.

[0045] To adapt to feature extraction in different situations, two attention modules are designed. The first attention module does not perform downsampling (m = n), and it is called the non-downsampling attention module. It divides the neighborhood with each point as the center and calculates the attention: ; In the formula, represents element-wise multiplication in sequence, and the sum function adds the weighted point features within the neighborhood. The second attention module performs downsampling (m < n), and it is called the downsampling attention module. It divides the neighborhood with the downsampled points as the center and calculates the attention. In addition, to improve the stability of the model, a residual connection is added, and the sampled point features are mapped to the same feature channels through a convolution and then added to the output: ; In the formula, represents a one-dimensional convolution, and act is the activation function.

[0046] (2.1.3) Learnable, parametric global positional encoding (providing the spatial position information between neighborhoods) and local positional encoding (providing the positional relationship of points within a neighborhood) are obtained from the central point coordinates and the relative coordinates of neighbor points through the MLP: ; In the formula, and are both MLPs composed of a convolutional layer, batch normalization, and a Swish activation function.

[0047] Considering that the global positional encoding is obtained from the central point coordinates of the neighborhood, and only Q among Q, K, and V is obtained from the central point features of the neighborhood. Therefore, the global positional encoding is added when calculating the attention weight to provide the spatial position information between neighborhoods for the learning of feature weights; the local positional encoding It is added after the V vector to provide local position information for points in the neighborhood.

[0048] (2.2) Construct a deep learning network based on local neighborhood attention (see Figure 4 ): (2.2.1) The deep learning network based on local neighborhood attention can be used for point cloud segmentation and point cloud classification. It consists of four modules: pre-embedding, downsampling feature extraction, residual block, and upsampling feature extraction. Its main feature aggregation operators are the non-downsampling attention module and the downsampling attention module.

[0049] (2.2.2) Pre-embedding. Before extracting deep features, we first use a pre-embedding module to expand the point cloud feature channel and extract shallow features of the point cloud. Pre-embedding consists of two consecutive non-subsampling attention modules.

[0050] (2.2.3) Downsampling feature extraction. The encoder part of the network downsamples the point cloud layer by layer, expanding the number of feature channels while reducing the number of points, and learning the deep features of the point cloud. Downsampling feature extraction is the main component of the encoder, which consists of a non-downsampling attention module and a downsampling attention module. The non-downsampling attention module is used to extract fine structure information, and the downsampling attention module is used to aggregate neighborhood features while downsampling the point cloud.

[0051] (2.2.4) Residual block. For the segmentation task, it requires deeper structural features for learning the geometric structure of the point cloud, so a residual block is added after each layer of downsampled feature extraction in the segmentation task. It contains a non-downsampled attention module, which connects its input residual to the output and then passes through an activation function. In fact, according to different tasks or datasets, the number of residual blocks added after the downsampled feature extraction can be adjusted to obtain better results.

[0052] (2.2.5) Upsampling feature extraction. For the segmentation task, the encoder part downsamples the point cloud. In order to finally obtain the classification score of each point, the decoder needs to upsample the point cloud. The point cloud features from the previous layer are upsampled using trilinear interpolation, concatenated with the output features of the corresponding layer in the encoder, and then a non-downsampled attention module is used to extract the fused point cloud features and send them to the next layer.

[0053] (2.2.6) Network architecture for point cloud segmentation and classification. For the segmentation task, the encoder part of the network consists of pre-embedding, downsampling feature extraction and residual blocks. The decoder part uses upsampling feature extraction to upsample the features obtained in the previous layer and connects the output of the corresponding layer of the encoder. The last layer of the decoder will obtain the features of each point, and finally use MLP to map it to the number of segmentation categories to obtain the score of each point. For the classification task, the encoder part removes the residual block compared to the segmentation task, because the deep feature extraction of the residual block will cause the network to over-learn the details of the point cloud, resulting in model overfitting. After the last downsampling feature extraction, two non-downsampling attention modules are used to further extract the point cloud features. The decoder part performs maximum pooling on the obtained features to obtain the global features of the point cloud, and finally uses MLP to obtain the classification score.

[0054] (3) Use the training set to train the network, use the validation set to validate the network during the training process, and optimize the model; (3.1) Use the training set to train the network, input the training set into the network, calculate the cross entropy loss, and update the network parameters; (3.2) Use the validation set to test and select the optimal network model: (3.2.1) Input the validation set into the model, and the data processing includes normalization and standardization, and then obtain the predicted label. For segmentation tasks, the mean intersection over union (mIoU) is used as the evaluation indicator, mIoU=TP / (TP+FP+FN), where TP represents true positive examples, FP represents false positive examples, and FN represents false negative examples. For classification tasks, the overall accuracy is used as the evaluation indicator. Compare the predicted labels with the actual labels to calculate the validation set accuracy, and save the optimal model on the validation set during the training process.

[0055] (4) Input the test set into the optimal deep network model to obtain the semantic segmentation result of the 3D point cloud.

[0056] (4.1) The test set is input into the model selected in step (3). Data processing includes normalization and standardization, and then the predicted label is obtained.

[0057] (4.2) Calculate the accuracy of the test set.

[0058] A point cloud semantic segmentation system based on local neighborhood attention provided by an embodiment of the present invention includes: The 3D point cloud data preprocessing module is used to divide the data set into training set, validation set and test set according to a certain ratio, preprocess the data set and normalize and standardize the point cloud data; Point cloud semantic segmentation network construction module, which is used to build a local neighborhood attention module that integrates local position encoding and global position encoding, and to build a deep learning network based on local neighborhood attention; Point cloud semantic segmentation network training module, used to train the network using the training set, verify the network using the verification set during the training process and select the network model; The point cloud semantic segmentation module includes point cloud classification and point cloud segmentation, which is used to input the test set into the selected optimal classification or segmentation model to obtain the classification or segmentation results of the point cloud.

[0059] In order to prove the creativity and technical value of the technical solution of the present invention, this section provides application examples of the claimed technical solution on specific products or related technologies.

[0060] As a preferred embodiment, Figure 2 As shown, the point cloud semantic segmentation method based on local neighborhood attention provided by the embodiment of the present invention specifically includes the following steps: (1) Divide the 3D point cloud data into training set, validation set, and test set, and preprocess the point cloud data set; As a preferred embodiment, the present invention obtains two public point cloud data sets for point cloud classification and point cloud segmentation. The data set for point cloud classification contains 12311 point cloud files, which are divided into 40 different categories. Among these point cloud files, 9843 are used as training sets and 2468 are used as test sets. The validation set is set to be consistent with the test set. The data set for point cloud component segmentation contains 16880 point cloud files, which have 50 different part category labels. Among these point cloud files, 14006 are used as training sets and 2874 are used as test sets. The validation set is set to be consistent with the test set.

[0061] Then the point cloud is preprocessed. For the point cloud classification task, 1024 points are randomly sampled from each point cloud file as model input, and for the point cloud component task, 2048 points are sampled. The coordinates of all points in the point cloud dataset are normalized to the interval [0, 1] in order to divide the neighborhood range and perform standardization.

[0062] (2) Construct a local neighborhood attention module that integrates local position encoding and global position encoding, and construct a deep learning network based on local neighborhood attention, including: (2.1) Construct a local neighborhood attention module that combines local position encoding and global position encoding, including: (2.1.1) The local neighborhood attention module, which combines local position encoding and global position encoding, uses vector attention to calculate point cloud features and aggregates them to the center point. Two types of position encoding are used to provide the model with spatial position relationships between points within the neighborhood and between neighborhoods.

[0063] (2.1.2) The local neighborhood attention module that combines local position encoding and global position encoding is expressed as follows: For a point cloud P containing n points, its features and coordinates are represented as and . Sample m points from the point cloud P as the center point, and their features and coordinates are expressed as and . Directly move the center point feature After linear mapping, we get the query vector Q, then: ; In the formula, is a linear layer, which reduces the number of feature channels from Mapping .

[0064] In order to obtain the key vector and value vector, first input the point cloud feature After two different linear mappings, the center point The ball query is used to divide the neighborhood into m subsets, each of which contains k points. The key vector K and value vector V are obtained respectively, and the corresponding neighbor point coordinates are obtained. So that the position encoding can be calculated later: ; In the formula, is a linear layer, which reduces the number of feature channels from Mapping ; It means taking b as the center and dividing the neighborhood on c to group feature a; k represents the number of points in each neighborhood.

[0065] Subtract the Q vector from the K vector, pass it through a multi-layer perceptron (MLP) and normalize it to get the attention weight. However, in order to help the model learn the structural relationship between neighborhoods, a global position encoding is added In addition, in order to prevent the network from overfitting, the attention weights are randomly lost: ; In the formula, represents an MLP consisting of a convolutional layer, batch normalization, and a Swish activation function, represents the softmax normalization function, and drop represents the random loss function.

[0066] In addition to the structural relationship between neighborhoods, the positional relationship of points within a neighborhood is also crucial for point cloud learning. Therefore, a local positional encoding is added to the V vector. After that, the attention weight W is used for weighted summation to aggregate the features of points within each neighborhood.

[0067] To adapt to feature extraction in different situations, two attention modules are designed. The first attention module does not perform downsampling (m = n), and it is called the non-downsampling attention module. It divides the neighborhood with each point as the center and calculates the attention: ; In the formula, represents element-wise multiplication in sequence, and the sum function adds the weighted point features within the neighborhood. The second attention module performs downsampling (m < n), and it is called the downsampling attention module. It divides the neighborhood with the downsampled points as the center and calculates the attention. In addition, to improve the stability of the model, a residual connection is added. The sampled point features are mapped to the same feature channels through a convolution and then added to the output: ; In the formula, represents one-dimensional convolution, and act is the activation function.

[0068] As a preferred embodiment, when the downsampling attention module is used in the present invention, the number of downsampled point clouds is 1 / 4 of that before sampling, that is, m = n / 4. The number of points k in the subset divided by ball query is 32, and act is the Swish activation function.

[0069] (2.1.3) Learnable, parametric global positional encoding (providing the spatial positional information between neighborhoods) and local positional encoding (providing the positional relationship of points within a neighborhood) are respectively obtained from the central point coordinates and the relative coordinates of neighbor points through MLP: ; In the formula, and are both MLP composed of a convolutional layer, batch normalization, and the Swish activation function.

[0070] Considering that the global positional encoding is obtained from the central point coordinates of the neighborhood, and only Q among Q, K, and V is obtained from the central point features of the neighborhood. Therefore, the global positional encoding is added when calculating the attention weight to provide the spatial positional information between neighborhoods for the learning of feature weights; local positional encoding It is added after the V vector to provide local position information for points in the neighborhood.

[0071] (2.2) Construct a deep learning network based on local neighborhood attention, including: (2.2.1) The deep learning network based on local neighborhood attention can be used for point cloud segmentation and point cloud classification. It consists of four modules: pre-embedding, downsampling feature extraction, residual block, and upsampling feature extraction. Its main feature aggregation operators are the non-downsampling attention module and the downsampling attention module.

[0072] (2.2.2) Pre-embedding. Before extracting deep features, we first use a pre-embedding module to expand the point cloud feature channel and extract shallow features of the point cloud. Pre-embedding consists of two consecutive non-subsampling attention modules.

[0073] (2.2.3) Downsampling feature extraction. The encoder part of the network downsamples the point cloud layer by layer, expanding the number of feature channels while reducing the number of points, and learning the deep features of the point cloud. Downsampling feature extraction is the main component of the encoder, which consists of a non-downsampling attention module and a downsampling attention module. The non-downsampling attention module is used to extract fine structure information, and the downsampling attention module is used to aggregate neighborhood features while downsampling the point cloud.

[0074] (2.2.4) Residual block. For the segmentation task, it requires deeper structural features for learning the geometric structure of the point cloud, so a residual block is added after each layer of downsampled feature extraction in the segmentation task. It contains a non-downsampled attention module, which connects its input residual to the output and then passes through an activation function. In fact, according to different tasks or datasets, the number of residual blocks added after the downsampled feature extraction can be adjusted to obtain better results.

[0075] (2.2.5) Upsampling feature extraction. For the segmentation task, the encoder part downsamples the point cloud. In order to finally obtain the classification score of each point, the decoder needs to upsample the point cloud. The point cloud features from the previous layer are upsampled using trilinear interpolation, concatenated with the output features of the corresponding layer in the encoder, and then a non-downsampled attention module is used to extract the fused point cloud features and send them to the next layer.

[0076] (2.2.6) Network architecture for point cloud segmentation and classification. For the segmentation task, the encoder part of the network consists of pre-embedding, downsampling feature extraction and residual blocks. The decoder part uses upsampling feature extraction to upsample the features obtained in the previous layer and connects the output of the corresponding layer of the encoder. The last layer of the decoder will obtain the features of each point, and finally use MLP to map it to the number of segmentation categories to obtain the score of each point. For the classification task, the encoder part removes the residual block compared to the segmentation task, because the deep feature extraction of the residual block will cause the network to over-learn the details of the point cloud, resulting in model overfitting. After the last downsampling feature extraction, two non-downsampling attention modules are used to further extract the point cloud features. The decoder part performs maximum pooling on the obtained features to obtain the global features of the point cloud, and finally uses MLP to obtain the classification score.

[0077] As a preferred embodiment, the present invention uses 4 downsampling feature extraction modules in the classification model; in the segmentation model, 4 downsampling feature extraction modules are used in the encoder part, and a residual block is used after each downsampling feature extraction module, and 4 upsampling feature extraction modules are used in the decoder part. The point cloud file of the input model contains the coordinate information x, y, z and normal vector information of each point , , , the feature dimension extracted by the pre-embedded module is 32-dimensional. After each downsampling feature extraction, the number of point cloud feature channels doubles; and after each upsampling feature extraction, the number of point cloud feature channels is halved.

[0078] (3) Use the training set to train the network, use the validation set to validate the network during the training process, and optimize the model, including: (3.1) Use the training set to train the network, input the training set into the network, calculate the cross entropy loss, and update the network parameters; As a preferred embodiment, during training, for point cloud classification, the present invention sets the batch size to 13, performs 400 rounds of training, the initial learning rate is 0.0001, uses the Adam optimizer to adjust the model parameters, and uses the cosine annealing algorithm to adjust the learning rate; for point cloud component segmentation, the batch size is set to 4, performs 120 rounds of training, the initial learning rate is 0.0001, and the learning rate is adjusted to 0.2 times in the 55th, 77th and 110th rounds, and the Adam optimizer is used to adjust the model parameters.

[0079] (3.2) Use the validation set to test and select the optimal network model, including: (3.2.1) Input the validation set into the model, and the data processing includes normalization and standardization, and then obtain the predicted label. For segmentation tasks, the mean intersection over union (mIoU) is used as the evaluation indicator, mIoU=TP / (TP+FP+FN), where TP represents true positive examples, FP represents false positive examples, and FN represents false negative examples. For classification tasks, the overall accuracy is used as the evaluation indicator. Compare the predicted labels with the actual labels to calculate the validation set accuracy, and save the optimal model on the validation set during the training process.

[0080] (4) Input the test set into the optimal deep network model to obtain the semantic segmentation results of the 3D point cloud, including: (4.1) Input the test set into the model selected in step (3), perform data processing including normalization and standardization, and then obtain the predicted label; (4.2) Calculate the accuracy of the test set.

[0081] The evaluation criteria for evaluating the point cloud semantic segmentation method based on local neighborhood attention provided by the embodiment of the present invention is the accuracy of the test set. For point cloud classification, the overall accuracy is used as the evaluation index, and for point cloud segmentation, the instance average intersection-over-union ratio is used as the evaluation criterion. In the point cloud classification task, the present invention is compared with three methods PointNet++, PointCNN, and Point-PlaneNet; in the point cloud segmentation task, the present invention is compared with three methods PointNet, SO-Net, and SPLATNet 3D. As shown in Table 1, the experimental results show that compared with previous methods, the point cloud semantic segmentation method based on local neighborhood attention provided by the embodiment of the present invention has achieved better experimental results on the test set on the public point cloud dataset, indicating that the point cloud semantic segmentation method of the present invention has good generalization ability and robustness.

[0082] The embodiments of the present invention may be implemented by hardware, software or a combination of the two. The hardware components may be constructed by specialized logic circuits; the software portion may be stored in a storage device and operated by appropriate hardware such as a microprocessor or specially designed hardware. A person of ordinary skill in the art will appreciate that the above-mentioned device and method may be implemented by computer executable instructions or processor control codes, which may be stored on a storage medium such as a disk, CD or DVD-ROM, or provided in a data medium such as a read-only memory, a programmable memory, an optical or electronic signal carrier, etc. The device and its modules of the present invention may be implemented by semiconductor circuits such as ultra-large-scale integrated circuits, gate arrays, logic chips, transistors, etc., or by software executed by various types of processors, or by a combination of the aforementioned hardware circuits and software.

[0083] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A point cloud semantic segmentation method and system based on local neighborhood attention, characterized in that: The following steps are involved: S101, dividing the three-dimensional point cloud data into training, verification, and test sets and preprocessing; S102, build a deep learning network based on local neighborhood attention, and use the training set to train and the validation set to obtain the optimal model; S103, input the test set into the optimal model to obtain the semantic segmentation result.

2. According to the method of claim 1, the point cloud semantic segmentation method based on local neighborhood attention is characterized in that: The point cloud semantic segmentation method based on local neighborhood attention comprises the following steps: Step 1: Divide the 3D point cloud data into training set, validation set, and test set, and preprocess the point cloud data set; Step 2: Construct a local neighborhood attention module that integrates local position encoding and global position encoding, and build a deep learning network based on local neighborhood attention; Step 3: Use the training set to train the network, use the validation set to validate the network during the training process, and optimize the model; Step 4: Input the test set into the optimal deep network model to obtain the semantic segmentation result of the 3D point cloud.

3. The point cloud semantic segmentation method based on local neighborhood attention according to claim 2, characterized in that: The step 1 of preprocessing the point cloud data set includes: The coordinates of the points in each point cloud file in the point cloud dataset are normalized to the interval [0, 1] in order to divide the neighborhood range and perform standardization.

4. The point cloud semantic segmentation method based on local neighborhood attention according to claim 2, characterized in that: The step 2 constructs a local neighborhood attention module that integrates local position encoding and global position encoding, and constructs a deep learning network based on local neighborhood attention, including: (1) Construct a local neighborhood attention module that combines local position encoding and global position encoding; The local neighborhood attention module, which combines local position encoding and global position encoding, uses vector attention to calculate point cloud features and aggregates them to the center point. Two types of position encoding are used to provide the model with the spatial position relationship between points within the neighborhood and between neighborhoods respectively; For a point cloud P containing n points, its features and coordinates are expressed as and . Sample m points from the point cloud P as the center point, and their features and coordinates are expressed as and . Directly move the center point feature After linear mapping, we get the query vector Q, then: ； In the formula, is a linear layer, which reduces the number of feature channels from Mapping ; In order to obtain the key vector and value vector, first input the point cloud feature After two different linear mappings, the center point The ball query is used to divide the neighborhood into m subsets based on the center. Each subset contains k points. The key vector K and value vector V are obtained respectively, and the corresponding neighbor point coordinates are obtained. In order to calculate the position code later; ； In the formula, is a linear layer, which reduces the number of feature channels from Mapping ; It means taking b as the center and dividing the neighborhood on c to group feature a; k represents the number of points in each neighborhood; Subtract the Q vector from the K vector, pass it through a multi-layer perceptron (MLP) and normalize it to get the attention weight. However, in order to help the model learn the structural relationship between neighborhoods, a global position encoding is added ,In addition, in order to prevent the network from overfitting, the attention weights are randomly lost; ； In the formula, represents an MLP consisting of a convolutional layer, batch normalization, and a Swish activation function, represents the softmax normalization function, and drop represents the random loss function; In addition to the structural relationship between neighborhoods, the positional relationship of points within a neighborhood is also crucial for point cloud learning, so a local position encoding is added to the V vector Then use the attention weight W for weighted summation to aggregate the features of each point in the neighborhood; In order to adapt to feature extraction in different situations, two attention modules are designed. The first attention module does not perform downsampling (m=n), which is called the non-downsampling attention module. It divides the neighborhood with each point as the center and calculates the attention; ； In the formula, represents element-wise multiplication in sequence, and the sum function adds the weighted point features in the neighborhood. The second attention module performs downsampling (m < n), which is called the downsampling attention module. It divides the neighborhood centered on the points obtained by downsampling and calculates the attention. In addition, to improve the stability of the model, a residual connection is added. The sampled point features are mapped to the same feature channels through a convolution and then added to the output. ； In the formula, represents one-dimensional convolution, act is the activation function; Learnable, parametric global position encoding (provides spatial location information between neighbors) and local location coding (provides the positional relationship of points in the neighborhood) respectively by the center point coordinates Relative coordinates of neighboring points Obtained through MLP; ； In the formula, and Both are MLPs consisting of convolutional layers, batch normalization, and Swish activation functions; Taking into account the global position encoding It is obtained from the coordinates of the neighborhood center point. Among Q, K, and V, only Q is obtained from the features of the neighborhood center point, so the global position encoding is added when calculating the attention weight , providing spatial location information between neighborhoods for learning feature weights; local position encoding It is added after the V vector to provide local position information for points in the neighborhood; (2) Constructing a deep learning network based on local neighborhood attention; The deep learning network based on local neighborhood attention can be used for point cloud segmentation and point cloud classification. It consists of four modules: pre-embedding, downsampling feature extraction, residual block, and upsampling feature extraction. Its main feature aggregation operators are non-downsampling attention module and downsampling attention module. Pre-embedding. Before extracting deep features, we first use a pre-embedding module to expand the point cloud feature channel and extract shallow features of the point cloud. Pre-embedding consists of two consecutive non-subsampling attention modules; Downsampling feature extraction. The encoder part of the network downsamples the point cloud layer by layer, expanding the number of feature channels while reducing the number of points, and learning the deep features of the point cloud. Downsampling feature extraction is the main component of the encoder. It consists of a non-downsampling attention module and a downsampling attention module. The non-downsampling attention module is used to extract fine structure information, and the downsampling attention module is used to aggregate neighborhood features while downsampling the point cloud; Residual block. For the segmentation task, it requires deeper structural features for learning the geometric structure of the point cloud, so a residual block is added after each layer of downsampled feature extraction in the segmentation task. It contains a non-downsampled attention module, which connects its input residual to the output and then passes through an activation function. In fact, according to different tasks or datasets, the number of residual blocks added after the downsampled feature extraction can be adjusted to obtain better results; Upsampling feature extraction. For the segmentation task, the encoder part downsamples the point cloud. In order to finally get the classification score of each point, the decoder needs to upsample the point cloud. The point cloud features from the previous layer are upsampled using trilinear interpolation, concatenated with the output features of the corresponding layer in the encoder, and then a non-downsampled attention module is used to extract the fused point cloud features and send them to the next layer; Network architecture for point cloud segmentation and classification. For segmentation tasks, the encoder part of the network consists of pre-embedding, downsampling feature extraction and residual blocks. The decoder part uses upsampling feature extraction to upsample the features obtained in the previous layer and connects the output of the corresponding layer of the encoder. The last layer of the decoder will obtain the features of each point, and finally use MLP to map it to the number of segmentation categories to obtain the score of each point. For classification tasks, compared with segmentation tasks, the encoder part removes the residual block because the deep feature extraction of the residual block will cause the network to over-learn the details of the point cloud, resulting in model overfitting. After the last downsampling feature extraction, two non-downsampling attention modules are used to further extract point cloud features. The decoder part performs maximum pooling on the obtained features to obtain the global features of the point cloud, and finally uses MLP to obtain the classification score.

5. The point cloud semantic segmentation method based on local neighborhood attention according to claim 2, characterized in that: The step 3 of training the network using the training set, validating the network using the validation set during the training process, and optimizing the model includes: (1) Use the training set to train the network, input the training set into the network, calculate the cross entropy loss, and update the network parameters; (2) Use the validation set to test and select the optimal network model; The validation set is input into the model, and the data processing includes normalization and standardization, and then the predicted label is obtained. For segmentation tasks, the mean intersection over union (mIoU) is used as the evaluation indicator, mIoU=TP / (TP+FP+FN), where TP represents true positive examples, FP represents false positive examples, and FN represents false negative examples. For classification tasks, the overall accuracy is used as the evaluation indicator. The predicted label is compared with the actual label to calculate the validation set accuracy, and the optimal model on the validation set during the training process is saved.

6. The point cloud semantic segmentation method based on local neighborhood attention according to claim 2, characterized in that: In step 4, the test set is input into the obtained optimal deep network model to obtain the semantic segmentation result of the three-dimensional point cloud, including: (1) Input the test set into the model selected in step 3, perform data processing including normalization and standardization, and then obtain the predicted label; (2) Calculate the accuracy of the test set.

7. A point cloud semantic segmentation system based on local neighborhood attention according to any one of claims 1 to 6, characterized in that: include: The 3D point cloud data preprocessing module is used to divide the data set into training set, validation set and test set according to a certain ratio, preprocess the data set and normalize and standardize the point cloud data; Point cloud semantic segmentation network building module, which is used to build a local neighborhood attention module that integrates local position encoding and global position encoding, and build a deep learning network based on local neighborhood attention; Point cloud semantic segmentation network training module, used to train the network using the training set, verify the network using the verification set during the training process and select the network model; The point cloud semantic segmentation module includes point cloud classification and point cloud segmentation, which is used to input the test set into the selected optimal classification or segmentation model to obtain the classification or segmentation results of the point cloud.

8. A point cloud semantic segmentation system based on local neighborhood attention, characterized in that: The invention comprises a computer device having a storage and executable point cloud semantic segmentation system program based on local neighborhood attention and a processor capable of executing any one of the point cloud semantic segmentation methods based on local neighborhood attention as claimed in claims 1 to 6.

9. A point cloud semantic segmentation system based on local neighborhood attention, characterized in that: It includes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes any one of the point cloud semantic segmentation methods based on local neighborhood attention as claimed in claims 1 to 6.

10. A point cloud semantic segmentation system based on local neighborhood attention, characterized in that: It comprises an information data processing terminal, and the information data processing terminal is used to implement the system.

Images