A point cloud semantic segmentation method based on global feature enhancement

Abstract

This invention discloses a point cloud semantic segmentation method based on global feature enhancement. The invention includes the following steps: 1. Point cloud data acquisition and preprocessing: Acquire point cloud data, preprocess it, and divide it into training and test sets for training and testing the model. 2. Establishment of the point cloud semantic segmentation model: Provide a point cloud semantic segmentation method based on global feature enhancement to enhance the model's ability to understand and utilize global point cloud features. In the encoder-decoder structure point cloud semantic segmentation network, a second-order covariance enhanced channel attention module (CECA) and a pseudo-Non-Local spatial attention module (PSA) are cascaded. The former learns the interdependencies between channels by modeling the second-order covariance statistics of each feature channel to enhance the representational ability of channel attention; the latter can effectively capture the correlation between points in the entire space. Both aim to capture the global feature information of the point cloud data. 3. Training the point cloud semantic segmentation model: Train the established model using the training set. 4. Testing the point cloud semantic segmentation model: Input the point cloud data from the test set into the trained model and infer the semantic category prediction values ​​of the test point clouds. This invention focuses on the global features of point cloud scenes, and improves the semantic segmentation accuracy of point cloud scenes by reasoning and enhancing the global features.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a point cloud semantic segmentation method based on global feature enhancement. Background Technology

[0002] Point clouds are scene data acquired through sensors such as laser scanners and radar, characterized by irregularity, disorder, and sparseness. Point cloud semantic segmentation assigns semantic category labels to each point in the point cloud data, enabling semantic understanding of different regions of the scene, identifying and classifying different objects in the environment. It is widely used in fields such as robotics, autonomous driving, smart city construction, and geographic information systems.

[0003] Currently, in the field of point cloud semantic segmentation, mainstream methods are based on deep learning techniques to extract features and then classify them point by point. These include projection-based, voxel-based, and point-based methods. Projection-based and voxel-based methods convert point clouds into multi-view images or regular voxel grids, and then apply traditional convolutional neural networks for feature learning. However, they generally suffer from problems such as information loss, high computational cost, and high memory consumption. Point-based methods, such as the classic PointNet network model, perform operations directly on the point cloud, extracting feature information point by point through multiple MLPs, and then obtaining global features of the point cloud through Max-Pooling. However, the PointNet network cannot capture local features. Later, PointNet++ improved by using local dependency and hierarchical feature learning to extract local features from different scales, making it suitable for complex point cloud scenarios.

[0004] PointNet++ introduces hierarchical learning to capture local features, which improves segmentation accuracy compared to PointNet. However, it still uses stacked MLPs to learn global features of point clouds, which cannot capture global features of long-distance dependencies between points. This leads to problems such as confusion between objects of different semantic categories, inaccurate segmentation boundaries, and low overall segmentation accuracy when the model performs semantic segmentation of objects in a scene. For example, objects such as tables, chairs, and sofas are confused in indoor scenes.

[0005] Therefore, it is necessary to design a point cloud semantic segmentation method that captures long-distance, global dependencies and learns more discriminative feature representations to improve the accuracy of point cloud semantic segmentation. Summary of the Invention

[0006] The purpose of this invention is to solve the problem of weak global feature discrimination in existing point cloud semantic segmentation technologies, and to provide a point cloud semantic segmentation method based on global feature enhancement. This method can effectively capture long-distance dependencies in scene point clouds and improve the segmentation accuracy of different semantic regions in complex scenes.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] A point cloud semantic segmentation method based on global feature enhancement, the method comprising:

[0009] Step 1: Point cloud data acquisition and preprocessing;

[0010] Step 2: Establishing a point cloud semantic segmentation model;

[0011] Step 3: Training the point cloud semantic segmentation model;

[0012] Step 4: Testing the point cloud semantic segmentation model.

[0013] Preferably, step 1, point cloud data acquisition and preprocessing, is characterized in that:

[0014] 1a) Obtain point cloud data of objects or scenes through sensors such as laser scanners and radar, or use public point cloud datasets, divide the point cloud dataset into k subsets, set aside one subset as the test set each time, and use the remaining k-1 subsets for training.

[0015] 1b) Given a point cloud training set, sample the input point cloud into a uniform number of N points, where the point cloud is represented as... It contains N points, each with D features, including position features {x, y, z} and color features {R, G, B}, etc.

[0016] Preferably, the distinguishing feature of establishing the point cloud semantic segmentation model in step 2 is that it uses an attention mechanism to acquire point cloud feature channels and the correlation between point positions to further capture long-distance global dependencies in the point cloud, enhancing the model's understanding of the overall scene structure and achieving more refined semantic segmentation. Specifically, as follows:

[0017] 2a) Extract point cloud features X, and use a point cloud semantic segmentation network with an encoder-decoder structure as the point cloud feature extraction network to construct a semantic feature representation for each point in the point cloud;

[0018] 2b) Establish a second-order covariance enhancement channel attention module (CECA) to further enhance the point cloud feature X, obtaining the enhanced point cloud feature E. The specific process is as follows:

[0019] Obtain output feature maps from point cloud feature extraction networks. Perform matrix multiplication between the transposed X and X' to obtain a channel attention map. Represented as:

[0020] A = X T ·X

[0021] Here, T represents the matrix transpose operation, and X represents the matrix multiplication operation. Then, A is subjected to Softmax normalization, and matrix multiplication with X is performed to obtain the feature map. Represented as:

[0022] B = X·Softmax(A)

[0023] Then, the second-order covariance matrix between each channel in the feature map X is calculated, and X is adjusted to a normalized matrix. Perform a second-order covariance operation on C to obtain the covariance matrix. Represented as:

[0024]

[0025] in, I is an N×N dimensional identity matrix, where 1 = [1, ..., 1]. T It is an N-dimensional vector. Next, to preserve the structural information of the covariance matrix and define a cross-channel operator to capture the relationships between adjacent channels, a row-wise convolution is performed on the covariance matrix, treating each row as a group in the group convolution. Then, a one-dimensional convolution with a kernel size of h is used to capture the relationships between adjacent channels, outputting a C′×1-dimensional weight vector. Each channel of the feature map X is then multiplied by the corresponding element in the weight vector to obtain the feature map. Represented as:

[0026] D = sigmoid(C1D) h (Group C′×1 (M)))◎X

[0027] Where sigmoid(·) represents the activation function, C1D h (·) represents a one-dimensional convolution with kernel size h, Group C′×1 (·) represents a group convolution with a kernel size of C′×1, and ◎ represents multiplication along the channel dimension.

[0028] Finally, a learnable weight parameter λ is set to generate a second-order covariance-enhanced channel attention map.

[0029] E = λB + X + D

[0030] 2c) Establish a pseudo-Non-Local spatial attention module (PSA) to further enhance the point cloud features E, obtaining the enhanced point cloud features R. The specific process is as follows:

[0031] The output feature map E is obtained from the second-order covariance enhancement channel attention module mentioned above, and E is fed into three different 1×1 convolutional layers to obtain different feature maps. and

[0032] Matrix multiplication is performed between the transposed E and F, followed by Softmax normalization to enhance the feature representation, resulting in a feature map. Represented as:

[0033] K = Softmax(E T ·F)

[0034] Here, represents the matrix multiplication operation. Then, the transpose of K is fed into the bottleneck transformation layer to obtain the feature map. Represented as:

[0035] Q = W k2 ReLU(LN(W k1 K T ))

[0036] Among them, W k2 ReLU(LN(W k1 (·))) represents the bottleneck transition layer, LN represents the normalization layer, ReLU is the activation function, and W k1 and W k2 It consists of two 1×1 convolutional layers, W k1 Reduce the channel dimension of feature map K to C′ / 16, W k2 Restore the channel dimension to the original dimension C′;

[0037] Then, matrix multiplication is performed on G and K to obtain the pseudo-feature map. Furthermore, to enhance the feature representation capability of O, after performing a Softmax normalization operation on H, a broadcast element-wise multiplication operation is performed with O to obtain an enhanced pseudo-feature map. Represented as:

[0038] P = Softmax((H)☉O

[0039] Here, ⊙ represents the element-wise multiplication operation in broadcast.

[0040] Finally, learnable weight parameters α and β are set to generate a pseudo-Non-Local spatial attention map. Represented as:

[0041] R = αP + βQ + E

[0042] 2d) Feature fusion: The point cloud features R and X output by the cascaded second-order covariance enhanced channel attention and pseudo-Non-Local spatial attention are spliced ​​together along the channel dimension.

[0043] 2e) Output the prediction results. After attention feature fusion, a 1×1 convolutional layer and a fully connected layer are used to predict the semantic category p of each point.

[0044] Preferably, the training of the point cloud semantic segmentation model in step 3 is characterized by:

[0045] 3a) Input the training set into the constructed point cloud semantic segmentation network and perform supervised training on the network using the Loss loss function; in each round, the performance of the model in that round will be evaluated and compared with the evaluation results of the previous round, and the model with higher accuracy will be saved.

[0046]

[0047] Where N represents the number of sample points, K represents the number of label values, and p i，j y represents the probability that the i-th sample is predicted to be the j-th label value. i，j This represents the actual category of the label; it is 1 if it is the j-th category, and 0 otherwise.

[0048] 3b) After training, the network model with the highest accuracy is taken as the optimal network model obtained from training.

[0049] Preferably, the testing of the point cloud semantic segmentation model in step 4 is characterized by inputting the test set into the optimal network model trained in the previous step for semantic segmentation to obtain the semantic segmentation result of each point.

[0050] The beneficial effects of this invention are as follows:

[0051] The method proposed in this invention utilizes a global feature enhancement module to address the insufficient global feature extraction problem in point cloud semantic segmentation, improving accuracy compared to other algorithms. Specifically: 1) It introduces an attention module cascaded with a second-order covariance enhanced channel attention module (CECA) and a pseudo-Non-Local spatial attention module (PSA) to enhance global point cloud features and improve segmentation accuracy; 2) Compared to previous attention enhancement modules based on the Non-Local principle, this invention provides a pseudo-Non-Local module to capture the correlation between points in the spatial domain, reducing the parameters of the attention module and making the model easier to train. Attached Figure Description

[0052] Figure 1 This is a flowchart of a point cloud semantic segmentation method based on global feature enhancement according to the present invention.

[0053] Figure 2 This is a schematic diagram of the second-order covariance enhanced channel attention module CECA of the present invention.

[0054] Figure 3 This is a schematic diagram of the pseudo-Non-Local spatial attention module (PSA) of the present invention.

[0055] Figure 4This is a visualization comparing the results of testing the present invention on a public dataset with other existing technologies. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the invention.

[0058] Example 1:

[0059] like Figure 1 As shown, this invention provides a point cloud semantic segmentation method based on global feature enhancement, the specific steps of which are as follows:

[0060] Step 1: Point cloud data acquisition and preprocessing;

[0061] Scene point cloud data is acquired using sensors such as laser scanners and radar, or by employing publicly available point cloud datasets. Taking the S3DIS point cloud dataset as an example, S3DIS contains point cloud data from 6 areas of 271 different rooms. In this embodiment, area 5 is selected as the test set, and the remaining 5 areas are used as the training set. Given the point cloud training set as input data, the input point cloud is sampled into a uniform number of 4096 points, where the point cloud is represented as... It contains 4096 points, each with 6 features, including position features {x, y, z} and color features {R, G, B}.

[0062] Step 2: Establishing a point cloud semantic segmentation model;

[0063] Considering only the positional features {x, y, z} of the point cloud for subsequent feature mapping, PointNet++, an encoder-decoder structured point cloud semantic segmentation network, is used as the point cloud feature extraction network to process the input point cloud data, constructing a semantic feature representation for each point in the point cloud, thus obtaining point cloud feature X. Then, point cloud feature X is used as input to the second-order covariance enhancement channel attention module CECA for point cloud feature enhancement. The specific process is as follows:

[0064] like Figure 2 As shown, the output feature map is obtained from the PointNet++ point cloud feature extraction network. N represents the number of points, and C′ represents the feature dimension of each point. Performing matrix multiplication directly on the transposed X and X' yields a channel attention map. Represented as:

[0065] A = X T ·X

[0066] Here, T represents the matrix transpose operation, and X represents the matrix multiplication operation. Then, A is subjected to Softmax normalization, and matrix multiplication with X is performed to obtain the feature map. Represented as:

[0067] B = X·Softmax(A)

[0068] Then, the second-order covariance matrix between each channel in the feature map X is calculated, and X is adjusted to a normalized matrix. Perform a second-order covariance operation on C to obtain the covariance matrix. Represented as:

[0069]

[0070] in, It is an N×N dimensional identity matrix, 1 = [1, ..., ] T It is an N-dimensional vector. Next, to preserve the structural information of the covariance matrix and define a cross-channel operator to capture the relationships between adjacent channels, a row-wise convolution is performed on the covariance matrix M, treating each row as a group in the group convolution. Then, a one-dimensional convolution with a kernel size of h is used to capture the relationships between adjacent channels, outputting a C′×1-dimensional weight vector. Each channel of the feature map X is then multiplied by the corresponding element in the weight vector to obtain the feature map. Represented as:

[0071] D = sigmoid(C1D) h (Group C′×1 (M)))◎X

[0072] Where sigmoid(·) represents the activation function, C1D h (·) represents a one-dimensional convolution with kernel size h, where h is set to 5. C′×1 (·) represents a group convolution with a kernel size of C′×1, and ◎ represents multiplication along the channel dimension.

[0073] Finally, the learnable weight parameter λ = 0.1 is set to generate a second-order covariance-enhanced channel attention map.

[0074] E j =λB j +X j +Dj

[0075] After the second-order covariance enhancement channel attention module CECA enhances the features of X, the point cloud features E are input into the pseudo-Non-Local spatial attention module PSA for further enhancement of the point cloud features. The specific process is as follows:

[0076] like Figure 3 As shown, feature map E is obtained from the second-order covariance enhancement channel attention module, and E is fed into three different 1×1 convolutional layers to obtain different feature maps. and

[0077] Matrix multiplication is performed between the transposed E and F, followed by Softmax normalization to enhance the feature representation, resulting in a feature map. Represented as:

[0078] K = Softmax(E T ·F)

[0079] Here, represents the matrix multiplication operation. Then, the transpose of K is fed into the bottleneck transformation layer to obtain the feature map. Represented as:

[0080] Q = W k2 ReLU(LN(W k1 K T ))

[0081] Among them, W k2 ReLU(LN(W k1 (·))) represents the bottleneck transition layer, LN represents the normalization layer, ReLU is the activation function, and W k1 and W k2 It consists of two 1×1 convolutional layers, W k1 Reduce the channel dimension of K to C′ / 16, W k2 Restore the channel dimension to the original dimension C′;

[0082] Then, matrix multiplication is performed on G and K to obtain the pseudo-feature map. Furthermore, to enhance the feature representation capability of O, a broadcast element-wise multiplication operation is performed on O and H to obtain an enhanced pseudo-feature map. Represented as:

[0083] P = Softmax(H) ⊙O

[0084] Here, ⊙ represents the element-wise multiplication operation in broadcast.

[0085] Finally, learnable weight parameters α and β are set and initialized to 0 to generate a pseudo-Non-Local spatial attention map. Represented as:

[0086] R = αP + βQ + E

[0087] After obtaining the enhanced point cloud features R, feature fusion is performed, and the point cloud features R output by the cascaded second-order covariance enhanced channel attention and pseudo-Non-Local spatial attention are concatenated with the point cloud features X along the channel dimension.

[0088] After performing attention feature fusion, a 1×1 convolutional layer and a fully connected layer are used to predict the semantic category p of each point.

[0089] Step 3: Training the point cloud semantic segmentation model;

[0090] The training set is used as point cloud data and input into the constructed point cloud semantic segmentation model. The network is then subjected to 50 rounds of supervised training using a loss function. In each round, the model's segmentation performance on the point cloud scene is evaluated and compared with the accuracy of the previous round. The model with higher accuracy is selected and saved. After 50 rounds of supervised training, the model with the highest accuracy is selected as the final point cloud semantic segmentation model.

[0091]

[0092] Where N represents the number of sample points, K represents the number of label values, and p i，j y represents the probability that the i-th sample is predicted to be the j-th label value. i，j The value represents the category corresponding to the actual label. If it is the j-th category, the value is 1, otherwise it is 0. In this embodiment, the S3DIS dataset used has a total of 13 category labels, that is, K=13.

[0093] Step 4: Testing the point cloud semantic segmentation model;

[0094] The test set is input into the trained network model for semantic segmentation. All points are used during testing to obtain the final semantic segmentation result.

[0095] To evaluate the effectiveness of the method proposed in this invention, a simulation experiment was conducted on the S3DIS dataset in this embodiment. The technical effects of this invention will be explained below in conjunction with the simulation experiment:

[0096] The dataset used in this embodiment is the S3DIS dataset, which contains point clouds of 271 different rooms in 6 regions. Each point has corresponding coordinate information and color information, and each point has one of 13 semantic category labels.

[0097] This embodiment was trained for 50 epochs on an RTX=3060 GPU with a batch size of 32, using the Adam optimizer with an initial learning rate of 0.001, and implemented on a PyTorch=1.10.1 platform using Ubuntu=18.04. After training the network to obtain the model using the training set, the model performance was evaluated using the test set, and mIoU was selected as the evaluation metric.

[0098]

[0099] Where mIoU represents the average class intersection-union ratio, p ii p represents the point that is correctly predicted as belonging to class i. ij p represents a point that belongs to category i but is classified as category j. ji This represents a point that is predicted to be in class j but is actually classified as class i.

[0100] This embodiment and the existing technology PointNet++ were used to train and test point cloud semantic segmentation on the S3DIS dataset. The IoU (%) and mIoU (%) of each category on the S3DIS dataset are shown in Table 1. Ceiling, Floor, etc. represent the categories to be semantically segmented in the dataset.

[0101] Table 1. Comparison of segmentation results of this embodiment and existing technology PointNet++ on the S3DIS dataset.

[0102]

[0103] As can be seen from Table 1, the segmentation accuracy of this embodiment on the S3DIS dataset exceeds that of the existing technology PointNet++ by 3.2%, indicating that the point cloud semantic segmentation effect of this method is stronger than that of PointNet++.

Claims

1. A point cloud semantic segmentation method based on global feature enhancement, characterized in that... Includes the following steps: Step 1: Point cloud data acquisition and preprocessing; 1a) Obtain point cloud data of objects or scenes through laser scanners or radar sensors, or use public point cloud datasets, divide the point cloud dataset into k subsets, set aside one subset as a test set each time, and use the remaining k-1 subsets for training. 1b) Given a point cloud training set, sample the input point cloud into a uniform number of N points, where the point cloud is represented as... It contains N points, each point has D features, including location features. and color characteristics {Red, Green, Blue}; Step 2: Establishing a point cloud semantic segmentation model; 2a) Extract point cloud features X, and use a point cloud semantic segmentation network with an encoder-decoder structure as the point cloud feature extraction network to construct a semantic feature representation for each point in the point cloud; 2b) Establish a second-order covariance enhancement channel attention module CECA to further enhance the point cloud feature X, and obtain the enhanced point cloud feature E; 2c) Establish a pseudo-Non-Local spatial attention module (PSA) to further enhance the point cloud feature E, resulting in the enhanced point cloud feature R; 2d) Feature fusion: The point cloud features R and X output by the cascaded second-order covariance enhanced channel attention and pseudo-Non-Local spatial attention are spliced ​​together along the channel dimension. 2e) Output the prediction results, and after attention feature fusion, use... Convolutional layers and fully connected layers predict the semantic category p for each point; Step 3: Training the point cloud semantic segmentation model; 3a) Input the training set into the constructed point cloud semantic segmentation network and perform supervised training on the network using the Loss loss function; in each round, the performance of the model in that round will be evaluated and compared with the evaluation results of the previous round, and the model with higher accuracy will be saved. Where N represents the number of sample points, and K represents the number of label values. This represents the probability that the i-th sample is predicted to be the j-th label value. This indicates the actual category of the label; it takes a value of 1 if it is the j-th category, and 0 otherwise. 3b) After training, the network model with the highest accuracy is taken as the optimal network model obtained from the training. Step 4) Testing the point cloud semantic segmentation model: The test set is input into the optimal network model trained in the previous step for semantic segmentation, and the semantic segmentation result of each point is obtained. Specifically, the second-order covariance enhanced channel attention module CECA established in step 2b) operates as follows: First, a point cloud semantic segmentation network with an encoder-decoder structure is used as the point cloud feature extraction network to construct a semantic feature representation for each point in the point cloud. Perform matrix multiplication between the transposed X and X' to obtain a channel attention map. , is represented as: Where T represents the matrix transpose operation. This represents a matrix multiplication operation; then, after performing a Softmax normalization operation on A, matrix multiplication is performed with X to obtain feature map B, represented as: Then, the second-order covariance matrix between each channel in the feature map X is calculated, and X is adjusted to a normalized matrix. Perform a second-order covariance operation on C to obtain the covariance matrix. , is represented as: in, I is a An identity matrix of dimension 1 It is an N-dimensional vector; Next, to preserve the structural information of the covariance matrix and define a cross-channel operator to capture the relationship between adjacent channels, a row-wise convolution is performed on the covariance matrix M. In the group convolution, each row is treated as a group. Then, a one-dimensional convolution with a kernel size of h is used to capture the relationship between adjacent channels, outputting a... The feature map X is obtained by multiplying each channel of the feature map X with the corresponding element in the weight vector. , is represented as: Where sigmoid(·) represents the activation function. This represents a one-dimensional convolution with kernel size h. Indicates the kernel size as Group convolution, This indicates multiplication along the channel dimension; Finally, a second-order covariance-enhanced channel attention map is generated. , is represented as: in, These are learnable weight parameters, initialized to 0.1; The pseudo-Non-Local Spatial Attention Module (PSA) established in step 2c) is operated as follows: First, feature map E is obtained from the second-order covariance enhancement channel attention module mentioned above. E is then fed into three different 1×1 convolutional layers to obtain different feature maps. , and ; Matrix multiplication is performed on F and its transpose E, followed by Softmax normalization to enhance the feature representation, resulting in the feature map. , is represented as: in, Represents matrix multiplication operation; The transpose of K is then fed into the bottleneck transformation layer to obtain the feature map. , is represented as: in, LN(·) represents the bottleneck transformation layer, LN(·) represents the normalization layer, and ReLU(·) is the activation function. and It consists of two 1×1 convolutional layers. Reduce the channel dimension of feature map K to , Restore the channel dimension to its original dimension. ; Then, matrix multiplication is performed on G and K to obtain the pseudo-feature map. Furthermore, to enhance the feature representation capability of O, after performing a Softmax normalization operation on H, a broadcast element-wise multiplication operation is performed with O to obtain an enhanced pseudo-feature map. , is represented as: in, This indicates an element-wise multiplication operation in broadcast; Finally, a pseudo-Non-Local spatial attention map is generated. , is represented as: in, and These are learnable weight parameters, initialized to 0.

Images