A Point Cloud Semantic Segmentation Method Based on 3D Object Context Representation

Abstract

The present invention discloses a point cloud semantic segmentation method based on three-dimensional object context representation, belonging to the field of computer vision technology. First, a data set composed of the original point cloud and the point cloud semantic annotation is constructed and divided into a training set, a validation set, and a test set; then, a three-dimensional sparse network model introducing three-dimensional object context representation is constructed and trained on the training set; finally, the trained model is tested on the test data to obtain the semantic annotation of the test point cloud. The method provided by the present invention introduces three-dimensional object context representation on the basis of a three-dimensional sparse convolutional neural network, enabling the consideration of the mutual relationship between each voxel and different category regions during classification, effectively aggregating large-scale semantic context information, and helping to improve the accuracy of semantic segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to the field of 3D point cloud semantic segmentation based on deep learning. Background Art

[0002] In recent years, deep learning technology has developed rapidly and been successfully applied to various tasks in computer vision. With the wide application of 3D sensors in fields such as robotics, deep learning based on 3D point clouds has gradually become a research hotspot. Similar to image semantic segmentation, 3D point cloud semantic segmentation is to assign a semantic label to each 3D point in the point cloud to achieve semantic understanding of the entire scene. However, compared with image data, 3D point clouds have the characteristics of sparsity, disorder, and non-structurality, which makes the processing using neural networks different. Currently, there are mainly three mainstream methods for point cloud semantic segmentation. The first is to convert the 3D point cloud into a range image and then use a 2D convolutional neural network for processing. This method can make full use of the mature 2D convolutional neural network, but it loses some 3D geometric information and affects the segmentation performance. The second is to directly process the 3D point cloud using a point cloud network (PointNet). Since this type of method requires iterative sampling and neighborhood search for a large number of point clouds, the computational cost is large and it cannot be applied in real time to large-scale scenes. The third is to divide the 3D space into 3D grids, convert the point cloud into sparse voxels, and use a sparse 3D convolutional neural network for processing. This type of method retains the 3D geometric information and can also flexibly design the network structure and perform relatively fast calculations through the sparse convolutional network. Therefore, the present invention adopts a method based on a 3D sparse convolutional network for point cloud semantic segmentation.

[0003] In various computer vision tasks, context information plays an important role. In image semantic segmentation, researchers have proposed many methods for aggregating context information, such as ASPP (Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587 (2017)), PMP (Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR (2017)), etc. To extract non-local context information, the literature (Wang, X. et al. (2018) ‘Non-local Neural Networks’, CVPR2018, pp. 7794–7803.) proposed non-local neural networks, which are similar in function to the popular self-attention module in recent years and can effectively improve network performance. In the literature (Yuan, Y., Chen, X. and Wang, J. (2020) ‘Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation’, Lecture Notes in Computer Science), 12351 LNCS, pp. 173–190. doi:10.1007 / 978-3-030-58539-6_11.), the authors pointed out that since the label of each pixel is the category of the object to which the pixel belongs, when extracting the context information of a certain pixel, the relationship between it and each category region should be focused on, and an object contextual representation (OCR) was proposed, which achieved better performance than other existing methods on multiple semantic segmentation annotation datasets. However, due to the characteristics of 3D point clouds, this idea has not been applied in point cloud segmentation. Inspired by OCR, the present invention extends the object contextual representation to sparse 3D point clouds, combines it with a sparse convolutional neural network, proposes a 3D object contextual representation (3D-OCR), and applies it to 3D point cloud semantic segmentation to improve the performance of point cloud segmentation. Summary of the Invention

[0004] The object of the present invention is to improve the performance of 3D point cloud semantic segmentation by introducing 3D semantic context information.

[0005] To achieve the above object, the present invention proposes a three-dimensional sparse neural network point cloud segmentation method introducing three-dimensional target context representation, and the method includes the following steps:

[0006] S1. Construct a data set composed of the original point cloud and the point cloud semantic annotation, and divide it into a training set, a validation set and a test set;

[0007] S2. Construct a three-dimensional sparse network model introducing three-dimensional target context representation, and train it on the training set;

[0008] S3. Test the trained model on the test data to obtain the semantic annotation of the test point cloud.

[0009] The specific steps of the step S2 are as follows:

[0010] S21. Select an appropriate voxel size according to the requirements of the scene size, accuracy and speed, divide the labeled point cloud into three-dimensional voxels, and calculate the coordinate index c for the non-empty voxels i , voxel feature x i and label l i , and form an input sparse feature tensor X and a sparse label tensor L;

[0011] S22. Construct a three-dimensional sparse convolutional backbone network, extract the high-level features Y of the sparse voxels, and input them into an auxiliary linear classification network to obtain the scores S aux of all non-empty voxels belonging to K categories;

[0012] S23. According to the classification scores of each non-empty voxel, organize all non-empty voxels into K three-dimensional generalized target regions M1, M2,... M K , and for each three-dimensional generalized target region M k , extract the target region representation f k :

[0013]

[0014] wherein, is the set of non-empty voxels, y i is the high-level feature of the voxel i extracted by the backbone, a ki is the coefficient measuring the degree of the voxel i belonging to M k , and this coefficient is obtained by performing a spatial Softmax operation on the category S aux obtained by the auxiliary classifier:

[0015]

[0016] S24. To extract the relationship context between non-empty voxels and the above-mentioned generalized target region, the correlation between non-empty voxels and the target region representation is calculated through the following formula:

[0017]

[0018] where represents the correlation between y and f, and w ik is the correlation coefficient after normalization; and φ represent the feature transformation function, which is usually composed of linear mapping, batch normalization, and non-linear activation functions;

[0019] S25. By synthesizing the correlations between non-empty voxels and all generalized target regions, the three-dimensional target context representation Z of non-empty voxels is calculated:

[0020]

[0021] where ρ and δ are transformation functions, similar to and φ in S24, and are also composed of linear mapping, batch normalization, and non-linear activation functions.

[0022] S26. The target context representation Z is concatenated (concat) with the high-level feature Y and undergoes feature transformation to form a new voxel feature G:

[0023] g i = γ([y i ; z i )

[0024] where the feature transformation γ is similar to and φ in S24, and is also composed of linear mapping, batch normalization, and non-linear activation functions;

[0025] S27. G is input into the linear classification layer to calculate the final class score S for each non-empty voxel;

[0026] S28. Calculate the loss based on the auxiliary classification score and the final classification score:

[0027] loss = CE(S, L) + λ·CE(S aux , L)

[0028] where CE is the cross-entropy loss function, and λ is the weight parameter that adjusts the importance of the auxiliary classification task and the final classification task.

[0029] S29. Use the stochastic gradient descent algorithm to update the network parameters, iterate a certain number of epochs on the training set, and record the parameters with the best performance on the validation set.

[0030] ​

[0031] S31. Using the same voxelization method as in S21, voxelize the test point cloud to form the input sparse feature tensor X t ;

[0032] S32. Input X t into the 3D sparse convolutional neural network trained in step S2 to obtain the final class scores S of all non-empty voxels t ;

[0033] S33. Calculate the label of each voxel:

[0034]

[0035] where represents the class score of voxel i.

[0036] S34. Using the mapping relationship between the point cloud and the voxels in step S31, map the semantic labels of the voxels to all 3D points divided into this voxel to obtain the final semantic labels of the point cloud.

[0037] Compared with the prior art, the effective benefits of the present invention are:

[0038] Based on the 3D sparse convolutional neural network, the method provided by the present invention introduces 3D object context representation, enabling consideration of the mutual relationship between each voxel and different class regions during classification, effectively aggregating large-scale semantic context information, and helping to improve the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a schematic diagram of the calculation process for training and testing the method of the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] The present invention will be further described in detail below with reference to the drawings and embodiments.

[0041] In this example, the SemanticKITTI point cloud semantic segmentation data is used. The SemanticKITTI dataset is a point cloud semantic segmentation dataset constructed in the literature (Behley, Jens, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Juergen Gall. 2019. “SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences,” no. iii. http: / / arxiv.org / abs / 1904.01416.). This dataset annotates 28 semantic categories for 22 point cloud sequences of the KITTI Odometry dataset. Sequences 00 - 10 are the training set, and 11 - 21 are the test set. Similar to the literature (Zhu, Xinge, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. 2020. “Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation.” http: / / arxiv.org / abs / 2011.10033.), we use sequence 08 in the original training set as the validation set, and the other sequences as the training set. At the same time, we merge the categories with different motion states and ignore the categories with very few points, and finally retain 19 semantic categories.

[0042] S1. Use sequences 00 - 07 and 09 - 10 of the SemanticKITTI dataset as the training set, sequence 08 as the validation set, and sequences 11 - 21 as the test set;

[0043] S2. Construct a three - dimensional sparse network model that introduces three - dimensional object context representation and train it on the training set;

[0044] S3. Test the trained model on the test data to obtain the semantic annotations of the test point cloud.

[0045] The detailed steps of the second step above are as follows:

[0046] S21. Divide the labeled point cloud into 3D voxels according to the voxel size of 0.05. Referring to the sparse tensor quantization method in the literature (Choy, Christopher, Junyoung Gwak, and Silvio Savarese. 2019. “4D Spatio-Temporal Convnets: Minkowski Convolutional Neural Networks.” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019 - June: 3070 - 79. https: / / doi.org / 10.1109 / CVPR.2019.00319.), with the position [x, y, z] of the point cloud as the coordinates and the position and reflectivity [x, y, z, intensity] as the features, after sparse tensor quantization, obtain the coordinate index c of the non-empty voxels i , voxel feature x i and label l i , to form the input sparse feature tensor X and sparse label tensor L;

[0047] S22. Use MinkowskiUNet as the backbone network to extract the high-level features Y of the sparse voxels and input them into an auxiliary linear classification network to obtain the scores S of all non-empty voxels belonging to K = 19 classes aux ;

[0048] S23. According to the classification scores of each non-empty voxel, organize all non-empty voxels into K three-dimensional generalized target regions M1, M2,... M K , and for each three-dimensional generalized target region M k , extract the target region representation f k :

[0049]

[0050] where, is the set of non-empty voxels, y i is the high-level feature of voxel i extracted by the backbone, a ki is the coefficient measuring the degree to which voxel i belongs to M k , and this coefficient is obtained by performing a spatial Softmax operation on the class S aux obtained by the auxiliary classifier:

[0051]

[0052] S24. To extract the relationship context between non-empty voxels and the above-mentioned generalized target region, the correlation between non-empty voxels and the target region representation is calculated through the following formula:

[0053]

[0054] where represents the correlation between y and f, and w ik is the correlation coefficient after normalization; and φ represent the feature transformation function, which consists of a linear mapping-batch normalization-nonlinear activation function;

[0055] S25. By synthesizing the correlations between non-empty voxels and all generalized target regions, the three-dimensional target context representation Z of non-empty voxels is calculated:

[0056]

[0057] where ρ and δ are transformation functions, similar to and φ in S24, and also consist of a linear mapping-batch normalization-nonlinear activation function.

[0058] S26. The target context representation Z is concatenated (concat) with the high-level feature Y and then undergoes feature transformation to form a new voxel feature G:

[0059] g i = γ([y i ; z i )

[0060] where the feature transformation γ is similar to and φ in S24, and also consists of a linear mapping-batch normalization-nonlinear activation function;

[0061] S27. G is input into the linear classification layer to calculate the final class score S for each non-empty voxel;

[0062] S28. Calculate the loss based on the auxiliary classification score and the final classification score:

[0063] loss = CE(S, L) + λ · CE(S aux , L)

[0064] where CE is the cross-entropy loss function, and λ is the weight parameter that adjusts the importance of the auxiliary classification task and the final classification task.

[0065] S29. Use the stochastic gradient descent algorithm to update the network parameters, iterate a certain number of epochs on the training set, and record the parameters with the best performance on the validation set.

[0066] The specific steps of step S3 are as follows:

[0067] S31. Using the same voxelization method as in S21, voxelize the test point cloud to form the input sparse feature tensor X t ;

[0068] S32. Input X t into the 3D sparse convolutional neural network trained in step S2 to obtain the final class scores S of all non-empty voxels t ;

[0069] S33. Calculate the label of each voxel:

[0070]

[0071] where represents the class score of voxel i.

[0072] S34. Using the mapping relationship between the point cloud and the voxels in step S31, map the semantic labels of the voxels to all 3D points divided into this voxel to obtain the final semantic labels of the point cloud.

[0073] Although the present invention has been described by way of examples, the present invention is not limited to the embodiments described herein. All changes and variations made without departing from the spirit of the present invention still fall within the scope of the present invention.

Claims

1. A 3D sparse neural network point cloud segmentation method introducing 3D target context representation, characterized in that The method includes the following steps: S1. Construct a dataset composed of the original point cloud and the semantic annotation of the point cloud, and divide it into a training set, a validation set, and a test set; S2. Construct a three-dimensional sparse network model introducing three-dimensional object context representation, and train it on the training set; S3. Test the trained model on the test data to obtain the semantic annotation of the test point cloud; The specific steps of step S2 are as follows: S21. Select an appropriate voxel size according to the scene size, precision, and speed requirements, divide the labeled point cloud into three-dimensional voxels, and for non-empty voxels calculate their coordinate index c i , voxel feature x i and label l i , and form the input sparse feature tensor X and sparse label tensor L; S22. Construct a 3D sparse convolutional backbone network to extract high-level features Y of the sparse voxels and input them into an auxiliary linear classification network to obtain the scores S of all non-empty voxels belonging to K categories aux ; S23. Organize all non-empty voxels into K three-dimensional generalized target regions M1, M2, ..., M according to the classification scores of each non-empty voxel K , and for each three-dimensional generalized target region M k , extract the target region representation f k : wherein, is a non-empty voxel set, y i is the high-level feature of voxel i for backbone extraction, a ki is the coefficient for measuring the degree to which voxel i belongs to M k , and this coefficient is obtained by performing spatial Softmax operation on the category S aux obtained by the auxiliary classifier: S24. To extract the relationship context between non-empty voxels and the above-mentioned general target regions, calculate the correlation between non-empty voxels and the above-mentioned target region representation through the following formula: Among them represents the correlation between y and f, and w ik is the correlation coefficient after normalization; and φ represent the feature change function, which is usually composed of linear mapping-batch normalization-nonlinear activation functions; S25. Synthesize the correlations between non-empty voxels and all general target regions, and calculate the three-dimensional object context representation Z of non-empty voxels: where ρ and δ are transformation functions, similar to and φ in S24, and also consist of a linear mapping batch normalization-nonlinear activation function; S26. Concatenate (concat) the target context representation Z with the high-level feature Y and perform feature transformation to form a new voxel feature G: g i = γ([y i ; z i ) (5) Among them, the feature transformation γ is similar to that in S24 and φ, and is also composed of a linear mapping, batch normalization, and a non-linear activation function; S27. Input G into the linear classification layer to calculate the final class score S of each non-empty voxel; S28. Calculate the loss according to the auxiliary classification score and the final classification score: loss = CE(S, L) + λ·CE(S aux , L) (6) where CE is the cross-entropy loss function, and λ is the weight parameter adjusting the importance of the auxiliary classification task and the final classification task; S29. Use the stochastic gradient descent algorithm to update the network parameters, iterate a certain number of epochs on the training set, and record the parameters with the best performance on the validation set; The specific steps of step S3 are as follows: S31. Using the same voxelization method as S21, voxelize the test point cloud to form the input sparse feature tensor X t ; S32. Input X t into the three-dimensional sparse convolutional neural network trained in step S2 to obtain the final class scores S of all non-empty voxels t ; S33. Calculate the label of each voxel: wherein represents the class score of voxel i; S34. Use the mapping relationship between the point cloud and the voxel in step S31 to map the semantic label of the voxel to all three-dimensional points divided into this voxel to obtain the final point cloud semantic label.

Images