A multi-scale context feature-based laser point cloud semantic segmentation method and system

Abstract

The application discloses a kind of multi-scale context feature's laser point cloud semantic segmentation method and system, method includes: S1, obtains original laser point cloud, pre-processes point cloud, extracts point cloud initial feature, obtains effective point cloud to be segmented;S2, laser point cloud is transformed into distance image by spherical projection formula;S3, distance image is input into multi-scale context feature fusion module, uses multiple convolution kernels to carry out convolution operation to distance image, obtains 64-dimensional feature map f M ;S4, f M Encoding is carried out by encoder, and decoder is spliced distance image deep feature and edge feature to obtain output result;The probability of each semantic label of output result is calculated, and the maximum probability semantic label is taken as the semantic segmentation result of distance image;S5, result is inversely mapped back to three-dimensional space, and laser point cloud semantic segmentation is completed.The application better retains point cloud feature on low-dimensional space, improves point cloud semantic average intersection-over-union value, and improves the segmentation result of laser point cloud.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of laser point cloud processing, and particularly relates to a laser point cloud semantic segmentation method and system based on multi-scale context features. BACKGROUND

[0002] Semantic segmentation is a computer image processing technology, and its task is to divide each pixel or point cloud obtained by a sensor in a scene into a plurality of regions with specific semantic labels according to a rule or model prepared in advance.

[0003] Compared with an image sensor, a laser sensor can provide more abundant and real spatial structure information, and is more helpful for an unmanned system to understand the environment in which the unmanned system is located. With continuous iteration and update of laser sensor technology and deep learning technology, laser point cloud semantic segmentation has developed rapidly in recent years. Existing laser point cloud semantic segmentation methods can be divided into two categories, namely a direct point-based semantic segmentation method and a projection-based semantic segmentation method. The direct point-based semantic segmentation method represented by PointNet and PointNet++ uses the permutation invariance and rotation invariance of point cloud to calculate the spatial features corresponding to each point cloud, and uses a deep learning model to classify the point cloud spatial features to obtain a semantic segmentation result of the global point cloud. This method is time-consuming in calculation and is not suitable for large-scale scene point cloud segmentation outdoors. The projection-based semantic segmentation method mainly maps the original point cloud to a two-dimensional distance image according to a spherical projection formula, uses a two-dimensional image semantic segmentation method to obtain a semantic label of the distance image, and then obtains a semantic label of the laser point cloud through an inverse mapping method. However, the existing projection-based semantic segmentation method uses a small receptive field convolution kernel to process the point cloud when predicting the semantic label of the point cloud, which easily ignores the local information around the point cloud and reduces the accuracy of the point cloud semantic segmentation. SUMMARY

[0004] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a laser point cloud semantic segmentation method and system based on multi-scale context features.

[0005] In order to achieve the above purpose, the following technical scheme is adopted in the present application:

[0006] A laser point cloud semantic segmentation method based on multi-scale context features comprises the following steps:

[0007] S1, obtaining original laser point cloud from a point cloud data set or a laser sensor, preprocessing the laser point cloud, extracting initial features of the point cloud, and obtaining effective point cloud to be segmented;

[0008] S2, transforming the effective point cloud using a spherical projection formula to obtain a distance image corresponding to the effective point cloud;

[0009] S3, input the distance image into the multi-scale context feature fusion module, use multiple convolution kernels to perform convolution operation on the distance image to obtain a 64-dimensional intermediate feature map f M of the distance image;

[0010] S4, input the 64-dimensional intermediate feature map f M into an encoder-decoder module, the encoder encodes the feature map f M , and the decoder splices the distance image deep features and edge features to obtain an output result of the distance image; use a Softmax function to calculate the probability of each semantic label of the output result, and take the semantic label with the maximum probability as the semantic segmentation result of the distance image;

[0011] S5, inversely map the semantic segmentation result back to the three-dimensional space to obtain the semantic label of the laser point cloud, and complete the semantic segmentation of the laser point cloud.

[0012] Further, step S1 is specifically:

[0013] read the point cloud data from the point cloud data set or the laser sensor, remove the NaN points in the laser point cloud, obtain the remaining point cloud original coordinate information and laser reflection intensity information, then calculate the geometric distance of each point to the laser sensor, scale the point cloud laser reflection intensity range from 0 to 255 to 0 to 1, and obtain the effective point cloud characterized by the distance between the point and the laser sensor, the point cloud coordinate and the normalized point cloud laser reflection intensity.

[0014] Further, step S2 is specifically:

[0015] use the spherical projection formula to transform the effective point cloud, map the three-dimensional coordinates of each point cloud to the pixel coordinates of the two-dimensional image, and each pixel has 5 channels; the spherical projection formula is as follows:

[0016]

[0017] wherein, W and H are the width and height of the distance image respectively, f down is the minimum vertical angle of the laser sensor, f is the vertical angle range of the laser sensor, and r is the distance from the point cloud to the laser sensor;

[0018] After projection, the size of the distance image is WxHx5.

[0019] Further, if multiple point clouds are mapped to the same pixel point on the distance image, the distance image stores the point cloud with the smallest distance between the point cloud and the laser sensor, and retains the mapping relationship of the laser point cloud and the two-dimensional pixel coordinates projected to the pixel point, and the distance image also retains the three-dimensional coordinates and the laser reflection intensity corresponding to the point cloud with the smallest distance between the point cloud and the laser sensor.

[0020] Further, in step S3, the multi-scale context feature fusion module includes two groups of convolution kernels of different scales and a residual network structure;

[0021] In the first group of convolution kernels, three convolution kernels of sizes 3x3, 5x5 and 7x7 are included, which are used to perform convolution operations on the distance image respectively to obtain distance image feature maps of different scales, and the three feature maps are spliced and point-by-point convolved to obtain a feature map f 21 of dimension 32.

[0022] The residual network structure takes the feature map f 21 as input to increase the dimension of the distance image feature map and obtain a 64-dimensional feature map f 22 .

[0023] In the second group of convolution kernels, the feature map f 22 is taken as input, and three convolution kernels are used to perform convolution transformation on the feature map f 22 to obtain a 64-dimensional distance image intermediate feature map f M , wherein the size of the first convolution kernel is 3x3 and the expansion coefficient is 1; the size of the second convolution kernel is 3x3 and the expansion coefficient is 2; the size of the third convolution kernel is 2x2 and the expansion coefficient is 2.

[0024] Further, the encoder-decoder module is divided into three parts in sequence, the first part is an encoder group, which has a total of 3 layers of encoders, each layer of encoder is connected in sequence to achieve the purpose of reducing the size of the feature map and expanding the dimension information of the feature map;

[0025] The middle second part is an encoder without downsampling function, which is used to perform point-by-point convolution on the 256-dimensional feature map to transform the deep features of the distance image.

[0026] The third part is a decoder group, which has a total of 3 layers of decoders connected in sequence, which is used to restore the size of the input feature map f M and calculate the semantic features of the feature points according to the deep feature information.

[0027] Among them, each layer of encoder includes two layers of point-by-point convolution, a Conv module and an average pooling module; each layer of decoder includes a pixel recombination method, a Conv module and a layer of point-by-point convolution.

[0028] In the encoder and decoder hierarchy with the same feature dimension, the encoder uses a group of skip layer connection structure before average pooling, inputs the edge feature information obtained by the encoder into the decoder, and the decoder fuses the edge feature and the deep feature. The feature matrix obtained by combining the two feature maps is convolved to obtain the distance image decoding feature.

[0029] Further, in each layer of the encoder structure, first, the input feature map is pointwise convolutional operated using a convolution kernel with a size of 1*1 to obtain a feature map f e1 as the input of the Conv module;

[0030] In the Conv module, three convolution kernels with different sizes and different expansion rates are included, wherein the size of the first convolution kernel is 3*3 and the expansion coefficient is 1, the size of the second convolution kernel is 3*3 and the expansion coefficient is 2, and the size of the third convolution kernel is 2 and the expansion coefficient is 2. The three convolution kernels together form a convolution module with receptive field sizes of 3, 7, and 4, respectively; the first convolution kernel takes the feature map f e1 as the input, and the inputs of the second and third convolution kernels are the outputs of the first and second convolution kernels, respectively; in order to effectively combine the outputs of the three convolution kernels, after the output of the third convolution kernel, the Conv module uses a splicing function to splice the feature maps output by the three convolution kernels, so that the output of the entire Conv module contains feature information under different receptive fields;

[0031] The Conv module processes the feature map f e1 , and then pointwise adds the output result and the feature map f e1 to obtain f e2 ; f e2 is processed using an average pooling method to comprehensively retain the feature values of each feature point, reduce the size of the feature map, and form the encoding features of the input feature map.

[0032] Further, in order to restore the size of the feature map with deep features to the size of the input feature map f M , the decoder first expands the size of the feature map with deep features, uses a group of pixel reorganization methods to upsample the input feature map, obtains a high-resolution low-dimensional feature map through reorganization and convolution among multiple channels of the deep feature map, and then splices the feature map f e2 with the pre-feature map f e2 of the same dimension to obtain a feature map f d1 ; the Conv module containing multiple convolution layers is used to reduce the dimension of the feature map f d1 to obtain a feature map f d2 ; the feature map f d2 is pointwise convolutional to obtain a group of decoding features.

[0033] Further, after repeating the decoding operation three times, the encoder-decoder module obtains a group of feature maps with a size of 2048*64*32, and according to the number of semantic labels of the data set, uses a convolution kernel with a size of 1 and a step of 1 to reduce the dimension of the feature map from 32 to 20, and then uses a Softmax function to calculate the probability of each semantic label, and takes the semantic label with the maximum probability as the semantic segmentation result of the pixel point of the image.

[0034] The application also comprises a laser point cloud semantic segmentation system of multi-scale context features, which adopts the laser point cloud semantic segmentation method provided by the application, and comprises a point cloud preprocessing module, a point cloud spherical projection module, a multi-scale context feature fusion module, an encoder-decoder module and a point cloud inverse mapping module.

[0035] The point cloud preprocessing module is used for reading point clouds from a point cloud dataset or a laser sensor, pre-processing the point clouds, calculating the distance of the point clouds to the laser sensor, normalizing the laser reflection intensity of the point clouds, extracting initial features of the point clouds, and outputting effective point clouds.

[0036] The point cloud spherical projection module adopts a spherical projection formula to map the effective point clouds to a two-dimensional distance image.

[0037] The multi-scale context feature fusion module comprises a plurality of convolution kernels of different sizes and different expansion rates, outputs feature maps output by the different convolution kernels, and splices, reduces dimensions of, and point-by-point convolves the feature maps to generate a 64-dimensional intermediate feature map f M of the distance image.

[0038] The encoder-decoder module comprises an encoder and a decoder, the encoder takes the feature map f M as input, passes through an encoder module composed of a point-by-point convolution, a residual module and an average pooling method, obtains a distance image deep feature with smaller size and higher dimension, the decoder decodes after splicing the shallow edge feature and the deep feature connected by a skip layer structure, outputs a decoding feature of the distance image, and obtains a 20-dimensional output result, the encoder-decoder module also adopts a Softmax function to calculate the probability of each pixel point in the output result for each semantic label, and takes the semantic label with the maximum probability as the semantic segmentation result of the distance image at the pixel point.

[0039] The point cloud inverse mapping module is used for inversely mapping the semantic segmentation result back to a three-dimensional space to obtain a semantic label of the laser point cloud, and completes the laser point cloud semantic segmentation.

[0040] Compared with the prior art, the application has the following advantages and beneficial effects:

[0041] 1. The application uses convolution kernels of different sizes to form a multi-scale context feature fusion module, which can increase the receptive field information of the distance image and strengthen the relationship between the center point cloud and its adjacent point clouds.

[0042] 2. The application reduces the number of residual modules of the encoder, increases the input information feature dimension of the encoder-decoder structure, and optimizes the convolution process on the basis of the RangeNet++ model.

[0043] 3、The application uses an average pooling function in the encoder to down-sample feature points in the horizontal direction and the vertical direction, comprehensively retains feature information of each feature point, and effectively reduces the size of the feature map.

[0044] 4、The application uses a pixel recombination method in the decoder to replace transposed convolution to up-sample the feature map, avoids parameters introduced by the transposed matrix, and accelerates the size restoration speed of the feature map. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of the method of the application;

[0046] Figure 2 is a structural schematic diagram of the multi-scale context feature fusion module of the application;

[0047] Figure 3 is a structural schematic diagram of the encoder-decoder module of the application;

[0048] Figure 4 is a structural schematic diagram of the encoder of the application;

[0049] Figure 5 is a structural schematic diagram of the decoder of the application. DETAILED DESCRIPTION

[0050] The application will be further described in detail below in combination with embodiments and drawings, but the embodiments of the application are not limited thereto.

[0051] EMBODIMENT

[0052] As shown in the drawings, the application is a multi-scale context feature laser point cloud semantic segmentation method, comprising the following steps: Figure 1

[0053] S1, obtaining original laser point cloud from a point cloud data set or a laser sensor, preprocessing the laser point cloud, extracting initial features of the point cloud, and obtaining effective point cloud to be segmented, specifically:

[0054] After reading the point cloud data from the point cloud data set or the laser sensor, since a small amount of point cloud coordinates cannot be represented by numbers (Not A Number, NaN), the divisor will be 0 when calculating the yaw angle of the point cloud, which will cause the program to abnormally exit and will affect the calculation of the point cloud projection coordinates, therefore, the point cloud preprocessing module first removes the NaN point cloud using the PCL library function to obtain the remaining point cloud original coordinate information and laser reflection intensity information, and then calculates the geometric distance r of each point cloud to the laser sensor;

[0055] ​In order to improve the training speed and prediction speed of the semantic segmentation model, the laser reflection intensity of the effective point cloud needs to be normalized, and the laser reflection intensity range of the point cloud is scaled from 0-255 to 0-1.

[0056] After the above processing, the size of the effective point cloud obtained is Nxd.

[0057] Wherein, N represents the number of the current effective point cloud, and d represents the information dimension of the current effective point cloud. In the embodiment, the information dimension d=5, including the distance r of the point cloud to the laser sensor, the coordinates xyz of the point cloud and the normalized reflection intensity i of the point cloud.

[0058] S2, using the spherical projection formula to transform the effective point cloud, obtaining the distance image corresponding to the effective point cloud; specifically:

[0059] Using the spherical projection formula to transform the effective point cloud, mapping the three-dimensional coordinates of each point cloud to the pixel coordinates of the two-dimensional image, each pixel has five channels; the spherical projection formula is as follows:

[0060]

[0061] Wherein, W, H are the width and height of the distance image respectively, f down is the minimum vertical angle of the laser sensor, f is the vertical angle range of the laser sensor, and r is the distance from the point cloud to the laser sensor;

[0062] If multiple point clouds are mapped to the same pixel point on the distance image, the distance image stores the point cloud with the smallest distance between the point cloud and the laser sensor, and the mapping relationship of the remaining projected laser point cloud and two-dimensional pixel coordinates is retained, and the distance image also retains the three-dimensional coordinates and laser reflection intensity corresponding to the point cloud with the smallest distance between the point cloud and the laser sensor.

[0063] After projection, the size of the distance image is WxHx5.

[0064] S3, inputting the distance image into the multi-scale context feature fusion module, using multiple convolution kernels to perform convolution operation on the distance image, obtaining a 64-dimensional intermediate feature map f M of the distance image.

[0065] In the embodiment, as shown in the figure, Figure 2 The multi-scale context feature fusion module includes two groups of convolution kernels with different scales and a group of residual network structures.

[0066] In the first group of convolution kernels, three convolution kernels with sizes of 3x3, 5x5 and 7x7 are included, and convolution operations are performed on the distance image respectively to obtain distance image feature maps at different scales, and the three feature maps are spliced and point-by-point convolved to obtain a feature map f 21 with a dimension of 32

[0067] The residual network structure takes the feature map f 21 as input, increases the dimension of the distance image feature map, and obtains a 64-dimensional feature map f 22 .

[0068] In the second group of convolution kernels, the feature map f 22 is taken as input, and three convolution kernels are used to perform convolution transformation on the feature map f 22 to obtain a 64-dimensional distance image intermediate feature map f M , wherein the size of the first convolution kernel is 3x3 and the expansion coefficient is 1; the size of the second convolution kernel is 3x3 and the expansion coefficient is 2; the size of the third convolution kernel is 2x2 and the expansion coefficient is 2.

[0069] S4, input the 64-dimensional intermediate feature map f M to the encoder-decoder module, the encoder encodes the feature map f M , and the decoder splices the distance image deep feature and the edge feature to obtain the output result of the distance image; the Softmax function is used to calculate the probability of each semantic label of the output result, and the semantic label with the maximum probability is taken as the semantic segmentation result of the distance image;

[0070] In this embodiment, as shown in Figure 3 , the encoder-decoder module is divided into three parts in sequence, the first part is an encoder group, which has three layers of encoders connected in sequence, achieving the purpose of reducing the size of the feature map and expanding the dimension information of the feature map;

[0071] The middle second part is an encoder without downsampling function, which is used for point-by-point convolution on the 256-dimensional feature map to transform the deep features of the distance image;

[0072] The third part is a decoder group, which has three layers of decoders connected in sequence, which is used to restore the size of the input feature map f M and calculate the semantic features of the feature points according to the deep feature information;

[0073] As shown in Figure 4 , each layer of encoder includes two layers of point-by-point convolution, a Conv module and an average pooling module; as shown in Figure 5 , each layer of decoder includes a pixel reconstruction method, a Conv module and a layer of point-by-point convolution;

[0074] In encoder and decoder layers with the same feature dimension, the encoder uses a set of skip connection structures before average pooling to input the edge feature information obtained by the encoder into the decoder. The decoder fuses the edge features and deep features, and performs convolution on the feature matrix after combining the two feature maps to obtain the distance image decoding features.

[0075] like Figure 4 As shown, in each encoder layer structure, a 1×1 convolution kernel is first used to perform pointwise convolution operations on the input feature map to obtain a feature map f with the same dimension as the output feature map. e1 , as input to the Conv module;

[0076] The Conv module contains three convolutional kernels of different sizes and dilation rates. The first kernel is 3×3 with a dilation factor of 1, the second kernel is 3×3 with a dilation factor of 2, and the third kernel is 2 with a dilation factor of 2. These three kernels together form convolutional modules with receptive field sizes of 3, 7, and 4, respectively. The first kernel uses the feature map f... e1 The first and second convolutional kernels are the inputs of the first and second convolutional kernels, respectively. To effectively combine the outputs of the three convolutional kernels, after the output of the third convolutional kernel, the Conv module uses a concatenation function to concatenate the feature maps output by the three convolutional kernels, so that the output of the entire Conv module contains feature information from different receptive fields.

[0077] The Conv module processes the feature map f e1 Process the data, then combine the output with the feature map f. e1 Adding them point by point, we get f. e2 , for f e2 The average pooling method is used to comprehensively retain the feature values ​​of each feature point, reduce the size of the feature map, and form the encoded features of the input feature map.

[0078] like Figure 5 As shown, this method restores the size of the feature map with deep features back to the input feature map f. M To determine the size of the input feature map, the decoder first enlarges the feature map with deep features, then upsamples the input feature map using a set of pixel recombination methods. Through multi-channel recombination and convolution of the deep feature map, a high-resolution, low-dimensional feature map is obtained, which is then compared with the pre-image feature map of the same dimension. e2 By concatenating the features, we obtain feature map f. d1 Using a Conv module with multiple convolutional layers to reduce the feature map f d1 The dimension of the feature map f d2 Pointwise convolution yields a set of decoded features.

[0079] After repeating the decoding operation 3 times, the encoder-decoder module obtains a set of feature maps with a data size of 2048x64x32, and according to the number of semantic labels of the data set, uses a convolution kernel with a size of 1 and a step of 1 to reduce the dimension of the feature map from 32 to 20, and then uses a Softmax function to calculate the probability of each semantic label, and takes the semantic label with the maximum probability as the semantic segmentation result of the distance image pixel point. The calculation formula of the Softmax function is as follows:

[0080]

[0081] wherein, res i is the feature value of the encoder-decoder output result for each semantic label, C is the number of semantic labels of the data set, is the probability of each semantic label.

[0082] S5, using a GPU-based kNN clustering method, inversely mapping the semantic segmentation result back to the three-dimensional space to obtain the semantic label of the laser point cloud, and completing the semantic segmentation of the laser point cloud.

[0083] In this embodiment, the kNN clustering method is used as follows:

[0084] S51, using a sliding window with a size of 5 to traverse all pixel points of the distance channel of the distance image, and establishing a distance neighboring point matrix Neighboring point matrix The size of the neighboring point matrix is WxHx1, wherein W and H are the width and height of the distance image, respectively.

[0085] S52, assigning the same distance neighboring point to all effective point clouds projected to the same pixel point to form a distance neighboring point matrix of the effective point cloud Matrix The size of the matrix is Nx1, wherein N is the number of effective point clouds.

[0086] S53, using a sliding window with a size of 5 to traverse the semantic segmentation result of the distance image, and establishing a label neighboring point matrix assigning the same label neighboring point to all effective point clouds projected to the same pixel point to form a label neighboring point matrix of the effective point cloud Matrix The size of the matrix is Nx1, wherein N is the number of effective point clouds.

[0087] S54, calculating the distance of each neighboring point to the center point in each column of the matrix Distance difference matrix

[0088] S55, calculating the inverse Gaussian kernel for the distance difference matrix weighting, a weighted distance difference matrix is obtained

[0089] S56, each column element of the weighted distance difference matrix is sorted in ascending order, the first k points after sorting are extracted, and the label with the highest frequency of occurrence in the first k points is taken as the label value of the center point cloud of the column element.

[0090] To verify that the method of the application can effectively improve the average intersection over union of laser point cloud semantic segmentation, in this embodiment, the network model is trained using the SemanticKITTI training set and the SemanticPOSS dataset on the PyTorch platform of Linux, the model training times is 100, the batch size is 4, the SGD optimizer with a learning rate of 0.01 and a momentum of 0.9 is used as the optimizer, and the weight decay rate is 10 -4 After training the network model, the SemanticKITTI test set and the SemanticPOSS test set are subjected to semantic segmentation to obtain the semantic label label file of the test set, and the label file is uploaded to the server to evaluate the model performance. In this embodiment, the average intersection over union (mean Intersection over Union, mIoU) and the intersection over union of each semantic label are used as the model evaluation indicators. The average intersection over union represents the average ratio of the intersection and union of the real label and the predicted value of all classes in the data set, and the calculation formula is as shown in the following formula:

[0091]

[0092] Wherein, TP c represents the number of point clouds correctly classified as class c, FP c represents the number of point clouds of other classes incorrectly predicted as class c by the model, FN c represents the number of point clouds of class c incorrectly predicted as other classes, and n represents the total number of labels.

[0093] The comparison of the method of the application and other laser point cloud semantic segmentation methods on the SemanticKITTI test set is shown in Table 1 as follows:

[0094]

[0095]

[0096] Table 1

[0097] The comparison of the method of the application and other projection-based laser point cloud semantic segmentation methods on the SemanticPOSS test set is shown in Table 2 as follows:

[0098] ​

[0099] Table 2

[0100] In another embodiment, a laser point cloud semantic segmentation system of multi-scale context features is also provided, the system adopts the laser point cloud semantic segmentation method described in the above embodiment, and the system comprises a point cloud preprocessing module, a point cloud spherical projection module, a multi-scale context feature fusion module, an encoder-decoder module and a point cloud inverse mapping module;

[0101] The point cloud preprocessing module is configured to read a point cloud from a point cloud dataset or a laser sensor, pre-process the point cloud, calculate a distance of the point cloud to the laser sensor, normalize a laser reflection intensity of the point cloud, extract initial features of the point cloud, and output valid point clouds.

[0102] The point cloud spherical projection module is configured to map the valid point clouds to a two-dimensional distance image by using a spherical projection formula.

[0103] The multi-scale context feature fusion module comprises a plurality of convolution kernels of different sizes and different expansion rates, outputs feature maps output by the different convolution kernels, and performs splicing, dimension reduction and point-by-point convolution on the feature maps to generate a 64-dimensional intermediate feature map f M of the distance image.

[0104] The encoder-decoder module comprises an encoder and a decoder, the encoder takes the feature map f M as input, passes through an encoder module composed of a point-by-point convolution, a residual module and an average pooling method, and obtains a distance image deep feature of a smaller size and a higher dimension, the decoder decodes after splicing a shallow edge feature and a deep feature connected by a skip layer structure, outputs a decoding feature of the distance image, and obtains a 20-dimensional output result, the encoder-decoder module further adopts a Softmax function to calculate a probability of each pixel point in the output result for each semantic label, and takes a semantic label with a maximum probability as a semantic segmentation result of the distance image at the pixel point.

[0105] The point cloud inverse mapping module is configured to inversely map the semantic segmentation result back to a three-dimensional space to obtain a semantic label of the laser point cloud, and complete the laser point cloud semantic segmentation.

[0106] It should be further noted that, in the present specification, terms such as “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement “comprises a” does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0107] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Numerous modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without the use of the inventive faculty. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for semantic segmentation of a laser point cloud with multi-scale contextual features, the method comprising: The method comprises the following steps: S1, obtaining an original laser point cloud from a point cloud dataset or a laser sensor, preprocessing the laser point cloud, extracting initial features of the point cloud, and obtaining an effective point cloud to be segmented; S2, transforming the effective point cloud using a spherical projection formula to obtain a distance image corresponding to the effective point cloud; S3, input the distance image to the multi-scale context feature fusion module, use multiple convolution kernels to perform convolution operation on the distance image to obtain a 64-dimensional intermediate feature map f of the distance image M The multi-scale context feature fusion module comprises two groups of convolution kernels with different scales and a group of residual network structures. In the first group of convolution kernels, 3 convolution kernels with sizes of 3x3, 5x5 and 7x7 are included, and convolution operations are performed on the distance image respectively to obtain distance image feature maps at different scales, and the three feature maps are spliced and point-by-point convolved to obtain a feature map f with a dimension of 32 21 ; The residual network structure takes a feature map f 21 as input, increases the dimension of the distance image feature map, and obtains a 64-dimensional feature map f 22 ; In the second group of convolution kernels, taking the feature map f 22 as input, the feature map f 22 is respectively convoluted using 3 convolution kernels to obtain a 64-dimensional distance image intermediate feature map f M , wherein the size of the first convolution kernel is 3x3, the expansion coefficient is 1; the size of the second convolution kernel is 3x3, the expansion coefficient is 2; the size of the third convolution kernel is 2x2, the expansion coefficient is 2; S4, the 64-dimensional intermediate feature map f M is input to an encoder-decoder module, the encoder encodes the feature map f M , the decoder splices the deep features and edge features of the distance image to obtain an output result of the distance image; a Softmax function is used to calculate the probability of each semantic label of the output result, and the semantic label with the maximum probability is taken as the semantic segmentation result of the distance image; S5, inversely mapping the semantic segmentation result back to a three-dimensional space to obtain a semantic label of the laser point cloud, and completing semantic segmentation of the laser point cloud.

2. The method of claim 1, wherein, Step S1 is specifically as follows: Point cloud data is read from a point cloud dataset or a laser sensor, NaN points in the laser point cloud are removed, original coordinate information and laser reflection intensity information of the remaining point cloud are obtained, then the geometric distance of each point to the laser sensor is calculated, the laser reflection intensity range of the point cloud is scaled from 0-255 to 0-1, and an effective point cloud is obtained with the distance between the point and the laser sensor, the point cloud coordinates and the normalized laser reflection intensity of the point cloud as features.

3. The method of claim 1, wherein, Step S2 is specifically as follows: The spherical projection formula is used to transform the effective point cloud, and the three-dimensional coordinates of each point cloud are mapped to the pixel coordinates of a two-dimensional image, each pixel having five channels; the spherical projection formula is as follows: wherein W, H are the width and height of the image respectively, f down is the minimum view angle of the laser sensor in the vertical direction, f is the vertical view angle range of the laser sensor, and r is the distance of the point cloud to the laser sensor. After projection, the size of the distance image is WxHx5.

4. The method of claim 1, wherein, If multiple point clouds are mapped to the same pixel point on the distance image, the distance image stores the point cloud with the smallest distance between the point cloud and the laser sensor, and retains the mapping relationship of the laser point cloud and the two-dimensional pixel coordinates projected to the pixel point, and the distance image also retains the three-dimensional coordinates and the laser reflection intensity of the point cloud with the smallest distance between the point cloud and the laser sensor.

5. The method of claim 1, wherein, The encoder-decoder module is sequentially divided into three parts, the first part is an encoder group, which has three layers of encoders, each layer of encoder is sequentially connected to reduce the size of the feature map and expand the dimension information of the feature map; The second part in the middle is an encoder without downsampling function, which is used for point-by-point convolution of the 256-dimensional feature map to transform the deep features of the distance image; The third part is a decoder group, which has three layers of decoders connected in sequence, for recovering the size of the input feature map f M and calculating the semantic features of the feature points according to the deep feature information. Each layer of encoder includes two layers of point-by-point convolution, a Conv module and an average pooling module; each layer of decoder includes a pixel recombination method, a Conv module and a layer of point-by-point convolution; In the same feature dimension encoder and decoder hierarchy, the encoder uses a set of skip connection structure before average pooling, inputs the edge feature information obtained by the encoder into the decoder, and fuses the edge feature and the deep feature by the decoder, and convolves the combined feature matrix of the two kinds of feature maps to obtain the distance image decoding feature.

6. The method of claim 5, wherein, In each layer encoder structure, first, the input feature map is pointwise convolution operation using a convolution kernel with size of 1x1, to obtain a feature map f with the same dimension and output feature as the input e1 , as the input of the Conv module; In the Conv module, three convolution kernels with different sizes and different expansion rates are included, wherein the size of the first convolution kernel is 3*3, the expansion coefficient is 1, the size of the second convolution kernel is 3*3, the expansion coefficient is 2, and the size of the third convolution kernel is 2, and the expansion coefficient is 2. The three convolution kernels together form a convolution module with receptive field sizes of 3, 7 and 4 respectively; the first convolution kernel takes the feature map f e1 as input, and the inputs of the second and third convolution kernels are the outputs of the first and second convolution kernels respectively; in order to effectively combine the outputs of the three convolution kernels, after the output result of the third convolution kernel, the Conv module uses a splicing function to splice the feature maps output by the three convolution kernels, so that the output of the entire Conv module contains feature information under different receptive fields. The Conv module processes the feature map f e1 and adds the output result and the feature map f e1 point by point to obtain f e2 , and f e2 uses the average pooling method to integrate and retain the feature values of each feature point, reduce the size of the feature map, and form the encoded features of the input feature map.

7. The method of claim 6, wherein, To restore the size of the feature map with deep features back to the size of the input feature map f M , the decoder first expands the size of the feature map with deep features, up-samples the input feature map using a set of pixel reorganization methods, obtains a high-resolution low-dimensional feature map through reorganization and convolution among multi-channels of the deep feature map, and then splices the feature map with the same dimension as the front feature map f e2 to obtain a feature map f d1 ; uses a Conv module containing multiple convolution layers to reduce the dimension of the feature map f d1 to obtain a feature map f d2 ; and point-by-point convolves the feature map f d2 to obtain a set of decoding features.

8. The method of claim 5, wherein, After repeating the decoding operation for three times, the encoder-decoder module obtains a feature map with a data size of 2048x64x32, and according to the number of semantic labels of the dataset, uses a convolution kernel with a size of 1 and a step of 1 to reduce the dimension of the feature map from 32 to 20, then uses a Softmax function to calculate the probability of each semantic label, and takes the semantic label with the maximum probability as the semantic segmentation result of the pixel point of the distance image.

9. A system for semantic segmentation of a laser point cloud with multi-scale contextual features, the system comprising: The system adopts the laser point cloud semantic segmentation method according to any one of claims 1-8, and the system comprises a point cloud preprocessing module, a point cloud spherical projection module, a multi-scale context feature fusion module, an encoder-decoder module and a point cloud inverse mapping module. The point cloud preprocessing module is used for reading point clouds from a point cloud data set or a laser sensor, pre-processing the point clouds, calculating distances of the point clouds to the laser sensor, normalizing laser reflection intensities of the point clouds, extracting initial features of the point clouds, and outputting effective point clouds. The point cloud spherical projection module maps the effective point clouds to a two-dimensional distance image by using a spherical projection formula. The multi-scale context feature fusion module comprises a plurality of convolution kernels with different sizes and different expansion rates, outputs feature maps output by the different convolution kernels, and generates a 64-dimensional intermediate feature map f of the distance image by splicing, dimension reduction and point-by-point convolution on the feature maps. M ; An encoder-decoder module comprises an encoder and a decoder, the encoder takes the feature map f M The input is input into an encoder module composed of a point-by-point convolution, a residual module and an average pooling method to obtain a distance image deep feature with smaller size and higher dimension. The decoder decodes the shallow edge features and the deep features connected through the skip connection structure, outputs decoding features of the distance image, and obtains a 20-dimensional output result; the encoder-decoder module also calculates probabilities of each pixel point in the output result for each semantic label by using a Softmax function, and takes a semantic label with a maximum probability as a semantic segmentation result of the distance image at the pixel point. The point cloud inverse mapping module is used for inversely mapping the semantic segmentation result back to a three-dimensional space, obtaining semantic labels of laser point clouds, and completing the semantic segmentation of the laser point clouds.

Images