Real-time laser radar semantic segmentation method based on multi-scale asymmetric convolution

Abstract

The invention provides a real-time laser radar semantic segmentation method based on multi-scale asymmetric convolution, and the method comprises the steps: obtaining Range image data, and also comprises the following steps: extracting local detail features; extracting global features; extracting multi-scale features; performing auxiliary segmentation; and training the neural network model through a multi-function supervision strategy. According to the real-time laser radar semantic segmentation method based on the multi-scale asymmetric convolution, a laser radar point cloud semantic segmentation task is converted into a 2D Range image segmentation task through spherical projection, in the aspect of feature extraction, a backbone feature network commonly used in an existing method is abandoned, and the real-time laser radar semantic segmentation method based on the multi-scale asymmetric convolution is obtained. The method is replaced by a small number of convolutional layers and a meticulously-designed multi-scale feature extraction module with only 0.04 M parameter quantity.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a real-time laser radar semantic segmentation method based on multi-scale asymmetric convolution. Background Art

[0002] LiDAR sensors are widely used in the field of autonomous driving perception because they can provide rich three-dimensional information and are robust to changes in lighting and weather conditions. However, existing technologies often cannot guarantee high accuracy and real-time performance when processing LiDAR data, especially on mobile computing platforms with limited computing resources. Therefore, how to balance the speed and accuracy of the algorithm on a mobile computing platform is one of the key and difficult issues in the current field of LiDAR point cloud semantic segmentation.

[0003] At present, methods based on raw point clouds and sparse voxels usually have huge model parameters, slow inference speed and high model complexity, which to some extent limit their application on mobile computing platforms. Specifically, three-dimensional point clouds exist in the form of a series of three-dimensional points and lack structured topological relationships. This requires that methods based on raw point clouds must first construct neighbor relationships between points. However, the number of points in large-scale point cloud scenes is usually in the millions, and the neighbor relationship search process will consume a lot of computing resources. Sparse voxel-based methods can use perfect three-dimensional convolutional neural networks to complete semantic segmentation tasks. However, the computational complexity of three-dimensional convolution increases cubically with the increase of voxel resolution. Although some projection-based methods can achieve real-time operation on mobile computing platforms, this is achieved at the expense of segmentation accuracy.

[0004] The invention patent application with application publication number CN113392840A discloses a real-time semantic segmentation method based on multi-scale segmentation fusion, including: establishing and training an adaptive multi-scale segmentation fusion network model; the adaptive multi-scale segmentation fusion network includes a backbone network, a classification layer, an alignment module and a fusion module; the image to be processed is sent to the backbone network for feature extraction, and feature maps of multiple scales are output; the classification layer is used to perform pixel-level classification on the output feature maps of each scale to obtain segmentation maps of different scales; the alignment module is used to unify the segmentation maps of different scales to the same resolution to obtain segmentation maps of the same size; the segmentation maps of the same size are sent to the fusion module, and the segmentation context information of different levels is fused based on a specific target category to obtain a fused segmentation map; the fused segmentation map is refined once using a convolution operation to obtain the final segmentation result. The disadvantage of this method is that its generalization ability is limited for data with different distributions, and it cannot be directly applied to lidar data.

[0005] The invention patent application with application publication number CN112213721A discloses a real-time semantic segmentation method based on a multi-scale structure. First, the semantic information branch is subjected to high-dimensional feature extraction; then the context semantic branch and the spatial branch are established; finally, the semantic features and spatial features are input into the feature fusion module for feature fusion, and finally the corresponding prediction map is output to achieve the semantic segmentation task. The disadvantage of this method is that it is applicable to 2D image data and has limited generalization ability on 3D lidar data. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a real-time lidar semantic segmentation method based on multi-scale asymmetric convolution. The lidar point cloud semantic segmentation task is converted into a 2D Range image segmentation task through spherical projection. In terms of feature extraction, the backbone feature network commonly used in existing methods is abandoned and replaced by a multi-scale feature extraction module with a small number of convolutional layers and a carefully designed parameter amount of only 0.04M.

[0007] The present invention provides a real-time laser radar semantic segmentation method based on multi-scale asymmetric convolution, comprising acquiring Range image data, and further comprising the following steps:

[0008] Step 1: Extract local detail features;

[0009] Step 2: Extract global features;

[0010] Step 3: Extract multi-scale features;

[0011] Step 4: Perform auxiliary segmentation;

[0012] Step 5: Train the neural network model through a multi-function supervision strategy.

[0013] Preferably, the obtaining of Range image data comprises projecting each unstructured LiDAR point cloud p 3D =(x,y,z),p∈P is projected from the Cartesian coordinate system to the two-dimensional image coordinate system to generate a Range image Among them, R is a set of real numbers, P is a set of lidar point clouds, and W ri and H ri is the width and height of the Range image, and C1 is the number of channels of the low-level features associated with each projection point.

[0014] In any of the above schemes, it is preferred that the spherical projection process is formally described as

[0015]

[0016] Where (u, v) is the pixel coordinate of the Range image, f = f u+f d is the vertical field of view of the laser radar, f u is the upper limit of the vertical field of view, f d is the lower bound of the vertical field of view,

[0017] Get a size of (H ri ,W ri ,5) Range image, each pixel contains 5 channels (x, y, z, d, r), where is the depth information of each point cloud, and r is the intensity information of the point cloud.

[0018] In any of the above solutions, preferably, the step 1 includes extracting the local detail information by stacking a small number of convolutional layers and a multi-scale feature extraction module.

[0019] In any of the above schemes, preferably, step 1 also includes stacking 3 ordinary convolution blocks with channel numbers of 32, 48 and 64 respectively and 1 multi-scale feature extraction module as a local detail feature encoder, and extracting multi-scale detail features by setting asymmetric convolutions of receptive fields of different sizes with expansion factors of [1, 1, 2, 3] respectively, and the step size of all convolution operations is 1.

[0020] In any of the above schemes, preferably, step 2 includes extracting global features through three bottleneck blocks and a multi-scale feature extraction module, each bottleneck block includes three inverted residual structures, the convolution step size of the first inverted residual structure in each bottleneck block is 2, and the remaining step sizes are 1, and the global features are extracted by setting asymmetric convolutions of receptive fields of different sizes with expansion factors of [1, 2, 3, 5] respectively.

[0021] In any of the above schemes, preferably, step 2 also includes upsampling the feature maps of different sizes and resolutions obtained by the three bottleneck blocks through bilinear interpolation to obtain feature maps with the same resolution as the Range image but different levels of semantic information, fusing feature maps of different semantic levels of information through feature splicing, and splicing the local detail feature map and the global feature map containing different levels to perform feature fusion.

[0022] In any of the above solutions, preferably, step 3 includes performing multi-scale feature extraction through a 1×1 convolution branch and four asymmetric convolution branches with different receptive fields, including the following sub-steps:

[0023] Step 31: Use a 1×1 convolution branch to extract detail information and obtain a detail feature map;

[0024] Step 32: Each asymmetric convolution branch with a different expansion factor is used to extract semantic feature information at different levels, and the feature maps of the four asymmetric convolution branches are fused to obtain a multi-level semantic feature map;

[0025] Step 33: Concatenate the obtained detail feature map and multi-level semantic feature map to enhance the feature information.

[0026] In any of the above schemes, preferably, step 4 includes the following sub-steps:

[0027] Step 41: Upsample the three feature maps of different sizes to the same size as the input Range image through bilinear interpolation sampling;

[0028] Step 42: Send the three feature maps including different levels of semantic information to auxiliary segmentation heads 1, 2, and 3 respectively to obtain semantic segmentation. Figure 1 , 2 ,3, calculate the semantic segmentation loss through the weighted cross entropy loss function;

[0029] Step 43: Semantic Segmentation Figure 1 , 2, 3 are sent to detail heads 1, 2, 3 respectively, and the boundary detail loss is calculated through the boundary loss function.

[0030] Preferably, in any of the above solutions, the generation process of the boundary detail feature map of the semantic label and the semantic prediction map is as follows:

[0031]

[0032] Among them, y gt and pd They are semantic labels and semantic prediction graphs, and is the boundary detail feature map corresponding to the semantic label and the semantic prediction map, pool() is the maximum pooling operation, and θ0 is the size of the sliding window.

[0033] In any of the above solutions, it is preferred that a weighted cross entropy function is used to alleviate category imbalance. The weighted cross entropy function is formally described as follows:

[0034]

[0035] Among them, y c is a semantic label, is the prediction graph, F c is the frequency of each category, α c is the weight of the cth category, C is the total number of categories, and the value of ∈ is 0.001.

[0036] In any of the above schemes, it is preferred to use the Lov′asz-Softmax loss function L ls To optimize the intersection-union ratio, its formal description is as follows:

[0037]

[0038] in represents the Lovasz expansion of the Jaccard index, m(c) represents the error vector, and m i (c) represents the error of pixel i in category c, is the segmentation map predicted by the network model and the semantic label of the i-th pixel of category c.

[0039] In any of the above schemes, it is preferred that L is introduced bd As the boundary loss function of auxiliary segmentation, it is formally described as follows:

[0040]

[0041] Among them, p c and R c Represent the predicted boundary graph y pd Relative to the semantic label y of the cth class gt The precision and recall of and y represent the predicted segmentation map and semantic label for each category.

[0042] In any of the above schemes, preferably, the total loss is a combination of three loss functions:

[0043] L=λ1L wce +λ2L ls +λ3L bd

[0044] Among them, L wce is the weighted cross entropy loss, L ls is the Lov′asz-Softmax loss function, L bd is the auxiliary boundary loss, and λ1, λ2, and λ3 are the corresponding weights.

[0045] The present invention proposes a real-time LiDAR semantic segmentation method based on multi-scale asymmetric convolution, which can significantly improve the processing speed while maintaining high segmentation accuracy to meet the application requirements on mobile computing platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The present invention is a flowchart of a preferred embodiment of a real-time lidar semantic segmentation method based on multi-scale asymmetric convolution according to the present invention.

[0047] Figure 2The present invention is a flowchart of another preferred embodiment of the real-time lidar semantic segmentation method based on multi-scale asymmetric convolution according to the present invention.

[0048] Figure 3 It is a schematic diagram of an embodiment of a local detail feature extraction process of a real-time lidar semantic segmentation method based on multi-scale asymmetric convolution according to the present invention.

[0049] Figure 4 It is a schematic diagram of an embodiment of the global feature extraction process of the real-time lidar semantic segmentation method based on multi-scale asymmetric convolution according to the present invention.

[0050] Figure 5 It is a schematic diagram of an embodiment of a multi-scale feature extraction process of a real-time lidar semantic segmentation method based on multi-scale asymmetric convolution according to the present invention.

[0051] Figure 6 It is a schematic diagram of an embodiment of the auxiliary segmentation process of the real-time lidar semantic segmentation method based on multi-scale asymmetric convolution according to the present invention. DETAILED DESCRIPTION

[0052] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0053] Embodiment 1

[0054] like Figure 1 As shown, step 100 is executed to obtain Range image data, including projecting each unstructured LiDAR point cloud p onto a spherical surface. 3D =(x,y,z),p∈P is projected from the Cartesian coordinate system to the two-dimensional image coordinate system to generate a Range image Among them, R is a set of real numbers, P is a set of lidar point clouds, and C1 is the number of channels of low-level features associated with each projection point.

[0055] The formal description of the spherical projection process is

[0056]

[0057] Among them, (u, v) is the pixel coordinate of the Range image, W ri and H ri is the width and height of the Range image, f=f u +f d is the vertical field of view of the laser radar, f u is the upper limit of the vertical field of view, f d is the lower bound of the vertical field of view.

[0058] Get a size of (H ri ,Wri ,5) Range image, each pixel contains 5 channels (x, y, z, d, r), where is the depth information of each point cloud, and r is the intensity information of the point cloud.

[0059] Execute step 110 to extract local detail features by stacking a small number of convolutional layers and multi-scale feature extraction modules to extract the local detail information.

[0060] Three ordinary convolution blocks with 32, 48 and 64 channels respectively and one multi-scale feature extraction module are stacked as the local detail feature encoder. Multi-scale detail features are extracted by setting asymmetric convolutions with different receptive fields with expansion factors of [1, 1, 2, 3], and the step size of all convolution operations is 1.

[0061] Execute step 120 to extract global features. The global features are extracted by three bottleneck blocks and a multi-scale feature extraction module. Each bottleneck block includes three inverted residual structures. The convolution step size of the first inverted residual structure in each bottleneck block is 2, and the remaining step sizes are 1. The global features are extracted by setting asymmetric convolutions of receptive fields of different sizes with expansion factors of [1, 2, 3, 5] respectively.

[0062] The feature maps of different sizes and resolutions obtained by the three bottleneck blocks are upsampled by bilinear interpolation to obtain feature maps with the same resolution as the Range image but different levels of semantic information. Feature maps with different semantic levels of information are fused through feature splicing, and local detail feature maps and global feature maps containing different levels are spliced ​​to perform feature fusion.

[0063] Execute step 130 to extract multi-scale features. The multi-scale feature extraction is performed through a 1×1 convolution branch and four asymmetric convolution branches with different receptive fields, including the following sub-steps:

[0064] Execute step 131, use the 1×1 convolution branch to extract detail information, and obtain a detail feature map;

[0065] Execute step 132, each asymmetric convolution branch with a different expansion factor is used to extract semantic feature information of different levels, and the feature maps of the four asymmetric convolution branches are feature fused to obtain a multi-level semantic feature map;

[0066] Execute step 13 to concatenate the obtained detail feature map and the multi-level semantic feature map to enhance the feature information.

[0067] Execute step 140 to perform auxiliary segmentation, including the following sub-steps:

[0068] Execute step 141 to upsample the three feature maps of different sizes to the same size as the input Range image through bilinear interpolation sampling;

[0069] Execute step 142 to send the three feature maps including semantic information of different levels to auxiliary segmentation heads 1, 2, and 3 respectively to obtain semantic segmentation. Figure 1 , 2 ,3, calculate the semantic segmentation loss through the weighted cross entropy loss function;

[0070] Execute step 143, semantic segmentation Figure 1 , 2, 3 are sent to detail heads 1, 2, 3 respectively, and the boundary detail loss is calculated through the boundary loss function.

[0071] The generation process of the boundary detail feature map of the semantic label and semantic prediction map is as follows:

[0072]

[0073] Among them, y gt and pd They are semantic labels and semantic prediction graphs, and is the boundary detail feature map corresponding to the semantic label and the semantic prediction map, pool() is the maximum pooling operation, and θ0 is the size of the sliding window.

[0074] Execute step 150 to train the neural network model using a multi-function supervision strategy.

[0075] The weighted cross entropy function is used to alleviate category imbalance. The formal description of the weighted cross entropy function is as follows:

[0076]

[0077] where y c is a semantic label, is the prediction graph, F c is the frequency of each category, α c is the weight of the cth category, C is the total number of categories, and the value of ∈ is 0.001.

[0078] Use the Lov′asz-Softmax loss function L ls To optimize the intersection-union ratio, its formal description is as follows:

[0079]

[0080] in represents the Lovasz expansion of the Jaccard index, m(c) represents the error vector, and m i (c) represents the error of pixel i in category c, is the segmentation map predicted by the network model and the semantic label of the i-th pixel of category c.

[0081] Introducing L bd As the boundary loss function of auxiliary segmentation, it is formally described as follows:

[0082]

[0083] Among them, p c and R c Represent the predicted boundary graph y pd Relative to the semantic label y of the cth class gt The precision and recall of and y represent the predicted segmentation map and semantic label for each category.

[0084] The total loss is a combination of three loss functions:

[0085] L=λ1L wce +λ2L ls +λ3L bd

[0086] Among them, L wce is the weighted cross entropy loss, L ls is the Lov′asz-Softmax loss function, L bd is the auxiliary boundary loss, and λ1, λ2, and λ3 are the corresponding weights.

[0087] The present invention aims to propose a new LiDAR semantic segmentation architecture that can significantly improve the processing speed while maintaining high segmentation accuracy to meet the application requirements on mobile computing platforms.

[0088] Embodiment 2

[0089] The patent application of this invention discloses a real-time LiDAR point cloud semantic segmentation method based on multi-scale asymmetric convolution, which can perform semantic segmentation of LiDAR data in real time on a mobile computing platform while maintaining competitive segmentation accuracy and without the need for a large amount of computing resources. The LiDAR point cloud semantic segmentation task is converted into a 2D Range image segmentation task through spherical projection. In terms of feature extraction, the backbone feature network commonly used in existing methods is abandoned and replaced by a small number of convolutional layers and a carefully designed multi-scale feature extraction module with only 0.04M parameters, which achieves a significant reduction in model parameters and computational complexity while maintaining highly competitive segmentation accuracy, and has a better trade-off between accuracy and speed than existing algorithms.

[0090] like Figure 2 As shown, a real-time lidar point cloud semantic segmentation method based on multi-scale asymmetric convolution includes the following steps:

[0091] 1. Range image data acquisition

[0092] Each unstructured LiDAR point cloud p is transformed into 3D =(x,y,z),p∈P is projected from the Cartesian coordinate system to the two-dimensional image coordinate system to generate a Range image The 3D point cloud segmentation problem is then transformed into a 2D image segmentation task, and then efficient 2D convolution operations are used for fast training and reasoning. The formal description of the projection process is as follows:

[0093]

[0094] Where (u, v) is the pixel coordinate of the Range image, W ri and H ri is the width and height of the Range image, f=f u +f d is the vertical field of view of the laser radar. The final result is (H ri ,W ri ,5) Range image, each pixel contains 5 channels (x, y, z, d, r), where is the depth information of each point cloud, and r is the intensity information of the point cloud.

[0095] 2. Local detail feature extraction

[0096] Local detail information is extracted by stacking a small number of convolutional layers and multi-scale feature extraction modules. Specifically, three ordinary convolutional blocks with 32, 48 and 64 channels and one multi-scale feature extraction module are stacked as local detail feature encoders. In order to obtain rich detail semantic information, for the multi-scale feature extraction module, asymmetric convolutions with different receptive fields and expansion factors of [1, 1, 2, 3] are set to extract multi-scale detail features. The step size of all the above convolution operations is 1.

[0097] 3. Global feature extraction

[0098] Global features are extracted through three bottleneck blocks and a multi-scale feature extraction module, each of which includes three inverted residual structures. Specifically, the convolution step size of the first inverted residual structure in each bottleneck block is 2, and the rest of the steps are 1. For the multi-scale feature extraction module, global features are extracted by setting asymmetric convolutions with different receptive fields with expansion factors of [1, 2, 3, 5] respectively. In order to alleviate the feature loss in the downsampling process, the feature maps of different resolutions obtained by the three bottleneck blocks are upsampled by bilinear interpolation to obtain feature maps with the same resolution as the Range image but different levels of semantic information, and then feature concatenation is used to fuse feature maps with different semantic levels of information. Finally, the local detail feature map and the global feature map containing different levels are concatenated to perform feature fusion. The local detail feature map contains rich detail information, and the global feature map contains contextual semantic information at different levels, thereby achieving feature enhancement.

[0099] 4. Multi-scale feature extraction

[0100] Multi-scale feature extraction is performed through a 1×1 convolution branch and four asymmetric convolution branches with different receptive fields. The 1×1 convolution branch is used to extract detail information and obtain a detail feature map. Each asymmetric convolution branch with different expansion factors is used to extract semantic feature information at different levels, and then the feature maps of the four asymmetric convolution branches are feature fused to obtain a multi-level semantic feature map. Finally, the obtained detail feature map and multi-level semantic feature map are spliced ​​to strengthen the feature information. Specifically, each asymmetric convolution branch includes three asymmetric convolution layers, and each asymmetric convolution layer includes an n×1 and 1×n convolution. This method decomposes the 3×3 convolution into a 3×1 and 1×3 convolution. Compared with 3×3 convolution, using asymmetric convolution can save about 33% of parameters and calculations under the same number of convolution kernels. For each asymmetric convolution branch, the feature maps output by the three asymmetric convolution layers are spliced ​​to further strengthen the feature information and serve as the final output feature map of each branch. In addition, the multi-scale feature extraction process can flexibly set different sizes of receptive fields for each asymmetric convolution branch by setting different expansion factors.

[0101] 5. Assisted segmentation

[0102] First, three feature maps of different sizes are upsampled to the same size as the input Range image through bilinear interpolation sampling. Then, the three feature maps containing semantic information at different levels are sent to auxiliary segmentation heads 1, 2, and 3 respectively to obtain semantic segmentation. Figure 1 , 2, 3, the semantic segmentation loss is calculated through the weighted cross entropy loss function. On this basis, the semantic segmentation Figure 1, 2, 3 are sent to detail heads 1, 2, 3 respectively, and the boundary detail loss is calculated by the boundary loss function. Auxiliary segmentation is only used in the training phase of the algorithm and is discarded in the inference phase. Therefore, auxiliary segmentation does not require additional computing resources while effectively improving segmentation accuracy. The generation process of boundary detail feature maps of semantic labels and semantic prediction maps is as follows:

[0103]

[0104] where y gt and pd They are semantic labels and semantic prediction graphs, and is the boundary detail feature map corresponding to the semantic label and the semantic prediction map, pool() is the maximum pooling operation, θ0 is the size of the sliding window, and the value of θ0 is set to 3 here.

[0105] 6. Training neural network models through multi-function supervision strategies

[0106] The weighted cross entropy function is used to alleviate the problem of category imbalance. The formal description of the weighted cross entropy function is as follows:

[0107]

[0108] where y c is a semantic label, is the prediction graph, F c is the frequency of each category, α c is the weight of the cth class. C is the total number of classes and the value of ∈ is 0.001.

[0109] Use the Lov′asz-Softmax loss function L ls To optimize the intersection-union ratio, its formal description is as follows:

[0110]

[0111]

[0112] in represents the Lovasz expansion of the Jaccard index, m(c) represents the error vector, and m i (c) represents the error of pixel i in category c, is the segmentation map predicted by the network model and the semantic label of the i-th pixel of category c.

[0113] In order to alleviate the boundary fuzziness problem, L bd As the boundary loss function of auxiliary segmentation, it is formally described as follows:

[0114]

[0115] Among them, p c and R c Represent the predicted boundary graph y pd Relative to the semantic label y of the cth class gt precision and recall. and y represent the predicted segmentation map and semantic label for each category.

[0116] The final total loss is the combination of these three loss functions:

[0117] L=λ1L wce +λ2L ls +λ3L bd

[0118] Where L wce is the weighted cross entropy loss, L ls is the Lov′asz-Softmax loss function, L bd is the auxiliary boundary loss. λ1, λ2, and λ3 are the corresponding weights, which are set to 1.0, 1.5, and 1.0 respectively.

[0119] The present invention uses a large number of lightweight convolutions to replace ordinary convolutions, such as asymmetric convolutions and depth-wise separable convolutions. Therefore, the model parameter volume of the present method is only 1.54M, and the method has high computational efficiency and fast inference speed.

[0120] The present invention constructs a multi-scale feature extraction module through a clever combination of asymmetric and dilated convolutions, which can flexibly set different receptive fields and thus efficiently capture semantic features of different scales, making this method particularly suitable for the segmentation of sparse point cloud scenes and small target dynamic instances.

[0121] Embodiment 3

[0122] The schematic diagram of the local detail feature extraction process is as follows: Figure 3 As shown in the figure. Three ordinary convolution blocks with 32, 48 and 64 channels and a multi-scale feature extraction module are stacked as the local detail feature information encoder. For the multi-scale feature extraction module, multi-scale detail features are extracted by setting asymmetric convolutions with different receptive fields with expansion factors of [1, 1, 2, 3]. The step size of the above convolution operations is 1.

[0123] Embodiment 4

[0124] The schematic diagram of the global feature extraction process is as follows: Figure 4As shown. The convolution step size of the first inverted residual structure in each bottleneck block is 2, and the rest of the step sizes are 1. For the multi-scale feature extraction module, the global feature information is extracted by setting asymmetric convolutions with different sizes of receptive fields with expansion factors of [1, 2, 3, 5]. The feature maps of different sizes and resolutions obtained by the three bottleneck blocks are upsampled by bilinear interpolation to obtain feature maps with the same resolution as the Range image but different levels of semantic information. Feature maps with different semantic levels of information are then fused by feature splicing. Finally, the detail feature map output by the local detail information module and the context feature map containing different levels output by the global feature information module are spliced ​​to perform feature fusion and achieve feature information enhancement.

[0125] Embodiment 5

[0126] The schematic diagram of the multi-scale feature extraction process is as follows: Figure 5 As shown. Each asymmetric convolution branch includes 3 asymmetric convolution layers, and each asymmetric convolution layer includes an n×1 and 1×n convolution. This method decomposes the 3×3 convolution into a 3×1 and 1×3 convolution. Compared with the ordinary 3×3 convolution, the use of asymmetric convolution can save about 33% of the parameters and computational complexity with the same number of convolution kernels. For each asymmetric convolution branch, the feature maps output by the three asymmetric convolution layers are concatenated to enhance the feature information and serve as the final output feature map of each branch. In addition, the multi-scale feature extraction module can flexibly set different sizes of receptive fields for each asymmetric convolution branch by setting different expansion factors. The step size of all the above convolution operations is 1.

[0127] Embodiment 6

[0128] The schematic diagram of the auxiliary segmentation process is as follows: Figure 6 As shown. First, three feature maps of different sizes are upsampled to the same size as the input Range image through bilinear interpolation sampling, and then the three feature maps including semantic information at different levels are sent to segmentation heads 1, 2, and 3 to obtain semantic segmentation maps. The loss is calculated through the weighted cross entropy loss function, so as to better learn semantic features at different levels. On this basis, the three semantic segmentation maps are sent to detail heads 1, 2, and 3, and the boundary detail loss is calculated through the boundary loss function, so as to better learn the boundary detail information of the segmented target.

[0129] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A real-time lidar semantic segmentation method based on multi-scale asymmetric convolution, comprising obtaining Range image data, characterized in that: The following steps are also included: Step 1: Extract local detail features; Step 2: Extract global features; Step 3: Extract multi-scale features; Step 4: Perform auxiliary segmentation; Step 5: Train the neural network model through a multi-function supervision strategy.

2. The real-time lidar semantic segmentation method based on multi-scale asymmetric convolution as claimed in claim 1, characterized in that: The acquisition of Range image data includes projecting each unstructured LiDAR point cloud p 3D =(x,y,z),p∈P is projected from the Cartesian coordinate system to the two-dimensional image coordinate system to generate a Range image Among them, R is a set of real numbers, P is a set of lidar point clouds, and W ri and H ri is the width and height of the Range image, and C1 is the number of channels of the low-level features associated with each projection point.

3. The real-time lidar semantic segmentation method based on multi-scale asymmetric convolution as claimed in claim 2, characterized in that: The formal description of the spherical projection process is Where (u, v) is the pixel coordinate of the Range image, f = f u +f d is the vertical field of view of the laser radar, f u is the upper limit of the vertical field of view, f d is the lower bound of the vertical field of view, Get a size of (H ri ,W ri ,5) Range image, each pixel contains 5 channels (x, y, z, d, r), where is the depth information of each point cloud, and r is the intensity information of the point cloud.

4. The real-time lidar semantic segmentation method based on multi-scale asymmetric convolution as claimed in claim 3, characterized in that: The step 1 includes extracting the local detail information by stacking a small number of convolutional layers and a multi-scale feature extraction module.

5. The real-time lidar semantic segmentation method based on multi-scale asymmetric convolution as claimed in claim 4, characterized in that: The step 1 also includes stacking three common convolution blocks with channel numbers of 32, 48 and 64 respectively and a multi-scale feature extraction module as a local detail feature encoder, extracting multi-scale detail features by setting asymmetric convolutions of receptive fields of different sizes with expansion factors of [1, 1, 2, 3] respectively, and the step size of all convolution operations is 1.

6. The real-time lidar semantic segmentation method based on multi-scale asymmetric convolution according to claim 5 is characterized in that: The step 2 includes extracting global features through three bottleneck blocks and a multi-scale feature extraction module, each bottleneck block includes three inverted residual structures, the convolution step size of the first inverted residual structure in each bottleneck block is 2, and the remaining step sizes are 1, and the global features are extracted by setting asymmetric convolutions of receptive fields of different sizes with expansion factors of [1, 2, 3, 5] respectively.

7. The real-time lidar semantic segmentation method based on multi-scale asymmetric convolution as claimed in claim 6, characterized in that: The step 2 also includes upsampling the feature maps of different sizes and resolutions obtained by the three bottleneck blocks through bilinear interpolation to obtain feature maps with the same resolution as the Range image but different levels of semantic information, fusing the feature maps of different semantic levels of information through feature splicing, and fusing the local detail feature map and the global feature map containing different levels by splicing.

8. The real-time LiDAR semantic segmentation method based on multi-scale asymmetric convolution as claimed in claim 7, characterized in that: The step 3 includes performing multi-scale feature extraction through a 1×1 convolution branch and four asymmetric convolution branches with different receptive fields, including the following sub-steps: Step 31: Use a 1×1 convolution branch to extract detail information and obtain a detail feature map; Step 32: Each asymmetric convolution branch with a different expansion factor is used to extract semantic feature information at different levels, and the feature maps of the four asymmetric convolution branches are fused to obtain a multi-level semantic feature map; Step 33: Concatenate the obtained detail feature map and multi-level semantic feature map to enhance the feature information.

9. The real-time lidar semantic segmentation method based on multi-scale asymmetric convolution as claimed in claim 8, characterized in that: The step 4 includes the following sub-steps: Step 41: Upsample the three feature maps of different sizes to the same size as the input Range image through bilinear interpolation sampling; Step 42: Send three feature maps including semantic information of different levels to auxiliary segmentation heads 1, 2, and 3 respectively to obtain semantic segmentation maps 1, 2, and 3, and calculate the semantic segmentation loss by weighted cross entropy loss function; Step 43: The semantic segmentation images 1, 2, and 3 are respectively sent to detail heads 1, 2, and 3, and the boundary detail loss is calculated by the boundary loss function.

10. The real-time lidar semantic segmentation method based on multi-scale asymmetric convolution as claimed in claim 9, characterized in that: The generation process of the boundary detail feature map of the semantic label and semantic prediction map is as follows: Among them, y gt and pd They are semantic labels and semantic prediction graphs, and is the boundary detail feature map corresponding to the semantic label and the semantic prediction map, pool() is the maximum pooling operation, and θ0 is the size of the sliding window.

Images