Semantic segmentation method of point cloud in unstructured environment based on cross-modal semantic enhancement

Abstract

The present invention proposes a semantic segmentation method for point clouds in unstructured environments based on cross-modal semantic enhancement. This method aims to address the problem that most current point cloud segmentation algorithms lack semantic information such as image texture and color and are difficult to simultaneously meet the requirements of accuracy and real-time performance in unstructured environments. An unstructured environment semantic segmentation network with deep fusion of images and lidar is constructed. The steps are as follows: first, a point cloud segmentation module based on spherical projection is designed. Secondly, an image segmentation module based on residual cross-layer connection is designed. Then, a two-dimensional pseudo-semantic enhancement module based on GAN is designed to compensate for the missing semantic information such as color and texture of the point cloud. Finally, the network is trained using a sample set to obtain network parameters, thereby achieving efficient and reliable segmentation of the semantics of three-dimensional point clouds in unstructured environments.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent vehicle environment perception, and specifically to an unstructured environment point cloud semantic segmentation method based on cross-modal semantic enhancement. Background Art

[0002] Three-dimensional scene understanding is a key technology in the field of ground unmanned systems and a prerequisite for safe and reliable passage in structured and unstructured environments. Currently, the more mature technologies are mainly designed for urban structured environments. There are fewer studies on unstructured environments (such as emergency rescue scenarios), and the technology is not yet mature. In unstructured environments, there are no structural features such as lanes, pavement, and guardrails. The boundaries of the drivable area are fuzzy and the textures are diverse. At the same time, affected by terrain, shrubs and other vegetation, the characteristics of obstacles are complex and changeable, and there are occlusions and missing situations. Therefore, the existing algorithms designed for structured environments are difficult to directly apply to unstructured environments.

[0003] At present, deep learning-based semantic segmentation tasks mostly use cameras and lidar as their main sensor data input sources. The color or texture features used in camera-based semantic segmentation methods for unstructured environments are easily disturbed by lighting and weather, and the lack of three-dimensional structural information also limits the adaptability of such algorithms in different scenarios. LiDAR-based semantic segmentation algorithms are difficult to distinguish between low-resolution objects or objects with similar geometric features due to the sparsity, disorder, and uneven distribution of point cloud data, as well as the fact that the depth and geometric information they rely on are too single. However, methods based on image and radar fusion (input-level, feature-level, and decision-level fusion) are highly dependent on the quality of each input signal or existing prior knowledge, and cannot be applied to complex and changeable unstructured scenes. Therefore, it is necessary to propose a robust method to fuse data from different sensor modalities to achieve high-precision and effective three-dimensional semantic segmentation in unstructured environments. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a semantic segmentation method of point cloud in unstructured environment based on cross-modal semantic enhancement. The method first designs a point cloud segmentation module based on spherical projection, then designs an image segmentation module based on residual cross-layer connection, and then designs a two-dimensional pseudo semantic enhancement module based on GAN. Finally, the network is trained using a sample set to obtain network parameters, thereby achieving efficient and reliable semantic segmentation of three-dimensional point cloud in unstructured environment. The method proposed in the present invention uses spherical projection, which can generate more compact and dense data compared to other projection methods (such as bird's-eye view projection) while reducing indexing and calculation time, which is beneficial to the extraction of point cloud features by deep neural networks; the use of residual structure and dilated convolution not only reduces the complexity of the network structure, but also can mine more effective information from the fusion features of different depths of the network, thereby ensuring that the network takes into account both the accuracy and real-time performance of point cloud segmentation; the use of the GAN style transfer algorithm allows the feature information between the image and the point cloud to learn from each other, converts the semantic features of the point cloud segmentation network into image semantic features and uses them as semantic enhancement information for the point cloud segmentation network, which can make up for the missing key semantic information such as color texture, and realize the deep fusion of point cloud and image features; in the testing phase, only lidar sensors are used for semantic segmentation of the scene, which is not affected by light and weather while taking into account the semantic feature information contained in the image, thereby improving the robustness of the segmentation algorithm. The overall design scheme and network schematic diagram are shown as follows: Figure 1 , Figure 2 As shown, it includes the following steps:

[0005] (1) Design of point cloud segmentation module based on spherical projection;

[0006] Sub-step 1: Radar point cloud preprocessing

[0007] The laser radar data point coordinate system OXYZ is established according to the right-hand rule with the radar center as the origin and the carrier forward direction as the OX axis; the azimuth The calculation formula of the vertex angle θ is as follows:

[0008]

[0009] Among them, (x, y, z) is the coordinate of each point in the radar point cloud in the Euclidean coordinate system; for each point in the point cloud, its (x, y, z) can be calculated. That is, project the points in the Euclidean coordinate system into the spherical coordinate system; this spherical coordinate system is actually a two-dimensional coordinate system, and its angles are differentiated to obtain a two-dimensional rectangular coordinate system:

[0010]

[0011] Among them, xi and xj are the horizontal and vertical coordinates of the two-dimensional rectangular coordinate system. Δθ corresponds to The resolution of θ;

[0012] Through this spherical projection transformation, any point (x, y, z) in the Euclidean space is projected to a point (xi, xj) in the two-dimensional coordinate system; three features of each point in the point cloud are extracted: Euclidean coordinates (x, y, z), thereby obtaining a tensor of size (H, W, C), where H is the width of the rendered image obtained after the spherical projection transformation, W is the length of the image, and C = 3; taking the KITTI dataset as an example, the dataset uses a 64-line laser radar, so H = 64; in the horizontal direction, due to the limitation of the dataset annotation range, the radar point cloud data at 90° in front is used and divided into 512 grids, that is, 512 points are sampled horizontally, so W = 512;

[0013] Sub-step 2: Design of encoder-decoder network based on residual dilated convolution

[0014] For the semantic segmentation task in unstructured environments, an encoder-decoder network based on residual dilated convolution is designed;

[0015] First, determine the convolution layer type, convolution kernel size, and convolution step size;

[0016] The convolution kernel size of the dilated convolution layer is designed to be 1×1 and 3×3, and the step size is set to 1; the convolution kernel size of the deconvolution layer is 3×3, and the step size is 2, so that the feature map after upsampling has the same resolution as the input;

[0017] Secondly, determine the pooling layer type, sampling size, and step size;

[0018] The maximum pooling operation is used to downsample the feature map, and the sampling size is set to 2×2 and the step size is set to 2;

[0019] Next, determine the dropout layer distribution;

[0020] Insert dropout only in the central layers of the encoder and decoder;

[0021] Finally, the different types of network layers involved above are combined, and the model selection is performed using the cross-validation method to determine the number of layers, the number of convolution kernels, and the dilation rate of the dilated convolution. The following optimal network architecture is obtained, in which each convolution operation is activated by the ReLU function:

[0022] Block 1: Use 32 1×1 convolution kernels with a dilation rate of 1 to convolve with the 512×64×3 input sample to obtain sub-features Figure 1, and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform batch normalization. After the feature maps are concatenated, they are convoluted with 1×1 convolution kernels with a dilation rate of 1 and perform BN. Figure 1 Add pixel by pixel to get a feature map with a dimension of 512×64×32;

[0023] Pooling layer 1: The feature map output by Block 2 is downsampled using a 2×2 maximum pooling layer with a step size of 2, resulting in a feature map of dimension 256×32×32;

[0024] Block2: Referring to Block1, convolution is performed with 64 1×1 convolution kernels with a dilation rate of 1 and the feature map output by pooling layer 1 to obtain sub-features. Figure 2 , and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. After the feature maps are concatenated, they are convoluted with 1×1 convolution kernels with a dilation rate of 1 and perform BN. Figure 2 Add pixel by pixel to get a feature map with a dimension of 256×32×64;

[0025] Pooling layer 2: After the feature map output by Block 2 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a feature map with a dimension of 128×16×64;

[0026] Block3: Referring to Block1, convolution is performed with 128 1×1 convolution kernels with a dilation rate of 1 and the feature map output by pooling layer 2 to obtain sub-features. Figure 3 , and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. After the feature maps are concatenated, they are convoluted with 1×1 convolution kernels with a dilation rate of 1 and perform BN. Figure 3 Add pixel by pixel to get a feature map with a dimension of 128×16×128;

[0027] Pooling layer 3: After the feature map output by Block 3 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a feature map with a dimension of 64×8×128;

[0028] Block4: Referring to Block1, convolve the feature map output by pooling layer 3 with 256 1×1 convolution kernels with a dilation rate of 1 to obtain sub-features. Figure 4 , and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. After the feature maps are concatenated, they are convoluted with 1×1 convolution kernels with a dilation rate of 1 and perform BN. Figure 4 Add pixel by pixel to get a feature map with a dimension of 64×8×256;

[0029] Deconvolution layer 1: Use 128 3×3 convolution kernels to convolve with the feature map output by Block4 and perform BN with a step size of 2. After dropout, add it pixel by pixel with the feature map output by Block3. After dropout, convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. After concatenating the feature maps, convolve with 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, after dropout, a feature map with a dimension of 128×16×128 is obtained.

[0030] Deconvolution layer 2: Referring to deconvolution layer 1, 64 3×3 convolution kernels are used to convolve with the feature map output by Block4 and perform BN with a step size of 2. After dropout, the feature map is added pixel by pixel with the feature map output by Block3. After dropout, the feature maps are convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, after dropout, a feature map with a dimension of 256×32×64 is obtained.

[0031] Deconvolution layer 3: Referring to deconvolution layer 1, 32 3×3 convolution kernels are used to convolve with the feature map output by Block4 and perform BN, with a step size of 2. The feature map output by Block3 is added pixel by pixel, and then convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, a feature map with a dimension of 512×64×32 is obtained.

[0032] Standard convolution layer: convolve the feature map output by deconvolution layer 3 with three 1×1 convolution kernels to obtain a feature map with a dimension of 512×64×3. The three channels correspond to the total number of semantic classes.

[0033] Soft-max layer: performs pixel-level classification on the feature map output by the standard convolutional layer to obtain the probability of each category and realize the 3D semantic segmentation of the scene;

[0034] Sub-step 3: Loss function design

[0035] The traditional cross entropy loss is weighted by the square root of the class frequency and combined with Lovasz-Softmax as the final loss function to maximize the IoU score of each class, as shown in the following formula:

[0036]

[0037] Among them, v i It is the number of points of various types. and P3D (i) are the true value and the corresponding predicted probability, J represents the Lovasz expansion of IoU, e(c k ) is class c k The error vector of

[0038] (2) Design of image segmentation module based on residual cross-layer connection;

[0039] Sub-step 1: Encoder-Decoder Network Design

[0040] In order to facilitate the use of the GAN algorithm in the two-dimensional pseudo-semantic enhancement module in step (3), the same encoding and decoding structure as the point cloud branch is used to extract image semantic features and obtain the classification probabilities;

[0041] Sub-step 2: Loss function design

[0042] The standard cross entropy function is used as the loss function of the supervised training model:

[0043]

[0044] in and P 2D (j) represent the true value and the corresponding predicted value respectively;

[0045] (3) Design of two-dimensional pseudo-semantic enhancement module based on GAN;

[0046] CycleGAN is selected as the generator of the two-dimensional pseudo semantic enhancement module. It is a cross-domain image generation GAN model with cycle consistency, that is, there are two generators G:2D→3D, F:3D→2D, which are inverse matrices of each other, and two discriminators D 2D , D 3D , where D 2D Used to distinguish original two-dimensional semantic features That is, the output of the encoder in step (2) and the two-dimensional pseudo-semantic feature converted from the three-dimensional semantic feature That is, the output of the encoder in step (1), D 3D Used to distinguish original 3D semantic features The three-dimensional pseudo semantic features obtained by converting the two-dimensional semantic features Make Cyclic 2D =F(G(feature 2D ))≈feature 2D , Cyclic 3D =G(F(feature 3D ))≈feature 3D ;

[0047] The adversarial loss of each mapping is as follows:

[0048]

[0049]

[0050] In order to further reduce the mapping function space, the cycle consistency loss is introduced to make the learned mapping function cycle consistent:

[0051]

[0052] Combining it with the adversarial losses in the two-dimensional and three-dimensional domains, the total loss function of the two-dimensional pseudo-semantic enhancement module is as follows:

[0053] L(G, F, D 2D , D 3D )=L GAN (G,D 3D , 2D, 3D)

[0054] +L TAN (F,D 2D , 2D, 3D)

[0055] +αL cycle (G, F) (7)

[0056] Among them, α is a hyperparameter that controls the relative importance of the two objectives;

[0057] (4) Network model training and point cloud semantic segmentation

[0058] Sub-step 1: Point cloud branch training

[0059] The Adam optimizer is used to iteratively train the network parameters. The initial learning rate of the optimizer is set to 0.01, the decay rate is set to 0.1 / 20K iterations, the dropout rate is set to 0.5, the batch size is set to 32, and the training epoch is set to 500;

[0060] First, a structured environment data set is used for pre-training, wherein the structured environment data set includes KITTI, to obtain pre-training weights, and then a small amount of collected unstructured environment data is used for further training to obtain network weights; in order to increase the amount of training data, the present invention performs horizontal flipping, random noise addition, and random rotation around the z-axis by [-5°, 5°] on the original point cloud;

[0061] Sub-step 2: Image branch training

[0062] Similar to the point cloud branch, we first use a structured environment dataset for pre-training, including KITTI, to obtain pre-training weights, and then use a small amount of collected unstructured environment data for further training to obtain network weights.

[0063] Sub-step 3: CycleGAN training

[0064] Using the network weights obtained in substeps 1 and 2, generate paired outputs of the point cloud and image branch encoders It is used to supervise the training of CycleGAN and realize the mutual conversion of two-dimensional and three-dimensional semantic features;

[0065] Sub-step 4: Point Cloud Network Retraining

[0066] Freeze the parameters of the point cloud branch encoder, and use the dataset to retrain the point cloud branch decoder after adding the 2D pseudo semantic enhancement module to obtain the final network weights;

[0067] Sub-step 5: Point cloud semantic segmentation

[0068] Using the network weights obtained in sub-step 4, the collected radar point cloud is tested online to obtain the semantic segmentation result of the unstructured environment 3D point cloud.

[0069] Beneficial Effects

[0070] 1. The spherical projection method used in the present invention is more conducive to the extraction of point cloud features while reducing indexing and calculation time.

[0071] 2. The present invention uses dilated convolution and residual structure, which not only reduces the complexity of the network structure, but also can mine more effective information from the fusion features of different depths of the network, thereby ensuring that the network takes into account both accuracy and real-time performance.

[0072] 3. The present invention uses the GAN style transfer algorithm to enable the feature information between the image and the point cloud to learn from each other, thereby making up for the key semantic information such as color and texture missing in the point cloud data, and realizing the deep fusion of point cloud and image features.

[0073] 4. The present invention only uses lidar sensors during the testing phase, which is not affected by light and weather. It also takes into account the three-dimensional structural information of the point cloud and the semantic feature information contained in the image, thereby improving the algorithm's adaptability to different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a flow chart of the overall scheme;

[0075] Figure 2 This is a schematic diagram of a point cloud segmentation network based on cross-modal semantic enhancement;

[0076] Figure 3 It is a schematic diagram of spherical projection;

[0077] Figure 4 It is a Block diagram;

[0078] Figure 5 It is a schematic diagram of the deconvolution layer;

[0079] Figure 6 This is a schematic diagram of CycleGAN. DETAILED DESCRIPTION

[0080] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0081] Three-dimensional scene understanding is a key technology in the field of ground unmanned systems and a prerequisite for safe and reliable passage in structured and unstructured environments. Currently, the more mature technologies are mainly designed for urban structured environments. There are fewer studies on unstructured environments (such as emergency rescue scenarios), and the technology is not yet mature. In unstructured environments, there are no structural features such as lanes, pavement, and guardrails. The boundaries of the drivable area are fuzzy and the textures are diverse. At the same time, affected by terrain, shrubs and other vegetation, the characteristics of obstacles are complex and changeable, and there are occlusions and missing situations. Therefore, the existing algorithms designed for structured environments are difficult to directly apply to unstructured environments.

[0082] At present, deep learning-based semantic segmentation tasks mostly use cameras and lidar as their main sensor data input sources. The color or texture features used in camera-based semantic segmentation methods for unstructured environments are easily disturbed by lighting and weather, and the lack of three-dimensional structural information also limits the adaptability of such algorithms in different scenarios. LiDAR-based semantic segmentation algorithms are difficult to distinguish between low-resolution objects or objects with similar geometric features due to the sparsity, disorder, and uneven distribution of point cloud data, as well as the fact that the depth and geometric information they rely on are too single. However, methods based on image and radar fusion (input-level, feature-level, and decision-level fusion) are highly dependent on the quality of each input signal or existing prior knowledge, and cannot be applied to complex and changeable unstructured scenes. Therefore, it is necessary to propose a robust method to fuse data from different sensor modalities to achieve high-precision and effective three-dimensional semantic segmentation in unstructured environments.

[0083] In order to solve the above problems, the present invention provides a semantic segmentation method of point cloud in unstructured environment based on cross-modal semantic enhancement. The method first designs a point cloud segmentation module based on spherical projection, then designs an image segmentation module based on residual cross-layer connection, and then designs a two-dimensional pseudo semantic enhancement module based on GAN. Finally, the network is trained using a sample set to obtain network parameters, thereby achieving efficient and reliable semantic segmentation of three-dimensional point cloud in unstructured environment. The method proposed in the present invention uses spherical projection, which can generate more compact and dense data compared to other projection methods (such as bird's-eye view projection) while reducing indexing and calculation time, which is beneficial to the extraction of point cloud features by deep neural networks; the use of residual structure and dilated convolution not only reduces the complexity of the network structure, but also can mine more effective information from the fusion features of different depths of the network, thereby ensuring that the network takes into account both the accuracy and real-time performance of point cloud segmentation; the use of the GAN style transfer algorithm allows the feature information between the image and the point cloud to learn from each other, converts the semantic features of the point cloud segmentation network into image semantic features and uses them as semantic enhancement information for the point cloud segmentation network, which can make up for the missing key semantic information such as color texture, and realize the deep fusion of point cloud and image features; in the testing phase, only lidar sensors are used for semantic segmentation of the scene, which is not affected by light and weather while taking into account the semantic feature information contained in the image, thereby improving the robustness of the segmentation algorithm. The overall design scheme and network schematic diagram are shown as follows: Figure 1 , Figure 2 As shown, the specific steps include:

[0084] (1) Design of point cloud segmentation module based on spherical projection

[0085] Sub-step 1: Radar point cloud preprocessing

[0086] The amount of laser radar point cloud data is huge, and due to the uneven sampling of the environment, the density of the point cloud varies greatly, making dense point search and indexing operations relatively time-consuming. A common solution is to project the point cloud into a two-dimensional image space, or to use a rasterization method to convert each frame of laser radar data into a pseudo image to generate a structured matrix form required for standard convolution operations. This method improves the efficiency of data processing, but loses a large amount of original point clouds and their geometric features, which is not conducive to feature extraction of subsequent networks. In order to ensure the real-time and effectiveness of semantic segmentation in unstructured environments, the present invention uses spherical projection to pre-process the radar point cloud to generate a more compact and dense rendering image.

[0087] The data point coordinate system OXYZ of the laser radar in the present invention is established according to the right-hand rule with the center of the radar as the origin and the carrier's forward direction as the OX axis. The calculation formula of the vertex angle θ is as follows:

[0088]

[0089] Among them, (x, y, z) is the coordinate of each point in the radar point cloud in the Euclidean coordinate system. For each point in the point cloud, its (x, y, z) can be calculated. That is, project the points in the Euclidean coordinate system into the spherical coordinate system. This spherical coordinate system is actually a two-dimensional coordinate system. For ease of understanding, differentiate its angles to obtain a two-dimensional rectangular coordinate system:

[0090]

[0091] Among them, xi and xj are the horizontal and vertical coordinates of the two-dimensional rectangular coordinate system. Δθ corresponds to The resolution of θ.

[0092] Through this spherical projection transformation, any point (x, y, z) in the Euclidean space can be projected to a point (xi, xj) in a two-dimensional coordinate system. The present invention extracts three features of each point in the point cloud: Euclidean coordinates (x, y, z), such as Figure 3 As shown. Thus, a tensor of size (H, W, C) is obtained, where H is the width of the rendered image after spherical projection transformation, W is the length of the image, and C = 3. Taking the KITTI dataset as an example, the dataset uses a 64-line laser radar, so H = 64; in the horizontal direction, due to the limitation of the dataset annotation range, the radar point cloud data at 90° in front is used and divided into 512 grids (that is, 512 points are sampled horizontally), so W = 512.

[0093] Sub-step 2: Design of encoder-decoder network based on residual dilated convolution

[0094] Existing fast segmentation networks (such as Enet, etc.) are designed with a deeper and more complex structure to achieve high-precision multi-semantic segmentation of structured scenes, and are not designed for unstructured environments. They have defects in algorithm structure and principle and are difficult to be directly applied to unstructured environments. To this end, the present invention designs an encoding-decoding network based on residual dilated convolution for the semantic segmentation task in unstructured environments. Specifically, Figure 2 The midpoint cloud branch is shown.

[0095] First, determine the convolution layer type, convolution kernel size, and convolution step size.

[0096] Generally speaking, the feature expression ability of the network increases with the increase in the number of standard convolutional layers, but stacking more convolutional layers increases the amount of calculation, reduces real-time performance, and increases the risk of overfitting in the network. A small number of convolutional layers makes it difficult to extract features with strong expression ability. Therefore, in the process of designing the network, the dilated convolution layer is introduced. Dilated convolution can greatly expand the receptive field of the network without introducing additional parameters. Reasonable design of the dilation rate of dilated convolution and combined use can enable the network to learn complete road and obstacle information. In order to restore the feature map to the original image size and avoid introducing a lot of noise, the deconvolution layer with learnable parameters is designed to decouple the deeply compressed features in the feature map layer by layer.

[0097] In order to avoid introducing a large number of parameters due to the large convolution kernel, the convolution kernel size of the dilated convolution layer is designed to be 1×1 and 3×3, and the step size is set to 1. The convolution kernel size of the deconvolution layer is 3×3, and the step size is 2, so that the feature map after upsampling has the same resolution as the input.

[0098] Next, determine the pooling layer type, sampling size, and stride.

[0099] In order to reduce the amount of network calculation and further increase the network's receptive field so that it can sense larger targets (such as drivable areas), the present invention introduces a pooling layer. Maximum pooling can retain the most significant features (such as edge features) within the sampling window, so the maximum pooling operation is used to downsample the feature map, and the sampling size is set to 2×2, and the step size is set to 2.

[0100] Next, determine the dropout layer distribution.

[0101] The basic features extracted by the lower network layers are edges and corners that are consistent in data distribution. Compared with structured scenes, the features that distinguish drivable areas, obstacles and backgrounds in unstructured scenes are relatively sparse. Therefore, dropout is only inserted in the central layers of the encoder and decoder, which is conducive to the formation of better high-level features in deeper layers of the network.

[0102] Finally, the different types of network layers mentioned above are combined to determine the number of layers, the number of convolution kernels, and the dilation rate of the dilated convolution.

[0103] For a long time, the structural design of neural networks has been designed by researchers according to their own application scenarios and needs, referring to the currently recognized models with excellent performance, to design a network architecture that meets specific needs. Considering that the task of semantic segmentation in unstructured environments is to segment the drivable area, obstacles and background (i.e., three categories), compared with most of the current multi-classification semantic segmentation networks for autonomous driving, its architecture should be simpler, so there is no need to stack a large number of convolutional layers to meet accuracy and real-time requirements. The present invention preliminarily determines that the value range of the number of convolutional layers is 6 to 12.

[0104] While reducing the amount of network computation, the pooling layer also loses some spatial position information. Therefore, its number needs to be controlled within a certain range so that the network can maintain a balance between minimizing the amount of computation and retaining as much spatial position information as possible. The initial range of the number of pooling layers is 2 to 4. Because the purpose of the deconvolution layer is to restore the reduced feature map (caused by downsampling in the pooling layer) to the size of the input image, the number of deconvolution layers is equal to the number of pooling layers.

[0105] Referring to the existing excellent semantic segmentation models, the number of convolution kernels is mostly set to an integer power of 2. The fewer the number of convolution kernels, the fewer features the network can extract, while a larger number of convolution kernels will introduce more parameters, increasing the difficulty of optimization. In order to maintain a balance between the two, the range of the number of convolution kernels is initially determined to be 2. 4 ~2 10 .

[0106] The expansion rate of the dilated convolution kernel is generally an integer power of 2. The larger the expansion rate, the larger the receptive field, but the greater the noise introduced, and even a "grid effect" (causing discontinuous segmentation or even missed segmentation) will occur. Therefore, the present invention preliminarily determines that the range of the expansion rate is 2 0 ~2 4 .

[0107] After determining the range of the dilation rate, the cross-validation method is used to select the model and obtain the model architecture corresponding to the best network performance. In the process of network design, not only the types of convolutional layers and pooling layers and the influence of their respective layers on the extraction of boundary features are comprehensively considered, but also the dilated convolution is introduced and the dilation rate is designed so that it can avoid the "grid effect" while obtaining a larger receptive field. In addition, with reference to ResNet (He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.) and UNet (Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical images segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.), a residual structure is designed for cross-layer connection, so that the network can mine more effective information from the fusion features of different depths, so that the network can learn meaningful and complete semantic features under the premise of ensuring real-time performance. The optimal network architecture obtained by model selection is as follows (each convolution operation is activated by the ReLU function):

[0108] Block 1: Use 32 1×1 convolution kernels with a dilation rate of 1 to convolve with the 512×64×3 input sample to obtain sub-features Figure 1 , and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform batch normalization (BN). After concatenation, the feature maps are convolved with 1×1 convolution kernels with a dilation rate of 1 and perform BN. Figure 1 Do element-wise addition to get a feature map with a dimension of 512×64×32, as shown below: Figure 4 As shown;

[0109] Pooling layer 1: The feature map output by Block 2 is downsampled using a 2×2 maximum pooling layer with a step size of 2, resulting in a feature map of dimension 256×32×32;

[0110] Block2: Referring to Block1, convolution is performed with 64 1×1 convolution kernels with a dilation rate of 1 and the feature map output by pooling layer 1 to obtain sub-features. Figure 2 , and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. After the feature maps are concatenated, they are convoluted with 1×1 convolution kernels with a dilation rate of 1 and perform BN. Figure 2 Add pixel by pixel to get a feature map with a dimension of 256×32×64;

[0111] Pooling layer 2: After the feature map output by Block 2 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a feature map with a dimension of 128×16×64;

[0112] Block3: Referring to Block1, convolution is performed with 128 1×1 convolution kernels with a dilation rate of 1 and the feature map output by pooling layer 2 to obtain sub-features. Figure 3 , and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. After the feature maps are concatenated, they are convoluted with 1×1 convolution kernels with a dilation rate of 1 and perform BN. Figure 3 Add pixel by pixel to get a feature map with a dimension of 128×16×128;

[0113] Pooling layer 3: After the feature map output by Block 3 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a feature map with a dimension of 64×8×128;

[0114] Block4: Referring to Block1, convolve the feature map output by pooling layer 3 with 256 1×1 convolution kernels with a dilation rate of 1 to obtain sub-features. Figure 4 , and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. After the feature maps are concatenated, they are convoluted with 1×1 convolution kernels with a dilation rate of 1 and perform BN. Figure 4 Add pixel by pixel to get a feature map with a dimension of 64×8×256;

[0115] Deconvolution layer 1: Use 128 3×3 convolution kernels to convolve with the feature map output by Block4 and perform BN with a step size of 2. After dropout, add it pixel by pixel with the feature map output by Block3. After dropout, convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. After concatenating the feature maps, convolve with 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, after dropout, a feature map with a dimension of 128×16×128 is obtained. Figure 5As shown;

[0116] Deconvolution layer 2: Referring to deconvolution layer 1, 64 3×3 convolution kernels are used to convolve with the feature map output by Block4 and perform BN with a step size of 2. After dropout, the feature map is added pixel by pixel with the feature map output by Block3. After dropout, the feature maps are convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, after dropout, a feature map with a dimension of 256×32×64 is obtained.

[0117] Deconvolution layer 3: Referring to deconvolution layer 1, 32 3×3 convolution kernels are used to convolve with the feature map output by Block4 and perform BN, with a step size of 2. The feature map output by Block3 is added pixel by pixel, and then convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, a feature map with a dimension of 512×64×32 is obtained.

[0118] Standard convolution layer: convolve the feature map output by deconvolution layer 3 with three 1×1 convolution kernels to obtain a feature map with a dimension of 512×64×3. The three channels correspond to the total number of semantic classes.

[0119] Soft-max layer: Perform pixel-level classification on the feature map output by the standard convolutional layer to obtain the probability of each category and realize 3D semantic segmentation of the scene.

[0120] Sub-step 3: Loss function design

[0121] The present invention weights the traditional cross entropy loss with the square root of the class frequency and combines it with Lovasz-Softmax as the final loss function to maximize the IoU score of each class, as shown in the following formula:

[0122]

[0123] Among them, v i It is the number of points of various types. and P 3D (i) are the true value and the corresponding predicted probability, J represents the Lovasz expansion of IoU, e(c k ) is class c k The error vector of

[0124] (2) Design of image segmentation module based on residual cross-layer connection

[0125] Sub-step 1: Encoder-Decoder Network Design

[0126] In order to facilitate the use of the GAN algorithm in the two-dimensional pseudo-semantic enhancement module in step (3), the same encoding and decoding structure as the point cloud branch is used to extract image semantic features and obtain the classification probabilities;

[0127] Sub-step 2: Loss function design

[0128] The present invention adopts the standard cross entropy function as the loss function of the supervised training model:

[0129]

[0130] in and P 2D (j) represent the true value and the corresponding predicted value respectively;

[0131] (3) Design of 2D pseudo-semantic enhancement module based on GAN

[0132] The present invention selects CycleGAN as the generator of the two-dimensional pseudo semantic enhancement module, which is a cross-domain image generation GAN model with cycle consistency. Figure 6 As shown, there are two generators G: 2D→3D, F: 3D→2D, and two discriminators D 2D , D 3D , where D 2D Used to distinguish original two-dimensional semantic features (i.e., the output of the encoder in step (2)) and the two-dimensional pseudo-semantic features obtained by converting the three-dimensional semantic features (i.e., the output of the encoder in step (1)), D 3D Used to distinguish original 3D semantic features The three-dimensional pseudo semantic features obtained by converting the two-dimensional semantic features Make Cyclic 2D =F(G(feature 2D ))≈feature 2D , Cyclic 3D =G(F(feature 3D ))≈feature 3D ;

[0133] The adversarial loss of each mapping is as follows:

[0134]

[0135]

[0136] In order to further reduce the possible mapping function space, a cycle consistency loss is introduced to make the learned mapping function cycle consistent:

[0137]

[0138] Combining it with the adversarial losses in the two-dimensional and three-dimensional domains, the total loss function of the two-dimensional pseudo-semantic enhancement module is as follows:

[0139] L(G, F, D 2D , D 3D )=L GAN (G,D 3D , 2D, 3D)

[0140] +L GAN (F,D 2D , 2D, 3D)

[0141] +αL cycle (G, F) (7)

[0142] Among them, α is a hyperparameter that controls the relative importance of the two objectives;

[0143] (4) Network model training and point cloud semantic segmentation

[0144] Sub-step 1: Point cloud branch training

[0145] The present invention uses the Adam optimizer to iteratively train the network parameters. The initial learning rate of the optimizer is set to 0.01, the decay rate is set to 0.1 / 20K iterations, the dropout rate is set to 0.5, the batch size is set to 32, and the training epoch is set to 500;

[0146] Use structured environment data sets (such as KITTI) for pre-training to obtain pre-training weights, and then use a small amount of collected unstructured environment data for further training to obtain network weights; in order to increase the amount of training data, the present invention performs horizontal flipping, random noise addition, and random rotation around the z-axis [-5°, 5°] on the original point cloud;

[0147] Sub-step 2: Image branch training

[0148] Similar to the point cloud branch, we first use a structured environment dataset (such as KITTI) for pre-training to obtain pre-training weights, and then use a small amount of collected unstructured environment data for further training to obtain network weights;

[0149] Sub-step 3: CycleGAN training

[0150] Using the network weights obtained in substeps 1 and 2, generate paired outputs of the point cloud and image branch encoders It is used to supervise the training of CycleGAN and realize the mutual conversion of two-dimensional and three-dimensional semantic features;

[0151] Sub-step 4: Point Cloud Network Retraining

[0152] Freeze the parameters of the point cloud branch encoder, and use the dataset to retrain the point cloud branch decoder after adding the 2D pseudo semantic enhancement module to obtain the final network weights;

[0153] Sub-step 5: Point cloud semantic segmentation

[0154] Using the network weights obtained in sub-step 4, the collected radar point cloud is tested online to obtain the semantic segmentation result of the unstructured environment 3D point cloud.

[0155] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A semantic segmentation method for point cloud in unstructured environments based on cross-modal semantic enhancement, characterized by: The following steps are included: (1) Design of point cloud segmentation module based on spherical projection; Sub-step 1: Radar point cloud preprocessing The laser radar data point coordinate system OXYZ is established according to the right-hand rule with the radar center as the origin and the carrier forward direction as the OX axis; the azimuth The calculation formula of the vertex angle θ is as follows: Among them, (x, y, z) is the coordinate of each point in the radar point cloud in the Euclidean coordinate system; for each point in the point cloud, its (x, y, z) can be calculated. That is, project the points in the Euclidean coordinate system into the spherical coordinate system; this spherical coordinate system is actually a two-dimensional coordinate system, and its angles are differentiated to obtain a two-dimensional rectangular coordinate system: Among them, xi and xj are the horizontal and vertical coordinates of the two-dimensional rectangular coordinate system. Δθ corresponds to The resolution of θ; Through this spherical projection transformation, any point (x, y, z) in the Euclidean space is projected to a point (xi, xj) in the two-dimensional coordinate system; three features of each point in the point cloud are extracted: Euclidean coordinates (x, y, z), thereby obtaining a tensor of size (H, W, C), where H is the width of the rendered image obtained after the spherical projection transformation, W is the length of the image, and C = 3; taking the KITTI dataset as an example, the dataset uses a 64-line laser radar, so H = 64; in the horizontal direction, due to the limitation of the dataset annotation range, the radar point cloud data at 90° in front is used and divided into 512 grids, that is, 512 points are sampled horizontally, so W = 512; Sub-step 2: Design of encoder-decoder network based on residual dilated convolution For the semantic segmentation task in unstructured environments, an encoder-decoder network based on residual dilated convolution is designed; First, determine the convolution layer type, convolution kernel size, and convolution step size; The convolution kernel size of the dilated convolution layer is designed to be 1×1 and 3×3, and the step size is set to 1; the convolution kernel size of the deconvolution layer is 3×3, and the step size is 2, so that the feature map after upsampling has the same resolution as the input; Secondly, determine the pooling layer type, sampling size, and step size; The maximum pooling operation is used to downsample the feature map, and the sampling size is set to 2×2 and the step size is set to 2; Next, determine the dropout layer distribution; Insert dropout only in the central layers of the encoder and decoder; Finally, the different types of network layers involved above are combined, and the model selection is performed using the cross-validation method to determine the number of layers, the number of convolution kernels, and the dilation rate of the dilated convolution. The following optimal network architecture is obtained, in which each convolution operation is activated by the ReLU function: Block 1: Use 32 1×1 convolution kernels with a dilation rate of 1 to convolve with the input sample of 512×64×3 to obtain sub-feature map 1, and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform batch normalization. After the feature maps are concatenated, they are convolved with a 1×1 convolution kernel with a dilation rate of 1 and perform BN, and then added pixel by pixel with sub-feature map 1 to obtain a feature map with a dimension of 512×64×32; Pooling layer 1: The feature map output by Block 2 is downsampled using a 2×2 maximum pooling layer with a step size of 2, resulting in a feature map of dimension 256×32×32; Block2: Referring to Block1, 64 1×1 convolution kernels with a dilation rate of 1 are used to convolve with the feature map output by pooling layer 1 to obtain sub-feature map 2. Then, convolutions are performed with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, and BN is performed. After the feature maps are concatenated, they are convolved with a 1×1 convolution kernel with a dilation rate of 1 and BN is performed. Then, the sub-feature map 2 is added pixel by pixel to obtain a feature map with a dimension of 256×32×64. Pooling layer 2: After the feature map output by Block 2 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a feature map with a dimension of 128×16×64; Block3: Referring to Block1, 128 1×1 convolution kernels with a dilation rate of 1 are used to convolve with the feature map output by pooling layer 2 to obtain sub-feature map 3. Then, convolution is performed with convolution kernels with a dilation rate of 3×3 and a dilation rate of 1 and 3×3 and a dilation rate of 2 respectively, and BN is performed. After the feature maps are concatenated, they are convolved with a convolution kernel with a dilation rate of 1 and BN is performed. Then, the sub-feature map 3 is added pixel by pixel to obtain a feature map with a dimension of 128×16×128. Pooling layer 3: After the feature map output by Block 3 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a feature map with a dimension of 64×8×128; Block4: Referring to Block1, 256 1×1 convolution kernels with a dilation rate of 1 are used to convolve with the feature map output by pooling layer 3 to obtain sub-feature map 4. Then, convolutions are performed with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, and BN is performed. After the feature maps are concatenated, they are convolved with a 1×1 convolution kernel with a dilation rate of 1, and BN is performed. Then, pixel-by-pixel addition is performed with sub-feature map 4 to obtain a feature map with a dimension of 64×8×256. Deconvolution layer 1: Use 128 3×3 convolution kernels to convolve with the feature map output by Block4 and perform BN with a step size of 2. After dropout, add it pixel by pixel with the feature map output by Block3. After dropout, convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. After concatenating the feature maps, convolve with 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, after dropout, a feature map with a dimension of 128×16×128 is obtained. Deconvolution layer 2: Referring to deconvolution layer 1, 64 3×3 convolution kernels are used to convolve with the feature map output by Block4 and perform BN with a step size of 2. After dropout, the feature map is added pixel by pixel with the feature map output by Block3. After dropout, the feature maps are convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, after dropout, a feature map with a dimension of 256×32×64 is obtained. Deconvolution layer 3: Referring to deconvolution layer 1, 32 3×3 convolution kernels are used to convolve with the feature map output by Block4 and perform BN, with a step size of 2. The feature map output by Block3 is added pixel by pixel, and then convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, a feature map with a dimension of 512×64×32 is obtained. Standard convolution layer: convolve the feature map output by deconvolution layer 3 with three 1×1 convolution kernels to obtain a feature map with a dimension of 512×64×3. The three channels correspond to the total number of semantic classes. Soft-max layer: performs pixel-level classification on the feature map output by the standard convolutional layer to obtain the probability of each category and realize 3D semantic segmentation of the scene; Sub-step 3: Loss function design The traditional cross entropy loss is weighted by the square root of the class frequency and combined with Lovasz-Softmax as the final loss function to maximize the IoU score of each class, as shown in the following formula: Among them, v i It is the number of points of various types. and P 3D (i) are the true value and the corresponding predicted probability, J represents the Lovasz expansion of IoU, e(c k ) is class c k The error vector of (2) Design of image segmentation module based on residual cross-layer connection; Sub-step 1: Encoder-Decoder Network Design In order to facilitate the use of the GAN algorithm in the two-dimensional pseudo-semantic enhancement module in step (3), the same encoding and decoding structure as the point cloud branch is used to extract image semantic features and obtain the classification probabilities; Sub-step 2: Loss function design The standard cross entropy function is used as the loss function of the supervised training model: in and P 2D (j) represent the true value and the corresponding predicted value respectively; (3) Design of two-dimensional pseudo-semantic enhancement module based on GAN; CycleGAN is selected as the generator of the two-dimensional pseudo semantic enhancement module. It is a cross-domain image generation GAN model with cycle consistency, that is, there are two generators G:2D→3D, F:3D→2D, which are inverse matrices of each other, and two discriminators D 2D , D 3D , where D 2D Used to distinguish original two-dimensional semantic features That is, the output of the encoder in step (2) and the two-dimensional pseudo-semantic feature converted from the three-dimensional semantic feature That is, the output of the encoder in step (1), D 3D Used to distinguish original 3D semantic features The three-dimensional pseudo semantic features obtained by converting the two-dimensional semantic features Make Cyclic 2D =F(G(feature 2D ))≈feature 2D , Cyclic 3D =G(F(feature 3D ))≈feature 3D ; The adversarial loss of each mapping is as follows: In order to further reduce the mapping function space, the cycle consistency loss is introduced to make the learned mapping function cycle consistent: Combining it with the adversarial losses in the two-dimensional and three-dimensional domains, the total loss function of the two-dimensional pseudo-semantic enhancement module is as follows: L(G,F,D 2D ,D 3D )=L GAN (G,D 3D ,2D,3D)+L GAN (F,D 2D ,2D,3D)+αL cycle (G,F) (7) Among them, α is a hyperparameter that controls the relative importance of the two objectives; (4) Network model training and point cloud semantic segmentation Sub-step 1: Point cloud branch training The Adam optimizer was used to iteratively train the network parameters. The initial learning rate of the optimizer was set to 0.01, the decay rate was set to 0.1 / 20K iterations, the dropout rate was set to 0.5, the batch size was set to 32, and the training epoch was set to 500. First, a structured environment data set is used for pre-training, wherein the structured environment data set includes KITTI, to obtain pre-training weights, and then a small amount of collected unstructured environment data is used for further training to obtain network weights; in order to increase the amount of training data, the present invention performs horizontal flipping, random noise addition, and random rotation around the z-axis by [-5°, 5°] on the original point cloud; Sub-step 2: Image branch training Similar to the point cloud branch, we first use a structured environment dataset for pre-training, including KITTI, to obtain pre-training weights, and then use a small amount of collected unstructured environment data for further training to obtain network weights. Sub-step 3: CycleGAN training Using the network weights obtained in substeps 1 and 2, generate paired outputs of the point cloud and image branch encoders It is used to supervise the training of CycleGAN and realize the mutual conversion of two-dimensional and three-dimensional semantic features; Sub-step 4: Point cloud network retraining Freeze the parameters of the point cloud branch encoder, and use the dataset to retrain the point cloud branch decoder after adding the 2D pseudo semantic enhancement module to obtain the final network weights; Sub-step 5: Point cloud semantic segmentation Using the network weights obtained in sub-step 4, the collected radar point cloud is tested online to obtain the semantic segmentation result of the unstructured environment 3D point cloud.

Images