Three-dimensional point-cloud semantic segmentation method based on multi-level boundary enhancement for unstructured environment
By introducing spherical projection and edge coding structures into the lidar semantic segmentation algorithm, combining attention fusion and discriminator, the problem of edge segmentation difficulties in unstructured environments is solved, and efficient and accurate three-dimensional point cloud semantic segmentation is achieved, improving the improvement of Edge perception and segmentation accuracy are achieved.
Patent Information
- Application Number
- PCT/CN2023/118672
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2023-09-14
- Publication Date
- 2024-11-14
AI Technical Summary
Existing semantic segmentation algorithms based on lidar are difficult to accurately obtain the edges between obstacles, passable areas and backgrounds in complex and changeable unstructured environments, resulting in poor segmentation effects, especially in the absence of structural features and complex terrain. In case.
The semantic-edge dual-coded structure design based on spherical projection is adopted, combined with the edge attention fusion module, decoder and edge discriminator, and the network model is optimized through the loss function to realize three-dimensional point cloud semantic segmentation, improving edge perception ability and segmentation accuracy.
Efficient and accurate segmentation of obstacles and passable areas in a complex and changeable unstructured environment, improving the real-time and robustness of the algorithm, making up for the shortcomings of traditional methods in edge feature extraction.
Smart Images

Figure CN2023118672_14112024_PF_FP_ABST
Abstract
Description
Semantic segmentation method of 3D point cloud in unstructured environment based on multi-level edge enhancement Technical Field
[0001] The present invention belongs to the technical field of intelligent vehicle environment perception, and mainly relates to a semantic segmentation method of three-dimensional point clouds in unstructured environments based on multi-level edge enhancement. Background Art
[0002] Efficient and reliable environmental perception and understanding are crucial for autonomous vehicles and robots in intelligent transportation systems, and are a prerequisite for positioning, decision-making, and navigation. Semantic segmentation, as a key approach to achieving refined environmental understanding, has become a research hotspot in recent years and has garnered widespread attention. LiDAR, a common environmental perception sensor, is virtually unaffected by changes in light and weather, and can provide precise distance and geometric structure information. It is widely used in various scenarios, and LiDAR-based semantic segmentation algorithms are also evolving.
[0003] Existing lidar-based methods are mostly limited to small-scale indoor scenes or typical urban traffic scenarios. Research on complex and changing unstructured environments (such as off-road environments and emergency rescue scenarios) is limited, the technology is still immature, and there are various problems. Methods based on traditional feature extraction (radial and lateral features) have achieved good detection results for obstacle and traversable area segmentation. However, these methods rely on small-scale point cloud feature extraction algorithms and are inefficient in large-scale unstructured environments. Furthermore, they are designed only for specific unstructured road scenarios (such as off-road or rural roads). When applied to complex and changing unstructured scenes, the limited geometric feature descriptions make these methods difficult to achieve ideal segmentation results. With the development of supervised and semi-supervised learning, some studies have used rich urban traffic scene datasets and a small number of labeled unstructured samples to perform scene understanding in off-road scenes. However, unstructured scenes lack structural features such as lanes, road surfaces, and guardrails, and the boundaries of the drivable area are fuzzy and the textures are diverse. At the same time, affected by terrain, shrubs, and other vegetation, the obstacle features are complex and changeable, and there are occlusions and missing areas. These sparse and complex environmental features lead to the inaccurate edge distinction between obstacles, drivable areas, and backgrounds in existing algorithms, making it difficult to directly apply them to complex and changeable unstructured environments.
[0004] Summary of the Invention
[0005] The present invention addresses the problem that existing methods in complex and changeable unstructured scenes have difficulty in clearly and accurately obtaining the edges between obstacles, traversable areas, and backgrounds, and cannot achieve real-time and accurate segmentation results. A method for semantic segmentation of 3D point clouds in unstructured environments based on multi-level edge enhancement is provided. The method includes six steps: designing a semantic-edge dual encoding structure based on spherical projection, designing an edge attention fusion module, designing a decoder, designing an edge discriminator, designing a loss function, and training a network model for point cloud semantic segmentation. First, an efficient semantic-edge dual encoding structure based on spherical projection is constructed. Second, an edge attention fusion module is designed to deeply and automatically fuse the intermediate features of the dual encoders. Third, a decoder is designed to restore the feature map to its original input size. Next, an edge discriminator is proposed to further improve the ability to recognize edges at the output end. Then, a loss function is designed. Finally, the network is trained using a sample set to obtain network parameters, thereby achieving efficient and reliable semantic segmentation of 3D point clouds in unstructured environments. The method of the present invention combines the advantages of traditional methods and deep learning algorithms to improve the perception of edges, thereby achieving efficient and high-precision 3D semantic segmentation in unstructured environments.
[0006] To achieve the above objectives, the present invention adopts a technical solution: a method for semantic segmentation of 3D point clouds in unstructured environments based on multi-level edge enhancement, comprising the following steps:
[0007] S1, Semantic-Edge Dual Encoding Structure Design Based on Spherical Projection: Spherical projection is used to preprocess the radar point cloud to generate a rendered image. A semantic-edge dual encoding structure based on residual dilated convolution is then established to simultaneously extract semantic and edge features at different scales. The semantic encoder and edge encoder use the same structural design, with all convolutional layers being dilated convolutional layers. Max pooling is used to downsample the feature maps, and dropout layers are inserted only in the center layer of the encoder.
[0008] S2, edge attention fusion module design: The edge attention fusion module is used to deeply fuse semantics and edge features of different scales; the edge attention fusion module is:
[0009] Among them, F out is the fused feature; F in_main 、F in_boundary Represent the intermediate features of the backbone semantic encoder and edge encoder respectively; It means element-by-element addition; Represents element-wise multiplication;
[0010] S3, decoder design: Design a deconvolution layer with learnable parameters to decouple the deeply compressed features in the feature map layer by layer. The convolution kernel size of the deconvolution layer is 3×3 and the stride is 2, so that the upsampled feature map has the same resolution as the input. The number of deconvolution layers is equal to the number of pooling layers.
[0011] S4, edge discriminator design: The edge discriminator combines the Markov discriminator with the global discriminator to constrain the edge output;
[0012] S5, loss function design: The loss function is a combination of weighted cross entropy loss, Lovasz-Softmax and edge discriminator loss in step S4, specifically:
[0013] Among them, v i It is the number of points of various types. and P 3D (i) are the true value and the corresponding predicted probability, J represents the Lovasz expansion of IoU, e(c k ) is class c k The error vector of
[0014] S6, Network model training and point cloud semantic segmentation: The SGD optimizer is used to iteratively train the network parameters to obtain the network weights. Based on the network weights, the collected radar point cloud is tested online to obtain the semantic segmentation results of the 3D point cloud in the unstructured environment.
[0015] As an improvement of the present invention, the preprocessing of the radar point cloud in step S1 is specifically as follows: each point in the point cloud in the Euclidean coordinate system is projected into the spherical coordinate system to obtain a two-dimensional rectangular coordinate system; with the radar center as the origin and the carrier forward direction as the OX axis, the azimuth of each point is established according to the right-hand rule. The calculation formula for the vertex angle θ is as follows:
[0016] Among them, (x, y, z) is the coordinate of each point in the point cloud in the Euclidean coordinate system; the angle is differentiated to obtain a two-dimensional rectangular coordinate system:
[0017] Among them, xi and xj are the horizontal and vertical coordinates of the two-dimensional rectangular coordinate system, Δθ corresponds to The resolution of θ.
[0018] As an improvement of the present invention, in the semantic-edge dual coding structure based on residual dilated convolution in step S1, the number of convolution layers ranges from 6 to 12; the number of pooling layers ranges from 2 to 4; the number of convolution kernels ranges from 24 ~2 10 ; The range of expansion rate is 2 0 ~2 4 .
[0019] As an improvement of the present invention, the structure of the optimal network architecture in step S1 is:
[0020] Block 1: Convolve 32 1×1 convolution kernels with a dilation rate of 1 with the 512×64×3 input sample to obtain sub-feature map 1. Then, convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, respectively, and perform batch normalization. The concatenated feature maps are convolved with a 1×1 convolution kernel with a dilation rate of 1, batch normalized, and then pixel-wise added to sub-feature map 1 to obtain a feature map of dimension 512×64×32.
[0021] Pooling layer 1: The feature map output by Block 2 is downsampled using a 2×2 max pooling layer with a stride of 2, resulting in a feature map of dimension 256×32×32.
[0022] Block 2: Referring to Block 1, convolve the feature map output by pooling layer 1 with 64 1×1 convolution kernels with a dilation rate of 1 to obtain sub-feature map 2. Then, convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, respectively, and perform batch normalization. After concatenating the feature maps, convolve with a 1×1 convolution kernel with a dilation rate of 1, perform batch normalization, and then add pixel-by-pixel to sub-feature map 2 to obtain a feature map with a dimension of 256×32×64.
[0023] Pooling layer 2: After the feature map output by Block 2 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a stride of 2 to obtain a feature map of dimension 128×16×64;
[0024] Block 3: Referring to Block 1, 128 1×1 convolution kernels with a dilation rate of 1 are convolved with the feature map output by pooling layer 2 to obtain sub-feature map 3. Subsequently, sub-feature map 3 is convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, and batch normalized. The concatenated feature maps are convolved with a 1×1 convolution kernel with a dilation rate of 1 and batch normalized, and then pixel-by-pixel added to sub-feature map 3 to obtain a feature map with a dimension of 128×16×128.
[0025] Pooling layer 3: After the feature map output by Block 3 is dropped out, it is downsampled using a 2×2 max pooling layer with a stride of 2, resulting in a feature map of dimension 64×8×128.
[0026] Block 4: Referring to Block 1, convolve the feature map output by pooling layer 3 with 256 1×1 convolution kernels with a dilation rate of 1 to obtain sub-feature map 4. Then, convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, respectively, and perform batch normalization. After concatenating the feature maps, convolve with a 1×1 convolution kernel with a dilation rate of 1, perform batch normalization, and then add pixel-by-pixel to sub-feature map 4 to obtain a feature map with a dimension of 64×8×256.
[0027] Each convolution operation in the optimal network architecture is activated by the ReLU function.
[0028] As an improvement of the present invention, the multi-scale channel attention module in step S2 is specifically:
[0029] Among them, σ represents the Sigmoid activation function; G(F) and L(F) represent the global channel context information and local channel context information, respectively.
[0030] As another improvement of the present invention, the structure of the decoder in step S3 is:
[0031] Deconvolution layer 1: 128 3×3 convolution kernels are convolved with the feature map fused after Block 4 and batch normalized with a stride of 2. After dropout, the kernel is added pixel by pixel to the feature map output by Block 3. After dropout, the kernel is convolved with 3×3 kernels with a dilation rate of 1 and 3×3 kernels with a dilation rate of 2, and batch normalized. The concatenated feature maps are convolved with a 1×1 kernel with a dilation rate of 1 and batch normalized. Finally, dropout is performed to obtain a feature map of dimension 128×16×128.
[0032] Deconvolution layer 2: Referring to deconvolution layer 1, 64 3×3 convolution kernels are used to convolve with the feature map output by deconvolution layer 1 and batch normalized with a step size of 2. After dropout, the feature map output by Block 2 is added pixel by pixel. After dropout, the feature map is convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and batch normalized. The concatenated feature maps are convolved with a 1×1 convolution kernel with a dilation rate of 1 and batch normalized. Finally, dropout is performed to obtain a feature map with a dimension of 256×32×64.
[0033] Deconvolution layer 3: Referring to deconvolution layer 1, 32 3×3 convolution kernels are used to convolve with the feature map output by deconvolution layer 2 and perform batch normalization. The stride is 2, and the feature map output by Block 1 is added pixel by pixel. Then, the convolution kernels are respectively convolved with 3×3 kernels with a dilation rate of 1 and 3×3 kernels with a dilation rate of 2 and batch normalized. The concatenated feature maps are convolved with a 1×1 kernel with a dilation rate of 1 and batch normalized. Finally, a feature map of dimension 512×64×32 is obtained.
[0034] Standard convolutional layer: convolution is performed with three 1×1 convolution kernels and the feature map output by deconvolution layer 3, resulting in a feature map of dimension 512×64×3, where the three channels correspond to the total number of semantic classes.
[0035] Soft-max layer: performs pixel-level classification on the feature map output by the standard convolutional layer to obtain the probability of each category.
[0036] As another improvement of the present invention, the edge discriminator in step S4 uses the edge feature map (represented as B and B respectively) extracted from the semantic label predicted by the network and the true value semantic label. pred and B GT ) as the output, and distinguish between semantic labels and true value semantic labels; the first three layers are feature sharing layers, each of which is composed of a 4×4 convolution kernel, batch normalization processing and LeakyReLU in series, which is used to learn common underlying features; then it is divided into two branches: the first branch ends with a fully connected layer and Sigmoid activation to obtain a binary classification output, taking into account the global information of the entire image to determine whether it is true or false; the second branch uses the same Markov discriminator as CycleGAN, and its output is a two-dimensional matrix, each value of the matrix represents the judgment of the local area of the input to enhance the attention to local texture details; the loss L DB Calculated by two branches:
[0037] in, They are the global discriminator and Markov discriminator losses respectively.
[0038] As a further improvement of the present invention, in the network model training step S6, the original point cloud is horizontally flipped, random noise is added, and randomly rotated around the z-axis by [-5°, 5°].
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] (1) The spherical projection method used in the present invention is more conducive to the extraction of point cloud features while reducing indexing and calculation time.
[0041] (2) The present invention uses dilated convolution and residual structure, which not only reduces the complexity of the network structure, but also can mine more effective information from the fusion features of different depths of the network, thereby ensuring that the network takes into account both accuracy and real-time performance.
[0042] (3) The edge encoder added in the present invention not only makes up for the missing edge structure information of the backbone network and realizes edge enhancement at the input end, but also provides rich intermediate features for subsequent feature-level edge enhancement.
[0043] (4) The edge attention fusion module used in the present invention automatically assigns multi-task learning weights through supervised learning, so that the feature information between the semantic encoder and the edge encoder is deeply integrated to achieve feature-level edge enhancement.
[0044] (5) The edge discriminator used in the present invention uses an end-to-end approach to achieve edge enhancement at the output level without the need for additional precise edge labels, further improving the algorithm's ability to perceive edges and its robustness to complex and changing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] FIG1 is a flow chart of the steps of the method of the present invention;
[0046] FIG2 is a schematic diagram of a point cloud segmentation network based on multi-level edge enhancement in a semantic-edge dual encoding structure based on residual dilated convolution in step S1 of the present invention;
[0047] 3 is a schematic diagram of spherical projection in the semantic-edge dual coding structure design based on spherical projection in step S1 of the method of the present invention;
[0048] FIG4 is a schematic diagram of a block in the semantic-edge dual coding structure design based on spherical projection in step S1 of the method of the present invention;
[0049] FIG5 is a schematic diagram of the structure of the edge attention fusion module in step S2 of the method of the present invention;
[0050] FIG6 is a schematic diagram of a multi-scale channel attention module in step S2 of the method of the present invention;
[0051] FIG7 is a schematic diagram of the structure of the deconvolution layer in step S3 of the method of the present invention;
[0052] FIG8 is a schematic diagram of the structure of the edge discriminator in step S4 of the method of the present invention. DETAILED DESCRIPTION
[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.
[0054] Example 1
[0055] The semantic segmentation method of 3D point cloud in unstructured environment based on multi-level edge enhancement is shown in Figure 1 and specifically includes the following steps:
[0056] Step S1: Design of semantic-edge dual coding structure based on spherical projection
[0057] Sub-step 1: Radar point cloud preprocessing
[0058] LiDAR point cloud data is massive, and due to uneven environmental sampling, point cloud density varies greatly, making dense point search and indexing operations relatively time-consuming. Common solutions involve projecting the point cloud into a two-dimensional image space or using rasterization to convert each frame of LiDAR data into a pseudo-image to generate the structured matrix required for standard convolution operations. While these methods improve data processing efficiency, they also lose a significant amount of the original point cloud and its geometric features, hindering subsequent feature extraction in the network.
[0059] To ensure the real-time and effectiveness of semantic segmentation in unstructured environments, the present invention uses spherical projection to pre-process the radar point cloud to generate a more compact and dense rendered image, as shown in Figure 3.
[0060] The data point coordinate system OXYZ of the laser radar is based on the radar center as the origin and the carrier's forward direction as the OX axis, and is established according to the right-hand rule. The calculation formula for the vertex angle θ is as follows:
[0061] Among them, (x, y, z) is the coordinate of each point in the point cloud in the Euclidean coordinate system. For each point in the point cloud, its (x, y, z) can be calculated. That is, projecting the points in the Euclidean coordinate system into the spherical coordinate system. This spherical coordinate system is actually a two-dimensional coordinate system. To make it easier to understand, differentiate its angles to obtain a two-dimensional rectangular coordinate system:
[0062] Among them, xi and xj are the horizontal and vertical coordinates of the two-dimensional rectangular coordinate system, Δθ corresponds to The resolution of θ.
[0063] Through this spherical projection transformation, any point (x, y, z) in the Euclidean space can be projected to a point (xi, xj) in the two-dimensional coordinate system. The present invention extracts four features of each point in the point cloud: Euclidean coordinates (x, y, z, r), where This results in a tensor of size (H, W, C), where H is the width of the rendered image after spherical projection transformation, W is the height, and C = 4. Taking the KITTI dataset as an example, this dataset uses a 64-line lidar, so H = 64. In the horizontal direction, due to the limitations of the subsequent discriminator, the radar point cloud data at 90° in front is used and divided into 512 grids (i.e., 512 points are sampled horizontally), so W = 512.
[0064] Next, the canny operator is used to extract edges from the generated (H, W, C) point cloud tensor to obtain the (H, W, C) edge tensor.
[0065] Sub-step 2: Design of semantic-edge dual encoding structure based on residual dilated convolution
[0066] Existing fast segmentation networks (such as SalsaNext, etc.) are designed with deeper and more complex structures to achieve high-precision multiple semantic segmentation of structured scenes. They are not designed for unstructured environments and ignore the important influence of edge information on the segmentation effect. There are defects in the algorithm structure and principle, and it is difficult to be directly applied to unstructured environments. To this end, the present invention designs a semantic-edge dual encoding network based on residual dilated convolution for the semantic segmentation task of unstructured environments to simultaneously extract semantic and edge features of different scales and achieve input-level edge enhancement, as shown in Figure 2. Since the dual encoders use the same structural design, the semantic encoder is taken as an example below.
[0067] First, determine the convolutional layer type, convolution kernel size, and convolution stride.
[0068] Generally speaking, a network's feature expression capability increases with the number of standard convolutional layers. However, stacking more convolutional layers increases computational complexity, reduces real-time performance, and increases the risk of overfitting. Fewer convolutional layers also make it difficult to extract highly expressive features. Therefore, dilated convolutional layers were introduced during network design. Dilated convolutions can significantly expand the network's receptive field without introducing additional parameters. By properly designing the dilation rates and combining them, the network can learn complete road and obstacle information.
[0069] In order to avoid introducing a large number of parameters due to the large convolution kernel, the convolution kernel sizes of the dilated convolution layer are designed to be 1×1 and 3×3, and the step size is set to 1.
[0070] Next, determine the pooling layer type, sampling size, and stride.
[0071] To reduce the network's computational load and further increase its receptive field, enabling it to perceive larger objects (e.g., drivable areas), this paper introduces a pooling layer. Max pooling preserves the most significant features (e.g., edge features) within the sampling window. Therefore, a max pooling operation is used to downsample the feature map, with a sampling size of 2×2 and a stride of 2.
[0072] Next, determine the dropout layer distribution.
[0073] The basic features extracted by the lower network layers are features such as edges and corners that are consistent in the data distribution. Compared with structured scenes, the features that distinguish the drivable area, obstacles and background in unstructured scenes are relatively sparse. Therefore, only inserting dropout in the center layer of the encoder is conducive to the formation of better high-level features in the deeper layers of the network.
[0074] Finally, the different types of network layers mentioned above are combined to determine the number of layers, the number of convolution kernels, and the dilation rate of the dilated convolution.
[0075] For a long time, the structural design of neural networks has been done by researchers based on their own application scenarios and needs, referring to currently recognized models with excellent performance, to design a network architecture that meets specific needs. Considering that the task of semantic segmentation in unstructured environments is to segment the drivable area, obstacles, and background (i.e., three categories), compared to most current multi-classification semantic segmentation networks for autonomous driving, its architecture should be simpler, so there is no need to stack a large number of convolutional layers to meet accuracy and real-time requirements. The present invention preliminarily determines that the number of convolutional layers ranges from 6 to 12.
[0076] While reducing the amount of network computation, the pooling layer also loses some spatial location information. Therefore, its number needs to be controlled within a certain range so that the network can maintain a balance between minimizing the amount of computation and retaining as much spatial location information as possible. The number of pooling layers is preliminarily determined to range from 2 to 4.
[0077] Referring to the existing excellent semantic segmentation models, the number of convolution kernels is mostly set to an integer power of 2. The fewer the number of convolution kernels, the fewer features the network can extract, while a larger number of convolution kernels will introduce more parameters, increasing the difficulty of optimization. In order to maintain a balance between the two, the range of the number of convolution kernels is initially determined to be 2. 4 ~2 10 .
[0078] The expansion rate of the dilated convolution kernel is generally an integer power of 2. The larger the expansion rate, the larger the receptive field, but the greater the noise introduced, and even a "grid effect" (causing discontinuous segmentation or even missed segmentation) may occur. Therefore, the present invention preliminarily determines that the expansion rate range is 2 0 ~24 .
[0079] After determining the range of dilation rates, cross-validation was used to select the model architecture that achieves optimal network performance. During network design, the impact of the types and number of convolutional and pooling layers on edge feature extraction was comprehensively considered. Dilated convolutions were also introduced and the dilation rate was carefully tailored to achieve a larger receptive field while avoiding the "grid effect." Furthermore, based on ResNet (He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.) and UNet (Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.), a residual structure was designed to implement cross-layer connections. This allows the network to extract more effective information from fused features at different depths, enabling the network to learn meaningful and complete semantic features while ensuring real-time performance. The optimal network architecture obtained by model selection is as follows (each convolution operation is activated by the ReLU function):
[0080] Block 1: Convolve 32 1×1 convolution kernels with a dilation rate of 1 with the 512×64×4 input sample to obtain sub-feature map 1. Then, convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, respectively, and perform batch normalization (BN). After concatenation, the feature maps are convolved with a 1×1 convolution kernel with a dilation rate of 1, and BN is performed. Finally, they are element-wise added to sub-feature map 1 to obtain a feature map of dimension 512×64×32, as shown in Figure 4.
[0081] Pooling layer 1: The feature map output by Block 2 is downsampled using a 2×2 max pooling layer with a stride of 2, resulting in a feature map of dimension 256×32×32.
[0082] Block 2: Referring to Block 1, convolve the feature map output by pooling layer 1 with 64 1×1 convolution kernels with a dilation rate of 1 to obtain sub-feature map 2. Then, convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, and perform batch normalization. After concatenating the feature maps, convolve with a 1×1 convolution kernel with a dilation rate of 1, perform batch normalization, and then add pixel-by-pixel with sub-feature map 2 to obtain a feature map with a dimension of 256×32×64.
[0083] Pooling layer 2: After the feature map output by Block 2 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a stride of 2 to obtain a feature map of dimension 128×16×64;
[0084] Block 3: Referring to Block 1, 128 1×1 convolution kernels with a dilation rate of 1 are convolved with the feature map output by pooling layer 2 to obtain sub-feature map 3. Subsequently, sub-feature map 3 is convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, and batch normalization is performed. The concatenated feature maps are convolved with a 1×1 convolution kernel with a dilation rate of 1 and batch normalization is performed. Finally, the concatenated feature maps are added pixel by pixel with sub-feature map 3 to obtain a feature map with a dimension of 128×16×128.
[0085] Pooling layer 3: After the feature map output by Block 3 is dropped out, it is downsampled using a 2×2 max pooling layer with a stride of 2, resulting in a feature map of dimension 64×8×128.
[0086] Block 4: Referring to Block 1, convolve the feature map output by pooling layer 3 with 256 1×1 convolution kernels with a dilation rate of 1 to obtain sub-feature map 4. Then, convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2, and perform batch normalization. After concatenating the feature maps, convolve with a 1×1 convolution kernel with a dilation rate of 1, perform batch normalization, and then add pixel-by-pixel with sub-feature map 4 to obtain a feature map with a dimension of 64×8×256.
[0087] Step S2: Edge attention fusion module design
[0088] Through the dual encoder structure of step S1, low-level and high-level semantic and edge information can be obtained respectively. Considering that most existing feature fusion methods use direct splicing or point-by-point addition, it is difficult to balance the texture information and edge information of two independent branches. Therefore, in order to better deeply fuse the features of these two branches in a controllable and intelligent manner, the present invention proposes an edge attention fusion module to achieve feature-level edge enhancement, as shown in Figure 5.
[0089] The edge attention fusion module can be expressed as:
[0090] Among them, F out is the fused feature; F in_main 、F in_boundary Represent the intermediate features of the backbone semantic encoder and edge encoder respectively; It means element-by-element addition; represents element-by-element multiplication; M is a multi-scale channel attention module, as shown in Figure 6, which can be expressed as:
[0091] Among them, σ represents the Sigmoid activation function; G(F) and L(F) represent the global channel context information and local channel context information, respectively.
[0092] Taking into account the number of network parameters and complexity, this embodiment only fuses the intermediate features of the semantic branch and edge branch after pooling layer 2 and Block 4.
[0093] Step S3: Decoder design
[0094] To restore the feature map to its original image size while avoiding the introduction of excessive noise, a deconvolution layer with learnable parameters is designed to perform a layer-by-layer hierarchical decoupling of the deeply compressed features in the feature map. To avoid the introduction of a large number of parameters due to overly large convolution kernels, the deconvolution layer uses a 3×3 kernel with a stride of 2, ensuring that the upsampled feature map has the same resolution as the input. Because the purpose of the deconvolution layer is to restore the reduced feature map (caused by the downsampling of the pooling layer) to the input image size, the number of deconvolution layers is equal to the number of pooling layers. The specific design is as follows:
[0095] Deconvolution layer 1: 128 3×3 convolution kernels are used to convolve with the feature map fused after Block 4 and perform BN with a stride of 2. After dropout, the convolution kernel is added pixel by pixel with the feature map output by Block 3. After dropout, the convolution kernels are respectively convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform BN. The concatenated feature maps are convolved with a 1×1 convolution kernel with a dilation rate of 1 and perform BN. Finally, after dropout, a feature map of dimension 128×16×128 is obtained, as shown in Figure 7.
[0096] Deconvolution layer 2: Referring to deconvolution layer 1, 64 3×3 convolution kernels are used to convolve with the feature map output by deconvolution layer 1 and perform batch normalization (BN) with a step size of 2. After dropout, the feature map output by Block 2 is added pixel by pixel. After dropout, the convolution kernels are used with 3×3 kernels with a dilation rate of 1 and 3×3 kernels with a dilation rate of 2, and batch normalization is performed. The concatenated feature maps are convolved with a 1×1 kernel with a dilation rate of 1 and batch normalization is performed. Finally, after dropout, a feature map with a dimension of 256×32×64 is obtained.
[0097] Deconvolution layer 3: Referring to deconvolution layer 1, 32 3×3 convolution kernels are used to convolve with the feature map output by deconvolution layer 2 and perform batch normalization (BN) with a stride of 2. The convolution kernel is then added pixel by pixel to the feature map output by Block 1. The convolution kernels are then convolved with 3×3 kernels with a dilation rate of 1 and 3×3 kernels with a dilation rate of 2 and perform batch normalization (BN). The concatenated feature maps are then convolved with a 1×1 kernel with a dilation rate of 1 and perform batch normalization (BN) to obtain a feature map with a dimension of 512×64×32.
[0098] Standard convolutional layer: convolution is performed with three 1×1 convolution kernels and the feature map output by deconvolution layer 3, resulting in a feature map of dimension 512×64×3, where the three channels correspond to the total number of semantic classes.
[0099] Soft-max layer: performs pixel-level classification on the feature map output by the standard convolutional layer to obtain the probability of each category.
[0100] Step S4: Edge Discriminator Design
[0101] Since there is no open-source, accurate point cloud edge label dataset, traditional multi-task supervised learning algorithms cannot be used to further constrain and learn edge information. Inspired by the image style transfer algorithms GAN (I. Goodfellow, et al., “Generative adversarial nets,” in NIPS, Montreal, Quebec, Canada, 2014.) and CycleGAN (J. Zhu, et al., “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in ICCV, Venice, Italy, 2017, pp. 2223-2232.), this paper designs an edge discriminator to further constrain edge output, thereby improving edge perception and achieving output-level edge enhancement.
[0102] In view of the deficiency of global or local information in the existing discriminators, the present invention combines the Markov discriminator with the global discriminator, as shown in Figure 8, which not only takes into account the local texture information but also ensures global consistency. The discriminator uses the semantic labels predicted by the network and the edge feature maps extracted by the true semantic labels (respectively represented as B pred and B GT ) as output and aims to distinguish them. The first three layers are feature sharing layers, each of which is composed of a 4×4 convolution kernel, BN and LeakyReLU in series, which are used to learn common underlying features; then it is divided into two branches: the first branch ends with a fully connected layer and Sigmoid activation to obtain a binary classification output, taking into account the global information of the entire image to determine whether it is true or false; the second branch uses the same Markov discriminator as CycleGAN, and its output is a two-dimensional matrix. Each value of the matrix represents the judgment of the local area of the input to enhance the attention to local texture details. Loss L DB Calculated by two branches:
[0103] in, They are the global discriminator and Markov discriminator losses respectively.
[0104] Step S5: Loss function design
[0105] In most publicly available datasets, there is an extreme imbalance between classes. For example, in autonomous driving scenarios, point clouds belonging to pedestrians are much less common than those belonging to roads or vehicles. This imbalance makes the network more sensitive to classes with more sample points in the training set, resulting in poor segmentation results. Therefore, this paper weights the traditional cross-entropy loss by the square root of the class frequency to enhance the contribution of classes that are less common in the dataset to the loss function.
[0106] In addition, since IoU (intersection-over-union) is a discrete and non-differentiable metric, it cannot be directly used as the network loss. Lovasz-softmax loss (Berman, M.; Triki, A.R.; Blaschko, M.B. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4413–4421) is a Lovasz smoothed extension of IoU loss. When IoU is used as the evaluation metric, it is more suitable for multi-class segmentation tasks and has been shown to outperform traditional cross-entropy loss methods for small objects.
[0107] Therefore, the present invention combines the weighted cross entropy loss, Lovasz-Softmax and the edge discriminator loss of step S4 as the final loss function to maximize the IoU scores of various categories, as shown in the following formula:
[0108] Among them, v i It is the number of points of various types. and P 3D (i) are the true value and the corresponding predicted probability, J represents the Lovasz expansion of IoU, e(c k ) is class c k The error vector.
[0109] Step S6: Network model training and point cloud semantic segmentation
[0110] Sub-step 1: Point cloud training
[0111] The present invention uses the SGD optimizer to iteratively train the network parameters. The optimizer's initial learning rate is set to 0.01, the decay rate is set to 0.1 / 20K iterations, the dropout rate is set to 0.5, the batch size is set to 8, and the training epoch is set to 150.
[0112] Due to the lack of publicly available datasets for unstructured environments, we pre-trained the model using a structured dataset (e.g., KITTI) to obtain pre-trained weights. This was then followed by further training using an unstructured dataset (e.g., RELLIS-3D) to obtain network weights. To increase the amount of training data, we horizontally flipped the original point cloud, added random noise, and randomly rotated it around the z-axis by [-5°, 5°].
[0113] Sub-step 2: Point cloud semantic segmentation
[0114] Using the network weights obtained in sub-step 1, the collected radar point cloud is tested online to obtain the semantic segmentation results of the unstructured environment 3D point cloud.
[0115] In summary, the method proposed in the present invention uses spherical projection, which can generate more compact and dense data than other projection methods (such as bird's-eye view projection) while reducing index and calculation time, which is beneficial to the extraction of point cloud features by deep neural networks; the addition of the edge encoder not only makes up for the missing edge structure information of the backbone network and realizes edge enhancement at the input end, but also provides rich intermediate features for subsequent feature-level edge enhancement; the addition of the edge attention fusion module automatically allocates multi-task learning weights through supervised learning, so that the feature information between the semantic encoder and the edge encoder is deeply fused; the use of the edge discriminator uses an end-to-end method to realize output-level edge enhancement without the need for additional precise edge labels, thereby improving the edge perception ability and the robustness of the segmentation algorithm.
[0116] It should be noted that the above content merely illustrates the technical idea of the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.
Claims
1. A semantic segmentation method of 3D point cloud in unstructured environment based on multi-level edge enhancement. It is characterized in that The steps include: S1, semantic-edge dual encoding structure design based on spherical projection: Spherical projection is used to preprocess the radar point cloud to generate a rendered image, and a semantic-edge dual encoding structure based on residual dilated convolution is established to extract semantic and edge features of different scales at the same time; The semantic encoder and edge encoder adopt the same structural design, the convolutional layers included are all dilated convolutional layers, the maximum pooling operation is used to downsample the feature map, and the dropout layer is only inserted in the center layer of the encoder; S2, edge attention fusion module design: semantics and edge features of different scales are deeply integrated through the edge attention fusion module; The edge attention fusion module is: Among them, F out is the fused feature; F in_main 、F in_boundary Represent the intermediate features of the backbone semantic encoder and edge encoder respectively; It means element-by-element addition; Represents element-wise multiplication; S3, decoder design: Design a deconvolution layer with learnable parameters to decouple the deeply compressed features in the feature map layer by layer. The convolution kernel size of the deconvolution layer is 3×3 and the step size is 2, so that the feature map after upsampling has the same resolution as the input. The number of deconvolution layers is equal to the number of pooling layers. S4, edge discriminator design: the edge discriminator combines the Markov discriminator with the global discriminator to constrain the edge output; S5, loss function design: The loss function is a combination of weighted cross entropy loss, Lovasz-Softmax and edge discriminator loss in step S4, specifically: Among them, v i It is the number of points of various types. and P 3D (i) are the true value and the corresponding predicted probability, J represents the Lovasz expansion of IoU, e(c k ) is class c k The error vector of S6, network model training and point cloud semantic segmentation: The SGD optimizer is used to iteratively train the network parameters to obtain the network weights. Based on the network weights, the collected radar point clouds are tested online to obtain the semantic segmentation results of the unstructured environment 3D point clouds.
2. The method for semantic segmentation of three-dimensional point clouds in unstructured environments based on multi-level edge enhancement according to claim 1, Features: The preprocessing of the radar point cloud in step S1 is specifically as follows: projecting each point in the point cloud in the Euclidean coordinate system into the spherical coordinate system to obtain a two-dimensional rectangular coordinate system; taking the radar center as the origin and the carrier forward direction as the OX axis, the azimuth of each point is established according to the right-hand rule. The calculation formula of the vertex angle θ is as follows: Among them, (x, y, z) is the coordinate of each point in the point cloud in the Euclidean coordinate system; the angle is differentiated to obtain a two-dimensional rectangular coordinate system: Among them, xi and xj are the horizontal and vertical coordinates of the two-dimensional rectangular coordinate system. Δθ corresponds to The resolution of θ.
3. The method for semantic segmentation of three-dimensional point cloud in unstructured environment based on multi-level edge enhancement as claimed in claim 2, Features: In the semantic-edge dual encoding structure based on residual dilated convolution in step S1, the number of convolution layers ranges from 6 to 12; the number of pooling layers ranges from 2 to 4; the number of convolution kernels ranges from 2 4 ~2 10 ; The range of expansion rate is 2 0 ~2 4 .
4. The method for semantic segmentation of three-dimensional point cloud in an unstructured environment based on multi-level edge enhancement according to claim 2 or 3, Features: The structure of the optimal network architecture in step S1 is: Block 1: Use 32 1×1 convolution kernels with a dilation rate of 1 to convolve with the 512×64×3 input sample to obtain sub-feature map 1, and then convolve with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and perform batch normalization. The feature maps are concatenated and convolved with 1×1 convolution kernels with a dilation rate of 1. The kernel is convolved and batch normalized, and then added pixel by pixel with sub-feature map 1 to obtain a feature map with a dimension of 512×64×32; Pooling layer 1: The feature map output by Block 2 is downsampled using a 2×2 maximum pooling layer with a step size of 2, resulting in a feature map of dimension 256×32×32; Block2: Referring to Block1, 64 1×1 convolution kernels with a dilation rate of 1 are used to convolve the feature map output by pooling layer 1 to obtain sub-feature map 2. Then, convolution kernels with a dilation rate of 3×3 and a dilation rate of 1 and a dilation rate of 2 are used to perform batch normalization. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and batch normalization. Then, the sub-feature map 2 is added pixel by pixel to obtain a feature map with a dimension of 256×32×64. Pooling layer 2: After the feature map output by Block 2 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a feature map with a dimension of 128×16×64; Block3: Referring to Block1, 128 1×1 convolution kernels with a dilation rate of 1 are used to convolve with the feature map output by pooling layer 2 to obtain sub-feature map 3. Then, convolution is performed with convolution kernels with a dilation rate of 3×3 and a dilation rate of 1 and 3×3 and a dilation rate of 2 respectively, and batch normalization is performed. After the feature maps are concatenated, they are convolved with a convolution kernel with a dilation rate of 1 and batch normalization is performed. Then, pixel-by-pixel addition is performed with sub-feature map 3 to obtain a feature map with a dimension of 128×16×128. Pooling layer 3: After the feature map output by Block 3 is dropped out, it is downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a feature map with a dimension of 64×8×128; Block4: Referring to Block1, 256 1×1 convolution kernels with a dilation rate of 1 are used to convolve with the feature map output by pooling layer 3 to obtain sub-feature map 4. Then, convolution with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 are performed respectively and batch normalization is performed. After the feature maps are concatenated, they are convolved with a 1×1 convolution kernel with a dilation rate of 1 and batch normalization is performed. Then, pixel-by-pixel addition is performed with sub-feature map 4 to obtain a feature map with a dimension of 64×8×256. Each convolution operation in the optimal network architecture is activated by the ReLU function.
5. The method for semantic segmentation of three-dimensional point cloud in unstructured environment based on multi-level edge enhancement as claimed in claim 4, Features: The multi-scale channel attention module in step S2 is specifically: Among them, σ represents the Sigmoid activation function; G(F) and L(F) represent the global channel context information and local channel context information respectively.
6. The method for semantic segmentation of three-dimensional point cloud in an unstructured environment based on multi-level edge enhancement as claimed in claim 5, Features: The structure of the decoder in step S3 is: Deconvolution layer 1: 128 3×3 convolution kernels are used to perform convolution with the feature map fused after Block4 and batch normalization with a step size of 2. After dropout, the feature map is added pixel by pixel with the feature map output by Block3. After dropout, the feature maps are convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and batch normalization. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and batch normalization. Finally, after dropout, a feature map with a dimension of 128×16×128 is obtained. Deconvolution layer 2: Referring to deconvolution layer 1, 64 3×3 convolution kernels are used to convolve with the feature map output by deconvolution layer 1 and batch normalized with a step size of 2. After dropout, the feature map is added pixel by pixel with the feature map output by Block 2. After dropout, the feature map is convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and batch normalized. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and batch normalized. Finally, after dropout, a feature map with a dimension of 256×32×64 is obtained. Deconvolution layer 3: Referring to deconvolution layer 1, 32 3×3 convolution kernels are used to convolve with the feature map output by deconvolution layer 2 and perform batch normalization, with a step size of 2. The feature map output by Block 1 is added pixel by pixel, and then convolved with 3×3 convolution kernels with a dilation rate of 1 and 3×3 convolution kernels with a dilation rate of 2 and batch normalization. The feature maps are concatenated and convolved with a 1×1 convolution kernel with a dilation rate of 1 and batch normalization is performed. After quantization, a feature map with a dimension of 512×64×32 is obtained. Standard convolution layer: convolve the feature map output by deconvolution layer 3 with three 1×1 convolution kernels to obtain a feature map with a dimension of 512×64×3. The three channels correspond to the total number of semantic classes. Soft-max layer: Perform pixel-level classification on the feature map output by the standard convolutional layer to obtain the probability of each category.
7. The method for semantic segmentation of three-dimensional point cloud in an unstructured environment based on multi-level edge enhancement according to claim 6, Features: The edge discriminator in step S4 uses the semantic labels predicted by the network and the edge feature maps extracted by the true semantic labels (respectively represented as B pred and B GT ) as output, and distinguish semantic labels from true value semantic labels; the first three layers are feature sharing layers, each of which is composed of a 4×4 convolution kernel, batch normalization processing and LeakyReLU in series, which is used to learn common underlying features; It then splits into two branches: the first branch ends with a fully connected layer and Sigmoid activation to obtain a binary classification output, taking into account the global information of the entire image to determine whether it is real or fake; the second branch uses the same Markov discriminator as CycleGAN, and its output is a two-dimensional matrix, each value of the matrix represents a judgment on the local area of the input to enhance the attention to local texture details; the loss L DB Calculated by two branches: in, They are the global discriminator and Markov discriminator losses respectively.
8. The method for semantic segmentation of three-dimensional point cloud in an unstructured environment based on multi-level edge enhancement as claimed in claim 7, Features: In the step S6 network model training, the original point cloud is horizontally flipped, random noise is added, and randomly rotated around the z-axis by [-5°, 5°].
Citation Information
Patent Citations
Semantic image segmentation method and system based on edge enhancement
CN111462126A
Non-structured environment point cloud semantic segmentation method based on cross-modal semantic enhancement
CN114549537A
Unstructured environment three-dimensional point cloud semantic segmentation method based on multi-level edge enhancement
CN116503602A
Low level sensor fusion based on lightweight semantic segmentation of 3D point clouds
US20220269900A1
System and method for proposal-free and cluster-free panoptic segmentation system of point clouds
WO2023036228A1
Cited By
AI-driven cement hydration microstructure evolution method and online virtual experiment platform
CN119180222A
Building deformation analysis method and system based on three-dimensional laser scanning and deep learning
CN119197368A
Editable condition printing image generation method
CN119227549A
Tooth image segmentation method based on tooth root edge weighted loss
CN119229129A
Wood tree species identification method based on multi-feature fusion
CN119229301A
