A semantic segmentation method for unstructured 3D point clouds based on multi-level edge enhancement

Abstract

This invention discloses a semantic segmentation method for 3D point clouds in unstructured environments based on multi-level edge enhancement. First, an efficient semantic-edge dual-encoding structure based on spherical projection is constructed. Second, an edge attention fusion module is designed to perform deep and automatic fusion of the intermediate features of the dual encoders. Third, a decoder is designed to restore the feature map to its original input size. Next, an edge discriminator is proposed to further improve the edge recognition capability at the output. Then, a loss function is designed. Finally, the network is trained using a sample set to obtain network parameters, thereby achieving efficient and reliable semantic segmentation of 3D point clouds in unstructured environments. This invention combines the advantages of traditional methods and deep learning algorithms to enhance edge perception capabilities, achieving efficient and high-precision 3D semantic segmentation of unstructured environments.

Description

Technical Field

[0001] This invention belongs to the field of intelligent vehicle environmental perception technology, and mainly relates to a semantic segmentation method for unstructured 3D point cloud of environment based on multi-level edge enhancement. Background Technology

[0002] Efficient and reliable environmental perception and understanding are crucial for autonomous vehicles and robots in intelligent transportation systems, and are necessary prerequisites for localization, decision-making, and navigation. Semantic segmentation, as an important means to achieve refined environmental understanding, has become a research hotspot in recent years and has received widespread attention. LiDAR, as a common environmental perception sensor, is almost unaffected by changes in light and weather, and can provide accurate distance and geometric structure information, making it widely used in various scenarios. Semantic segmentation algorithms based on LiDAR are also constantly evolving.

[0003] Existing LiDAR-based methods are mostly limited to small-scale indoor scenes or typical urban traffic scenarios. Research on complex and variable unstructured environments (such as off-road environments and emergency rescue scenarios) is limited, the technology is still immature, and various problems exist. Methods based on traditional feature extraction (radial and lateral features) have achieved good detection results for obstacle and passable area segmentation; however, these methods rely on small-scale point cloud feature extraction algorithms, which are inefficient in large-scale unstructured environments. Furthermore, they are only designed for specific unstructured road scenarios (such as specific off-road or rural road scenarios). When applied to complex and variable unstructured scenarios, the limited geometric feature descriptions make it difficult for these methods to achieve ideal segmentation results. With the development of supervised and semi-supervised learning, some studies have used rich urban traffic scene datasets and a small number of labeled unstructured samples to achieve scene understanding in off-road scenarios. However, unstructured scenes lack structural features such as lanes, roads, and guardrails, and the boundaries of drivable areas are blurred and have diverse textures. At the same time, due to the influence of terrain, shrubs, and other vegetation, obstacle features are complex and varied, and there are cases of occlusion and missing parts. These sparse and complex environmental features lead to the inaccurate differentiation of the edges between obstacles, drivable areas, and background by existing algorithms, making it difficult to directly apply them to complex and varied unstructured environments. Summary of the Invention

[0004] This invention addresses the problem that existing methods struggle to clearly and accurately capture the edges between obstacles, passable areas, and the background in complex and ever-changing unstructured scenes, resulting in inaccurate real-time segmentation. It provides a multi-level edge enhancement-based method for 3D point cloud semantic segmentation in unstructured environments. The method comprises six steps: a semantic-edge dual-encoding structure based on spherical projection, an edge attention fusion module, a decoder, an edge discriminator, a loss function, and network model training for point cloud semantic segmentation. First, an efficient semantic-edge dual-encoding structure based on spherical projection is constructed. Second, an edge attention fusion module is designed to perform deep and automatic fusion of intermediate features from the dual encoders. Third, a decoder is designed to restore the feature map to its original input size. Next, an edge discriminator is proposed to further improve edge recognition capabilities at the output. Then, a loss function is designed. Finally, the network is trained using a sample set to obtain network parameters, thereby achieving efficient and reliable 3D point cloud semantic segmentation in unstructured environments. This invention combines the advantages of traditional methods and deep learning algorithms to enhance edge perception, achieving efficient and high-precision 3D semantic segmentation in unstructured environments.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a semantic segmentation method for unstructured 3D point clouds based on multi-level edge enhancement, comprising the following steps:

[0006] S1, Design of semantic-edge dual encoding structure based on spherical projection: The radar point cloud is preprocessed using spherical projection to generate a rendered image, and a semantic-edge dual encoding structure based on residual dilated convolution is established. At the same time, semantic and edge features at different scales are extracted. The semantic encoder and the edge encoder adopt the same structural design. All the convolutional layers included are dilated convolutional layers. Max pooling is used to downsample the feature map, and dropout layers are inserted only in the central layer of the encoder.

[0007] S2, Edge Attention Fusion Module Design: The edge attention fusion module deeply fuses semantics and edge features at different scales; the edge attention fusion module is as follows:

[0008]

[0009] Among them, F out Features after fusion; F in_main F in_boundary These represent the intermediate features of the backbone semantic encoder and the edge encoder, respectively. This indicates element-wise addition; This indicates element-wise multiplication;

[0010] S3, Decoder Design: Design a learnable deconvolution layer to decouple the deeply compressed features in the feature map layer by layer. The kernel size of the deconvolution layer is 3×3 and the stride is 2, so that the upsampled feature map has the same resolution as the input. The number of deconvolution layers is equal to the number of pooling layers.

[0011] S4, Edge Discriminator Design: The edge discriminator combines a Markov discriminator with a global discriminator to constrain edge output;

[0012] S5, Loss Function Design: The loss function is a combination of weighted cross-entropy loss, Lovasz-Softmax, and the edge discriminator loss from step S4, specifically:

[0013]

[0014] Among them, v i It refers to the points for each type of item. and P 3D (i) represent the true value and the corresponding predicted probability, respectively, where J denotes the Lovasz extension of IoU, e(c k ) is class C k The error vector;

[0015] S6, Network Model Training and Point Cloud Semantic Segmentation: The SGD optimizer is used to iteratively train the network parameters to obtain the network weights. Based on the network weights, the collected radar point clouds are tested online to obtain the semantic segmentation results of the unstructured environment 3D point clouds.

[0016] As an improvement to the present invention, the preprocessing of the radar point cloud in step S1 specifically involves: projecting each point in the point cloud from the Euclidean coordinate system to the spherical coordinate system to obtain a two-dimensional rectangular coordinate system; establishing the azimuth angle of each point according to the right-hand rule, with the radar center as the origin and the forward direction of the carrier as the OX axis. The formula for calculating the vertex angle θ is as follows:

[0017]

[0018] Where (x, y, z) are the coordinates of each point in the point cloud in the Euclidean coordinate system; by differentiating their angles, a two-dimensional Cartesian coordinate system is obtained:

[0019]

[0020] Where xi and xj are the x and y coordinates of a two-dimensional rectangular coordinate system. Δθ respectively correspond to The resolution of θ.

[0021] As an improvement of the present invention, in the semantic-edge dual-coding structure based on residual dilated convolution in step S1, the number of convolutional layers ranges from 6 to 12; the number of pooling layers ranges from 2 to 4; and the number of convolutional kernels ranges from 2. 4 ~2 10 The expansion rate ranges from 2. 0 ~2 4 .

[0022] As an improvement to the present invention, the structure of the optimal network architecture in step S1 is as follows:

[0023] Block 1: Convolve 32 1×1 kernels with an expansion ratio of 1 with the 512×64×3 input samples to obtain sub-features. Figure 1 Then, convolutions are performed with 3×3 kernels with an expansion rate of 1 and 3×3 kernels with an expansion rate of 2, respectively, followed by batch normalization. The concatenated feature maps are then convolved with a 1×1 kernel with an expansion rate of 1, followed by batch normalization, and finally convolved with the sub-features. Figure 1 By performing pixel-by-pixel summation, a feature map with dimensions of 512×64×32 is obtained;

[0024] Pooling layer 1: The feature map output by Block2 is downsampled using a 2×2 max pooling layer with a stride of 2, resulting in a feature map with dimensions of 256×32×32;

[0025] Block 2: Referring to Block 1, 64 1×1 convolutional kernels with an expansion ratio of 1 are convolved with the feature map output from pooling layer 1 to obtain sub-features. Figure 2 Then, convolutions are performed with 3×3 kernels with an expansion rate of 1 and 3×3 kernels with an expansion rate of 2, respectively, followed by batch normalization. The concatenated feature maps are then convolved with a 1×1 kernel with an expansion rate of 1, followed by batch normalization, and finally convolved with the sub-features. Figure 2 By performing pixel-by-pixel summation, a feature map with dimensions of 256×32×64 is obtained;

[0026] Pooling layer 2: After the feature map output by Block2 is dropped out, it is downsampled by a 2×2 max pooling layer with a stride of 2 to obtain a feature map with dimensions of 128×16×64.

[0027] Block 3: Referring to Block 1, convolve 128 1×1 convolutional kernels with an expansion ratio of 1 with the feature map output from pooling layer 2 to obtain sub-features. Figure 3 Then, convolutions are performed with 3×3 kernels with an expansion rate of 1 and 3×3 kernels with an expansion rate of 2, respectively, followed by batch normalization. The concatenated feature maps are then convolved with a 1×1 kernel with an expansion rate of 1, followed by batch normalization, and finally convolved with the sub-features. Figure 3By performing pixel-by-pixel summation, a feature map with dimensions of 128×16×128 is obtained;

[0028] Pooling layer 3: After the feature map output by Block 3 is dropped out, it is downsampled by a 2×2 max pooling layer with a stride of 2, resulting in a feature map with dimensions of 64×8×128.

[0029] Block 4: Referring to Block 1, convolve 256 1×1 convolutional kernels with an expansion rate of 1 with the feature map output from pooling layer 3 to obtain sub-features. Figure 4 Then, convolutions are performed with 3×3 kernels with an expansion rate of 1 and 3×3 kernels with an expansion rate of 2, respectively, followed by batch normalization. The concatenated feature maps are then convolved with a 1×1 kernel with an expansion rate of 1, followed by batch normalization, and finally convolved with the sub-features. Figure 4 By performing pixel-by-pixel summation, a feature map with dimensions of 64×8×256 is obtained;

[0030] Each convolution operation in the optimal network architecture is activated by the ReLU function.

[0031] As an improvement of the present invention, the multi-scale channel attention module in step S2 is specifically as follows:

[0032]

[0033] Where σ represents the Sigmoid activation function; G(F) and L(F) represent global channel context information and local channel context information, respectively.

[0034] As another improvement of the present invention, the structure of the decoder in step S3 is as follows:

[0035] Deconvolutional layer 1: Convolves 128 3×3 convolutional kernels with the feature map output from Block 4 and performs batch normalization with a stride of 2. After dropout, it is added pixel by pixel with the feature map output from Block 3. After dropout, it is convolved with 3×3 kernels with a dilation rate of 1 and 3×3 kernels with a dilation rate of 2 respectively and then batch normalized. The feature maps are concatenated and then convolved with 1×1 kernels with a dilation rate of 1 and batch normalized. Finally, after dropout, a feature map with dimensions of 128×16×128 is obtained.

[0036] Deconvolutional layer 2: Referring to deconvolutional layer 1, 64 3×3 convolutional kernels are used to convolve with the feature map output by deconvolutional layer 1 and batch normalization is performed. The stride is 2. After dropout, it is added pixel by pixel with the feature map output by Block 2. After dropout, it is convolved with 3×3 convolutional kernels with a dilation rate of 1 and 3×3 convolutional kernels with a dilation rate of 2 respectively and batch normalization is performed. The feature maps are concatenated and then convolved with 1×1 convolutional kernels with a dilation rate of 1 and batch normalization is performed. Finally, after dropout, a feature map with a dimension of 256×32×64 is obtained.

[0037] Deconvolutional layer 3: Referring to deconvolutional layer 1, 32 3×3 convolutional kernels are used to convolve with the feature map output by deconvolutional layer 2 and batch normalization is performed. The stride is 2. The feature map output by Block 1 is added pixel by pixel. Then, it is convolved with 3×3 convolutional kernels with a dilation rate of 1 and 3×3 convolutional kernels with a dilation rate of 2 respectively and batch normalization is performed. The feature maps are concatenated and then convolved with 1×1 convolutional kernels with a dilation rate of 1 and batch normalization is performed. Finally, a feature map with a dimension of 512×64×32 is obtained.

[0038] Standard convolutional layer: Convolves three 1×1 convolutional kernels with the feature map output by the deconvolutional layer 3 to obtain a feature map with dimensions of 512×64×3, with the three channels corresponding to the total number of semantic classes;

[0039] Soft-max layer: Performs pixel-level classification on the feature map output by the standard convolutional layer to obtain the probability of each class.

[0040] As another improvement of the present invention, the edge discriminator in step S4 uses the semantic labels predicted by the network and the edge feature maps extracted from the ground truth semantic labels (referred to as B respectively). pred and B GT The output is a semantic label, distinguishing between semantic labels and ground truth semantic labels. The first three layers are feature-sharing layers, each consisting of a 4×4 convolutional kernel, batch normalization, and LeakyReLU concatenation, used to learn common low-level features. Then it splits into two branches: the first branch terminates in a fully connected layer with Sigmoid activation, yielding a binary classification output that considers global information from the entire image to determine whether it is true or false; the second branch uses the same Markov discriminator as CycleGAN, its output being a two-dimensional matrix where each value represents a judgment on a local input region, enhancing attention to local texture details; the loss L... DB Calculated from two branches:

[0041]

[0042]

[0043] in, These are the losses for the global discriminator and the Markov discriminator, respectively.

[0044] As a further improvement of the present invention, in step S6, network model training, the original point cloud is horizontally flipped, random noise is added, and it is randomly rotated around the z-axis [-5°, 5°].

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The spherical projection method used in this invention reduces indexing and computation time while being more conducive to the extraction of point cloud features.

[0047] (2) The present invention uses dilated convolution and residual structure, which not only reduces the complexity of the network structure, but also can extract more effective information from the fusion features of different depths of the network, thereby ensuring that the network can simultaneously achieve both accuracy and real-time performance.

[0048] (3) The edge encoder added in this invention not only makes up for the missing edge structure information in the backbone network and realizes edge enhancement at the input end, but also provides rich intermediate features for subsequent feature-level edge enhancement.

[0049] (4) The edge attention fusion module used in this invention automatically allocates multi-task learning weights through supervised learning, so as to achieve deep fusion of feature information between semantic encoder and edge encoder and realize feature-level edge enhancement.

[0050] (5) The edge discriminator used in this invention achieves edge enhancement at the output level through an end-to-end method without the need for additional precise edge labels, which further improves the algorithm’s ability to perceive edges and its robustness to complex and variable scenarios. Attached Figure Description

[0051] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0052] Figure 2 This is a schematic diagram of a point cloud segmentation network based on multi-level edge enhancement in the semantic-edge dual coding structure based on residual dilated convolution in step S1 of the present invention.

[0053] Figure 3 This is a schematic diagram of the spherical projection in step S1 of the method of the present invention, which is based on the semantic-edge dual coding structure design of spherical projection.

[0054] Figure 4 This is a schematic diagram of the Block in the semantic-edge dual-encoding structure design based on spherical projection in step S1 of the method of the present invention;

[0055] Figure 5This is a schematic diagram of the edge attention fusion module in step S2 of the method of the present invention;

[0056] Figure 6 This is a schematic diagram of the multi-scale channel attention module in step S2 of the method of the present invention;

[0057] Figure 7 This is a schematic diagram of the deconvolution layer in step S3 of the method of the present invention;

[0058] Figure 8 This is a schematic diagram of the edge discriminator in step S4 of the method of the present invention. Detailed Implementation

[0059] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0060] Example 1

[0061] A semantic segmentation method for unstructured 3D point clouds based on multi-level edge enhancement, such as Figure 1 As shown, the specific steps include the following:

[0062] Step S1: Design of a semantic-edge dual-encoding structure based on spherical projection

[0063] Sub-step 1: Radar point cloud preprocessing

[0064] LiDAR point cloud data is massive, and due to uneven environmental sampling, the density of the point cloud varies greatly, making dense point search and indexing operations relatively time-consuming. Common solutions include projecting the point cloud into a two-dimensional image space or using rasterization to convert each frame of LiDAR data into a pseudo-image to generate the structured matrix form required for standard convolution operations. These methods improve data processing efficiency but lose a large amount of the original point cloud and its geometric features, which is detrimental to feature extraction in subsequent networks.

[0065] To ensure the real-time performance and effectiveness of semantic segmentation in unstructured environments, this invention employs spherical projection to preprocess radar point clouds, generating more compact and denser rendered images. Specifically, as follows... Figure 3 As shown.

[0066] The lidar data point coordinate system OXYZ is established with the lidar center as the origin and the vehicle's forward direction as the OX axis, according to the right-hand rule. Azimuth angle. The formula for calculating the vertex angle θ is as follows:

[0067]

[0068] Where (x, y, z) are the coordinates of each point in the point cloud in the Euclidean coordinate system. For each point in the point cloud, its coordinates can be calculated from its (x, y, z) coordinates. This involves projecting points from the Euclidean coordinate system onto a spherical coordinate system. This spherical coordinate system is actually a two-dimensional coordinate system; for ease of understanding, its angles are differentiated to obtain a two-dimensional Cartesian coordinate system.

[0069]

[0070] Where xi and xj are the x and y coordinates of a two-dimensional rectangular coordinate system. Δθ respectively correspond to The resolution of θ.

[0071] This spherical projection transformation allows any point (x, y, z) in Euclidean space to be projected onto a point (xi, xj) in a two-dimensional coordinate system. This invention extracts four features from each point in the point cloud: Euclidean coordinates (x, y, z, r), where... This results in a tensor of size (H, W, C), where H is the width of the rendered image after spherical projection transformation, W is the length of the image, and C = 4. Taking the KITTI dataset as an example, this dataset uses a 64-line LiDAR, so H = 64; in the horizontal direction, due to limitations of the subsequent discriminator, radar point cloud data at a 90° angle directly in front is used and divided into 512 grids (i.e., 512 points are sampled horizontally), so W = 512.

[0072] Next, the edges are extracted from the generated point cloud tensor (H, W, C) using the Canny operator to obtain the edge tensor (H, W, C).

[0073] Sub-step 2: Design of a semantic-edge dual-encoding structure based on residual dilated convolution

[0074] Existing fast segmentation networks (such as SalsaNext) employ deep and complex structures to achieve high-precision multi-semantic segmentation in structured scenes, neglecting the need for design in unstructured environments and overlooking the significant impact of edge information on segmentation results. These shortcomings in algorithm structure and principle make them difficult to apply directly to unstructured environments. Therefore, this invention proposes a semantic-edge dual-encoding network based on residual dilated convolution for unstructured semantic segmentation tasks. This network simultaneously extracts semantic and edge features at different scales, achieving input-level edge enhancement. Specifically, as follows... Figure 2 As shown in the diagram. Since both encoders use the same structural design, the semantic encoder will be used as an example below.

[0075] First, determine the type of convolutional layer, the kernel size, and the stride.

[0076] Generally, the feature representation capability of a network increases with the number of standard convolutional layers. However, stacking more convolutional layers increases computational cost, reduces real-time performance, and increases the risk of overfitting. Conversely, a small number of convolutional layers makes it difficult to extract highly expressive features. Therefore, dilated convolutional layers are introduced in the network design process. Dilated convolutions can greatly expand the receptive field of the network without introducing additional parameters. By properly designing the dilation rate and combining them, the network can learn complete road and obstacle information.

[0077] To avoid introducing a large number of parameters due to excessively large convolution kernels, the dilated convolutional layers are designed with kernel sizes of 1×1 and 3×3, and the stride is set to 1 for both.

[0078] Secondly, determine the pooling layer type, sampling size, and step size.

[0079] To reduce the computational cost of the network and further increase its receptive field, enabling it to detect larger targets (e.g., drivable areas), this invention introduces a pooling layer. Max pooling can preserve the most salient features (e.g., edge features) within the sampling window; therefore, max pooling is used to downsample the feature map, with a sampling size of 2×2 and a stride of 2.

[0080] Next, determine the dropout layer distribution.

[0081] The basic features extracted by lower network layers are edges and corners that are consistent in data distribution. However, compared to structured scenes, the features that distinguish drivable areas, obstacles and backgrounds are relatively sparse in unstructured scenes. Therefore, inserting dropout only in the central layer of the encoder is beneficial for the network to form better high-level features in deeper layers.

[0082] Finally, the different types of network layers mentioned above are combined to determine the number of layers, the number of convolutional kernels, and the dilation rate of the dilated convolutions.

[0083] Traditionally, neural network architecture design has involved researchers tailoring their networks to specific application scenarios and needs, referencing currently recognized high-performing models to create architectures that meet particular requirements. Considering that the task of unstructured environment semantic segmentation involves dividing drivable areas, obstacles, and background (i.e., three-class classification), its architecture should be simpler than most current multi-class semantic segmentation networks designed for autonomous driving. This avoids the need to stack a large number of convolutional layers to meet accuracy and real-time performance requirements. This invention initially determines the range of the number of convolutional layers to be 6–12.

[0084] While pooling layers reduce the computational cost of the network, they also lose some spatial location information. Therefore, their number needs to be controlled within a certain range so that the network can maintain a balance between minimizing computational cost and preserving as much spatial location information as possible. The initial range for the number of pooling layers is determined to be 2 to 4.

[0085] Referring to existing excellent semantic segmentation models, the number of convolutional kernels is mostly set to an integer power of 2. Fewer kernels mean fewer features the network can extract, while a larger number of kernels introduces more parameters, increasing optimization difficulty. To maintain a balance, the initial range for the number of kernels is determined to be 2. 4 ~2 10 .

[0086] The dilation rate of the dilated convolution kernel is generally taken as an integer power of 2. A larger dilation rate results in a larger receptive field, but also introduces more noise and can even produce a "grid effect" (causing discontinuous segmentation or even missed segmentation). Therefore, this invention initially determines the range of dilation rate to be 2. 0 ~2 4 .

[0087] After determining the range of dilation rates, cross-validation was used to select the model architecture that yielded the optimal network performance. During network design, the impact of the types and numbers of convolutional and pooling layers on edge feature extraction was comprehensively considered. Dilated convolutions were introduced, and the dilation rate was designed to achieve a larger receptive field while avoiding the "grid effect." Furthermore, referencing ResNet (He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.) and UNet (Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical imagesegmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.), residual structures were designed for cross-layer connections. This allows the network to extract more effective information from fused features at different depths, enabling it to learn meaningful and complete semantic features while maintaining real-time performance. The optimal network architecture obtained by model selection is as follows (where each convolutional operation is activated by the ReLU function):

[0088] Block 1: Convolve 32 1×1 convolutional kernels with an expansion ratio of 1 with the 512×64×4 input samples to obtain sub-features. Figure 1 Then, convolutions are performed with 3×3 kernels with an expansion rate of 1 and 3×3 kernels with an expansion rate of 2, followed by batch normalization (BN). The feature maps are then concatenated and convolved with a 1×1 kernel with an expansion rate of 1, followed by BN, and finally convolved with the sub-features. Figure 1 Element-wise addition is performed to obtain a feature map with dimensions of 512×64×32, as shown below. Figure 4 As shown;

[0089] Pooling layer 1: The feature map output by Block2 is downsampled using a 2×2 max pooling layer with a stride of 2, resulting in a feature map with dimensions of 256×32×32;

[0090] Block 2: Referring to Block 1, 64 1×1 convolutional kernels with an expansion ratio of 1 are convolved with the feature map output from pooling layer 1 to obtain sub-features. Figure 2 Then, convolve the data with 3×3 kernels with an expansion rate of 1 and 3×3 kernels with an expansion rate of 2, and perform batch normalization (BN). Concatenate the feature maps and convolve them with a 1×1 kernel with an expansion rate of 1, then perform BN again. Finally, convolve the data with the sub-features. Figure 2 By performing pixel-by-pixel summation, a feature map with dimensions of 256×32×64 is obtained;

[0091] Pooling layer 2: After the feature map output by Block2 is dropped out, it is downsampled by a 2×2 max pooling layer with a stride of 2 to obtain a feature map with dimensions of 128×16×64.

[0092] Block 3: Referring to Block 1, convolve 128 1×1 convolutional kernels with an expansion ratio of 1 with the feature map output from pooling layer 2 to obtain sub-features. Figure 3 Then, convolve the data with 3×3 kernels with an expansion rate of 1 and 3×3 kernels with an expansion rate of 2, and perform batch normalization (BN). Concatenate the feature maps and convolve them with a 1×1 kernel with an expansion rate of 1, then perform BN again. Finally, convolve the data with the sub-features. Figure 3 By performing pixel-by-pixel summation, a feature map with dimensions of 128×16×128 is obtained;

[0093] Pooling layer 3: After the feature map output by Block 3 is dropped out, it is downsampled by a 2×2 max pooling layer with a stride of 2, resulting in a feature map with dimensions of 64×8×128.

[0094] Block 4: Referring to Block 1, convolve 256 1×1 convolutional kernels with an expansion rate of 1 with the feature map output from pooling layer 3 to obtain sub-features. Figure 4 Then, convolve the data with 3×3 kernels with an expansion rate of 1 and 3×3 kernels with an expansion rate of 2, and perform batch normalization (BN). Concatenate the feature maps and convolve them with a 1×1 kernel with an expansion rate of 1, then perform BN again. Finally, convolve the data with the sub-features. Figure 4 By adding each pixel, a feature map with dimensions of 64×8×256 is obtained.

[0095] Step S2: Design of the Edge Attention Fusion Module

[0096] Through the dual encoder structure in step S1, low-level and high-level semantic and edge information can be obtained separately. Considering that most existing feature fusion methods rely on direct concatenation or point-by-point addition, it is difficult to balance the texture and edge information of the two independent branches. Therefore, to better fuse the features of these two branches in a controllable and intelligent manner, this invention proposes an edge attention fusion module to achieve feature-level edge enhancement, specifically as follows... Figure 5 As shown.

[0097] The edge attention fusion module can be represented as:

[0098]

[0099] Among them, F out Features after fusion; F in_main F in_boundary These represent the intermediate features of the backbone semantic encoder and the edge encoder, respectively. This indicates element-wise addition; This represents element-wise multiplication; M is a multi-scale channel attention module, specifically as follows: Figure 6 As shown, it can be represented as:

[0100]

[0101] Where σ represents the Sigmoid activation function; G(F) and L(F) represent global channel context information and local channel context information, respectively.

[0102] Considering the number and complexity of network parameters, this embodiment only fuses the intermediate features of semantic branches and edge branches after pooling layer 2 and Block 4.

[0103] Step S3: Decoder Design

[0104] To restore the feature map to its original image size while avoiding the introduction of excessive noise, learnable deconvolution layers are designed to decouple the deeply compressed features in the feature map layer by layer. To avoid introducing too many parameters due to excessively large kernels, the deconvolution layer uses a 3×3 kernel size with a stride of 2, ensuring that the upsampled feature map has the same resolution as the input. Since the purpose of the deconvolution layer is to restore the scaled feature map (caused by downsampling in the pooling layer) to the input image size, the number of deconvolution layers equals the number of pooling layers. The specific design is as follows:

[0105] Deconvolutional layer 1: 128 3×3 convolutional kernels are convolved with the feature map output from Block 4, followed by batch normalization (BN) with a stride of 2. After dropout, the feature map is added pixel-by-pixel to the feature map output from Block 3. After dropout again, it is convolved with 3×3 kernels with a dilation of 1 and 3×3 kernels with a dilation of 2, followed by BN. The concatenated feature maps are then convolved with a 1×1 kernel with a dilation of 1, followed by BN. Finally, after dropout, a feature map with dimensions of 128×16×128 is obtained. (Details follow...) Figure 7 As shown;

[0106] Deconvolutional layer 2: Referring to deconvolutional layer 1, 64 3×3 convolutional kernels are used to convolve with the feature map output by deconvolutional layer 1 and perform BN with a stride of 2. After dropout, the feature map is added pixel by pixel with the feature map output by Block 2. After dropout, it is convolved with 3×3 kernels with a dilation rate of 1 and 3×3 kernels with a dilation rate of 2 and performed BN. The feature maps are concatenated and then convolved with 1×1 kernels with a dilation rate of 1 and performed BN. Finally, after dropout, a feature map with dimensions of 256×32×64 is obtained.

[0107] Deconvolutional layer 3: Referring to deconvolutional layer 1, 32 3×3 convolutional kernels are used to convolve with the feature map output by deconvolutional layer 2 and perform BN, with a stride of 2. The feature map output by Block 1 is added pixel by pixel, and then convolved with 3×3 kernels with a dilation rate of 1 and 3×3 kernels with a dilation rate of 2 respectively and performed BN. The feature maps are concatenated and then convolved with 1×1 kernels with a dilation rate of 1 and performed BN. Finally, a feature map with dimensions of 512×64×32 is obtained.

[0108] Standard convolutional layer: Convolves three 1×1 convolutional kernels with the feature map output by the deconvolutional layer 3 to obtain a feature map with dimensions of 512×64×3, with the three channels corresponding to the total number of semantic classes;

[0109] Soft-max layer: Performs pixel-level classification on the feature map output by the standard convolutional layer to obtain the probability of each class.

[0110] Step S4: Edge Discriminator Design

[0111] Since there is currently no open-source, accurate point cloud edge label dataset, traditional multi-task supervised learning algorithms cannot be used to further constrain and learn edge information. Inspired by image style transfer algorithms GAN (I. Goodfellow, et al., “Generative adversarial nets,” in NIPS, Montreal, Quebec, Canada, 2014.) and CycleGAN (J. Zhu, et al., “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in ICCV, Venice, Italy, 2017, pp. 2223-2232.), this invention designs an edge discriminator to further constrain edge output, thereby improving edge perception capability and achieving output-level edge enhancement.

[0112] To address the shortcomings of existing discriminators in providing global or local information, this invention combines a Markov discriminator with a global discriminator, specifically as follows: Figure 8 As shown, this approach considers both local texture information and ensures global consistency. The discriminator utilizes the semantic labels predicted by the network and the edge feature maps extracted from the ground truth semantic labels (denoted as B, B, and C, respectively). pred and B GT The first three layers are feature-sharing layers, each consisting of a 4×4 convolutional kernel, Batch Normalization (BN), and Leaky ReLU concatenated to learn common low-level features. Then, it splits into two branches: the first branch terminates in a fully connected layer with a Sigmoid activation, producing a binary classification output that considers global information from the entire image to determine whether it is real or fake; the second branch uses the same Markov discriminator as CycleGAN, whose output is a two-dimensional matrix where each value represents a judgment on a local input region, enhancing attention to local texture details. The loss L... DB Calculated from two branches:

[0113]

[0114]

[0115] in, These are the losses for the global discriminator and the Markov discriminator, respectively.

[0116] Step S5: Loss Function Design

[0117] In most publicly available datasets, there is an extreme imbalance between classes. For example, in autonomous driving scenarios, there are far fewer point clouds representing pedestrians than points representing roads or vehicles. This imbalance makes the network more sensitive to classes with more sample points in the training set, resulting in poor segmentation performance. Therefore, this invention weights the traditional cross-entropy loss function with the square root of the class frequency to enhance the effect of less frequent classes on the loss function.

[0118] Furthermore, since loU (intersection-over-union) is a discrete and non-differentiable metric, it cannot be directly used as the network loss. Lovasz-softmax loss (Berman, M.; Triki, AR; Blaschko, MB. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18-23 June 2018; pp. 4413-4421), as a Lovasz smoothing extension of loU loss, is more suitable for multi-class segmentation tasks when loU is used as the evaluation metric, and has been shown to outperform traditional cross-entropy loss methods on small targets.

[0119] Therefore, this invention combines the weighted cross-entropy loss, Lovasz-Softmax, and the edge discriminator loss from step S4 above as the final loss function to maximize the IoU score for each class, as shown in the following formula:

[0120]

[0121] Among them, v i It refers to the points for each type of item. and P 3D (i) represent the true value and the corresponding predicted probability, respectively, where J denotes the Lovasz extension of the loU, and e(c k ) is class C k The error vector.

[0122] Step S6: Network Model Training and Point Cloud Semantic Segmentation

[0123] Sub-step 1: Point cloud training

[0124] This invention employs the SGD optimizer to iteratively train the network parameters. The optimizer's initial learning rate is set to 0.01, the decay rate is set to 0.1 / 20K iterations, the dropout rate is set to 0.5, the batch size is set to 8, and the training epochs are set to 150.

[0125] Due to the lack of publicly available unstructured environment datasets, it is necessary to use structured environment datasets (such as KITTI) for pre-training to obtain pre-trained weights, and then use unstructured environment datasets (such as RELLIS-3D) for further training to obtain network weights. To increase the amount of training data, this invention performs horizontal flipping, adds random noise, and randomly rotates the original point cloud around the z-axis by [-5°, 5°].

[0126] Sub-step 2: Point cloud semantic segmentation

[0127] Using the network weights obtained in sub-step 1, online testing is performed on the collected radar point cloud to obtain the semantic segmentation results of the unstructured environment 3D point cloud.

[0128] In summary, the method proposed in this invention uses spherical projection, which reduces indexing and computation time while generating more compact and dense data compared to other projection methods (such as bird's-eye view projection), thus facilitating the extraction of point cloud features by deep neural networks. The addition of the edge encoder not only compensates for the missing edge structure information in the backbone network, achieving edge enhancement at the input end, but also provides rich intermediate features for subsequent feature-level edge enhancement. The addition of the edge attention fusion module automatically allocates multi-task learning weights through supervised learning, enabling deep fusion of feature information between the semantic encoder and the edge encoder. The use of the edge discriminator achieves edge enhancement at the output level using an end-to-end method without the need for additional precise edge labels, improving edge perception capability and the robustness of the segmentation algorithm.

[0129] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for semantic segmentation of unstructured environment 3D point cloud based on multi-level edge enhancement, characterized in that, Comprising the following steps: Step S1, semantic-edge dual coding structure design based on spherical projection: adopting spherical projection to preprocess the radar point cloud, generating a rendering image, and establishing a semantic-edge dual coding structure based on residual dilated convolution, while extracting semantic and edge features of different scales; The semantic encoder and the edge encoder adopt the same structure design, and the convolution layers included are all dilated convolution layers, the feature maps are down-sampled by using the maximum pooling operation, and a dropout layer is only inserted in the center layer of the encoder; Step S2, edge attention fusion module design: the semantic and edge features of different scales are deeply fused through the edge attention fusion module; The edge attention fusion module is: ； wherein, is the fused feature; , respectively represent the intermediate feature of the backbone semantic encoder and the edge encoder; denotes element-wise addition; denotes element-wise multiplication; Step S3, decoder design: the parameters of the learning-based deconvolution layer are used to decouple the deep compressed features in the feature map layer by layer, the convolution kernel size of the deconvolution layer is 3x3, and the step is 2, so that the feature map after up-sampling is the same as the input resolution, and the number of deconvolution layers is equal to the number of pooling layers; Step S4, edge discriminator design: the edge discriminator combines Markov discriminator and global discriminator to constrain the edge output; Step S5, loss function design: the loss function is a combination of weighted cross-entropy loss, Lovasz-Softmax and the edge discriminator loss in step S4, specifically: ； where are the counts of each class, and are the true values and the corresponding predicted probabilities, respectively, denotes the Lovasz extension of IoU, is the error vector for class . Step S6, network model training and point cloud semantic segmentation: the network parameters are iteratively trained using the SGD optimizer to obtain network weights, and the collected radar point cloud is tested online according to the network weights to obtain the unstructured environment three-dimensional point cloud semantic segmentation result.

2. The multi-level edge enhancement based unstructured environment 3D point cloud semantic segmentation method of claim 1, wherein: The step S1 of preprocessing the radar point cloud specifically comprises: projecting each point in the point cloud in an Euclidean coordinate system into a spherical coordinate system to obtain a two-dimensional rectangular coordinate system; taking the radar center as an origin and a forward direction of the carrier as a Y-axis, establishing a right-hand rule, and calculating an azimuth angle and a zenith angle of each point according to the following formulas. and a zenith angle of each point according to the following formulas. ； wherein, is the coordinate of each point in the point cloud in the Euclidean coordinate system; and the angle is differentiated to obtain a two-dimensional rectangular coordinate system: ； wherein, are the horizontal and vertical coordinates of a two-dimensional rectangular coordinate system, respectively correspond to the resolution.

3. The multi-level edge enhancement based unstructured environment 3D point cloud semantic segmentation method of claim 2, wherein: The number of convolution layer in the semantic-edge dual coding structure based on residual dilated convolution in the step S1 is 6 12; the number of pooling layer is 2 4; the number of convolution kernel is .

4. The multi-level edge enhancement based unstructured environment 3D point cloud semantic segmentation method of claim 2 or 3, wherein: The structure of the optimal network architecture in step S1 is: Block1: 32 1x1, dilated rate 1 convolution kernels are used to convolve with 512x64x3 input samples to obtain sub-feature map 1, then 3x3, dilated rate 1 and 3x3, dilated rate 2 convolution kernels are used to convolve and batch normalization is performed, the feature maps are spliced, 1x1, dilated rate 1 convolution kernel is used to convolve and batch normalization is performed, and then pixel-by-pixel addition is performed with sub-feature map 1 to obtain a feature map with a dimension of 512x64x32; Pooling layer 1: the feature map output by Block2 is down-sampled by using a 2x2 maximum pooling layer with a step of 2 to obtain a feature map with a dimension of 256x32x32; Block2: referring to Block1, 64 1x1, dilated rate 1 convolution kernels are used to convolve with the feature map output by pooling layer 1 to obtain sub-feature map 2, then 3x3, dilated rate 1 and 3x3, dilated rate 2 convolution kernels are used to convolve and batch normalization is performed, the feature maps are spliced, 1x1, dilated rate 1 convolution kernel is used to convolve and batch normalization is performed, and then pixel-by-pixel addition is performed with sub-feature map 2 to obtain a feature map with a dimension of 256x32x64; Pooling layer 2: the feature map output by Block2 is down-sampled by using a 2x2 maximum pooling layer with a step of 2 to obtain a feature map with a dimension of 128x16x64; Block3: refer to Block1, using 128 1x1, dilated rate 1 convolution kernel and the feature map output by the pooling layer 2 to do convolution, get sub-feature map 3, then respectively with 3x3, dilated rate 1 and 3x3, dilated rate 2 convolution kernel do convolution and do batch standardization processing, the feature map is spliced with 1x1, dilated rate 1 convolution kernel and batch standardization processing, then with sub-feature map 3 do pixel by pixel addition, get the feature map of dimension 128x16x128; Pooling layer 3: the feature map output by Block3 is down-sampled by using 2x2 maximum pooling layer after dropout, step is 2, get the feature map of dimension 64x8x128; Block4: refer to Block1, using 256 1x1, dilated rate 1 convolution kernel and the feature map output by the pooling layer 3 to do convolution, get sub-feature map 4, then respectively with 3x3, dilated rate 1 and 3x3, dilated rate 2 convolution kernel do convolution and do batch standardization processing, the feature map is spliced with 1x1, dilated rate 1 convolution kernel and batch standardization processing, then with sub-feature map 4 do pixel by pixel addition, get the feature map of dimension 64x8x256; Each convolution operation in the optimal network architecture is activated by a ReLU function.

5. The multi-level edge enhancement based unstructured environment 3D point cloud semantic segmentation method of claim 4, wherein: The step S2 The multi-scale channel attention module is specifically: ； wherein, denotes a Sigmoid activation function; and denote global and local channel context information, respectively.

6. The multi-level edge enhancement based unstructured environment 3D point cloud semantic segmentation method of claim 5, wherein: The structure of the decoder in step S3 is: Deconvolution layer 1: using 128 3x3 convolution kernel and the feature map output by Block4 after fusion to do convolution and batch standardization processing, step is 2, after dropout, add the feature map output by Block3 pixel by pixel, then after dropout, respectively with 3x3, dilated rate 1 and 3x3, dilated rate 2 convolution kernel do convolution and do batch standardization processing, the feature map is spliced with 1x1, dilated rate 1 convolution kernel and batch standardization processing, finally after dropout, get the feature map of dimension 128x16x128; Deconvolution layer 2: refer to deconvolution layer 1, using 64 3x3 convolution kernel and the feature map output by deconvolution layer 1 to do convolution and batch standardization processing, step is 2, after dropout, add the feature map output by Block2 pixel by pixel, then after dropout, respectively with 3x3, dilated rate 1 and 3x3, dilated rate 2 convolution kernel do convolution and do batch standardization processing, the feature map is spliced with 1x1, dilated rate 1 convolution kernel and batch standardization processing, finally after dropout, get the feature map of dimension 256x32x64; Deconvolution layer 3: refer to deconvolution layer 1, using 32 3x3 convolution kernel and the feature map output by deconvolution layer 2 to do convolution and batch standardization processing, step is 2, add the feature map output by Block1 pixel by pixel, then respectively with 3x3, dilated rate 1 and 3x3, dilated rate 2 convolution kernel do convolution and do batch standardization processing, the feature map is spliced with 1x1, dilated rate 1 convolution kernel and batch standardization processing, finally get the feature map of dimension 512x64x32; Standard convolution layer: use 3 1x1 convolution kernels to do convolution with the feature map output by the deconvolution layer 3, get the feature map with the dimension of 512x64x3, 3 channels correspond to the total number of semantic classes; Soft-max layer: do pixel-level classification on the feature map output by the standard convolution layer, get the probability of each class.

7. The multi-level edge enhancement based unstructured environment 3D point cloud semantic segmentation method of claim 6, wherein: The edge discriminator of the step S4 extracts edge feature maps using semantic labels obtained by network prediction and true value semantic labels and as output, and distinguishes semantic labels and true value semantic labels; the first three layers are feature sharing layers, each of which is composed of a 4x4 convolution kernel, batch normalization processing and LeakyReLU in series, for learning common underlying features; After that, the network is split into two branches: the first branch ends with a fully connected layer with Sigmoid activation, resulting in a binary output that considers the global information of the whole image to determine whether it is real or fake; the second branch uses the same Markov discriminator as CycleGAN, whose output is a two-dimensional matrix, where each value of the matrix represents the judgment of the input local region to enhance the attention to local texture details; the loss is calculated by the two branches: ； ； wherein, , are the global discriminator, Markov discriminator loss, respectively.

8. The multi-level edge enhancement based unstructured environment 3D point cloud semantic segmentation method of claim 7, wherein: The step S6 network model training includes horizontal flipping, adding random noise and random rotation around the z-axis to the original point cloud processing.

Images