A radar segmentation method integrating spatial attention and self-attention transformer network

Abstract

This invention discloses a radar segmentation method that integrates spatial attention and a self-attention transformer network. The original point cloud is injected with instances to obtain instance-enhanced point cloud data. The enhanced point cloud data extracts features from each point, retaining the point with the largest eigenvalue for each voxel as the voxel feature. A sparse tensor is constructed from the voxel features and voxel coordinates. Voxel features are then extracted using a UNET-like network that integrates sparse spatial attention and a Transformer. The network outputs voxel semantic information, which is converted into semantic information for the point cloud and outputs the semantic segmentation results. This method improves the accuracy of the network in predicting edge categories.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation technology, and in particular relates to a radar segmentation method integrating spatial attention and self-attention transformation networks. Background Art

[0002] Environmental perception is a primary challenge for mobile robots and autonomous vehicles, and visual-based environmental perception methods have achieved excellent results. Compared to visual sensors, 3D lidar (LiDAR) can quickly and accurately acquire distance information from the surrounding environment, unaffected by lighting conditions. In scenarios like autonomous driving, semantic segmentation of radar point clouds is a crucial step in environmental perception. Semantic segmentation algorithms can predict the category of each point cloud in the scene, such as road, vehicle, pedestrian, or plant. Accurate segmentation of the surrounding environment is a prerequisite for reliable autonomous driving.

[0003] Currently, there are three different point cloud representation methods. Point-based methods do not require pre-processing of the point cloud, but instead operate directly on the point cloud. Point-based methods can achieve semantic segmentation of point clouds with fewer network parameters, but are very inefficient in extracting local features and cannot achieve high accuracy. Projection-based methods are also the most widely studied methods, with the main projection methods being spherical projection and bird's-eye view projection. Projection-based methods convert irregular, disordered point cloud data into regular two-dimensional projection images, then perform semantic segmentation on the projection images using mature two-dimensional convolution methods, and finally back-project the segmented projection images back into three-dimensional space. However, these methods lose geometric information about scene objects during the three-dimensional to two-dimensional projection process, and cannot achieve high segmentation accuracy. Voxel-based methods divide point clouds into different voxels, but to improve accuracy, the voxel resolution must be continuously increased, which increases memory and computing resource consumption.

[0004] Existing methods primarily focus on network improvements, with little attention paid to the input data itself. However, in real-world datasets, there is a significant imbalance in the number of point clouds of different categories. This can cause features from less-populated categories to be suppressed by features from more-populated categories during training, resulting in poor prediction results for categories with fewer point clouds. Summary of the Invention

[0005] This application proposes a radar segmentation method that integrates spatial attention and self-attention transformation networks to alleviate point cloud category imbalance and improve the accuracy of point cloud semantic segmentation.

[0006] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0007] A radar segmentation method that integrates spatial attention and self-attention transformer networks, including:

[0008] S1. Use the instance injection method to inject the original point cloud into the point cloud to obtain instance-enhanced point cloud data.

[0009] S2. Extract the features of each point from the enhanced point cloud data through an MLP module, and retain the point with the largest eigenvalue for each voxel as the feature of the voxel;

[0010] S3. Construct voxel features and voxel coordinates into a sparse tensor, input the constructed semantic segmentation network for training, output voxel semantic information, and convert it into semantic information of point cloud, and output the semantic segmentation result. The semantic segmentation network includes multiple stages, and each stage includes an encoder and a decoder from the input side to the output side. The encoder of the first stage also includes a linear layer with expanded dimension before it, and the decoder of the first stage also includes a SegHead layer after it; the encoder of each stage combines the asymmetric residual convolution module and the spatial attention mechanism to encode the input features, strengthen the semantic information, and obtain the output features of the encoder; the output features of the encoder of other stages except the last stage are passed to the encoder of the next stage after the downsampling layer; the output features of the encoder of the last stage are passed to the self-attention transform network Transformer; the output features of the encoder of each stage and the output features of the decoder of the next stage are input to the decoder of this stage after the upsampling layer; the output features of the decoder of the first stage are output through the SegHead layer to output the probability of each voxel in the point cloud belonging to each class.

[0011] Furthermore, the input features of the MLP module pass through the linear layer, BatchNorm layer and ReLU activation function from the input side to the output side.

[0012] Furthermore, the input features of the asymmetric residual convolution module pass through two branches. The first branch passes through a 1x3x3 convolution layer, a BatchNorm layer, a ReLU activation function, a 3x1x3 convolution layer, a BatchNorm layer, and a ReLU activation function from the input side to the output side; the second branch passes through a 3x1x3 convolution layer, a BatchNorm layer, a ReLU activation function, a 3x1x3 convolution layer, a BatchNorm layer, and a ReLU activation function from the input side to the output side. The asymmetric residual convolution module adds the above two branches to obtain the output features.

[0013] Furthermore, the input features of the spatial attention mechanism are sequentially passed through a 3x3x3 convolution layer, a BatchNorm layer, and a ReLU activation function to obtain feature 1, and then passed through a 1x1x1 convolution layer, a BatchNorm layer, and a sigmoid activation function to obtain feature 2. The feature 1 and the feature 2 elements are added to feature 3, and feature 3 is multiplied by the spatial attention mechanism input element to obtain the output of the spatial attention mechanism.

[0014] Furthermore, the self-attention transformation network first performs position encoding calculation on the position information to obtain the position relationship features between points, and then fuses it with the feature information. The fused information passes through two layerNorm layers, the self-attention module and the MLP layer, and the jump connection adds the features of the input and self-attention modules; the input features of the self-attention module pass through three linear layers to query, key and value features respectively, and then multiply the query features and key features by the transposed matrix to obtain the attention map features, and finally the attention map features are multiplied by the value features after softmax as the output features of the module.

[0015] Furthermore, the overall network loss function of the semantic segmentation network is represented by the weighted cross entropy loss function L wce , lovasz-softmax loss function L ls and TVLoss loss function L tv Log-linear fusion, where:

[0016] The weighted cross entropy loss function for:

[0017]

[0018] in y i and Represents the true value and predicted value of each category, f i Indicates the frequency of occurrence of the i-th category. In the semantic segmentation of point clouds, it indicates the proportion of the number of i-th category point clouds to the total number of point clouds. i is the weight coefficient for each category; p(y i )and Represents the probability of the true value and predicted value of each category respectively;

[0019] The lovasz-softmax loss function L ls for:

[0020]

[0021]

[0022] Where |C| represents the number of categories, J is a piecewise linear function with a global minimum, and represents the Lovasz expansion of Iou. m(c) represents the error vector of category c. i (c) and y i (c) represents the predicted value and true value of the i-th point cloud of category c respectively;

[0023] The TVLoss loss function L tv for:

[0024]

[0025] in Represents an exclusive OR operation, y and Represent the true value and predicted value of the point cloud respectively. i, j and k represent the three dimensions of the voxel. and Represents the edge errors in three dimensions i, j, and k respectively;

[0026] The overall loss function of the network is:

[0027] L total =L wce +L ls +L tv ;

[0028] Among them, L total Represents the overall loss function. L wce , L ls and L tv They represent the weighted cross entropy loss function, lovasz-softmax loss function and TVLoss loss function respectively.

[0029] Furthermore, the SegHead layer includes a 3x3x3 convolutional layer, a BatchNorm layer and a ReLU activation function from the input side to the output side.

[0030] This application proposes a radar segmentation method that integrates spatial attention and self-attention transformer networks. Using instance injection, it mitigates the imbalanced distribution of dataset categories. To enhance the network's ability to extract both global and local features, a sparse convolution-based spatial attention mechanism is proposed to extract key information from feature maps. The addition of a Transformer module enhances the network's ability to extract global information, expands the network's receptive field, and improves network performance. Finally, a new TV loss is proposed to strengthen the network's supervision of point cloud edges, thereby improving the accuracy of point cloud semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Flowchart of the radar segmentation method that integrates spatial attention and self-attention transformer networks for this application;

[0032] Figure 2 Schematic diagram of the semantic segmentation network structure;

[0033] Figure 3 Schematic diagram of the asymmetric residual module and spatial attention mechanism structure;

[0034] Figure 4 Schematic diagram of the self-attention transformation network. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0036] like Figure 1 As shown in the figure, a radar segmentation method integrating spatial attention and self-attention transformer network is provided, including:

[0037] Step S1: The original point cloud is injected into the instance to obtain instance-enhanced point cloud data.

[0038] During instance injection, we first filter and save instance data from the dataset based on the instance information provided (including six categories: bicycles, motorcycles, trucks, other vehicles, pedestrians, cyclists, and motorcyclists). During training, this saved instance data is injected above the road in each frame of the point cloud. To avoid overlap with other point clouds on the road, the instance injection process uses a spatial conflict detection method. Different probabilities are assigned to instances of different categories based on the number of instances in that category, ensuring that each instance category is evenly injected into the training data.

[0039] Step S2: The enhanced point cloud data is passed through the MLP module to extract the features of each point, and the point with the largest eigenvalue is retained as the feature of each voxel.

[0040] This application can use the MLP module to extract the features of each point, and further retain the point with the largest eigenvalue of each voxel as the feature of the voxel. Figure 1 In the embodiment shown, three MLP modules are used, but only one MLP module may be used.

[0041] Each MLP module includes a linear layer (Linear), a BatchNorm layer (BN), and a ReLU activation function from the input side to the output side.

[0042] The point cloud data passes through three MLP modules in sequence to extract the features of each point, and further retains the point with the largest eigenvalue for each voxel as the feature of the voxel.

[0043] Step S3, construct voxel features and voxel coordinates into a sparse tensor, input the constructed semantic segmentation network for training, output voxel semantic information, and convert it into semantic information of point cloud, and output the semantic segmentation result. The semantic segmentation network includes multiple stages, and each stage includes an encoder and a decoder from the input side to the output side. The encoder of the first stage also includes a linear layer with expanded dimension before the encoder, and the decoder of the first stage also includes a SegHead layer after the decoder; the encoder of each stage combines the asymmetric residual convolution module and the spatial attention mechanism to encode the input features, strengthen the semantic information, and obtain the output features of the encoder; the output features of the encoders of other stages except the last stage are passed to the encoder of the next stage after the downsampling layer; the output features of the encoder of the last stage are passed to the self-attention transform network Transformer; the output features of the encoder of each stage and the output features of the decoder of the next stage are input to the decoder of this stage after the upsampling layer; the output features of the decoder of the first stage output the probability of each voxel in the point cloud belonging to each class through the SegHead layer.

[0044] like Figure 1 As shown in the figure, three MLP layers extract semantic features from the point cloud. Each voxel retains the point cloud feature with the largest eigenvalue as the voxel feature, and a sparse tensor is constructed from the voxel features and voxel coordinates. The sparse tensor is input to the semantic segmentation network, which outputs voxel semantic information, which is converted into semantic information of the point cloud and outputs the semantic segmentation result.

[0045] like Figure 2 As shown, the semantic segmentation network consists of multiple stages ( Figure 2 Each line in the figure is a stage, providing four stages), each stage includes an encoder and a decoder from the input side to the output side, the encoder of the first stage also includes a linear layer with expanded dimension before it, and the decoder of the first stage also includes a SegHead layer; the encoder of each stage combines the asymmetric residual convolution module and the spatial attention mechanism to encode the input features, strengthen the semantic information, and obtain the output features of the encoder; the output features of the encoder of other stages except the last stage are passed to the encoder of the next stage after the downsampling layer; the output features of the encoder of the last stage are passed into the self-attention transformation network Transformer; the output features of the encoder of each stage and the output features of the decoder of the next stage are input into the decoder of this stage after the upsampling layer; the output features of the decoder of the first stage are output through the SegHead layer to output the probability of each voxel in the point cloud belonging to each class.

[0046] In the encoder of the first stage, the output features of the encoder of the first stage are directly passed to the decoder of the first stage. The output features of the encoder of the first stage are also passed to the encoder of the second stage after passing through the downsampling layer. The output features of the encoder of the second stage are directly passed to the decoder of the second stage. The output features of the encoder of the second stage are also passed to the encoder of the third stage after passing through the downsampling layer. The output features of the encoder of the third stage are directly passed to the decoder of the third stage. The output features of the encoder of the third stage are also passed to the encoder of the fourth stage after passing through the downsampling layer. The decoder of the fourth stage passes the output features of the encoder of the fourth stage through the self-attention transformation network Transformer and then passes it to the decoder of the third stage through the upsampling layer. The decoder of the third stage passes the output features of the encoder of the third stage through the self-attention transformation network Transformer. The output features are merged and added to the output features of the fourth-stage decoder for the first time. After the first merging and adding, they are passed to the second-stage decoder through the upsampling layer. The second-stage decoder merges and adds the output features of the second-stage encoder and the output features of the third-stage decoder for the second time. After the second merging and adding, they are passed to the first-stage decoder through the upsampling layer. The first-stage decoder merges and adds the output features of the first-stage encoder and the output features of the second-stage decoder for the third time. After the third merging and adding, the first-stage decoder outputs the output features of the first-stage decoder. The output features of the first-stage decoder are output through SegHead to output the probability of each voxel belonging to each class, and finally converted into the probability of each class of each point cloud.

[0047] like Figure 3 As shown, the input features of the asymmetric residual module in the encoder pass through two branches. The first branch, from the input side to the output side, passes through a 1x3x3 convolutional layer, a BatchNorm layer, a ReLU activation function, a 3x1x3 convolutional layer, a BatchNorm layer, and a ReLU activation function. The second branch, from the input side to the output side, passes through a 3x1x3 convolutional layer, a BatchNorm layer, a ReLU activation function, a 3x1x3 convolutional layer, a BatchNorm layer, and a ReLU activation function. The asymmetric residual convolution module adds these two branches together to obtain the output features.

[0048] The output of the asymmetric residual module serves as the input to the spatial attention mechanism module. The input features of the spatial attention mechanism are sequentially processed from the input side to the output side through a 3x3x3 convolutional layer, a BatchNorm layer, and a ReLU activation function to obtain feature 1. They are then processed through a 1x1x1 convolutional layer, a BatchNorm layer, and a sigmoid activation function to obtain feature 2. Feature 1 and feature 2 are element-wise added to obtain feature 3, which is then element-wise multiplied by the spatial attention mechanism input to obtain the output of the spatial attention mechanism.

[0049] The SegHead layer in this embodiment includes a 3x3x3 convolutional layer, a BatchNorm layer, and a ReLU activation function from the input side to the output side.

[0050] The output features of the fourth stage decoder are passed through the self-attention transformer network and then passed to the third stage decoder through the upsampling layer. Figure 3 In the Transformer module, the position information (coordsN*3) is first encoded (position embedding) to calculate the position relationship features between points, which are then fused with the feature information (featsN*C). The fused information passes through two layerNorm layers, a self-attention module, and an MLP layer. The skip connection adds the features of the input and self-attention modules.

[0051] The self-attention module passes input features through three linear layers to generate query, key, and value features. The attention map features are then multiplied by the transposed matrices of the query and key features. Finally, the attention map features are softmaxed and multiplied by the value features to produce the module's output features.

[0052] After the semantic segmentation network is trained using the training data, the trained semantic segmentation network is used to perform semantic segmentation on the radar point cloud and output the semantic segmentation result, which will not be described in detail here.

[0053] In a specific embodiment, the present application also calculates the overall network loss function when training a semantic segmentation network, performs backpropagation, and updates network parameters. The overall network loss function is a linear fusion of the weighted cross entropy loss function, the lovasz-softmax loss function, and the TVLoss loss function.

[0054] Among them, the weighted cross entropy loss function is:

[0055]

[0056] in y i and Represents the true value and predicted value of each category, f i Indicates the frequency of occurrence of the i-th category. In the semantic segmentation of point clouds, it indicates the proportion of the number of i-th category point clouds to the total number of point clouds. p(y i )and Represents the probability of the true value and predicted value of each category respectively.

[0057] The lovasz-softmax loss function is:

[0058]

[0059]

[0060] Where |C| represents the number of categories, J is a piecewise linear function with a global minimum, represents the Lovasz expansion of Iou, m(c) represents the error vector of category c, and x i (c) and y i (c) represent the predicted value and true value of the i-th point cloud of category c, respectively.

[0061] The TVLoss loss function is:

[0062]

[0063] in Represents the exclusive OR operation, y and They represent the true value and predicted value of the point cloud respectively, and i, j and k represent the three dimensions of the voxel.

[0064] The overall loss function is:

[0065] L total =L wce +L ls +L tv .

[0066] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A radar segmentation method integrating spatial attention and self-attention transformer network, characterized in that The radar segmentation method integrating spatial attention and self-attention transformation network includes: S1. The original point cloud is injected into the instance to obtain instance-enhanced point cloud data. S2. Extract the features of each point from the enhanced point cloud data through an MLP module, and retain the point with the largest eigenvalue for each voxel as the feature of the voxel; S3. Construct voxel features and voxel coordinates into a sparse tensor, input the constructed semantic segmentation network for training, output voxel semantic information, and convert it into semantic information of point cloud, and output the semantic segmentation result. The semantic segmentation network includes multiple stages, and each stage includes an encoder and a decoder from the input side to the output side. The encoder of the first stage also includes a linear layer with expanded dimension before it, and the decoder of the first stage also includes a SegHead layer after it; the encoder of each stage combines the asymmetric residual convolution module and the spatial attention mechanism to encode the input features, strengthen the semantic information, and obtain the output features of the encoder; the output features of the encoders of other stages except the last stage are passed to the encoder of the next stage after the downsampling layer; the output features of the encoder of the last stage are passed to the self-attention transformation network Transformer; the output features of the encoder of each stage and the output features of the decoder of the next stage are input to the decoder of this stage after the upsampling layer; the output features of the decoder of the first stage output the probability of each voxel in the point cloud belonging to each class through the SegHead layer; The SegHead layer includes a 3x3x3 convolutional layer, a BatchNorm layer, and a ReLU activation function from the input side to the output side. The input features of the asymmetric residual module pass through two branches. The first branch passes through a 1x3x3 convolution layer, a BatchNorm layer, a ReLU activation function, a 3x1x3 convolution layer, a BatchNorm layer, and a ReLU activation function from the input side to the output side; the second branch passes through a 3x1x3 convolution layer, a BatchNorm layer, a ReLU activation function, a 3x1x3 convolution layer, a BatchNorm layer, and a ReLU activation function from the input side to the output side. The asymmetric residual convolution module adds the above two branches by elements to obtain the output features. The input features of the spatial attention mechanism are sequentially passed through a 3x3x3 convolution layer, a BatchNorm layer, and a ReLU activation function to obtain feature 1, and then passed through a 1x1x1 convolution layer, a BatchNorm layer, and a sigmoid activation function to obtain feature 2. The feature 1 and the feature 2 are element-wise added to obtain feature 3, and feature 3 is multiplied by the spatial attention mechanism input element to obtain the output of the spatial attention mechanism; The self-attention transformation network first performs position encoding calculation on the position information to obtain the position relationship features between points, and then fuses them with the feature information. The fused information passes through two layerNorm layers, a self-attention module and an MLP layer, and the jump connection adds the input and self-attention module features. The input features of the self-attention module pass through three linear layers to obtain query, key and value features, and then multiply the query features and key features by the transposed matrix to obtain the attention map features. Finally, the attention map features are multiplied by the value features after softmax as the output features of the module.

2. The radar segmentation method integrating spatial attention and self-attention transformation network according to claim 1 is characterized in that The input features of the MLP module pass through the linear layer, BatchNorm layer and ReLU activation function from the input side to the output side.

3. The radar segmentation method integrating spatial attention and self-attention transformation network according to claim 1 is characterized in that The overall network loss function of the semantic segmentation network is composed of the weighted cross entropy loss function L wce , lovasz-softmax loss function L ls and TVLoss loss function L tv Log-linear fusion, where: The weighted cross entropy loss function for: in y i and Represents the true value and predicted value of each category, f i Indicates the frequency of occurrence of the i-th category. In the semantic segmentation of point clouds, it indicates the proportion of the number of i-th category point clouds to the total number of point clouds. i is the weight coefficient for each category; p(y i )and Represents the probability of the true value and predicted value of each category respectively; The lovasz-softmax loss function L ls for: Where |C| represents the number of categories, J is a piecewise linear function with a global minimum, represents the Lovasz expansion of Iou, m(c) represents the error vector of category c, and x i (c) and y i (c) represents the predicted value and true value of the i-th point cloud of category c respectively; The TVLoss loss function L tv for: in Represents the exclusive OR operation, y and Represent the true value and predicted value of the point cloud respectively, i, j and k represent the three dimensions of the voxel, and Represents the edge errors in three dimensions i, j, and k respectively; The overall loss function of the network is: L total =L wce +L ls +L tv ; Among them, L total Represents the overall loss function, L wce , L ls and L tv They represent the weighted cross entropy loss function, lovasz-softmax loss function and TVLoss loss function respectively.

Images