Laser radar semantic segmentation domain generalization method based on multi-layer aerial view assistance

Abstract

The invention discloses a laser radar semantic segmentation domain generalization method based on multi-layer aerial view assistance, and the method comprises the steps: firstly introducing a three-dimensional sparse convolution encoder-decoder neural network, and converting original point cloud data into three-dimensional sparse voxel features; then, a top-down auxiliary task is added to enhance the capability of learning domain invariant features of the backbone network; in the auxiliary task, a three-dimensional space is divided according to the height under a Cartesian coordinate system, and a multi-layer aerial view BEV projection is generated to compensate for height information lost in the two-dimensional projection process; then, supervising and training the network by using a point cloud truth value label, and jointly optimizing a three-dimensional segmentation head and a BEV segmentation head; according to the method, multi-layer BEV information is utilized, and the problem that when a three-dimensional sparse convolutional network is used for semantic segmentation, the domain transfer phenomenon occurs between a source domain and a target domain, and consequently the network performance is remarkably reduced can be solved.

Description

Technical Field

[0001] The invention relates to a laser radar semantic segmentation domain generalization method based on the assistance of a multi-layer bird's-eye view, and belongs to the field of laser radar semantic segmentation. Background Art

[0002] Point cloud semantic segmentation is a basic task in the field of computer vision and has wide applications in many fields such as robotics and automation. In particular, in the understanding of autonomous driving scenarios, the semantic segmentation of lidar point clouds is crucial. Currently, most methods rely on manually annotated point cloud data, which are supervised by deep neural network models and assume that the data distribution of the source domain is consistent with that of the target domain. However, when the data distribution of the source domain is inconsistent with that of the target domain, that is, domain shift occurs, the model performance often decreases significantly.

[0003] In order to solve the domain shift problem, more and more researchers have begun to pay attention to domain adaptation technology for LiDAR semantic segmentation. A common way to alleviate domain shift is to collect datasets from multiple domains for pre-training, but the construction of multi-domain LiDAR datasets is extremely complex and costly. Unsupervised Domain Adaptation (UDA) technology reduces the impact of domain shift by utilizing unlabeled data from the target domain without the need for labeled data from the target domain. However, in practical applications, target domain data is usually difficult to obtain completely, and the model needs to be retrained every time the scene changes, which is time-consuming and cumbersome. To solve this problem, semantic segmentation domain generalization methods have emerged. Their goal is to learn domain-invariant features under the premise of using only source domain data for training, so that the model can perform comparable performance to the source domain on an unknown target domain without using target domain data.

[0004] In view of this, the present invention proposes a domain generalization method for lidar semantic segmentation based on multi-layer bird's-eye view assistance, so as to enhance the point cloud semantic segmentation network trained only on the source domain to maintain good semantic segmentation performance on the unseen target domain; this method enhances the learning of domain invariant features by the backbone network by introducing the bird's-eye view auxiliary task branch; at the same time, a spatial partitioning strategy is adopted to generate multi-layer bird's-eye view (BEV) projections to compensate for the height information that may be lost in the projection process, thereby improving the domain generalization ability of the backbone network. Summary of the invention

[0005] The purpose of the present invention is to propose a domain generalization method for lidar semantic segmentation based on bird's-eye view assistance; the method converts raw point cloud data into sparse three-dimensional voxel features by introducing a three-dimensional sparse convolutional encoder-decoder neural network, and enhances the domain generalization ability of the network by adding a top-down prediction auxiliary task; in the auxiliary task branch, the three-dimensional space in the Euclidean coordinate system is first spatially divided according to different height ranges, and the three-dimensional voxel features output by the decoder are projected on each divided space plane along the height axis; then, the segmentation head is applied to the sparse three-dimensional features and multiple BEV features for processing; the training process of the entire network is supervised by the semantic truth value label of the 3D point cloud, and the loss function of the three-dimensional segmentation head and the BEV segmentation head are jointly optimized, so as to improve the performance of the network in the domain generalization task; the specific steps include:

[0006] Step 1: 3D semantic segmentation backbone network

[0007] The main learning task of the trunk branch is to perform semantic segmentation, that is, to learn the mapping function where p i ∈P j ; Each point cloud P j ~D by N j points and extracted from a point cloud distribution D; the set Represents the semantic category vocabulary, which contains K classification labels;

[0008] Sub-step 1: Sparse voxelization of the original point cloud

[0009] The input LiDAR point cloud P j Processed into a 3D voxel (occupancy) grid V j ; In this stage, the continuous 3D input space is uniformly sampled into discrete 3D units, and the corresponding points are assigned to the voxels to which they belong;

[0010] Based on a sparse convolutional backbone network, a three-dimensional encoder-decoder network g 3D Let's learn about 3D voxel grids V j The encoder is composed of a series of sparse 3D convolution and downsampling layers to extract and compress 3D features. The decoder upsamples the features through a sparse convolution upsampling layer to restore the initial resolution. A sparse batch normalization layer is used in the sampling process to improve the stability and performance of the model.

[0011] Sub-step 2: Voxel semantic label prediction

[0012] 3D encoder-decoder network g 3D The output is a set of voxel-level feature representations Each feature vector corresponds to a voxel unit; these voxel features are sent to a sparse segmentation head to predict the semantic information of these voxels, and finally the voxel features are calculated The category distribution of each voxel is obtained, specifically:

[0013]

[0014] In the formula, σ represents the Softmax activation function, h 3D Represents a 3D sparse segmentation head.

[0015] Step 2: BEV auxiliary task branch

[0016] In order to enhance the domain generalization ability of the network, the present invention aims to encourage the backbone network to learn domain-invariant features; since the scan results of different domains are more similar in the BEV space, this property helps to alleviate the domain shift phenomenon; based on this, the present invention converts the three-dimensional sparse voxel features into multiple BEV plane-based features and trains a new network to estimate the BEV semantic scene layout;

[0017] Sub-step 1: 3D space division

[0018] In order to compensate for the height information lost in two-dimensional projection, the present invention divides the three-dimensional space into two-dimensional projections according to the height intervals {t0, t1, t2, t3, ..., t q} to obtain the segmentation space E = {E1, E2, E3, ..., E q } , a BEV projection is performed in each space to retain the height information in the three-dimensional space as much as possible;

[0019] Sub-step 2: Multi-layer BEV feature projection

[0020] Given each voxel in space The center coordinates of Its corresponding BEV coordinates The relationship between them is:

[0021]

[0022] In the formula, r′ i =||(x i ,y i )||2∈[t q-1 , t q ], f′ is the maximum sensing range of the sensor, ω and h represent the width and height of the BEV image respectively; only the image at the projection boundary B 3D =[(-b x , b x ), (-b y , by )] will be projected, where b x and b y are the boundary values ​​of the x-axis and y-axis; adjust the projection coordinates to fit the height and width of the BEV feature map; initialize the BEV features to 0 and fill each cell with non-empty voxels To define BEV characteristics When multiple voxels projected to the same unit, one of them was randomly retained;

[0023] Sub-step 3: BEV ground truth label projection

[0024] Through The same process is used to obtain the source labels; each voxel in space With category labels Corresponding; using formula (1) and the same boundary B 3D To calculate the label coordinates of each layer of BEV image pixels The defined BEV tags are:

[0025]

[0026] Sub-step 4: BEV feature label prediction

[0027] By using a standard 2D segmentation head Predict BEV semantic posterior probability; the segmentation head uses the input BEV features To predict semantic categories in the two-dimensional BEV plane; first reduce the pooling operation The dimension of Input to h BEV In , BEV semantic prediction is generated; specifically, the formula of BEV semantic prediction is defined as:

[0028]

[0029] In the formula, σ represents the Softmax activation function;

[0030] Step 3: Loss function definition

[0031] For a given 3D semantic prediction With label And BEV semantic prediction With label The present invention adopts the Soft DICE loss function to jointly train the two segmentation heads of the network to reduce the negative impact of category imbalance; the loss function for the three-dimensional part is defined as:

[0032]

[0033] The loss function for the BEV part is defined as:

[0034]

[0035] in, q Indicates the total number of partitioned spaces;

[0036] Finally, the contributions of the two loss functions are averaged and the total loss function is defined as:

[0037]

[0038] Step 4: Network training

[0039] The network proposed in the present invention uses MinkowskiNet as a sparse convolutional backbone network during training, the voxel size is set to 0.05m, the total batch size is 16, the learning rate is 0.01, and the Adam optimizer is used; in terms of data enhancement, random rotation, scaling and downsampling strategies are used; the rotation angle range is set to [-π / 2, π / 2], the scaling ratio is set to [0.95, 1.05], and 80% of the point cloud data are randomly downsampled; according to the resolution of the input data, the 3D projection boundary B is set 3D ;

[0040] In the Synth4D-KITTI and SemanticKITTI datasets, and The boundary is set to 50m in the Synth4D-nuScenes and nuScenes datasets; in the Synth4D-nuScenes and nuScenes datasets, the boundary is set to 30m. Through spatial boundary calculation, BEV features with a resolution of 168×168 are generated; and the height intervals of Synth4D-KITTI and SemanticKITTI datasets and Synth4D-nuScenes and nuScenes datasets are set to {[-3m,-1.5m], [-1.5m, 0m], [0m, 1.5m], [1.5m, 5m]}, {[-4m, 0m], [0m, 4m], [4m, 8m], [8m, 12m]} respectively to generate four-layer BEV images; the downsampled density features are processed by a maximum pooling layer with a window size of 5, a stride of 3, and a padding of 1; the BEV decoder is implemented by three layers of 2D convolution, and the convolutional layers alternately apply batch normalization and ReLU activation functions, and finally semantic classification is performed through the Softmax activation function;

[0041] The beneficial effects of the present invention are that the bird's-eye view-assisted lidar semantic segmentation domain generalization method proposed in the present invention can utilize the BEV-assisted segmentation branch to enhance the domain generalization ability of the semantic segmentation of the backbone network, and designs a multi-layer BEV generation method to retain the height information lost by the BEV projection, thereby ensuring that the model can perform well in unseen target domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a network flow chart of the generalization method of lidar semantic segmentation domain assisted by multi-layer bird's-eye view. DETAILED DESCRIPTION

[0043] Point cloud semantic segmentation is a basic task in the field of computer vision. It has wide applications in many fields such as robotics and automation. In particular, in the understanding of autonomous driving scenarios, the semantic segmentation of LiDAR point clouds is crucial. Currently, most methods rely on manually annotated point cloud data, perform supervised training through deep neural network models, and assume that the data distribution of the source domain and the target domain are consistent. However, when the data distribution of the source domain and the target domain are inconsistent, that is, domain shift occurs, the model performance often decreases significantly.

[0044] In order to solve the domain shift problem, more and more researchers have begun to pay attention to domain adaptation technology for LiDAR semantic segmentation. A common way to alleviate domain shift is to collect datasets from multiple domains for pre-training, but the construction of multi-domain LiDAR datasets is extremely complex and costly. Unsupervised Domain Adaptation (UDA) technology reduces the impact of domain shift by utilizing unlabeled data from the target domain without the need for labeled data from the target domain. However, in practical applications, target domain data is usually difficult to obtain completely, and the model needs to be retrained every time the scene changes, which is time-consuming and cumbersome. To solve this problem, semantic segmentation domain generalization methods have emerged. Their goal is to learn domain-invariant features under the premise of using only source domain data for training, so that the model can perform comparable to the source domain on an unknown target domain without using target domain data.

[0045] In view of this, the present invention proposes a domain generalization method for lidar semantic segmentation based on multi-layer bird's-eye view auxiliary, so as to enhance the semantic segmentation performance of the point cloud semantic segmentation network trained only on the source domain on the unseen target domain; this method enhances the learning of domain invariant features of the backbone network by introducing the bird's-eye view auxiliary task branch; at the same time, a spatial partitioning strategy is adopted to generate multi-layer BEV projections to compensate for the height information that may be lost in the projection process, thereby improving the domain generalization ability of the backbone network;

[0046] The purpose of the present invention is to propose a domain generalization method for lidar semantic segmentation based on bird's-eye view assistance; this method converts raw point cloud data into sparse three-dimensional voxel features by introducing a three-dimensional sparse convolutional encoder-decoder neural network, and enhances the domain generalization ability of the network by adding a top-down prediction auxiliary task; in the auxiliary task branch, the three-dimensional space in the Euclidean coordinate system is first spatially divided according to different height ranges, and the three-dimensional voxel features output by the decoder are projected on each divided space plane along the height axis; then, the segmentation head is applied to the sparse three-dimensional features and multiple BEV features for processing; the training process of the entire network is supervised by the semantic truth value label of the 3D point cloud, and the loss function of the three-dimensional segmentation head and the BEV segmentation head are jointly optimized, so as to improve the performance of the network in the domain generalization task; Figure 1 As shown, the specific steps include:

[0047] Step 1: 3D semantic segmentation backbone network

[0048] The main learning task of the trunk branch is to perform semantic segmentation, that is, to learn the mapping function where p i ∈P j ; Each point cloud P j ~D by N j points and extracted from a point cloud distribution D; the set Represents the semantic category vocabulary, which contains K classification labels;

[0049] Sub-step 1: Sparse voxelization of the original point cloud

[0050] The input LiDAR point cloud P j Processed into a 3D voxel (occupancy) grid V j ; In this stage, the continuous 3D input space is uniformly sampled into discrete 3D units, and the corresponding points are assigned to the voxels to which they belong;

[0051] Based on a sparse convolutional backbone network, a three-dimensional encoder-decoder network g 3D Let's learn about 3D voxel grids V j The encoder is composed of a series of sparse 3D convolution and downsampling layers to extract and compress 3D features. The decoder upsamples the features through a sparse convolution upsampling layer to restore the initial resolution. A sparse batch normalization layer is used in the sampling process to improve the stability and performance of the model.

[0052] Sub-step 2: Voxel semantic label prediction

[0053] 3D encoder-decoder network g 3D The output is a set of voxel-level feature representations Each feature vector corresponds to a voxel unit; these voxel features are sent to a sparse segmentation head to predict the semantic information of these voxels, and finally the voxel features are calculated The category distribution of each voxel is obtained, specifically:

[0054]

[0055] In the formula, σ represents the Softmax activation function, h 3D Represents a 3D sparse segmentation head.

[0056] Step 2: BEV auxiliary task branch

[0057] In order to enhance the domain generalization ability of the network, the present invention aims to encourage the backbone network to learn domain-invariant features; since the scan results of different domains are more similar in the BEV space, this property helps to alleviate the domain shift phenomenon; based on this, the present invention converts the three-dimensional sparse voxel features into multiple BEV plane-based features and trains a new network to estimate the BEV semantic scene layout;

[0058] Sub-step 1: 3D space division

[0059] In order to compensate for the height information lost in two-dimensional projection, the present invention divides the three-dimensional space into two-dimensional projections according to the height intervals {t0, t1, t2, t3, ..., t q} to obtain the segmentation space E = {E1, E2, E3, ..., E q}, perform a BEV projection in each space to preserve the height information in the three-dimensional space as much as possible;

[0060] Sub-step 2: Multi-layer BEV feature projection

[0061] Given each voxel in space The center coordinates of Its corresponding BEV coordinates The relationship between them is:

[0062]

[0063] In the formula, r′ i =||(x i ,y i )||2∈[t q-1 , t q ], f′ is the maximum sensing range of the sensor, w and h represent the width and height of the BEV image respectively; only the image at the projection boundary B 3D =[(-b x , b x ), (-b y , b y)] will be projected, where b x and b y are the boundary values ​​of the x-axis and y-axis; adjust the projection coordinates to fit the height and width of the BEV feature map; initialize the BEV features to 0 and fill each cell with non-empty voxels To define BEV characteristics When multiple voxels projected to the same unit, one of them was randomly retained;

[0064] Sub-step 3: BEV ground truth label projection

[0065] Through The same process is used to obtain the source labels; each voxel in space With category labels Corresponding; using formula (1) and the same boundary B3D To calculate the label coordinates of each layer of BEV image pixels The defined BEV tags are:

[0066]

[0067] Sub-step 4: BEV feature label prediction

[0068] By using a standard 2D segmentation head Predict BEV semantic posterior probability; the segmentation head uses the input BEV features To predict semantic categories in the two-dimensional BEV plane; first reduce the pooling operation The dimension of Input to hBEV In , BEV semantic prediction is generated; specifically, the formula of BEV semantic prediction is defined as:

[0069]

[0070] In the formula, σ represents the Softmax activation function;

[0071] Step 3: Loss function definition

[0072] For a given 3D semantic prediction With label And BEV semantic prediction With label The present invention adopts the Soft DICE loss function to jointly train the two segmentation heads of the network to reduce the negative impact of category imbalance; the loss function for the three-dimensional part is defined as:

[0073]

[0074] The loss function for the BEV part is defined as:

[0075]

[0076] in, q Indicates the total number of partitioned spaces;

[0077] Finally, the contributions of the two loss functions are averaged and the total loss function is defined as:

[0078]

[0079] Step 4: Network training

[0080] The network proposed in the present invention uses MinkowskiNet as a sparse convolutional backbone network during training, the voxel size is set to 0.05m, the total batch size is 16, the learning rate is 0.01, and the Adam optimizer is used; in terms of data enhancement, random rotation, scaling and downsampling strategies are used; the rotation angle range is set to [-π / 2, π / 2], the scaling ratio is set to [0.95, 1.05], and 80% of the point cloud data are randomly downsampled; according to the resolution of the input data, the 3D projection boundary B is set 3D ;

[0081] In the Synth4D-KITTI and SemanticKITTI datasets, and The boundary is set to 50m in the Synth4D-nuScenes and nuScenes datasets; in the Synth4D-nuScenes and nuScenes datasets, the boundary is set to 30m. Through spatial boundary calculation, BEV features with a resolution of 168×168 are generated; and the height intervals of Synth4D-KITTI and SemanticKITTI datasets and Synth4D-nuScenes and nuScenes datasets are set to {[-3m,-1.5m],[-1.5m,0m],[0m,1.5m],[1.5m,5m]}, {[-4m,0m],[0m,4m],[4m,8m],[8m,12m]} respectively to generate four-layer BEV images; the downsampled density features are processed by a maximum pooling layer with a window size of 5, a stride of 3, and a padding of 1; the BEV decoder is implemented by three layers of 2D convolution, and the convolutional layers alternately apply batch normalization and ReLU activation functions, and finally semantic classification is performed through the Softmax activation function.

[0082] The bird's-eye view-assisted lidar semantic segmentation domain generalization method proposed in the present invention can utilize the BEV-assisted segmentation branch to enhance the domain generalization ability of the backbone network's semantic segmentation, and designs a multi-layer BEV generation method to retain the height information lost by the BEV projection, ensuring that the model can perform well in unseen target domains.

Claims

1. A method for generalizing the semantic segmentation domain of lidar based on multi-layer bird's-eye view assistance, characterized in that: By introducing a 3D sparse convolutional encoder-decoder neural network, the original point cloud data is converted into sparse 3D voxel features, and the domain generalization ability of the network is enhanced by adding top-down prediction auxiliary tasks. In the auxiliary task branch, the 3D space in the Euclidean coordinate system is first divided into different height ranges, and the 3D voxel features output by the decoder are projected on each divided space plane along the height axis. Subsequently, the segmentation head is applied to the sparse 3D features and multiple BEV features for processing. The training process of the entire network is supervised by the semantic truth label of the 3D point cloud, and the loss functions of the 3D segmentation head and the BEV segmentation head are jointly optimized, thereby improving the performance of the network in the domain generalization task. The specific steps include: Step 1: 3D semantic segmentation backbone network The main learning task of the trunk branch is to perform semantic segmentation, that is, to learn the mapping function where p i ∈P j ; Each point cloud P j ~D by N j points and extracted from a point cloud distribution D; the set Represents the semantic category vocabulary, which contains K classification labels; Sub-step 1: Sparse voxelization of the original point cloud The input LiDAR point cloud P j Processed into a three-dimensional voxel grid V j ; In this stage, the continuous 3D input space is uniformly sampled into discrete 3D units, and the corresponding points are assigned to the voxels to which they belong; Based on a sparse convolutional backbone network, a three-dimensional encoder-decoder network g aD Let's learn about 3D voxel grids V j The encoder is composed of a series of sparse 3D convolution and downsampling layers to extract and compress 3D features. The decoder upsamples the features through a sparse convolution upsampling layer to restore the initial resolution. A sparse batch normalization layer is used in the sampling process to improve the stability and performance of the model. Sub-step 2: Voxel semantic label prediction 3D encoder-decoder network g3 D The output is a set of voxel-level feature representations Each feature vector corresponds to a voxel unit; these voxel features are sent to a sparse segmentation head to predict the semantic information of these voxels, and finally the voxel features are calculated The category distribution of each voxel is obtained, specifically: In the formula, σ represents the Softmax activation function, h 3D represents the three-dimensional sparse segmentation head; Step 2: BEV auxiliary task branch In order to enhance the domain generalization ability of the network, the backbone network is encouraged to learn domain-invariant features. Since the scan results of different domains are more similar in BEV space, this feature helps to alleviate the domain shift phenomenon. Based on this, the 3D sparse voxel features are converted into multiple BEV plane-based features, and a new network is trained to estimate the BEV semantic scene layout. Sub-step 1: 3D space division In order to compensate for the height information lost in two-dimensional projection, the three-dimensional space is divided into two layers according to the height intervals {t0, t1, t2, t3, ..., t q } to obtain the segmentation space E = {E1, E2, E3, ..., E q }, perform a BEV projection in each space to preserve the height information in the three-dimensional space as much as possible; Sub-step 2: Multi-layer BEV feature projection Given each voxel in space The center coordinates of Its corresponding BEV coordinates The relationship between them is: In the formula, r′ i =||(x i ,y i )||2∈[t q-1 , t q ], f′ is the maximum sensing range of the sensor, w and h represent the width and height of the BEV image respectively; only the image at the projection boundary B 3D =[(-b x , b x ), (-b y , b y )] will be projected, where b x and b y are the boundary values ​​of the x-axis and y-axis; adjust the projection coordinates to fit the height and width of the BEV feature map; initialize the BEV features to 0 and fill each cell with non-empty voxels To define BEV characteristics When multiple voxels projected to the same unit, one of them was randomly retained; Sub-step 3: BEV ground truth label projection Through The same process is used to obtain the source labels; each voxel in space With category labels Corresponding; using formula (1) and the same boundary B3 D To calculate the label coordinates of each layer of BEV image pixels The defined BEV tags are: Sub-step 4: BEV feature label prediction By using a standard 2D segmentation head Predict BEV semantic posterior probability; the segmentation head uses the input BEV features To predict semantic categories in the two-dimensional BEV plane; first reduce the pooling operation The dimension of Input to h BEV In , BEV semantic prediction is generated; specifically, the formula of BEV semantic prediction is defined as: In the formula, σ represents the Softmax activation function; Step 3: Loss function definition For a given 3D semantic prediction With label And BEV semantic prediction With label The Soft DICE loss function is used to jointly train the two segmentation heads of the network to alleviate the negative impact of category imbalance; the loss function for the three-dimensional part is defined as: The loss function for the BEV part is defined as: Among them, q represents the total number of partitioned spaces; Finally, the contributions of the two loss functions are averaged and the total loss function is defined as: Step 4: Network training The network uses MinkowskiNet as the sparse convolution backbone network during training, with a voxel size of 0.05m, a total batch size of 16, a learning rate of 0.01, and an Adam optimizer. In terms of data enhancement, random rotation, scaling, and downsampling strategies are used. The rotation angle range is set to [-π / 2, π / 2], the scaling ratio is set to [0.95, 1.05], and 80% of the point cloud data is randomly downsampled. The 3D projection boundary B is set according to the resolution of the input data. 3D ; In the Synth4D-KITTI and SemanticKITTI datasets, and The boundary of the dataset is set to 50m; in the Synth4D-nuScenes and nuScenes datasets, the boundary is set to 30m; through spatial boundary calculation, BEV features with a resolution of 168×168 are generated; and the height intervals of the Synth4D-KITTI and SemanticKITTI datasets and the Synth4D-nuScenes and nuScenes datasets are set to {[-3m, -1.5m], [-1.5m , 0m], [0m, 1.5m], [1.5m, 5m]}, {[-4m, 0m], [0m, 4m], [4m, 8m], [8m, 12m]} to generate four-layer BEV images; the downsampled density features are processed by a maximum pooling layer with a window size of 5, a stride of 3, and a padding of 1; the BEV decoder is implemented by three layers of 2D convolution, and the convolutional layers alternately apply batch normalization and ReLU activation functions, and finally semantic classification is performed through the Softmax activation function.

Images