Automatic driving scene panoramic segmentation method based on multi-modal fusion perception

Abstract

The application discloses an automatic driving scene panorama segmentation method based on multi-modal fusion perception, which comprises four steps, first, the feature map of the camera image of the automatic driving scene is obtained by using an image semantic segmentation network, then the point cloud is projected onto the feature map to obtain the two-dimensional feature of the point cloud, then the feature is taken as the input of the backbone network together with the three-dimensional feature of the point cloud, second, the bottom layer of the backbone network is divided according to the camera field of view, and after being matched with the feature map, the sequence is tiled into a multi-head self-attention network to learn the correlation of the two different modal information, finally, the clustering information of the backbone network is output, and the cosine similarity of the clustering center is learned by using a multilayer perception to determine whether the multiple centers should be merged or not. The application solves the problem that the panorama segmentation task in the automatic driving scene cannot well use multi-modal information, and the learning of semantic information is greatly enhanced by using two levels of fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of perception of autonomous driving scenes, and particularly to a method for panoramic segmentation using multiple different modal data simultaneously. BACKGROUND

[0002] 3D vision perception has become a crucial task in a wide range of applications including autonomous driving, robotics, and virtual reality. In 3D vision perception, panoramic segmentation is a comprehensive perception task consisting of semantic segmentation, object detection, and instance segmentation, which is very challenging because it needs to predict the semantic label of each point from background classes such as trees, roads, etc., while also identifying instances of object classes such as cars, bicycles, and pedestrians.

[0003] Panoramic segmentation work can be roughly divided into two categories, namely detection-based methods and detection-free methods. Detection-based methods usually adopt a two-stage training process, which first generates target detection boxes with instance labels, and then merges these instance predictions into semantic segmentation. On the contrary, detection-free methods predict semantic class information and instance clustering information simultaneously. During training, it is directly supervised by annotations, and during inference, it uses a parameter-free post-processing to group points belonging to object classes into the nearest object center. Since two different supervision information is trained simultaneously and only a single model is used, detection-free methods have achieved better performance. However, both methods have under-segmentation or over-segmentation errors. Previous studies have shown that the reason is that laser point clouds are usually sparse and unevenly distributed, making the performance of deep neural networks significantly different between processing near point clouds and far point clouds. For most three-dimensional networks, far objects that only occupy a few points are too small to be effectively detected in the view, while nearby objects are often too large and prone to over-segmentation.

[0004] On the contrary, images can provide rich texture and color information. Because the pixels in the image are dense, even small objects have enough information to identify them. This has prompted the use of images to supplement the scene perception performance of laser point clouds. In addition, most autonomous driving systems are equipped with both camera cameras and point cloud cameras, making this multi-modal fusion idea have very broad development prospects. SUMMARY

[0005] The purpose of this invention is to provide a panoramic segmentation method for autonomous driving scenes based on multimodal fusion perception. This method simultaneously integrates local detail texture information and global semantic information, including input-level point-to-pixel fusion and high-level voxel-to-image fusion. Pixel fusion accurately utilizes the geometric relationship between the coordinate systems of the LiDAR sensor and the image camera, correcting for misalignment caused by asynchronous acquisition time of the LiDAR sensor and image camera due to vehicle motion. Global semantic information fusion involves multi-head self-attention feature learning on voxel features and image features in a deep network. Both are designed as lightweight, plug-and-play modules. Finally, during instance clustering, the model again references the pixel fusion information, using color and texture information for secondary clustering to address the over-segmentation problem of large objects.

[0006] The specific technical solution for achieving the objective of this invention is as follows:

[0007] A panoramic segmentation method for autonomous driving scenes based on multimodal fusion perception is characterized by the use of multimodal feature fusion to enhance network feature extraction capabilities and large object clustering effects. The method includes the following specific steps:

[0008] Step 1: Image Feature Generation and Projection

[0009] A1: A dataset is constructed using RGB image information of the surrounding environment received from vehicle-mounted sensors and LiDAR point cloud information; several camera images of a scene in the dataset are input together into a pre-trained image semantic segmentation network to obtain several minimum-size feature maps and half-size two-dimensional feature maps; all parameters of the image semantic segmentation network are fixed during training.

[0010] A2: Construct a transformation matrix based on the relative positional relationship of the sensors on the sampling vehicle, project the point cloud onto the corresponding half-size two-dimensional feature map, obtain the features on the feature map based on the pixel coordinates of the projected point cloud, and form the image features of the point cloud. Point clouds that cannot be projected onto the two-dimensional feature map are directly filtered out. For point clouds that can be projected onto multiple two-dimensional feature maps at the same time, randomly use one two-dimensional feature map to obtain the features.

[0011] Step 2: Backbone Network Feature Initialization and Training

[0012] B1: After obtaining the image features of the point cloud and the three-dimensional coordinate geometric features of the original point cloud data in step 1, the image features of the point cloud are passed through a multi-layer perception machine capable of reducing dimensions, and the geometric features of the point cloud are passed through a multi-layer perception machine capable of enlarging dimensions, so that the two features reach the same dimension, then the space grid is divided, and the inconsistency of the number of points in the two modal data is eliminated by separate pooling, and finally the two modal features of each grid are added together to obtain the network initialization input feature after multi-modal fusion, and a cylindrical grid in a polar coordinate system is used as the main network data structure for feature learning;

[0013] B2: The network semantic segmentation branch is trained using focal loss and cross-entropy loss, and the center point probability regression and center point coordinate deviation regression are trained using L2 loss;

[0014] Step 3: High-dimensional feature fusion at the bottom of the main network

[0015] C1: Take out the three-dimensional feature map at the bottom of the main network, use the orientation angle information of the camera on the vehicle to divide the three-dimensional feature map into several regions corresponding to the number of cameras according to the polar coordinate angle, each region corresponds to a two-dimensional feature map with the smallest size in A2, and the three-dimensional features and two-dimensional features in the corresponding region are tiled and connected together to form a serialized information;

[0016] C2: Use the absolute position of the three-dimensional grid center in the whole space and the pixel coordinates on the smallest two-dimensional feature map to form the position encoding of the serialized information, and then input the serialized information with the position encoding into a multi-head self-attention mechanism network with linear complexity to learn high-dimensional cross-modal information fusion while keeping the feature dimension unchanged;

[0017] Step 4: Network output result integration and correction

[0018] D1: Network output grid-level semantic segmentation prediction, label each grid in the three-dimensional space where the point cloud exists with a semantic class label, then label each point cloud with a semantic class label according to the grid class label of each point cloud, and thus obtain the class to which each point in the input point cloud data belongs, completing the point-level semantic segmentation prediction;

[0019] D2: The network outputs the center point probability prediction map in the bird's eye view perspective, filtered using the maximum pooling with a kernel size of 5 to obtain the coordinates of the candidate center points; according to the transformation matrix constructed in step A2, the candidate center points are projected onto the corresponding pixels of the corresponding half-size two-dimensional feature map to obtain the two-dimensional image features of the positions of the center points, and a multi-layer perception is used to learn the cosine similarity between each center point and other center points to merge the center points of the semantic segmentation result of large objects through a threshold to obtain the final instance target center, thereby improving the clustering effect of large objects; wherein the large objects are buses, trucks, fire engines and other engineering construction vehicles;

[0020] D3: The network outputs the coordinate deviation of each grid from the center point in the bird's eye view perspective. For all grids with points, the coordinates are added to the network output deviation to obtain new grid positions, and then the nearest center point is calculated. All grids belonging to the same center point are regarded as a single instance, and the semantic category of the instance is obtained by principal component voting of the semantic category labels of all points of the instance. The final output result is that the points of the semantic segmentation result of the background category are labeled with the corresponding category label, and the points of the semantic segmentation of the foreground are labeled with the category label and the instance serial number at the same time, and the serial number of the instance in the corresponding scene and category is predicted.

[0021] In step A2, the transformation matrix is constructed according to the relative position relationship of the sensors on the sampling vehicle, and the point cloud is projected onto the corresponding pixels of the corresponding half-size two-dimensional feature map. Specifically, an extrinsic parameter matrix is constructed through the position relationship of the laser radar camera and the RGB image camera, the point cloud is changed from the laser radar coordinate system to the RGB image camera coordinate system by multiplying the three-dimensional coordinates of the point cloud by the extrinsic parameter matrix, and then an intrinsic change matrix is constructed through the internal parameters of the RGB camera. The coordinates of the point cloud in the RGB image camera coordinate system are multiplied by the intrinsic matrix to obtain the specific position of the point cloud in the two-dimensional image in the three-dimensional space, thereby realizing the projection of the point cloud in the three-dimensional space onto the two-dimensional image.

[0022] In step B1, the cylindrical grid in the polar coordinate system is used as the data structure of the backbone network for feature learning. Specifically, the characteristics of the polar coordinate system, i.e., the arc length of the grid increases with the distance from the axis, are used to fit the characteristics of the laser radar, i.e., the near density and the far sparsity, to balance the difference in the number of internal points between the near distance grid and the far distance grid, and to improve the feature extraction capability of the network. Using the polar coordinate system can also directly correspond the angle axis to the angle of the image camera field of view, thereby limiting the fusion range of high-dimensional features to within one-sixth of the scene area and reducing the consumption of computing resources.

[0023] The step B2 uses focal loss and cross-entropy loss to train the network semantic segmentation branch, and specifically, when calculating the loss function, the cross-entropy loss function is used as the classification loss, a penalty item for high confidence categories is generated by the predicted category probability coefficient to form the focal loss, the proportion of categories that are very easy to judge in the loss function is reduced, the learning efficiency of difficult categories is increased, and the serious class imbalance problem in the autonomous driving scene segmentation task is solved.

[0024] The linear complexity multi-head self-attention mechanism network in the step C2 is specifically: the once square level calculation complexity multi-head self-attention operation is divided into two times, the first operation uses the parameters learned by the network to replace the input query, the input sequence length is compressed into a fixed value independent of the input sequence length, the second operation uses the output of the first operation as the key and the value, and the original input as the query, and the fixed value sequence length is restored to the original input size, so that the output is aligned with the input, while the feature learning ability is maintained, the calculation complexity is reduced from the square level to the linear.

[0025] In the step D2, the two-dimensional image features are learned by a multi-layer perception to obtain the cosine similarity between each center point and other center points, and a threshold is used to merge the center points of the semantic segmentation result to obtain the final instance target center, and specifically, after obtaining the two-dimensional image features of the center points, the three-dimensional features and the two-dimensional image features are added together and learned by a multi-layer perception to obtain the multi-modal features of the center points, the cosine similarity between all the center point multi-modal features is calculated, and the center points with a similarity greater than a threshold are regarded as the same instance, and a new center point is calculated as the final instance target center by using the mean value of the coordinates.

[0026] The beneficial effects of the present application are:

[0027] Two different levels of multi-modal fusion methods are used, which improves the feature extraction ability of the whole network and the final panoramic segmentation effect.

[0028] The cylindrical grid division structure based on the polar coordinate system is used, which solves the distribution imbalance problem caused by the near dense and far sparse of the laser radar data, and improves the network feature learning ability.

[0029] A secondary clustering method is proposed, which can use the color texture information that the laser radar does not have to merge multiple center points belonging to the same large object, and improve the clustering effect of large objects that are difficult to handle in the panoramic segmentation task. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1Flow chart for pixel-level fusion of the present application;

[0031] Figure 2 Flow chart for high-level semantic feature fusion of the present application;

[0032] Figure 3 Flow chart of the present application. Specific implementation method

[0033] In order to facilitate the understanding of the present application, the present application will be described in detail below in combination with the drawings and examples.

[0034] Referring to Figure 1 In step 1 of the present application, the picture data is input into the 2D image semantic segmentation network to obtain the minimum size feature map and the half size feature map, the minimum size feature map is used for subsequent network global feature fusion, and the half size feature map is used for pixel-level fusion.

[0035] S100: The backbone semantic segmentation network adopted by the image modal data, the network used at present is swiftnet18.

[0036] S110: The minimum size feature map is the feature map of the last layer of the network S100 after downsampling.

[0037] S120: The minimum half size feature map is the second last layer feature map after upsampling of the network S100 after downsampling.

[0038] Referring to Figure 1 In step 1 of the present application, according to the coordinate system transformation relationship between the image camera and the laser radar camera, the point cloud is projected onto the half size feature map to obtain the image feature.

[0039] S130: By participating in the three-dimensional coordinates of the camera inside and outside the point cloud, the position ratio of each point on the x-axis and y-axis of each image can be calculated, the legal range of this value is 0 to 1, if negative or greater than 1 value appears, it represents that the point is outside the image field of view, the same point is at most in the legal ratio range in two images, which means that it appears on two images at the same time, it can also not be in the range of any image, which means that the point is not in the range of 6 image cameras.

[0040] S140: After obtaining the x-axis and y-axis ratio of the point on the image, the image feature corresponding to the point can be obtained from the half size feature map, the features of one image are taken for the points in multiple images, and the points not in the image are directly ignored.

[0041] S150: The points ignored after S140 belong to a subset of the original point cloud quantity.

[0042] Referring to Figure 1, the image features of the obtained points and the geometric features of the original points in step 2 of the present application are respectively learned by respective multi-layer perceptrons, and then pooled into the same spatial grid division. The features are spliced in units of grids, and the grids where the point cloud exists are initialized as the input features of the backbone network.

[0043] S160: Multi-layer perceptron network, the image features of the point cloud in the camera field of view and the geometric position features of the original point cloud are independently learned by respective multi-layer perceptron networks.

[0044] S170: Limit the entire scene to a region of 0-50 meters, divide the entire space into grids according to 480 units of length axis, 360 units of angle axis, and 32 units of height axis, maximum pool the features learned in S160 into the corresponding grids, and then connect the features of the two modalities together in units of grids to obtain the three-dimensional backbone network initialization features S200.

[0045] Referring to Figure 2 In step 3 of the present application, the polar coordinate grid features of the bottom layer of the backbone network are taken out, the camera orientation angle given by the data set is used to divide the feature map into 6 regions according to the polar coordinate angle corresponding to the image, and the 2D feature map of the smallest size in step 1 is taken out, the features in the corresponding region are tiled and connected together to form a long sequence information.

[0046] S210-220: Both are backbone networks, the backbone network used in the present application is a U-shaped structure, which first performs downsampling S210 and then performs upsampling S220.

[0047] S230: The three-dimensional polar coordinate grid features obtained after S210 downsampling.

[0048] S240: According to the polar coordinate angle and the angle occupied by the camera field of view of the image, the three-dimensional polar coordinate features after downsampling are divided into 6 blocks according to the angle, and each block corresponds to an image.

[0049] S250: After tiling the corresponding grid and feature map, add the respective coordinate position coding to form a long sequence.

[0050] S260: Linear multi-head self-attention mechanism module, the specific process is to divide a traditional multi-head self-attention mechanism into two light-weight operations, in the first time, the key and value use the long sequence output by S250, and the query uses a network-learned fixed-length parameter to reduce the sequence length in the calculation. The query of the second operation uses the output of S250, and the key and value use the output of the first operation to return the features to the original length. Finally, the calculation amount is reduced from the sequence length square level to the linear level.

[0051] Referring to Figure 3 , the center point heat map predicted by the network is used to calculate the coordinates of the candidate center points in step 4, and then the candidate center points are projected onto the half-size feature map to obtain the image features of the candidate center points. The cosine similarity between them is learned through a multi-layer perception, and the center points with too high similarity are merged to achieve the effect of secondary clustering. After obtaining the center points, the displacement of each grid is calculated, and the nearest center point after displacement is obtained, that is, the grid set belonging to each center point is obtained, that is, the preliminary clustering result is obtained. According to the preliminary clustering result and the semantic category prediction result, the final panoramic segmentation result output by the network is determined.

[0052] S310: Center point heat map prediction result, for the bird's eye view, whether each pixel is a center point is predicted with a probability of 0-1, and the size is 480*360.

[0053] S320: Grid coordinate displacement result, for the bird's eye view, the displacement deviation of each pixel from its belonging center point is predicted, and the size is 480*360*2.

[0054] S330: Semantic segmentation prediction result, the probability of each grid belonging to each category is predicted, and the size is 480*360*32*category number.

[0055] S340: Center point prediction and secondary clustering process, the center point heat map predicted by the network is used to calculate the coordinates of the candidate center points, and then the candidate center points are projected onto the half-size feature map to obtain the image features of the candidate center points. The cosine similarity between them is learned through a multi-layer perception, and the center points with too high similarity are merged to achieve the effect of secondary clustering.

[0056] S350: Category-independent clustering process, specifically, after obtaining the center points, the displacement of each grid is calculated, and the nearest center point after displacement is obtained, that is, the grid set belonging to each center point is obtained, that is, the preliminary clustering result is obtained.

[0057] S360: Instance principal component voting, the specific process is as follows: after obtaining the preliminary clustering result, if the semantic categories of each point in each clustering set are not uniform, the final category of the set is determined according to the category with the most points in the set to ensure the uniformity of the final panoramic segmentation prediction result.

Claims

1. A panoramic segmentation method for autonomous driving scenes based on multimodal fusion perception, characterized in that, The method includes the following specific steps: Step 1: Image Feature Generation and Projection A1: A dataset is constructed using RGB image information of the surrounding environment received from vehicle-mounted sensors and LiDAR point cloud information; several camera images of a scene in the dataset are input together into a pre-trained image semantic segmentation network to obtain several minimum-size two-dimensional feature maps and half-size two-dimensional feature maps; all parameters of the image semantic segmentation network are fixed during training. A2: Construct a transformation matrix based on the relative positional relationship of the sensors on the sampling vehicle, project the point cloud onto the corresponding half-size two-dimensional feature map, obtain the features on the feature map based on the pixel coordinates of the projected point cloud, and form the image features of the point cloud. Point clouds that cannot be projected onto the two-dimensional feature map are directly filtered out. For point clouds that can be projected onto multiple two-dimensional feature maps at the same time, randomly use one two-dimensional feature map to obtain the features. Step 2: Backbone Network Feature Initialization and Training B1: After obtaining the image features of the point cloud and the three-dimensional coordinate geometric features inherent in the original point cloud data in step 1, the image features of the point cloud are passed through a multilayer perceptron that can reduce the dimension, and the geometric features of the point cloud are passed through a multilayer perceptron that can amplify the dimension, so that the two features reach the same dimension. Then, the data is divided according to the spatial grid, and separate pooling is used to eliminate the problem of inconsistent point counts in the two modal data. Finally, the two modal features of each grid are added together to obtain the network initialization input features after multimodal fusion. Cylindrical grids in polar coordinate system are used as the backbone network data structure for feature learning. B2: Use focal loss and cross-entropy loss to train the semantic segmentation branch of the network, and use L2 loss to train the center point probability regression and center point coordinate deviation regression. Step 3: Fusion of high-dimensional features at the bottom layer of the backbone network C1: Extract the three-dimensional feature map of the lowest layer of the backbone network, and use the orientation angle information of the camera on the vehicle to divide the three-dimensional feature map into several regions with the same number of cameras according to the polar coordinate angle. Each region corresponds to a two-dimensional feature map of the smallest size in A2. Tile and connect the three-dimensional features and two-dimensional features in the corresponding region to form a serialized information. C2: The positional encoding of the serialized information is constructed using the absolute position of the center of the three-dimensional grid in the overall space and the pixel coordinates on the smallest two-dimensional feature map. The serialized information is then fed into a multi-head self-attention mechanism network with linear complexity to learn high-dimensional cross-modal information fusion while keeping the feature dimension unchanged. Step 4: Integrating and correcting network output results D1: The network outputs a grid-level semantic segmentation prediction, which assigns a semantic category label to each grid containing a point cloud in the 3D space. Then, based on the grid category label of each point cloud, a semantic category label is assigned to each point cloud, thereby obtaining the category of each point in the input point cloud data and completing the point-level semantic segmentation prediction. D2: The network outputs a probability prediction map of the center point from a bird's-eye view. This map is filtered using max pooling with a kernel size of 5 to obtain the coordinates of candidate center points. Based on the transformation matrix constructed in step A2, the candidate center points are projected onto the corresponding pixels of the corresponding half-size 2D feature map to obtain the 2D image features of the center point's location. These 2D image features are then processed by a multilayer perceptron to learn the cosine similarity between each center point and every other center point. A threshold is used to merge the center points of large objects in the semantic segmentation results to obtain the final instance target centers, improving the clustering effect of large objects. The large objects referred to are buses, trucks, fire trucks, and other construction vehicles. D3: The coordinate deviation of each grid from its center point in the bird's-eye view of the network output. For all grids with points, the coordinates are added to the deviation output by the network to obtain the new grid position. Then, the nearest center point is calculated. All grids belonging to the same center point are regarded as a single instance. The semantic category of the instance is obtained by principal component voting of the semantic category labels of all points in the instance. The final output result is to assign the corresponding category label to the points whose semantic segmentation result is background, and to assign both the category label and the instance number to the points whose semantic segmentation result is foreground. The instance's index in its scene and category is predicted.

2. The panoramic segmentation method for autonomous driving scenes based on multimodal fusion perception according to claim 1, characterized in that, Step A2 describes constructing a transformation matrix based on the relative positions of the sensors on the sampling vehicle, and projecting the point cloud onto the corresponding pixels of the corresponding half-size two-dimensional feature map. Specifically, this involves constructing an extrinsic parameter matrix based on the positional relationship between the LiDAR camera and the RGB image camera, transforming the point cloud from the LiDAR coordinate system to the RGB image camera coordinate system by multiplying the three-dimensional coordinates of the point cloud with the extrinsic parameter matrix, and then constructing an intrinsic parameter transformation matrix based on the parameters inside the RGB camera. By multiplying the coordinates of the point cloud in the RGB image camera coordinate system with the intrinsic parameter matrix, the specific position of the point cloud in three-dimensional space on the two-dimensional image is obtained, thus realizing the projection of the point cloud in three-dimensional space onto the two-dimensional image.

3. The panoramic segmentation method for autonomous driving scenes based on multimodal fusion perception according to claim 1, characterized in that, Step B1 describes using a cylindrical grid in polar coordinates as the backbone network data structure for feature learning. Specifically, it utilizes the characteristic that the arc length of the grid in polar coordinates increases with the distance axis to fit the near-dense and far-sparse features of the LiDAR, balancing the difference in the number of points within the near-distance and far-distance grids, thereby improving the network's feature extraction capability. Using polar coordinates also allows the angle axis to directly correspond to the angle of the image camera's field of view, limiting the high-dimensional feature fusion range to one-sixth of the scene area and reducing computational resource consumption.

4. The panoramic segmentation method for autonomous driving scenes based on multimodal fusion perception according to claim 1, characterized in that, Step B2 describes training the semantic segmentation branch of the network using focal loss and cross-entropy loss. Specifically, when calculating the loss function, while using the cross-entropy loss function as the classification loss, a penalty term for high-confidence classes is generated by the predicted class probability coefficients to form a focal loss. This reduces the proportion of easily judged classes in the loss function, increases the learning efficiency for difficult classes, and solves the serious class imbalance problem in autonomous driving scene segmentation tasks.

5. The panoramic segmentation method for autonomous driving scenes based on multimodal fusion perception according to claim 1, characterized in that, The linear complexity multi-head self-attention mechanism network described in step C2 is specifically as follows: the multi-head self-attention operation with a quadratic computational complexity is split into two operations. The first operation uses the parameters learned by the network to replace the input query, compressing the length of the input sequence into a fixed value that is independent of the length of the input sequence. The second operation uses the output of the first operation as the key and value, and the original input as the query. The fixed sequence length is restored to the original input size, so that the output is aligned with the input. While maintaining the feature learning ability, the computational complexity is reduced from quadratic to linear.

6. The panoramic segmentation method for autonomous driving scenes based on multimodal fusion perception according to claim 1, characterized in that, In step D2, the two-dimensional image features are processed by a multilayer perceptron to learn the cosine similarity between each center point and other center points. A threshold is used to merge the center points of large objects in the semantic segmentation result to obtain the final instance target center. Specifically, after obtaining the two-dimensional image features of the center points, the three-dimensional features are added together with the two-dimensional image features and processed by a multilayer perceptron to obtain the multimodal features of the center points. The cosine similarity between all the multimodal features of the center points is calculated. Center points with similarity greater than the threshold are regarded as the same instance, and a new center point is calculated using the mean of their coordinates as the final instance target center.

Images