A semantic occupancy prediction method based on fusion of lidar point cloud and image

Abstract

The application provides a kind of semantic occupancy prediction method based on laser radar point cloud and image fusion, it is related to semantic occupancy prediction field, comprising: step 1, read laser radar point cloud data and corresponding binocular image data for training semantic occupancy prediction model;Step 2, after reading data, data preprocessing work is carried out, including image normalization, cropping, random flip of point cloud and obtaining corresponding initial voxel features by voxelizing point cloud;Step 3, obtain the final feature map with the same size as the scene;Step 4, obtain the final semantic occupancy prediction result using convolution layer in the head network, calculate the loss function, update model parameters, complete training, save training weight file;Step 5, prediction is carried out.The application improves the attention of the model to the scarce category and enhances the perception ability of the model.

Description

Technical Field

[0001] This invention relates to the field of semantic occupancy prediction, and more specifically to a semantic occupancy prediction method based on lidar point cloud and image fusion. Background Technology

[0002] In recent years, the importance of autonomous driving technology has gradually increased, and the safety and reliability of autonomous driving are the main focus of related research. Current autonomous driving systems mainly rely on 3D object detection and semantic occupancy prediction methods to perceive the surrounding environment and assist the system in making safe driving plans. 3D object detection primarily perceives moving objects in the driving scene, including pedestrians and vehicles, representing these objects as cuboid anchor boxes. However, 3D object detection methods still have some shortcomings. They only detect common moving objects in the scene, ignoring the perception of potential obstacles. Furthermore, the way objects are represented as anchor boxes is not precise enough. Semantic occupancy prediction can effectively compensate for the shortcomings of 3D object detection. The semantic occupancy prediction method first divides the driving space into a series of three-dimensional voxels with equal side lengths, and then assigns a semantic category to each voxel, typically including pedestrians, different vehicles, buildings, roads, vegetation, etc. The semantic occupancy prediction method comprehensively examines the objects that may appear in the driving space and, compared to 3D object detection, can perceive the shape details of objects more accurately. The overall precision of semantic occupancy prediction depends on the size of the divided voxels; the smaller the size, the more precise the scene perception.

[0003] Semantic occupancy prediction methods typically involve three steps: feature extraction and viewpoint transformation, encoding / decoding, and semantic prediction. The network first reads the data for the corresponding modality, extracts its features using a backbone network, and then transforms the features to a 3D spatial level or a BEV level. Next, the features are input into the encoder and decoder. The encoder continuously downsamples the features to extract deeper features, expanding the receptive field. The decoder continuously upsamples the features to restore resolution and uses skip connections to merge features of corresponding scales from the encoder and decoder to enrich the semantic information contained within the features. Finally, in the semantic prediction stage, the network predicts the probability distribution of all categories for each voxel, selecting the one with the highest probability as the semantic category of the current voxel, thus obtaining the semantic occupancy prediction for the entire voxelized scene.

[0004] Currently, semantic occupancy prediction mainly includes methods based on LiDAR point clouds and image-based methods.

[0005] LiDAR point clouds are collections of points distributed in three-dimensional space, acquired using LiDAR. Their attributes include Cartesian coordinates (x, y, z) and reflection intensity. LiDAR point clouds can accurately represent the spatial location of points and have wide applications in autonomous driving. Point cloud-based methods typically voxelize the input point cloud data to obtain sparse voxels, then use a 3D sparse convolutional network to extract the corresponding voxel features. Before inputting these features into the encoder, they are densified into ordinary 3D vector features. However, using a 3D convolutional encoder and decoder consumes significant computational resources, resulting in slow computation speed and high cost. Currently, most methods choose to transform the 3D features into 2D features at the bird's-eye view level, using a 2D convolutional network to construct the encoder and decoder, which can significantly reduce computational overhead and time. Using LiDAR point clouds for semantic occupancy prediction has certain limitations because the point cloud distribution is sparse on the object's surface, but absent inside the object.

[0006] When using images for semantic occupancy prediction, a challenge arises because images are two-dimensional (2D), while driving scenarios are three-dimensional (3D). Therefore, viewpoint transformation methods are needed to convert 2D image features into 3D features. Common methods include the frustum method and the projection method. The frustum method utilizes computer graphics principles, treating the camera's field of view as a cone in space. A neural network is used to estimate the depth of the image, and the predicted depth, combined with the frustum, assigns corresponding feature values ​​to each position within the frustum. The projection method uses the camera's parameter matrix to project each voxel onto the image, sampling at the corresponding positions to obtain 3D features. Subsequent processing after obtaining the 3D features is similar to point cloud-based methods. Image-based methods, when performing 3D occupancy prediction, suffer from a lack of depth information and the camera's susceptibility to occlusion, resulting in worse performance compared to LiDAR point clouds. Furthermore, the 2D-to-3D conversion is prone to bias and blurring, impacting performance. One approach is to use LiDAR and image fusion techniques for semantic occupancy prediction. Images contain rich and dense semantic information, which can compensate for the sparse nature of LiDAR point clouds. Simultaneously, LiDAR point clouds can also provide depth information for images. LiDAR point cloud and image fusion is an important research direction in the field of autonomous driving. Summary of the Invention

[0007] To address the aforementioned technical issues, this invention provides a semantic occupancy prediction method based on the fusion of lidar point clouds and images, which enhances the model's attention to scarce categories and strengthens the model's perception capabilities.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A semantic occupancy prediction method based on the fusion of lidar point cloud and image includes the following steps:

[0010] Step 1: Read the LiDAR point cloud data and corresponding binocular image data used to train the semantic occupancy prediction model;

[0011] Step 2: After reading the data, perform data preprocessing, including image normalization, cropping, random flipping of the point cloud, and voxelization of the point cloud to obtain the corresponding initial voxel features.

[0012] Step 3: In the point cloud branch, a 3D sparse convolutional backbone network is used to extract multi-level, multi-scale features. In the image branch, after the backbone network extracts features, a viewpoint transformation module is used to upscale the 2D image features to 3D. Subsequently, the network inputs the 3D point cloud and image features into the fusion module. In the fusion module, all features are transformed into a bird's-eye view and then fused based on an attention mechanism to obtain the fused multi-scale features of the point cloud and image. After the decoder performs fusion upsampling on the multi-scale features, the final feature map with the same size as the scene is obtained.

[0013] Step 4: Use convolutional layers in the head network to obtain the final semantic occupancy prediction result, calculate the loss function, update the model parameters, complete the training, and save the training weight file;

[0014] When calculating the loss function, the model randomly samples several semantic category labels that account for a large proportion of the dataset (e.g., more than 8%), reducing the proportion of the majority categories and making the model pay more attention to the scarce categories.

[0015] Step 5: Read the point cloud file and stereo image file used for prediction, complete the point cloud voxelization, read the model training weight file, and use the same network model as in Steps 3 and 4 to obtain the final semantic occupancy prediction result.

[0016] Beneficial effects:

[0017] 1) It can effectively improve the overall accuracy of semantic occupancy prediction; existing semantic occupancy prediction methods only use LiDAR point clouds or image single-modality, which has certain limitations. This invention uses a dual-modality fusion method of LiDAR point clouds and images, using images to supplement semantic information for point cloud features, improving the model's ability to recognize object interfaces and small objects, and improving the overall accuracy of semantic occupancy prediction;

[0018] 2) Simultaneously use both lidar point cloud and image data to complete the semantic occupancy prediction task;

[0019] 3) Use an attention-based fusion module to promote the fusion of LiDAR point clouds and images to obtain fusion features with richer semantic information.

[0020] 4) Effectively improves the accuracy of rare category recognition; the autonomous driving dataset uses LiDAR and cameras to collect data from real driving environments, thus exhibiting a class imbalance problem. In the dataset, categories such as roads and vehicles have a large proportion, while obstacles such as traffic signs and tree trunks have a smaller proportion. The model often overemphasizes the larger-proportion categories. This invention employs a label sampling method, randomly sampling the larger-proportion categories in most frames to increase the model's attention to rare categories and enhance its perceptual capabilities. Attached Figure Description

[0021] Figure 1 This is a flowchart of the semantic occupancy prediction model training process of the present invention;

[0022] Figure 2 This is a diagram of the semantic occupancy prediction network structure proposed in this invention;

[0023] Figure 3 This is the overall structure of the point cloud and image fusion model based on the attention mechanism proposed in this invention;

[0024] Figure 4 This is the overall structure of the attention-based skip connection upsampling network proposed in this invention;

[0025] Figure 5 To visualize the semantic occupancy prediction results for four different driving scenarios, we compared them with ground truth labels and other advanced methods (SSC-RS); where (a) is the first driving scenario, (b) is the second driving scenario, (c) is the third driving scenario, and (d) is the fourth driving scenario. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0027] The semantic occupancy prediction method based on lidar point cloud and image fusion of the present invention includes the following steps:

[0028] Step 1: Read the LiDAR point cloud data and corresponding stereo image data used to train the semantic occupancy prediction model. The point cloud data is in the format of N×4, where N is the number of points, 4 is the three-dimensional coordinates and reflection intensity of each point, and the image data is in the format of 2×3×H×W, where 2 represents the use of stereo images, 3 is the RGB channels of the image, and H and W are the height and width of the image.

[0029] Step 2: After reading the data, perform data preprocessing, including image normalization, cropping, random flipping of the point cloud, and voxelization of the point cloud to obtain the corresponding initial voxel features.

[0030] Step 3: In the point cloud branch, a 3D sparse convolutional backbone network is used to extract multi-level, multi-scale features. In the image branch, after feature extraction using the backbone network, the model uses a viewpoint transformation module to upscale the 2D image features to 3D. Subsequently, the network inputs the 3D point cloud and image features into the fusion module. In the fusion module, all features are transformed into a bird's-eye view and then fused using an attention-based mechanism to obtain fused multi-scale features of the point cloud and image. After the decoder performs fusion and upsampling on the multi-scale features, the final feature map with the same size as the scene is obtained.

[0031] Step 4: Use convolutional layers in the head network to obtain the final semantic occupancy prediction result. Calculate the loss function, update the model parameters, complete training, and save the training weight file.

[0032] When calculating the loss function, the model randomly samples the labels of several semantic categories that account for a large proportion of the dataset, reducing the proportion of the majority categories and making the model pay more attention to the scarce categories.

[0033] Step 5: Read the point cloud file and stereo image file used for prediction, complete the point cloud voxelization, read the model training weight file, and use the same network model as in Steps 3-4 to obtain the final semantic occupancy prediction result.

[0034] Specifically, such as Figure 1 As shown, training the semantic occupancy prediction model includes the following steps:

[0035] Step 1: Read the LiDAR point cloud and image datasets used to train the model;

[0036] Step 2: Perform data augmentation on the read data, including image normalization and random flipping of the point cloud, and voxelize the point cloud data;

[0037] Step 3: Input the voxelized point cloud features into the 3D sparse convolutional backbone network to extract features. At the same time, input the image into the image backbone network to extract features, and then input it into the viewpoint conversion module to obtain the three-dimensional image features.

[0038] Step 4: Input the 3D voxel features and image features into the fusion module, and convert them to the bird's-eye view level for feature fusion;

[0039] Step 5: Input the fused features into the decoder and upsample to restore the feature size;

[0040] Step 6: Input the features into the head network to obtain the semantic occupancy prediction output;

[0041] Step 7: Calculate the loss function and update the model until all training rounds are completed.

[0042] like Figure 2 As shown, the overall network structure of the semantic occupancy prediction model is similar to that of the semantic segmentation network, both being UNet structures based on an encoder and a decoder. The encoder extracts deep features through downsampling, while the decoder uses skip connections and upsampling modules to restore feature resolution for semantic classification.

[0043] like Figure 2 As shown, the network's encoder includes a point cloud branch and an image branch. The point cloud branch comprises a symmetric backbone network and a semantic branch for calculating the auxiliary loss function, both of which are composed of 3D sparse convolutions. The image branch includes an image backbone network and a viewpoint transformation module. Between the encoder and decoder, the network includes a fusion module, which takes as input the multi-scale features output from the point cloud and image branches. The fusion module uses an attention mechanism to fuse the point cloud and image features at multiple scales. The network's decoder receives the previously generated multi-scale features, uses skip connections and an upsampling network to restore the features to the same size as the voxelized scene, and then uses a prediction head to obtain the final output of semantic occupancy prediction.

[0044] like Figure 3 As shown, the attention-based point cloud and image fusion network contains a multi-layer fusion model, with inputs being point cloud features of the same size, image features, and the output of the previous fusion model.

[0045] During viewpoint transformation, the network predicts depth from the image and uses point clouds to supervise the depth prediction process. 3D image features obtained using depth and frustum methods may suffer from distortion and blurring. Common vector concatenation methods for aggregating point cloud and image features can lead to alignment issues, resulting in poor feature fusion performance. This invention proposes using an attention-based fusion network to aggregate features from LiDAR point clouds and images.

[0046] The model establishes fusion networks for point cloud and image features at different scales. The features input into the fusion network include the outputs of the symmetric backbone network, the semantic branch, the previous fusion network layer, and the image branch, all of which are two-dimensional features transformed to a bird's-eye view. The network first calculates the weights for the first three types of features using average pooling, convolutional layers, and activation functions. The corresponding features are then multiplied by the weights and concatenated to obtain the aggregated point cloud features.

[0047] ;

[0048] in, For the initial fusion of point cloud features, This represents the concatenation operation, and σ represents the activation function. Represents convolution. Indicates average pooling. This is the output of the previous layer fusion network. The semantic branch corresponds to the output of the k-th layer. This represents the output of the k-th layer of a symmetric backbone network.

[0049] Subsequently, a key-value pair attention mechanism is used to fuse point cloud features and image features, with point cloud features serving as the query and image features as the key. Finally, the number of channels is adjusted using a 2D convolution with a kernel size of 1 to obtain the final fused features.

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] in, This represents the key-value pair attention mechanism, where Q, K, and V are the corresponding query, key, and value. It is a normalized exponential function; This represents one of the heads in a multi-head attention mechanism. Indicates learnable parameters; As a multi-head attention mechanism, This indicates a splicing operation. These are learnable parameters; The features are obtained by fusing point cloud and image features through a multi-head attention mechanism. This is the output of the image branch corresponding to the k-th layer.

[0056] This invention employs an attention-based skip connection upsampling network. Features downsampled by the encoder contain rich semantic information, but their size is often reduced by eight to sixteen times. Therefore, the decoder needs to upsample the features to restore their size to the same size as the scene for the next step of semantic occupancy prediction.

[0057] This invention uses a skip connection-based upsampling network with an attention mechanism as the decoder. The input to each upsampling network layer is the output of the previous layer and features of the corresponding size from the fusion module. Using features from the fusion module is a common skip connection technique in encoder-decoder structures. In each upsampling network layer, the features from the previous layer are first upsampled to adjust their size. Then, the features from the fusion module are expanded using dilated convolutions to increase their receptive field. Finally, a multi-head attention mechanism is used to fuse the two features. In multi-head attention, the former serves as the query, and the latter as the key. Finally, a two-dimensional convolutional layer is used to obtain the output upsampled features.

[0058] ;

[0059] ;

[0060] ;

[0061] ;

[0062] The above formula illustrates the working principle of the k-th layer upsampling network. This indicates features that have undergone upsampling; upsample represents the upsampling operation. This is the output of the previous upsampling network. Features processed by dilated convolution, For the output of the fusion network corresponding to the k-th layer, Indicates dilated convolution, This indicates features that have undergone multi-head attention processing. This is the output of the upsampling network at this layer. This represents a convolution with a kernel size of 1.

[0063] like Figure 4 As shown, the inputs to each layer of the upsampling network are the output of the previous layer's upsampling model and the output of the corresponding fusion module.

[0064] The following describes the semantic prediction head of this invention. The features output by the decoder are two-dimensional features of the same size as the scene on the bird's-eye view layer. Therefore, this invention uses a two-dimensional convolution as the prediction head. The number of output channels is the product of the semantic category C and the scene height H, i.e., the shape of the output feature is C*H×L×W, where L and W are the length and width of the scene. The feature shape is then reshaped into a three-dimensional vector of C×H×L×W, and the output value represents the probability distribution of the C categories corresponding to each voxel. After performing a softmax operation on the semantic category channels, the category with the highest probability is selected as the predicted semantic category for that voxel.

[0065] The training of this invention is performed on the publicly available SemanticKITTI dataset. Most autonomous driving datasets, including this one, are sampled from real-world driving scenarios, thus exhibiting an imbalance in the distribution of semantic label categories. In real-world sampling, background voxels and environmental voxels, such as buildings and ground, have a high proportion, while moving targets, which are more important in autonomous driving, including vehicles and pedestrians, have a low proportion. This causes the model to overemphasize high-proportion categories, resulting in lower accuracy for low-proportion categories. Although existing methods use weighted functions when calculating the loss function during training, the results are still not ideal. Step 4 of this invention uses a label sampling module. When calculating the loss function, it randomly samples several categories with a high proportion (excluding roads, as roads are more important). If the number exceeds a certain threshold (e.g., greater than 8%), it randomly samples voxels corresponding to the larger categories in the frame's label, filtering out a certain proportion of voxels to ensure the model focuses more on those categories with a lower proportion in the dataset, thus improving the overall performance of the model. Specifically, we first count the number of voxels in several semantic categories that exceed 8% mentioned earlier in each frame of data, and then sample more than 2,000 frames. The sampling filtering ratio is calculated from the proportion of each category in the dataset. Approximately 60% of the voxels in these categories are filtered out and do not participate in the loss function calculation.

[0066] like Figure 5 The image shows a comparison of semantic occupancy prediction results with ground truth labels and other state-of-the-art methods (SSC-RS) after visualizing the results from four different driving scenarios. Figure 5 As can be seen from the area within the white circle, the model of this invention exhibits better perceptual performance on object interfaces, small objects, and categories with a smaller proportion, i.e., from... Figure 5 As can be seen from (a), this model performs well at the boundaries. Figure 5 (b) Figure 5 As can be seen from (d), this model performs well on small objects. Figure 5 As can be seen from (c), the present invention performs well on continuous object surfaces.

Claims

1. A semantic occupancy prediction method based on lidar point cloud and image fusion, characterized in that, Includes the following steps: Step 1: Read the LiDAR point cloud data and corresponding binocular image data used to train the semantic occupancy prediction model; Step 2: After reading the data, perform data preprocessing, including image normalization, cropping, random flipping of the point cloud, and voxelization of the point cloud to obtain the corresponding initial voxel features. Step 3: In the point cloud branch, a 3D sparse convolutional backbone network is used to extract multi-level and multi-scale features; in the image branch, after the backbone network extracts features, a viewpoint transformation module is used to upscale the two-dimensional image features to three dimensions; subsequently, the network inputs the three-dimensional point cloud and image features into the fusion module. In the fusion module, all features are transformed into a bird's-eye view and then fused based on an attention mechanism to obtain the fused multi-scale features of the point cloud and image. After the decoder fuses and upsamples the multi-scale features, a final feature map with the same size as the scene is obtained. Step 4: Use convolutional layers in the head network to obtain the final semantic occupancy prediction result, calculate the loss function, update the model parameters, complete the training, and save the training weight file; When calculating the loss function, the model randomly samples several semantic category labels that account for more than a threshold of the dataset, reducing the proportion of the majority category and making the model pay more attention to the scarce category. Step 5: Read the point cloud file and stereo image file used for prediction, complete the point cloud voxelization, read the model training weight file, and use the same network model as in Steps 3 and 4 to obtain the final semantic occupancy prediction result. The overall network structure of the semantic occupancy prediction model is based on a UNet structure of encoder and decoder; the encoder extracts deep features through downsampling, and the decoder uses skip connections and upsampling modules to restore feature resolution for semantic classification. The encoder includes a point cloud branch and an image branch. The point cloud branch consists of a symmetric backbone network and a semantic branch for calculating an auxiliary loss function, both of which are composed of three-dimensional sparse convolutions. The image branch includes an image backbone network and a viewpoint transformation module. Between the encoder and decoder, a fusion module is set up in the network. The input of the fusion module is the multi-scale features output from the point cloud branch and the image branch. The fusion module uses an attention mechanism to fuse the point cloud and image features at multiple scales. The decoder receives the previously generated multi-scale features, uses skip connections and an upsampling network to restore the features to the same size as the voxelized scene, and then uses a prediction head to obtain the final output of semantic occupancy prediction. The attention-based point cloud and image fusion network contains a multi-layer fusion model. The inputs are point cloud features of the same size, image features, and the output of the previous fusion model. During the viewpoint transition, the network predicts the depth of the image and uses the point cloud to supervise the depth prediction process. An attention-based fusion network is used to aggregate the features of the LiDAR point cloud and the image. The model establishes fusion networks for point cloud and image features at different scales. The features input into the fusion network include the outputs of the symmetric backbone network, the semantic branch, the previous fusion network layer, and the image branch, all of which are two-dimensional features transformed to a bird's-eye view. The network first calculates the weights for the first three types of features using average pooling, convolutional layers, and activation functions. The corresponding features are multiplied by the weights and then concatenated to obtain the aggregated point cloud features. ； in, For the initial fusion of point cloud features, This represents the concatenation operation, and σ represents the activation function. Represents convolution. Indicates average pooling. This is the output of the previous layer fusion network. The semantic branch corresponds to the output of the k-th layer. This represents the output of the k-th layer of a symmetric backbone network. Subsequently, a key-value pair attention mechanism is used to fuse point cloud features and image features, with point cloud features serving as the query and image features as the key; finally, the number of channels is adjusted through a 2D convolution with a kernel size of 1 to obtain the final fused features. ； ； ； ； ； in, This represents the key-value pair attention mechanism, where Q, K, and V are the corresponding query, key, and value. It is a normalized exponential function; This represents one of the heads in a multi-head attention mechanism. Indicates learnable parameters; As a multi-head attention mechanism, This indicates a splicing operation. These are learnable parameters; The features are obtained by fusing point cloud and image features through a multi-head attention mechanism. This is the output of the image branch corresponding to the k-th layer.

2. The semantic occupancy prediction method based on lidar point cloud and image fusion according to claim 1, characterized in that, The point cloud data is in the format of N×4, where N is the number of points, 4 is the three-dimensional coordinates and reflection intensity of each point, and the image data is in the format of 2×3×H×W, where 2 represents the use of a stereo image, 3 is the RGB channels of the image, and H and W are the height and width of the image.

3. The semantic occupancy prediction method based on lidar point cloud and image fusion according to claim 1, characterized in that, Training the semantic occupancy prediction model includes the following steps: Step 1: Read the LiDAR point cloud and image datasets used to train the model; Step 2: Perform data augmentation on the read data, including image normalization and random flipping of the point cloud, and voxelize the point cloud data; Step 3: Input the voxelized point cloud features into the 3D sparse convolutional backbone network to extract features. At the same time, input the image into the image backbone network to extract features, and then input it into the viewpoint conversion module to obtain the three-dimensional image features. Step 4: Input the 3D voxel features and image features into the fusion module, and convert them to the bird's-eye view level for feature fusion; Step 5: Input the fused features into the decoder and upsample to restore the feature size; Step 6: Input the features into the head network to obtain the semantic occupancy prediction output; Step 7: Calculate the loss function and update the model until all training rounds are completed.

4. The semantic occupancy prediction method based on lidar point cloud and image fusion according to claim 1, characterized in that, An attention-based skip connection upsampling network is used as the decoder. The input to each upsampling network layer is the output of the previous layer and features of the corresponding size from the fusion module. Using features from the fusion module is the skip connection commonly used in encoder-decoder structures. In each upsampling network layer, the features from the previous layer are first upsampled to adjust their size. Then, the features from the fusion module are expanded using dilated convolutions to increase their receptive field. Finally, a multi-head attention mechanism is used to fuse the two features. In multi-head attention, the former serves as the query, and the latter as the key. Finally, a two-dimensional convolutional layer is used to obtain the output upsampled features. ； ； ； ； The above formula illustrates the working principle of the k-th layer upsampling network. This indicates the features after upsampling. Indicates an upsampling operation. This is the output of the previous upsampling network. Features processed by dilated convolution, For the output of the fusion network corresponding to the k-th layer, Indicates dilated convolution, This indicates features that have undergone multi-head attention processing. This is the output of the upsampling network at this layer. This represents a convolution with a kernel size of 1. The inputs to each layer of the upsampling network are the output of the previous layer's upsampling model and the output of the corresponding fusion module.

5. The semantic occupancy prediction method based on lidar point cloud and image fusion according to claim 1, characterized in that, Using a 2D convolution as the prediction head, the number of output channels is the product of the semantic category C and the scene height H, meaning the shape of the output feature is C. H×L×W, where L and W are the length and width of the scene; then the feature shape is reshaped into a three-dimensional vector of C×H×L×W, and the output value is the probability distribution of C categories corresponding to each voxel; after performing a softmax operation on the semantic category channel, the category with the highest probability is selected as the semantic category predicted for that voxel.

6. The semantic occupancy prediction method based on lidar point cloud and image fusion according to claim 1, characterized in that, In step 4, the label sampling module is used to count the number of the major categories when calculating the loss function. If the number exceeds the threshold, the voxels corresponding to the major categories in the label of the frame are randomly sampled to filter out a certain proportion of voxels, so as to ensure that the model pays more attention to the categories that account for a small proportion in the dataset and improves the overall performance of the model.

Images