Three-dimensional occupancy prediction method based on multi-task learning

Abstract

This invention relates to a 3D occupancy prediction method based on multi-task learning, belonging to the field of autonomous driving environment perception and 3D vision technology. The method involves: extracting multimodal features from a surround view image and point cloud using an image / LiDAR encoder; projecting 2D image features onto a 3D voxel space using bilinear interpolation to compress multimodal BEV features; optimizing BEV features by fusing channel / spatial attention using a dynamic bidirectional attention module; and introducing a separable auxiliary 3D detection head during training, which is jointly optimized with the occupancy prediction head through channel-height conversion to achieve multi-task learning. This invention's method is lightweight, low-overhead, enhances the perception of irregular targets and scene geometry, and achieves high-precision and efficient 3D occupancy prediction in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving environment perception and 3D vision technology, specifically involving a 3D occupancy prediction method based on multi-task learning. Background Technology

[0002] With the rapid development of autonomous driving technology, comprehensive, accurate, and efficient perception of the vehicle's surrounding environment has become a core prerequisite for achieving safe and reliable autonomous driving. 3D occupancy prediction, as an emerging environmental representation method, can densely model scenes in voxel space, simultaneously characterizing geometric structure and semantic information, providing autonomous driving systems with a more complete environmental understanding than traditional 3D object detection. Especially when facing complex scenarios such as irregular objects and special road structures, 3D occupancy prediction exhibits stronger generalization ability and robustness.

[0003] Currently, 3D occupancy prediction methods are mainly divided into pure vision-based schemes and multimodal fusion-based schemes. Pure vision-based methods rely solely on camera images, which have low hardware costs and are easy to deploy. However, due to the lack of direct depth and geometric information, these methods generally rely on monocular depth estimation or complex deformable attention mechanisms to infer 3D structures, resulting in high computational overhead and limited prediction accuracy. Multimodal fusion-based methods, on the other hand, combine camera and LiDAR data, fully utilizing the complementarity of their semantic information and geometric structure, significantly improving the accuracy and robustness of environmental perception.

[0004] However, existing multimodal 3D occupancy prediction methods generally rely on high-resolution input and complex 3D convolutional network structures to achieve superior performance, resulting in a large number of model parameters, high computational complexity, and high deployment costs, making it difficult to meet the actual needs of autonomous driving systems for real-time performance and lightweight design. Although some works have attempted to reduce computational complexity through lightweight design, these are mostly focused on pure vision solutions, and research on lightweight occupancy prediction for LiDAR-camera fusion scenarios remains limited. Furthermore, existing methods often employ simple feature concatenation or cascading operations in the feature fusion stage, failing to fully utilize the complementarity and contextual relationships between multimodal features, thus limiting further improvements in feature representation capabilities. Simultaneously, how to introduce effective auxiliary supervision signals during the training phase to enhance the representation learning of the main task without adding additional computational overhead during the inference phase is also a pressing issue that needs to be addressed. Summary of the Invention

[0005] The purpose of this invention is to provide a 3D occupancy prediction method based on multi-task learning, which can effectively enhance the perception of irregular targets and scene geometry while maintaining a lightweight network structure and low computational overhead, and achieve high-precision and high-efficiency 3D occupancy prediction in complex environments.

[0006] To achieve the above objectives, the technical solution of the present invention is: a three-dimensional occupancy prediction method based on multi-task learning, comprising:

[0007] The system acquires images from surround-view cameras and point cloud data from LiDAR as input.

[0008] Multi-scale image features of the panoramic camera image are extracted by an image encoder, and sparse voxelized point cloud spatial features of the lidar point cloud data are extracted by a lidar encoder. The multi-scale image features are then projected onto a three-dimensional voxel space using bilinear interpolation to generate three-dimensional image features.

[0009] The three-dimensional image features and the sparse voxelized point cloud spatial features are compressed along the height dimension to generate multimodal bird's-eye view BEV features;

[0010] The multimodal bird's-eye view BEV features are fused using a dynamic bidirectional attention fusion module to obtain fused BEV features.

[0011] The fused BEV features are input into a fully convolutional BEV encoder for encoding, and the 3D occupied voxel representation is recovered through a channel-to-height conversion strategy.

[0012] During the training phase, a separable auxiliary 3D target detection head is introduced, and the loss of the auxiliary 3D target detection head is combined with the cross-entropy loss of 3D occupancy prediction as the total loss function to optimize the network parameters; during the inference phase, the auxiliary 3D target detection head is removed.

[0013] Furthermore, the image encoder uses a pre-trained ResNet50 combined with a feature pyramid network FPN, and the lidar encoder is based on a sparse convolutional structure. The lidar point cloud data is first voxelized and organized into sparse tensors, and then the sparse voxelized point cloud spatial features are extracted through an initial sparse convolutional layer and a multi-layer coding module.

[0014] Furthermore, the step of projecting multi-scale image features onto a three-dimensional voxel space using bilinear interpolation specifically includes: predefining a three-dimensional voxel grid with a size of [missing information]. Where Z is the number of voxels in the height direction, and H and W are the height and width of the image feature map; the center point of each voxel is projected onto the corresponding image feature plane through camera intrinsic and extrinsic parameters, and valid projection points that fall within the image feature plane and are located within the camera's field of view are selected and retained; image features are sampled at sub-pixel level using bilinear interpolation on the valid projection points to generate the image feature representation corresponding to each voxel; for voxels located in the overlapping area of ​​the fields of view of multiple cameras in a multi-camera surround view system, the image features from different cameras are averaged and fused to obtain three-dimensional image features.

[0015] Furthermore, the processing flow of the dynamic bidirectional attention fusion module includes: concatenating the input multimodal bird's-eye view BEV features along the channel dimension, performing preliminary fusion through a 3×3 two-dimensional convolutional layer to obtain a feature map x; performing global average pooling on the feature map x, and generating channel attention weights through a two-layer fully connected network. Spatial attention weights are generated by calculating spatial attention weights on feature map x using a 7×7 convolution kernel. Applying channel attention weights and spatial attention weights to feature map x respectively yields weighted features. The weighted features are fused with the original concatenated features after 1×1 convolution and batch normalization through residual connections, and then activated by ReLU to obtain the fused BEV features. .

[0016] Furthermore, the channel-to-height conversion strategy specifically involves converting the shape of the fully convolutional BEV encoder output to... The BEV features are rearranged in the channel dimension into a shape of The 3D occupancy status prediction score, where B is the batch size and C is the number of BEV feature channels. To predict the number of categories, W, H, and Z represent the number of voxels in the x, y, and z directions of three-dimensional space, respectively, and satisfy the following condition: .

[0017] Furthermore, the separable auxiliary 3D target detection head adopts the TransFusion detection head structure, borrows the DETR matching strategy, and uses the Hungarian algorithm to perform binary matching between the predicted result and the real object, reducing the matching cost. It is a weighted average of classification loss, regression loss, and IoU loss.

[0018] Furthermore, the classification loss The regression loss is a binary cross-entropy loss. To predict the normalized L1 loss between BEV centers and true centers, the IoU loss... The matching cost formula is as follows: (This is the IoU loss between the predicted bounding box and the ground truth bounding box.)

[0019]

[0020] in , , Here, represents the weighting coefficients for each loss term, and p represents the predicted classification probability. 'b' represents the true classification label, and 'b' represents the predicted BEV center coordinates or predicted bounding box parameters. These are the actual BEV center coordinates or actual bounding box parameters.

[0021] Furthermore, the total loss function is ,in The cross-entropy loss for 3D occupancy prediction. To assist the total loss of the 3D target detection head, it is composed of classification loss. Regression loss and IoU loss The weighted sum and composition, To balance the losses of auxiliary tasks.

[0022] Furthermore, the AdamW optimizer is used during the training phase, along with a cosine annealing learning rate scheduling strategy that includes a warm-up phase, with an initial learning rate of... The CBGS strategy was used for multi-cycle training, and the spatial range of the point cloud data was set to... Meters, voxel size set to Meters; Image input resolution set to .

[0023] Furthermore, the method was validated on the Occ3D-nuScenes and SurroundOcc datasets. The mean intersection-union ratio (mIoU) was used as the evaluation metric when a camera visibility mask was present, and the RayIoU evaluation metric was used when a camera visibility mask was absent. The RayIoU evaluation metric was calculated as follows: when the predicted class of a query ray is consistent with the true class, and the L1 error between the predicted depth and the true depth is less than a given threshold, the ray is determined to be a true instance (TP).

[0024] Compared to existing technologies, this invention offers the following advantages: The method of this invention achieves high prediction accuracy while maintaining a lightweight network structure through efficient modeling in a bird's-eye view space. Specifically, this invention employs bilinear interpolation to efficiently project two-dimensional image features into a three-dimensional voxel space, avoiding the additional computational overhead of monocular depth estimation and deformable attention mechanisms. Through a designed dynamic bidirectional attention fusion module, channel attention and spatial attention are combined to fully and efficiently fuse multimodal BEV features, resulting in only a minimal increase in model complexity while effectively enhancing the model's perception of small and irregularly shaped targets. Furthermore, this invention introduces a separable auxiliary three-dimensional target detection head during the training phase, significantly improving the network's geometric perception capability by introducing additional geometric and structural constraints, while avoiding any additional computational overhead during the inference phase. This solves the problem of existing methods struggling to balance model complexity and prediction performance. Systematic experiments on the Occ3D-nuScenes and SurroundOcc datasets demonstrate that the present invention outperforms existing methods on multiple evaluation metrics. In particular, it maintains stable and superior prediction accuracy even with low-resolution input and lightweight backbone network configuration, providing a feasible and efficient solution for the design of lightweight 3D occupancy prediction models. Attached Figure Description

[0025] Appendix Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention;

[0026] Appendix Figure 2 This is a schematic diagram of the dynamic bidirectional attention fusion module according to an embodiment of the present invention;

[0027] Appendix Figure 3 This is a visualization diagram of the occupancy prediction dataset according to an embodiment of the present invention. Detailed Implementation

[0028] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] This invention provides a three-dimensional occupancy prediction method based on multi-task learning, comprising:

[0030] The system acquires images from surround-view cameras and point cloud data from LiDAR as input.

[0031] Multi-scale image features of the panoramic camera image are extracted by an image encoder, and sparse voxelized point cloud spatial features of the lidar point cloud data are extracted by a lidar encoder. The multi-scale image features are then projected onto a three-dimensional voxel space using bilinear interpolation to generate three-dimensional image features.

[0032] The three-dimensional image features and the sparse voxelized point cloud spatial features are compressed along the height dimension to generate multimodal bird's-eye view BEV features;

[0033] The multimodal bird's-eye view BEV features are fused using a dynamic bidirectional attention fusion module to obtain fused BEV features.

[0034] The fused BEV features are input into a fully convolutional BEV encoder for encoding, and the 3D occupied voxel representation is recovered through a channel-to-height conversion strategy.

[0035] During the training phase, a separable auxiliary 3D target detection head is introduced, and the loss of the auxiliary 3D target detection head is combined with the cross-entropy loss of 3D occupancy prediction as the total loss function to optimize the network parameters; during the inference phase, the auxiliary 3D target detection head is removed.

[0036] The following is a detailed implementation process of the present invention.

[0037] like Figure 1 As shown, this invention presents a lightweight multimodal 3D occupancy prediction method based on multi-task learning. The method first extracts multimodal features from surround-view camera images and point cloud data using an image encoder and a LiDAR encoder, respectively. Then, bilinear interpolation is used to project the 2D image features into a 3D voxel space, compressing them along the height dimension to generate multimodal bird's-eye view (BEV) features. Next, a dynamic bidirectional attention fusion module efficiently fuses the multimodal BEV features, achieving synergistic enhancement of channel attention and spatial attention. Finally, the fused BEV features are used as input to the occupancy prediction network. A fully convolutional BEV encoder and a channel-to-height conversion strategy are used to recover the 3D occupancy voxel representation. A separable auxiliary 3D target detection head is introduced to apply additional geometric supervision during the training phase. The weighted sum of the cross-entropy loss and the auxiliary detection loss for 3D occupancy prediction is calculated, multi-task learning is performed, and high-precision 3D occupancy prediction results are obtained.

[0038] The method includes the following steps:

[0039] Step S1: Using the panoramic camera image and the LiDAR point cloud data as multimodal inputs, extract multi-scale image features and sparse voxelized point cloud spatial features through the image encoder and the LiDAR encoder, respectively, and project the two-dimensional image features into the three-dimensional voxel space through bilinear interpolation.

[0040] In step S1, image feature extraction and projection are obtained through the following specific steps:

[0041] Step S11: The image encoder uses a pre-trained ResNet50 as the backbone network and combines it with a Feature Pyramid Network (FPN) to process the panoramic camera images and extract image features at multiple scales. High-level features are used to enhance semantic expression, while low-level features preserve fine spatial structure information. The input image resolution is set to... .

[0042] Step S12: The lidar encoder models the original point cloud data based on a sparse convolutional structure. The original point cloud is first voxelized and organized into a sparse tensor, and the spatial extent of the point cloud is set to... Meters, voxel size set to The system then progressively extracts multi-scale spatial features through an initial sparse convolutional layer and a multi-layer coding module, ultimately generating an efficient voxel feature representation for LiDAR.

[0043] Step S13: Predefine a space with a size of [size missing] in three-dimensional space. A regular voxel grid, in which This represents the number of voxels along the height direction. The center point of each voxel is projected onto the corresponding image feature plane using known camera intrinsic and extrinsic parameters. Only valid projection points that simultaneously fall within the image feature plane and are within the camera's field of view are retained. For these valid projection points, image features are sampled at sub-pixel levels using bilinear interpolation, thereby generating a corresponding image feature representation for each voxel. In a multi-camera surround-view system, for voxels located within the overlapping areas of multiple camera fields of view, image features from different cameras are averaged and fused to obtain a more stable and consistent voxel feature representation.

[0044] Step S2: Compress the 3D image features and point cloud voxel features generated in step S1 along the height dimension to generate the corresponding multimodal bird's-eye view (BEV) feature representation.

[0045] Step S3: Efficiently fuse multimodal BEV features using a dynamic bidirectional attention fusion module. This module employs a dual-path attention mechanism, fusing channel attention and spatial attention to enhance feature representation, and retaining original information through residual connections, thereby improving the model's ability to perceive key targets and regions.

[0046] In step S3, the processing flow of the dynamic bidirectional attention fusion module is implemented through the following specific steps:

[0047] Step S31: Input multi-scale BEV feature map Perform splicing operations along the channel dimension, and through a The two-dimensional convolutional layers are used for initial fusion to extract rich contextual information and obtain feature maps. :

[0048] Step S32: Introduce a channel attention mechanism. The feature map \(x\) is processed using global average pooling, and then two fully connected layers are used to generate channel attention weights to enhance the feature responses of key channels and highlight important semantic information.

[0049] in, This indicates a global average pooling operation. and It consists of two fully connected layers. This represents the Sigmoid activation function.

[0050] Step S33: Introduce a spatial attention mechanism. (Using...) Convolution kernel pairs feature maps It performs computations to capture local spatial context information, with a receptive field approximately equivalent to a five-layer convolutional structure, capable of covering key local features of typical targets while maintaining high computational efficiency.

[0051]

[0052] Step S34: Apply the channel attention weights and spatial attention weights to the feature map respectively. Perform channel-wise and spatial location-wise weighted operations to obtain weighted features. :

[0053]

[0054] Step S35: Connect the weighted features with the intermediate features via residual join. The original concatenated features after convolution and batch normalization are fused and then processed by the ReLU activation function to obtain the final fused BEV features. :

[0055] in, This indicates the batch normalization operation.

[0056] Step S4: The fused BEV features are input into a fully convolutional BEV encoder for encoding. Deep feature extraction and contextual information aggregation are performed using an occupancy prediction head, and a channel-to-height conversion strategy is employed to recover the 3D occupancy voxel representation. During the training phase, a separable auxiliary 3D object detection head is introduced as an auxiliary supervision task, jointly optimized with the 3D occupancy prediction task, performing multi-task learning. During the inference phase, this auxiliary detection head is removed to maintain the model's lightweight design and efficient inference.

[0057] In step S4, the 3D occupancy prediction and auxiliary supervision are achieved through the following specific steps:

[0058] Step S41: The occupancy prediction head performs deep feature extraction and contextual information aggregation on the BEV features output by the BEV encoder to enhance the discriminative ability of the features. The occupancy prediction head adopts a multi-scale feature fusion module, which can expand the global receptive field of the network while maintaining the ability to express local details, enabling the model to have a more comprehensive understanding of the overall scene structure.

[0059] Step S42: The BEV features processed by the occupancy prediction head are input to the channel-to-height module. This module explicitly restores the 2D BEV features to a 3D voxel space representation by performing an efficient rearrangement operation in the channel dimension. Specifically, this strategy will transform the 2D BEV features into a 3D voxel space representation. The BEV feature map is a shape of The occupancy status prediction score, among which Indicates batch size, Indicates the number of channels. Indicates the number of categories. and Representing three-dimensional space respectively The number of voxels in the direction, and satisfying Channel dimension The implicit encoding of height information at different height levels allows for effective decoupling and reconstruction of the implicitly encoded height information in the channel dimension into an explicit 3D voxel representation, achieving efficient 3D occupancy prediction without introducing additional 3D convolution operations.

[0060] Step S43: During the training phase, a separable auxiliary 3D object detection head is introduced. This auxiliary detection head adopts the detection head structure proposed in TransFusion, borrows the DETR method, and uses the Hungarian algorithm to perform binary matching between the predicted results and the real objects. The matching cost... Defined by a weighted sum of classification loss, regression loss, and IoU loss:

[0061]

[0062] in, It is a binary cross-entropy loss. It is the L1 loss between the predicted BEV center and the true center (both are normalized to the range [0,1]). It is the IoU loss between the predicted bounding box and the ground truth bounding box. , , These are the weight coefficients for each loss term. For all matching sample pairs, a focus loss is calculated for the classification branch, while bounding box regression is only supervised using L1 loss for positive sample pairs. For heatmap prediction, the focus loss is reduced using the penalty from CenterPoint. Given the optimized BEV features... Two convolutional layers are used to generate key point heatmaps from the BEV perspective. And apply a Gaussian kernel to map the center points of all real 3D bounding boxes to the target heatmap. Above. The training objective, based on the focus loss of the Gaussian heatmap, is defined as follows:

[0063]

[0064] in, and These represent the locations of the heatmaps. The predicted score and the actual value at the location, It is the number of objects in the point cloud. and It is a hyperparameter of focus loss.

[0065] Total loss of auxiliary detection head Represented as:

[0066]

[0067] in, and The loss weighting coefficients are set to 0.25 and 0.25 respectively in this embodiment.

[0068] Step S44: The overall loss function is composed of the auxiliary 3D target detection loss. and 3D occupancy prediction loss Together they form a whole. The occupancy prediction task uses a simple cross-entropy loss. The final total loss function is defined as follows: in, This is the weight for the loss of the auxiliary 3D object detection task, which is set to 0.01 during training.

[0069] Step S45: During model training, the optimizer uses AdamW, along with a cosine annealing learning rate scheduling strategy that includes a warm-up phase. The initial learning rate is set to... Training was performed using a CBGS strategy for 6 training epochs with a batch size of 4. All experiments were conducted on a single NVIDIA RTX 4090 GPU.

[0070] The proposed method was validated on the Occ3D-nuScenes and SurroundOcc large-scale autonomous driving datasets. The mean intersection-union ratio (mIoU) was used as the evaluation metric when there was a camera visibility mask, and the RayIoU evaluation metric with semantic awareness was introduced when there was no camera visibility mask, so as to comprehensively evaluate the 3D occupancy prediction performance of the model in complex urban environments.

[0071] The RayIoU evaluation metric is calculated as follows: when the predicted class of a query ray is consistent with the true class, and the L1 error between the predicted depth and the true depth is less than a given threshold, the ray is determined to be a TruePositive (TP).

[0072] Example:

[0073] like Figure 1 As shown, this embodiment provides a three-dimensional occupancy prediction method based on multi-task learning.

[0074] This embodiment includes the following specific steps:

[0075] Step S1: Using the panoramic camera image and the LiDAR point cloud data as multimodal inputs, extract multi-scale image features and sparse voxelized point cloud spatial features through the image encoder and the LiDAR encoder, respectively, and project the two-dimensional image features into the three-dimensional voxel space through bilinear interpolation.

[0076] In step S1, image feature extraction and projection are obtained through the following specific steps:

[0077] Step S11: The image encoder uses a pre-trained ResNet50 as the backbone network and combines it with a Feature Pyramid Network (FPN) to process the panoramic camera images and extract image features at multiple scales. High-level features are used to enhance semantic expression, while low-level features preserve fine spatial structure information. The input image resolution is set to... .

[0078] Step S12: The lidar encoder models the original point cloud data based on a sparse convolutional structure. The original point cloud is first voxelized and organized into a sparse tensor, and the spatial extent of the point cloud is set to... Meters, voxel size set to The system then progressively extracts multi-scale spatial features through an initial sparse convolutional layer and a multi-layer coding module, ultimately generating an efficient voxel feature representation for LiDAR.

[0079] Step S13: Predefine a space with a size of [size missing] in three-dimensional space. A regular voxel grid, in which This represents the number of voxels along the height direction. The center point of each voxel is projected onto the corresponding image feature plane using known camera intrinsic and extrinsic parameters. Only valid projection points that simultaneously fall within the image feature plane and are within the camera's field of view are retained. For these valid projection points, image features are sampled at sub-pixel levels using bilinear interpolation, thereby generating a corresponding image feature representation for each voxel. In a multi-camera surround-view system, for voxels located within the overlapping areas of multiple camera fields of view, image features from different cameras are averaged and fused to obtain a more stable and consistent voxel feature representation.

[0080] Step S2: Compress the 3D image features and point cloud voxel features generated in step S1 along the height dimension to generate the corresponding multimodal bird's-eye view (BEV) feature representation.

[0081] Step S3: Efficiently fuse multimodal BEV features using a dynamic bidirectional attention fusion module. This module employs a dual-path attention mechanism, fusing channel attention and spatial attention to enhance feature representation, and retaining original information through residual connections, thereby improving the model's ability to perceive key targets and regions.

[0082] like Figure 2 As shown, in step S3, the processing flow of the dynamic bidirectional attention fusion module is implemented through the following specific steps:

[0083] Step S31: Input multi-scale BEV feature map Perform splicing operations along the channel dimension, and through a The two-dimensional convolutional layers are used for initial fusion to extract rich contextual information and obtain feature maps. :

[0084] Step S32: Introduce a channel attention mechanism. The feature map \(x\) is processed using global average pooling, and then two fully connected layers are used to generate channel attention weights to enhance the feature responses of key channels and highlight important semantic information.

[0085] in, This indicates a global average pooling operation. and It consists of two fully connected layers. This represents the Sigmoid activation function.

[0086] Step S33: Introduce a spatial attention mechanism. (Using...) Convolution kernel pairs feature maps It performs computations to capture local spatial context information, with a receptive field approximately equivalent to a five-layer convolutional structure, capable of covering key local features of typical targets while maintaining high computational efficiency.

[0087]

[0088] Step S34: Apply the channel attention weights and spatial attention weights to the feature map respectively. Perform channel-wise and spatial location-wise weighted operations to obtain weighted features. :

[0089]

[0090] Step S35: Connect the weighted features with the intermediate features via residual join. The original concatenated features after convolution and batch normalization are fused and then processed by the ReLU activation function to obtain the final fused BEV features. :

[0091] in, This indicates the batch normalization operation.

[0092] Step S4: The fused BEV features are input into a fully convolutional BEV encoder for encoding. Deep feature extraction and contextual information aggregation are performed using an occupancy prediction head, and a channel-to-height conversion strategy is employed to recover the 3D occupancy voxel representation. During the training phase, a separable auxiliary 3D object detection head is introduced as an auxiliary supervision task, jointly optimized with the 3D occupancy prediction task, performing multi-task learning. During the inference phase, this auxiliary detection head is removed to maintain the model's lightweight design and efficient inference.

[0093] In step S4, the 3D occupancy prediction and auxiliary supervision are achieved through the following specific steps:

[0094] Step S41: The occupancy prediction head performs deep feature extraction and contextual information aggregation on the BEV features output by the BEV encoder to enhance the discriminative ability of the features. The occupancy prediction head adopts a multi-scale feature fusion module, which can expand the global receptive field of the network while maintaining the ability to express local details, enabling the model to have a more comprehensive understanding of the overall scene structure.

[0095] Step S42: The BEV features processed by the occupancy prediction head are input to the channel-to-height module. This module explicitly restores the 2D BEV features to a 3D voxel space representation by performing an efficient rearrangement operation in the channel dimension. Specifically, this strategy will transform the 2D BEV features into a 3D voxel space representation. The BEV feature map is a shape of The occupancy status prediction score, among which Indicates batch size, Indicates the number of channels. Indicates the number of categories. and Representing three-dimensional space respectively The number of voxels in the direction, and satisfying Channel dimension The implicit encoding of height information at different height levels allows for effective decoupling and reconstruction of the implicitly encoded height information in the channel dimension into an explicit 3D voxel representation, achieving efficient 3D occupancy prediction without introducing additional 3D convolution operations.

[0096] Step S43: During the training phase, a separable auxiliary 3D object detection head is introduced. This auxiliary detection head adopts the detection head structure proposed in TransFusion, borrows the DETR method, and uses the Hungarian algorithm to perform binary matching between the predicted results and the real objects. The matching cost... Defined by a weighted sum of classification loss, regression loss, and IoU loss:

[0097]

[0098] in, It is a binary cross-entropy loss. It is the L1 loss between the predicted BEV center and the true center (both are normalized to the range [0,1]). It is the IoU loss between the predicted bounding box and the ground truth bounding box. , , These are the weight coefficients for each loss term. For all matching sample pairs, a focus loss is calculated for the classification branch, while bounding box regression is only supervised using L1 loss for positive sample pairs. For heatmap prediction, the focus loss is reduced using the penalty from CenterPoint. Given the optimized BEV features... Two convolutional layers are used to generate key point heatmaps from the BEV perspective. And apply a Gaussian kernel to map the center points of all real 3D bounding boxes to the target heatmap. Above. The training objective, based on the focus loss of the Gaussian heatmap, is defined as follows:

[0099]

[0100] in, and These represent the locations of the heatmaps. The predicted score and the actual value at the location, It is the number of objects in the point cloud. and It is a hyperparameter of focus loss.

[0101] Total loss of auxiliary detection head Represented as:

[0102] in, and The loss weighting coefficients are set to 0.25 and 0.25 respectively in this embodiment.

[0103] Step S44: The overall loss function is composed of the auxiliary 3D target detection loss. and 3D occupancy prediction loss Together they form a whole. The occupancy prediction task uses a simple cross-entropy loss. The final total loss function is defined as follows: in, This is the weight for the loss of the auxiliary 3D object detection task, which is set to 0.01 during training.

[0104] Step S45: During model training, the optimizer uses AdamW, along with a cosine annealing learning rate scheduling strategy that includes a warm-up phase. The initial learning rate is set to... Training was performed using a CBGS strategy for 6 training epochs with a batch size of 4. All experiments were conducted on a single NVIDIA RTX 4090 GPU.

[0105] The method was validated on the Occ3D-nuScenes and SurroundOcc large-scale autonomous driving datasets. The mean Intersection over Union (mIoU) was used as the evaluation metric when camera visibility masks were present, and a semantically aware RayIoU metric was introduced when camera visibility masks were absent to comprehensively evaluate the model's 3D occupancy prediction performance in complex urban environments. The RayIoU metric was calculated as follows: when the predicted class of a query ray matched the ground truth class, and the L1 error between the predicted depth and the ground truth depth was less than a given threshold, the ray was considered a TruePositive (TP).

[0106] This embodiment presents systematic experimental validation on the Occ3D-nuScenes and SurroundOcc large-scale autonomous driving datasets. On the Occ3D-nuScenes dataset, the camera visibility mask provided by the Occ3D-nuScenes dataset is used for evaluation. To verify the effectiveness of the Dynamic Bidirectional Attention Fusion Module (DBAFusion), the Separable Assisted 3D Object Detection Head (ADH), and the Bilinear Interpolation Method (BI), ablation experiments were performed on the Occ3D-nuScenes validation set, and the results are shown in Table 1.

[0107]

[0108] Figure 3 This is a visualization diagram of the occupancy prediction dataset according to an embodiment of the present invention.

[0109] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A three-dimensional occupancy prediction method based on multi-task learning, characterized in that, include: The system acquires images from surround-view cameras and point cloud data from LiDAR as input. Multi-scale image features of the panoramic camera image are extracted by an image encoder, and sparse voxelized point cloud spatial features of the lidar point cloud data are extracted by a lidar encoder. The multi-scale image features are projected onto a three-dimensional voxel space using bilinear interpolation to generate three-dimensional image features. The three-dimensional image features and the sparse voxelized point cloud spatial features are compressed along the height dimension to generate multimodal bird's-eye view BEV features; The multimodal bird's-eye view BEV features are fused using a dynamic bidirectional attention fusion module to obtain fused BEV features. The fused BEV features are input into a fully convolutional BEV encoder for encoding, and the 3D occupied voxel representation is recovered through a channel-to-height conversion strategy. During the training phase, a separable auxiliary 3D target detection head is introduced, and the loss of the auxiliary 3D target detection head is combined with the cross-entropy loss of 3D occupancy prediction as the total loss function to optimize the network parameters; during the inference phase, the auxiliary 3D target detection head is removed.

2. The method of claim 1, wherein, The image encoder uses a pre-trained ResNet50 combined with a feature pyramid network (FPN). The lidar encoder is based on a sparse convolutional structure. The lidar point cloud data is first voxelized and organized into sparse tensors, and then the sparse voxelized point cloud spatial features are extracted through an initial sparse convolutional layer and a multi-layer coding module.

3. The three-dimensional occupancy prediction method based on multi-task learning according to claim 1, characterized in that, The projecting the multi-scale image features to the three-dimensional voxel space by bilinear interpolation specifically comprises: predefining a three-dimensional voxel grid with a size of wherein Z is the number of voxels in the height direction, H and W are the height and width of the image feature map; projecting the center point of each voxel to the corresponding image feature plane through the camera internal and external parameters, screening and retaining the effective projection points that fall within the range of the image feature plane and are located within the camera field of view; using bilinear interpolation to sample the image features at the sub-pixel level position for the effective projection points, generating the image feature representation corresponding to each voxel; for the voxels located in the overlapping region of the fields of view of multiple cameras in the multi-camera surround view system, averaging and fusing the image features from different cameras to obtain the three-dimensional image features.

4. The three-dimensional occupancy prediction method based on multi-task learning according to claim 1, characterized in that, The processing flow of the dynamic bidirectional attention fusion module includes: splicing input multi-modal bird's eye view (BEV) features along a channel dimension, performing preliminary fusion through a 3*3 two-dimensional convolution layer to obtain a feature map x; performing global average pooling on the feature map x, generating channel attention weights through a two-layer fully connected network ; calculating the feature map x using a 7*7 convolution kernel to generate spatial attention weights ; applying the channel attention weights and the spatial attention weights to the feature map x respectively to obtain weighted features ; fusing the weighted features and original spliced features processed through 1*1 convolution and batch normalization through a residual connection to obtain fused BEV features after ReLU activation .

5. The three-dimensional occupancy prediction method based on multi-task learning according to claim 1, characterized in that, The channel-to-height conversion strategy is specifically: rearranging the BEV feature output by the full convolution BEV encoder in the shape of into a three-dimensional occupancy state prediction score in the shape of in the channel dimension, where B is the batch size, C is the number of BEV feature channels, is the number of prediction categories, W, H, Z are respectively the number of voxels in the x, y, z directions in the three-dimensional space, and satisfy .

6. The three-dimensional occupancy prediction method based on multi-task learning according to claim 1, characterized in that, The separable auxiliary three-dimensional target detection head adopts a TransFusion detection head structure, learns from the matching strategy of DETR, performs bipartite matching between the prediction result and the real object through a Hungarian algorithm, and the matching cost is weighted by the classification loss, the regression loss and the IoU loss.

7. The three-dimensional occupancy prediction method based on multi-task learning according to claim 6, characterized in that, The classification loss The regression loss is a binary cross-entropy loss. To predict the normalized L1 loss between BEV centers and true centers, the IoU loss... The matching cost formula is as follows: (This is the IoU loss between the predicted bounding box and the ground truth bounding box.) in , , Here, represents the weighting coefficients for each loss term, and p represents the predicted classification probability. 'b' represents the true classification label, and 'b' represents the predicted BEV center coordinates or predicted bounding box parameters. These are the actual BEV center coordinates or actual bounding box parameters.

8. The three-dimensional occupancy prediction method based on multi-task learning according to claim 1, characterized in that, The total loss function is: ,in The cross-entropy loss for 3D occupancy prediction. To assist the total loss of the 3D target detection head, it is composed of classification loss. Regression loss and IoU loss The weighted sum and composition, To balance the losses of auxiliary tasks.

9. The three-dimensional occupancy prediction method based on multi-task learning according to claim 1, characterized in that, The training phase employs the AdamW optimizer, combined with a cosine annealing learning rate scheduling strategy that includes a warm-up phase, with an initial learning rate of... The CBGS strategy was used for multi-cycle training, and the spatial range of the point cloud data was set to... Meters, voxel size set to Meters; Image input resolution set to .

10. The three-dimensional occupancy prediction method based on multi-task learning according to claim 1, characterized in that, The method was validated on the Occ3D-nuScenes and SurroundOcc datasets. The mean intersection-union ratio (mIoU) was used as the evaluation metric when a camera visibility mask was present, and the RayIoU evaluation metric was used when a camera visibility mask was absent. The RayIoU evaluation metric was calculated as follows: when the predicted class of a query ray is consistent with the true class, and the L1 error between the predicted depth and the true depth is less than a given threshold, the ray is determined to be a true instance (TP).

Images