Cross-dimension branch automatic driving semantic occupancy rate prediction method

Abstract

The invention provides a cross-dimensional branch automatic driving semantic occupancy rate prediction method, which is specially used for efficiently extracting features from a 2D image and then projecting the features to a high-dimensional space for optimization, can more effectively convert image data into 3D voxels for representation, and remarkably improves the accuracy of semantic occupancy prediction. The method mainly comprises the following five parts: (1) carrying out feature extraction on an input RGB (Red, Green and Blue) image by adopting an ImageEncoder component; (2) extracting 2D image features into a 3D environment by adopting an Image Trans component and keeping channel efficiency; (3) a View Transformation module is adopted to carry out conversion on the characteristics of the input image; (4) refining voxel features by adopting an XDB-BEVStream module; and (5) improving the performance of the model by adopting an integrated loss function suite. According to the method, SemanticKITTI and nuScenes data sets are used for verification, and compared with the most advanced SOTA method, the XDB-Occ framework is obviously superior to the method in the aspect of semantic occupancy prediction.

Description

Technical Field

[0001] The present invention relates to technical fields such as computer vision processing technology, autonomous driving visual perception, deep learning, and three-dimensional semantic occupancy prediction. Background Art

[0002] With the rapid development of technologies such as autonomous driving, intelligent monitoring, and robot navigation, environmental perception systems need to process multi-dimensional data from different sensors, such as lidar, cameras, depth sensors, etc. The information provided by these sensors varies in dimensions and features. How to effectively integrate and utilize these diverse data to improve the understanding and prediction capabilities of complex environments has become an urgent problem to be solved.

[0003] However, in practical applications, traditional occupancy prediction methods often rely on a single data source, resulting in poor performance in complex environments (such as lighting changes, dynamic obstacles and occlusions, etc.), and the robustness and adaptability of the model are limited. This limitation not only affects the performance of the system, but may also lead to safety hazards, especially in autonomous driving and robot navigation scenarios, where the occupancy status of the surrounding environment cannot be accurately judged in real time. Therefore, it is urgent to invent a cross-dimensional branching semantic occupancy prediction method for autonomous driving to solve these problems and enhance the adaptability and robustness of the model to complex environments.

[0004] The cross-dimensional branch occupancy method aims to improve the prediction accuracy of environmental occupancy status through cross-dimensional data fusion and branch network structure. This technology can process information from different dimensions in parallel and extract richer features, thereby enhancing the model's adaptability and robustness to complex environments. By solving the challenges of multi-source data integration and occupancy status prediction, the cross-dimensional branch occupancy technology provides intelligent perception systems with more accurate environmental understanding capabilities and promotes the development of automation and intelligent applications.

[0005] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a cross-dimensional branching method for predicting semantic occupancy in autonomous driving, so as to solve the cross-dimensional mapping problem from two-dimensional image feature extraction to three-dimensional semantic occupancy prediction in the field of autonomous driving.

[0007] The present invention provides a method for predicting semantic occupancy of autonomous driving with cross-dimensional branches, comprising:

[0008] (1) Use the ImageEncoder component to extract features from the input RGB image;

[0009] (2) Use the ImageTrans component to extract 2D image features into a 3D environment while maintaining channel efficiency;

[0010] (3) Use the View Transformation module to transform the features of the input image;

[0011] (4) Use XDB-BEVStream module to refine voxel features;

[0012] (5) Use an integrated loss function suite to improve model performance.

[0013] As a further configuration of the above invention, the ImageEncoder component extracts multi-scale features from the input RGB image through a convolutional neural network.

[0014] As a further configuration of the above invention, the ImageTrans component extracts 2D image features into a 3D environment and maintains channel efficiency, and the steps include:

[0015] (1) Design a hierarchical feature expansion module (HFEM) to extract the output feature maps from the image encoder at three different resolutions and then upscale them by factors of 0.5, 1, and 2, respectively;

[0016] (2) These upsampled feature maps are concatenated to produce a feature map with a resolution of 1 / 16 as the original input.

[0017] As a further configuration of the above invention, a View Transformation module is used to transform the features of the input image, and the steps include:

[0018] (1) Use the DepthNet neural network to estimate the depth information of each pixel in the input image;

[0019] (2) Use the ContextNet neural network to extract contextual features of the input image;

[0020] (3) The information obtained after processing by the DepthNet neural network and the ContextNet neural network is voxel pooled.

[0021] As a further setting of the above invention, XDB-BEVStream refines the voxel feature module, converts volume voxels into BEV features through a dual-path transformer encoder, and bifurcates into local and global paths, focusing on optimizing the BEV path, wherein the carefully designed lightweight separable convolution can expand the receptive field of the BEV feature, and this extension is integrated into the efficient pyramid branch (EPB), which can enhance the perception ability of XDB-Occ about the dependencies between objects; in addition, local residual learning (LRL) and global residual learning (GRL) are combined in EPB to enhance the robustness of the model.

[0022] As a further configuration of the above invention, in the BEVStream module, the perception capability of the model is enhanced by the EPB module, and the steps include:

[0023] (1) The EPB module is used to capture the extended receptive field and the interdependence between objects;

[0024] (2) Using a bottleneck structure in EPB to reduce the amount of computation;

[0025] (3) Residual and shortcut branches are introduced in EPB to transfer information at lower layers, thereby deepening the network.

[0026] As a further setting of the above invention, in the BEVStream module, features are extracted in the horizontal and vertical directions of the input feature map respectively to ensure that attention is focused on the key parts of the image. The separable convolution uses spatial dilation convolution with different dilation rates to further extract features. After a series of convolution operations, the separable convolution fuses the obtained features through the final convolution layer (1×1Conv) to generate the final attention map. The attention map is element-wise multiplied with the original input feature map. Each element of the original feature map is weighted according to the value of the attention map to highlight important features and suppress unimportant features.

[0027] Given an input feature map F∈R C×H×W , where C represents the number of channels, H and W represent the height and width of the feature map respectively, the convolution operation is represented by *, and the Hadamard product (element product) is represented by Indicates that is an intermediate feature map obtained by performing a series of dilated one-dimensional convolutions on F, using kernels of size (2d-1)×1 and 1x(2d-1), where d is the dilation rate; then, Z C Yes The feature map produced by the depthwise convolution performed uses a kernel size of and Where K is the kernel size. Attention map A C It is through Z CThe 1×1 convolution is applied to generate, as spatial weights to highlight the important areas in the feature map; finally, by paying attention to the map A C The output of the separable convolution mechanism is calculated by multiplying the Hadamard product of the original input feature map F by This process effectively enhances the representation of important features while attenuating less relevant information while maintaining computational efficiency.

[0028] The inference formula for separable convolution is as follows:

[0029]

[0030] A C =W 1×1 *Z C

[0031]

[0032] in, represents the intermediate feature map after the initial dilated convolution, Z C is the feature map after deep convolution, A C is the attention map generated by 1×1 convolution, is the final output feature map after applying the attention mechanism.

[0033] The parameters and FLOPs are calculated as follows:

[0034]

[0035] PLOPs = Param × H × W

[0036] As a further setting of the above invention, an integrated loss function suite is used to improve the performance of the model. The classification loss of CrossEntropyLoss is used to enhance foreground recognition by adjusting the class weights (background: 0.1, foreground: 1.0) with a weight of 2.0. Mask loss uses CrossEntropyLoss and Sigmoid activation to zero occupancy prediction with a weight of 5.0; the Dice loss with Sigmoid activation is also used to optimize the segmentation accuracy with the same weight of 5.0, ensuring the performance of the upper model across classification and occupancy prediction. The formula of the core key loss function of the method of the present invention, the integral loss function, is as follows:

[0037] L all =0.5×L mask +0.2×L class +0.5×L dice

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) ImageTrans in the XDB-Occ framework can promote 2D image features into a 3D context while maintaining channel efficiency. It ensures that the generated voxel features are enriched with higher dimensions, providing a more detailed and comprehensive understanding of the environment and significantly improving the accuracy of semantic occupancy prediction.

[0040] (2) Extensive experiments on the SemanticKITTI and nuScenes datasets show that XDB-Occ significantly outperforms the state-of-the-art (SOTA) methods; on SemanticKITTI, the proposed method improves the mIoU by 1.16, while on nuScenes, the proposed method improves the mIoU by 1.6. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the invented new framework XDB-Occ;

[0042] Figure 2 It is a schematic diagram of the new framework XDB-Occ of the present invention;

[0043] Figure 3 It is a detailed structural diagram of HFEM;

[0044] Figure 4 These are qualitative results on the SemanticKITTI validation set;

[0045] Figure 5 This is the performance comparison result on the nuScenes dataset;

[0046] Figure 6 These are qualitative results on the nuScenes validation set;

[0047] Figure 7 It is the comparison result of SSC mIoU and SC IoU on the SemanticKITTI dataset;

[0048] Figure 8 It is the comparison result of Mask loss and Dice loss on the SemanticKITTI dataset. DETAILED DESCRIPTION

[0049] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.

[0050] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.

[0051] As attached Figures 1 to 8 As shown, according to a cross-dimensional branching semantic occupancy prediction method for autonomous driving according to a specific embodiment of the present invention, the framework mainly consists of five parts:

[0052] (1) Use the ImageEncoder component to extract features from the input RGB image;

[0053] (2) Use the ImageTrans component to extract 2D image features into a 3D environment while maintaining channel efficiency;

[0054] (3) Use the View Transformation module to transform the features of the input image;

[0055] (4) Use XDB-BEVStream module to refine voxel features;

[0056] (5) Use an integrated loss function suite to improve model performance.

[0057] The first part is to extract features from the input RGB image.

[0058] The ImageEncoder component extracts multi-scale features from the input RGB image through a convolutional neural network.

[0059] In the second part, the ImageTrans component is used to extract 2D image features into a 3D environment while maintaining channel efficiency. The steps include:

[0060] (1) Design a hierarchical feature expansion module (HFEM) to extract the output feature maps from the image encoder at three different resolutions and then upscale them by factors of 0.5, 1, and 2, respectively;

[0061] (2) These upsampled feature maps are concatenated to produce a feature map with a resolution of 1 / 16 as the original input.

[0062] In the third part, the View Transformation module is used to transform the features of the input image, including:

[0063] (1) Use the DepthNet neural network to estimate the depth information of each pixel in the input image;

[0064] (2) Use the ContextNet neural network to extract contextual features of the input image;

[0065] (3) The information obtained after processing by the DepthNet neural network and the ContextNet neural network is voxel pooled.

[0066] Part 4: Refine voxel features using the XDB-BEVStream module

[0067] Volume voxels are converted into BEV features through a dual-path transformer encoder, which is bifurcated into local and global paths. The focus is on optimizing the BEV path, where the carefully designed lightweight separable convolution can expand the receptive field of BEV features. This extension is integrated into the efficient pyramid branch (EPB), which can enhance the perception ability of XDB-Occ about inter-object dependencies; in addition, local residual learning (LRL) and global residual learning (GRL) are combined in EPB to enhance the robustness of the model.

[0068] In the BEVStream module, the perception capability of the model is enhanced through the EPB module, and the steps include:

[0069] (1) The EPB module is used to capture the extended receptive field and the interdependence between objects;

[0070] (2) Using a bottleneck structure in EPB to reduce the amount of computation;

[0071] (3) Residual and shortcut branches are introduced in EPB to transfer information at lower layers, thereby deepening the network.

[0072] In the BEVStream module, features are extracted in the horizontal and vertical directions of the input feature map to ensure that attention is focused on the key parts of the image. The separable convolution uses spatial dilated convolutions with different dilation rates to further extract features. After a series of convolution operations, the separable convolution fuses the features obtained through the final convolution layer (1×1Conv) to generate the final attention map, which is element-wise multiplied with the original input feature map. Each element of the original feature map is weighted according to the value of the attention map to highlight important features and suppress unimportant features.

[0073] Given an input feature map F∈R C×H×W , where C represents the number of channels, H and W represent the height and width of the feature map respectively, the convolution operation is represented by *, and the Hadamard product (element product) is represented by Indicates that is an intermediate feature map obtained by performing a series of dilated one-dimensional convolutions on F, using kernels of size (2d-1)×1 and 1x(2d-1), where d is the dilation rate; then, ZC Yes The feature map produced by the depthwise convolution performed uses a kernel size of and Where K is the kernel size and the attention map A C It is through Z C The 1×1 convolution is applied to generate, as spatial weights to highlight the important areas in the feature map; finally, by paying attention to the map A C The output of the separable convolution mechanism is calculated by multiplying the Hadamard product of the original input feature map F by This process effectively enhances the representation of important features while attenuating less relevant information while maintaining computational efficiency.

[0074] The inference formula for separable convolution is as follows:

[0075]

[0076] A C =W 1×1 *Z C

[0077]

[0078] in, represents the intermediate feature map after the initial dilated convolution, Z C is the feature map after deep convolution, A C is the attention map generated by 1×1 convolution, is the final output feature map after applying the attention mechanism.

[0079] The parameters and FLOPs are calculated as follows:

[0080]

[0081] PLOPs = Param × H × W

[0082] Part 5: Improving model performance using an integrated loss function suite

[0083] Using the classification loss of CrossEntropyLoss, the foreground recognition is enhanced by adjusting the class weights (background: 0.1, foreground: 1.0) with a weight of 2.0. The Mask loss uses CrossEntropyLoss and Sigmoid activation to zero the occupancy prediction with a weight of 5.0; the Dice loss with Sigmoid activation is also used to optimize the segmentation accuracy with the same weight of 5.0, ensuring the performance of the upper model across classification and occupancy prediction. The formula of the integral loss function, the core key loss function of the method of the present invention, is as follows:

[0084] L all =0.5×L mask +0.2×L class +0.5×L dice

[0085] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A cross-dimensional branching method for predicting semantic occupancy rate of autonomous driving, characterized in that: include: (1) Use the ImageEncoder component to extract features from the input RGB image; (2) Use the ImageTrans component to extract 2D image features into a 3D environment while maintaining channel efficiency; (3) Use the View Transformation module to transform the features of the input image; (4) Use XDB-BEVStream module to refine voxel features; (5) Use an integrated loss function suite to improve model performance.

2. The cross-dimensional branching semantic occupancy prediction method for autonomous driving according to claim 1, characterized in that: Part (1) uses the ImageEncoder component to extract features from the input RGB image.

3. The method for predicting semantic occupancy of autonomous driving with cross-dimensional branches according to claim 1, characterized in that: The ImageTrans component of part (2) extracts 2D image features into a 3D environment while maintaining channel efficiency. Its steps include: (1) Design a hierarchical feature expansion module (HFEM) to extract the output feature maps from the image encoder at three different resolutions and then upscale them by factors of 0.5, 1, and 2, respectively; (2) These upsampled feature maps are concatenated to produce a feature map with a resolution of 1 / 16 as the original input.

4. The method for predicting semantic occupancy of autonomous driving with cross-dimensional branches according to claim 1, characterized in that: Part (3) uses the ViewTransformation module to transform the features of the input image, and the steps include: (1) Use the DepthNet neural network to estimate the depth information of each pixel in the input image; (2) Use the ContextNet neural network to extract contextual features of the input image; (3) The information obtained after processing by the DepthNet neural network and the ContextNet neural network is voxel pooled.

5. The cross-dimensional branching semantic occupancy prediction method for autonomous driving according to claim 1, characterized in that: The XDB-BEVStream refined voxel feature module in part (4) uses a dual-path transformer encoder to convert volume voxels into BEV features, and divides them into a local path and a global path, with the focus on optimizing the BEV path. The designed lightweight separable convolution can expand the receptive field of the BEV feature. This extension is integrated into the efficient pyramid branch (EPB), which can enhance the perception ability of XDB-Occ about the dependencies between objects. In addition, local residual learning (LRL) and global residual learning (GRL) are combined in EPB to enhance the robustness of the model.

6. The method for predicting semantic occupancy of autonomous driving with cross-dimensional branches according to claim 1, characterized in that: In the BEVStream module of part (4), the perception capability of the model is enhanced through the EPB module, and the steps include: (1) The EPB module is used to capture the extended receptive field and the interdependence between objects; (2) Using a bottleneck structure in EPB to reduce the amount of computation; (3) Residual and shortcut branches are introduced in EPB to transfer information at lower layers, thereby deepening the network.

7. The method for predicting semantic occupancy of autonomous driving with cross-dimensional branches according to claim 1, characterized in that: In the BEVStream module of part (4), features are extracted in the horizontal and vertical directions of the input feature map to ensure that attention is focused on the key parts of the image. Separable convolution uses spatial dilation convolution with different dilation rates to further extract features. After a series of convolution operations, the separable convolution fuses the features obtained through the final convolution layer (1×1Conv) to generate the final attention map, which is element-wise multiplied with the original input feature map. Each element of the original feature map is weighted according to the value of the attention map to highlight important features and suppress unimportant features. Given an input feature map F∈R C×H×W , where C represents the number of channels, H and W represent the height and width of the feature map respectively, the convolution operation is represented by *, and the Hadamard product (element product) is represented by Indicates that is an intermediate feature map obtained by performing a series of dilated one-dimensional convolutions on F, using kernels of size (2d-1)×1 and 1x(2d-1), where d is the dilation rate; then, Z C Yes The feature map produced by the depthwise convolution performed uses a kernel size of and Where K is the kernel size and the attention map A C It is through Z C The 1×1 convolution is applied to generate, as spatial weights to highlight the important areas in the feature map; finally, by paying attention to the map A C The output of the separable convolution mechanism is calculated by multiplying the Hadamard product of the original input feature map F by This process effectively enhances the representation of important features while attenuating less relevant information, while maintaining computational efficiency; The inference formula for separable convolution is as follows: AND C =In 1×1 *WITH C in, represents the intermediate feature map after the initial dilated convolution, Z C is the feature map after deep convolution, A C is the attention map generated by 1×1 convolution, is the final output feature map after applying the attention mechanism; the parameters and FLOPs are calculated as follows: PLOPs=Param×H×W.

8. The method for predicting semantic occupancy of autonomous driving with cross-dimensional branches according to claim 1, characterized in that: The integrated loss function suite of part (5) uses the classification loss of CrossEntropyLoss to enhance foreground recognition by adjusting the class weights (background: 0.1, foreground: 1.0) with a weight of 2.

0. Maskloss uses CrossEntropyLoss and Sigmoid activation to zero occupancy prediction with a weight of 5.

0. The Dice loss with Sigmoid activation also optimizes segmentation accuracy with the same weight of 5.0, ensuring the performance of the upper-level model across classification and occupancy prediction. The formula of the integral loss function, the core key loss function of the method of the present invention, is as follows: L all =0.5×L mask +0.2×L class +0.5×L dice 。

Images