3D semantic occupancy prediction method, system and medium based on scene visibility

Abstract

This invention discloses a 3D semantic occupancy prediction method, system, and medium based on scene visibility, belonging to the fields of computer vision and autonomous driving technology. The method includes: constructing a bilateral feature fusion module to perform local enhancement and self-attention global enhancement on voxel features respectively, followed by fusion output; sequentially connecting an image feature encoding module, a 2D-3D viewpoint conversion module, a bilateral feature fusion module, an occupancy decoding module, and a semantic decoding module to obtain a 3D semantic occupancy prediction model; constructing a visibility mask generation module to divide all voxels in a certain range of the 3D scene into visible and invisible voxels; constructing a one-to-one cross-entropy supervised loss structure for supervising visible voxel prediction results and invisible voxel prediction results, and a 3D neighborhood block distribution loss structure; and training the 3D semantic occupancy prediction model using the loss structure and a sample set for subsequent prediction. This improves the accuracy of semantic occupancy prediction for the current scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and autonomous driving technology, and more specifically, relates to a 3D semantic occupancy prediction method, system and medium based on scene visibility. Background Technology

[0002] In autonomous driving, accurate perception of obstacles in the surrounding environment is crucial, forming the foundation for subsequent path planning and motion control. Vision-based autonomous driving technology is becoming increasingly common due to the ease of image acquisition; image input provides more precise sensor data compared to LiDAR input. The vision-based semantic occupancy prediction task involves dividing a complete 3D scene into several cubic voxels of a specific length, inputting a surround view image, and outputting the semantic labels for all voxels, including whether they are occupied and the semantic category of obstacles.

[0003] Compared to common perception tasks such as 3D object detection, semantic occupancy prediction is more advantageous in perceiving irregular vehicles, static background obstacles, and blacklisted objects, making it highly effective in autonomous driving perception tasks and trending towards becoming a unified descriptor for multiple perception tasks. However, this task faces two major challenges: 1) Information gaps: Specifically, due to occlusion, the 3D space is divided into visible and invisible regions. The voxels in the visible region within the camera's viewpoint only account for a portion of all 3D voxels, while the voxels in the invisible region lack corresponding visual representation information; 2) Voxelization leads to high computational complexity: Compared to previous 2D forms or bird's-eye views, the computational complexity of semantic occupancy prediction increases from O(HW) to O(HWD), resulting in a significant computational burden.

[0004] Existing methods typically extract 2D panoramic image features, perform 2D-to-3D viewpoint transformation to obtain 3D voxel features, utilize a 3D network (such as 3D U-Net or a self-attention module) to enhance 3D contextual semantics, complete the feature representation of invisible regions, and finally use a unified loss function to supervise the prediction of all voxels. Existing methods have two drawbacks: 1) They lack the ability to process and fuse local and global features, failing to effectively complete the features of invisible parts. Specifically, occluded foreground obstacles (such as vehicles and pedestrians) require local context for completion, while large-area occluded background obstacles (such as buildings and curbs) require reasonable inference from global context information. The local receptive field of 3D U-Net limits its long-range modeling ability, and the self-attention module disrupts the original 3D geometry, ignoring local characteristics; 2) They ignore the significant differences in prediction patterns between visible and invisible regions, causing the network to overemphasize the prediction of the more difficult invisible regions, thus impairing the performance of visible regions and resulting in an overall suboptimal result. Overall, existing methods ignore the differences in feature processing and prediction patterns between visible and invisible regions caused by occlusion, thus greatly limiting overall performance. Summary of the Invention

[0005] In view of the shortcomings and improvement needs of the existing technology, the present invention provides a 3D semantic occupancy prediction method, system and medium based on scene visibility, which aims to improve the accuracy of the semantic occupancy prediction of the current scene.

[0006] To achieve the above objectives, according to one aspect of the present invention, a 3D semantic occupancy prediction method based on scene visibility is provided, comprising: constructing a bilateral feature fusion module, the bilateral feature fusion module including a local perception unit and a long-distance dependency unit set in parallel, to perform local enhancement and self-attention global enhancement on the input voxel features respectively and then fuse the output; sequentially connecting an image feature encoding module, a 2D-3D viewpoint conversion module, the bilateral feature fusion module, an occupancy decoding module and a semantic decoding module to construct a 3D semantic occupancy prediction model; constructing a visibility mask generation module, used to divide the input voxels into visible voxels and invisible voxels, constructing a one-to-one cross-entropy supervised loss structure to supervise the prediction results of visible voxels, and constructing a 3D neighborhood block distribution loss structure to supervise the prediction results of invisible voxels; training the 3D semantic occupancy prediction model with a sample set as the goal of convergence of the one-to-one cross-entropy supervised loss structure and the 3D neighborhood block distribution loss structure; in the prediction stage, directly inputting the target surround view image into the trained 3D semantic occupancy prediction model to obtain the semantic occupancy prediction result of the 3D scene.

[0007] Furthermore, the local sensing unit includes: a convolutional subunit comprising a first convolutional layer, a first batch of normalized layers, a first activation layer, a second convolutional layer, and a second batch of normalized layers connected in sequence; a downsampling subunit comprising a third convolutional layer and a third batch of normalized layers connected in sequence; and a second activation layer, wherein the sum of the outputs of the convolutional subunits and the outputs of the downsampling subunits is output after passing through the second activation layer.

[0008] Furthermore, the long-distance dependency unit includes: a feature compression layer, used to compress the voxel features within the same columnar section according to the intensity distribution of each voxel feature within the same columnar section, to obtain BEV features; a feature self-attention enhancement layer, used to perform attention calculation on the BEV features, to obtain BEV enhanced features; and a feature recombination layer, used to redistribute the BEV enhanced features according to the feature height distribution within the columnar section, to recover the enhanced voxel features.

[0009] Furthermore, the feature recombination layer is specifically used to: perform matrix multiplication calculation on the BEV enhanced features according to the feature height distribution within the columnar body, so as to redistribute the features in the height dimension and obtain the enhanced voxel features.

[0010] Furthermore, the visibility mask generation module includes: a ray occlusion judgment unit, used to perform preliminary visibility judgment on the input voxels, the judgment rule being: the input voxel is regarded as a starting voxel connected to multiple ending voxels to form multiple rays, for any ray, the starting voxel is sequentially judged as an initial visible voxel along the ray direction until an occupying voxel is encountered, and the voxels after the occupying voxel are judged as initial invisible voxels; a camera view range filtering unit, used to project the initial visible voxels onto the image coordinate system, if the coordinates of the projected voxels are not within the preset image range, the corresponding initial visible voxels are corrected to invisible voxels, and the final judgment result of visible and invisible voxels is obtained.

[0011] Furthermore, the 3D neighborhood block distribution loss structure is expressed as:

[0012]

[0013] Among them, L 3D-PatchDistribution The loss is the 3D neighborhood block distribution loss, where N is the number of invisible voxels, and D... KL () represents the divergence; n i The number of voxels within a neighborhood window centered on the invisible voxel i; y i For the truth label of the invisible voxel i, y j p is the truth label for voxel j. j The network prediction result for voxel j; The function is a symbolic function. If the condition inside the brackets [] is satisfied, the value is 1; otherwise, the value is 0.

[0014] Furthermore, the number of bilateral feature fusion modules in the 3D semantic occupancy prediction model is 1-4, and they are connected sequentially.

[0015] Furthermore, the prediction stage specifically includes: the image feature encoding module extracting 2D image features of the target panoramic image; the 2D-3D viewpoint conversion module converting the 2D image features to a 3D occupancy coordinate system to obtain initial voxel features; the bilateral feature fusion module performing feature enhancement on the initial voxel features; the occupancy decoding module sequentially decoding and binary classifying the enhanced voxel features to obtain an occupancy prediction map; the semantic decoding module sequentially performing semantic decoding and classification on the occupancy voxels in the occupancy prediction map to obtain a semantic prediction map; and combining the occupancy prediction map and the semantic prediction map as the semantic occupancy prediction result of the 3D scene.

[0016] According to another aspect of the present invention, a 3D semantic occupancy prediction system based on scene visibility is provided, comprising: a processor; and a memory storing a computer-executable program, which, when executed by the processor, causes the processor to perform the 3D semantic occupancy prediction method based on scene visibility as described above.

[0017] According to another aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the scene visibility-based 3D semantic occupancy prediction method as described above.

[0018] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0019] (1) A 3D semantic occupancy prediction method based on scene visibility is proposed. It focuses on the fact that the complete 3D scene is divided into visible and invisible areas due to occlusion. It utilizes the obvious differences between the two in terms of information basis and prediction mode, proposes a bilateral feature fusion module to encode features, and designs a 3D neighborhood block distribution loss structure to supervise the voxel prediction of invisible areas, thereby improving the accuracy of the current scene semantic occupancy prediction.

[0020] (2) The provided bilateral feature fusion module uses local perception units to maintain local geometric structure and long-distance dependency units to capture long-distance dependency relationships between cubes, thereby simultaneously performing effective feature completion for partially occluded foreground obstacles and large-area occluded background obstacles.

[0021] (3) The provided visibility mask generation module obtains the corresponding visibility mask from the current scene occupation label through the ray occlusion judgment unit and the camera view range filtering unit; the provided 3D neighborhood block distribution loss structure reduces the constraint requirements on the cube of the invisible area, from strictly requiring accurate position and category to having a correct prediction of the target category within a certain neighborhood range, thereby improving the overall performance. Attached Figure Description

[0022] Figure 1 A flowchart of a 3D semantic occupancy prediction method based on scene visibility provided in an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the architecture of the 3D semantic occupancy prediction model provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the architecture of the bilateral feature fusion module provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of a local sensing unit provided in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of the structure of a long-distance dependent unit provided in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the visibility masking results provided in an embodiment of the present invention;

[0028] Figure 7 This is a semantic occupancy prediction result diagram for an autonomous driving scenario provided by an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0030] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0031] Figure 1 A flowchart illustrating a 3D semantic occupancy prediction method based on scene visibility provided in an embodiment of the present invention. (See also...) Figure 1 , combined Figures 2-7The 3D semantic occupancy prediction method based on scene visibility in this embodiment is described in detail. The method includes operations S1-S5.

[0032] Operation S1 constructs a bilateral feature fusion module, which includes parallel local perception units and long-distance dependency units to perform local enhancement and self-attention global enhancement on the input voxel features and then fuse the output.

[0033] See Figure 3 The bilateral feature fusion module comprises a local perceptual unit and a long-range dependency unit. The local perceptual unit focuses on local contextual information, preserving the original 3D geometry. The long-range dependency unit performs self-attention global enhancement with affordable computation, obtaining the global receptive field, establishing long-range dependencies, and effectively completing completely occluded background obstacles. The output of the bilateral feature fusion module is the sum of the outputs of the local perceptual unit and the long-range dependency unit.

[0034] According to an embodiment of the present invention, the local sensing unit includes a convolutional sub-unit, a downsampling sub-unit, and a second activation layer. See also... Figure 4 The convolutional subunit comprises a first convolutional layer, a first batch of normalized layers, a first activation layer, a second convolutional layer, and a second batch of normalized layers connected in sequence; the downsampling subunit comprises a third convolutional layer and a third batch of normalized layers connected in sequence; the second activation layer is used to activate the sum of the outputs of the convolutional subunits and the downsampling subunits before outputting the result. Preferably, the kernel size of the first, second, and third convolutional layers is 3×3×3, and the strides are 2, 1, and 2, respectively. The sum of the outputs of the convolutional subunits and the downsampling subunits, after passing through the second activation layer, becomes the output of the local perceptual unit, resulting in an enhanced voxel feature map that preserves local characteristics.

[0035] See Figure 5 The long-distance dependency unit includes: a feature compression layer, a feature self-attention enhancement layer, and a feature reorganization layer.

[0036] The feature compression layer is used to compress voxel features within the same column based on the intensity distribution of voxel features, obtaining BEV features. That is, for all voxels within the same column, weights are assigned based on their feature intensity distribution, thus compressing them appropriately according to their importance to obtain BEV features. Specifically, firstly, F... 3d ∈R C×H×W×Z Rearranged into R CZ×H×W Using a 1×1 Conv, the corresponding height of the feature intensity distribution D∈R Z×H×W The weighted sum is calculated along the height using this distribution and then compressed to obtain F. com ∈R C×H×W .

[0037] The feature self-attention enhancement layer is used to perform attention calculations on the BEV features to obtain enhanced BEV features. Specifically, it is represented as follows: The feature self-attention enhancement layer consists of a multi-head self-attention network and a feedforward network.

[0038] The feature reconstruction layer is used to redistribute the BEV enhancement features based on the feature height distribution within the columnar section to recover the enhanced voxel features. Specifically, the feature reconstruction layer performs matrix multiplication calculations on the BEV enhancement features based on the feature height distribution within the columnar section to redistribute features along the height dimension, obtaining the enhanced voxel features. Specifically, it also utilizes the learned feature intensity distribution along the height. Time-enhanced BEV features Calculate matrix multiplication, perform high-dimensional feature redistribution, and obtain the voxel feature representation F. attn ∈R C×H×W×Z .

[0039] Operation S2 connects the image feature encoding module, 2D-3D viewpoint conversion module, bilateral feature fusion module, occupancy decoding module and semantic decoding module in sequence to construct a 3D semantic occupancy prediction model.

[0040] See Figure 2 This illustrates the architecture of the 3D semantic occupancy prediction model. The image feature encoding module extracts features from the six input panoramic images, providing the basis for subsequent voxel features. The image feature encoding module utilizes a 2D backbone network to extract four-level multi-scale features from the six panoramic images, downsampled at 1 / 2, 1 / 4, 1 / 8, and 1 / 16. Subsequently, a Second FPN network is used to fuse the multi-scale perspective features, denoted as F... 2d ∈R 6×c×h×w The feature retained here is a 1 / 8 downsampling scale.

[0041] The 2D-3D viewpoint transformation module projects 2D image features from the pixel coordinate system to the 3D occupancy coordinate system where the desired occupancy label is located, thereby obtaining the initial voxel features of the current scene. This coordinate system transformation involves using camera intrinsic parameters to transform image features from the pixel coordinate system to their respective camera coordinate systems, then using camera extrinsic parameters to transform the feature representation from the camera coordinate system to the vehicle coordinate system, then using the vehicle's pose to transform it to the world coordinate system, and finally transforming it to the 3D occupancy coordinate system. The unit of this 3D occupancy coordinate system is a grid, while the units of the other coordinate systems are meters.

[0042] Specifically, the 2D-3D perspective conversion module utilizes F 2dA semantic feature and a depth distribution are learned through a deep network and a semantic network, respectively. The depth distribution represents the actual depth position of the current pixel. The semantic feature and the depth distribution are then multiplied by an outer product to obtain the feature representations of several predicted 3D positions. Subsequently, the initial voxel features of the current scene, denoted as F, are obtained using camera intrinsic and extrinsic parameters and vehicle pose transformation. 3d ∈R C×H×W×Z .

[0043] In this embodiment of the invention, the 3D semantic occupancy prediction model contains 1-4 bilateral feature fusion modules, which are connected sequentially. Preferably, it includes 4 bilateral feature fusion modules.

[0044] The occupancy decoding module is used to decode the enhanced voxel features output by the bilateral feature fusion module to obtain decoded voxel features; and to perform binary classification on the decoded voxel features to obtain the occupancy prediction map.

[0045] The semantic decoding module is used to further perform semantic decoding on the occupancy voxels in the occupancy prediction map, and to classify the semantic decoding features to obtain the semantic prediction map; the semantic prediction map and the occupancy prediction map are combined to obtain the semantic occupancy prediction result of the 3D scene.

[0046] Operation S3 constructs a visibility mask generation module, which divides the input voxels into visible voxels and invisible voxels, constructs a one-to-one cross-entropy supervised loss structure to supervise the prediction results of visible voxels, and constructs a 3D neighborhood block distribution loss structure to supervise the prediction results of invisible voxels.

[0047] According to an embodiment of the present invention, the visibility mask generation module includes a ray occlusion judgment unit and a camera view range filtering unit, which can divide all voxels of the scene into visible voxels and invisible voxels.

[0048] The ray occlusion judgment unit is used to perform a preliminary visibility judgment on the input voxels. The judgment rule is as follows: the input voxel is regarded as a starting voxel connected to multiple ending voxels to form multiple rays. For any ray, the voxels from the starting voxel are successively judged as initially visible voxels along the ray direction until an occupying voxel is encountered. The voxels after the occupying voxel are judged as initially invisible voxels.

[0049] In the ray occlusion determination unit, the determination of whether the ray originates from the origin voxel (x) s y s , z s ) and any endpoint voxel (x e y e , z e The voxels are connected together. The starting voxel is unique and is the voxel where the vehicle is located; the ending voxel is a set V. end, where M represents the voxels on the six end faces of the 3D scene. Visibility mask M∈R H×W×Z Initializing all voxels to 0 indicates they are invisible. The rule for determining which voxels are visible is to start from the origin voxel and proceed along the ray direction, identifying each voxel as visible as M[v]. x ,v y ,z z The process continues until an occupying voxel is encountered, at which point the judgment terminates. Subsequent voxels on this ray are all invisible voxels. After judging all rays, the initial visibility mask M can be obtained.

[0050] The camera view range filtering unit is used to project the initial visible voxels onto the image coordinate system. If the projected voxel coordinates are not within the preset image range, the corresponding initial visible voxels are corrected to invisible voxels, and the final judgment result of visible and invisible voxels is obtained.

[0051] In the camera viewpoint filtering unit, voxels belonging to the visible region in the initial visibility mask M are projected onto the image coordinate system using camera intrinsic and extrinsic parameters. If the projected voxel coordinates do not belong to any image range, they are filtered out, and the mask attribute is corrected to invisible, thus obtaining the final visibility mask M. ′ ,like Figure 6 As shown.

[0052] The visible region one-to-one cross-entropy supervised loss structure is used to strictly supervise the prediction results of visible voxels, so that visible voxels have accurate location information and discriminative category information.

[0053] The 3D neighborhood block distribution loss structure for invisible regions is used to provide relatively relaxed neighborhood distribution supervision for the prediction results of invisible voxels, including the sampling neighborhood of the same class of sample units and the divergence loss structure of neighborhood prediction.

[0054] The sampling neighborhood consists of similar sample units. For each invisible voxel i, the selected candidate samples are those belonging to a voxel window of a certain size centered on that invisible voxel, for a total of n. i Take the neighborhood voxels that are of the same category.

[0055] The divergence loss structure for neighborhood prediction performs divergence supervision on the average prediction results of each invisible voxel i and all its candidate samples. Supervising the neighborhood mean reduces the optimization difficulty of invisible voxels and makes them more flexible in prediction.

[0056] In summary, the 3D neighborhood block distribution loss structure constructed in this embodiment is represented as follows:

[0057]

[0058] Among them, L 3D-Patch DistributionThe loss is the 3D neighborhood block distribution loss, where N is the number of invisible voxels, and D... KL () represents the divergence; n i The number of voxels within a neighborhood window centered on the invisible voxel i; y i For the truth label of the invisible voxel i, y j p is the truth label for voxel j. j The network prediction result for voxel j; The function is a symbolic function. If the condition inside the brackets [] is satisfied, the value is 1; otherwise, the value is 0.

[0059] Operation S4 aims to train a 3D semantic occupancy prediction model by combining the sample set with the convergence of the one-to-one cross-entropy supervised loss structure and the 3D neighborhood block distribution loss structure.

[0060] In operation S5, during the prediction phase, the target surround view image is directly input into the trained 3D semantic occupancy prediction model to obtain the semantic occupancy prediction result of the 3D scene.

[0061] According to an embodiment of the present invention, the prediction stage specifically includes: an image feature encoding module extracting 2D image features from the target panoramic image; a 2D-3D viewpoint conversion module converting the 2D image features to a 3D occupancy coordinate system to obtain initial voxel features; a bilateral feature fusion module enhancing the initial voxel features; an occupancy decoding module sequentially decoding and binary classifying the enhanced voxel features to obtain an occupancy prediction map; a semantic decoding module sequentially performing semantic decoding and classification on the occupancy voxels in the occupancy prediction map to obtain a semantic prediction map; and combining the occupancy prediction map and the semantic prediction map as the semantic occupancy prediction result of the 3D scene.

[0062] Taking the example of using ResNet-50 or ResNet-101 in the image feature encoding module, where the voxel feature dimension obtained after 2D-3D viewpoint conversion is 1 / 4 of the final output voxel dimension, and upsampling the voxel features by 4 times to restore them to the final resolution before inputting them into the semantic decoding module, this embodiment illustrates the method.

[0063] A 3D semantic occupancy prediction model was trained using the nuScene-Occupancy dataset. nuScene is a large-scale autonomous driving dataset that provides labels for foreground obstacles, while the panoptic nuScene dataset provides semantic segmentation labels for radar point clouds. This allows for the generation of dense and accurate semantic occupancy labels for nuScene-Occupancy. The dataset includes 850 scenes, 28,120 training images, and 6,019 validation images. The resolution of the semantic occupancy labels is 512×512×40, and the actual size of each voxel is 0.2m×0.2m×0.2m. A total of 18 categories were designed, including 16 semantic categories, 1 idle category, and 1 noise category.

[0064] During training, a subset of training data is selected for each iteration, and training on all training sets completes one round. This process is repeated until the maximum number of rounds is reached, resulting in a trained 3D semantic occupancy prediction model. The maximum number of rounds is preferably 15. In the above training process, one training iteration is as follows: the loss value and activation values ​​of the intermediate layers of the network are calculated using the forward propagation algorithm, and the gradients corresponding to the network parameters are obtained through backpropagation, thereby updating the parameters.

[0065] The visualization results of the trained 3D semantic occupancy prediction model perceiving surrounding obstacles in different scenarios are as follows: Figure 7 As shown, the six images on the left are the input RGB images, with the viewpoints from top to bottom and left to right being front left, front, front right, back left, back, and back right, respectively. The middle image shows the ground truth labels for semantic occupancy, and the right image shows the model's predicted output. It can be seen that the trained model has a relatively complete and accurate perceptual modeling of foreground obstacles (such as cars and trucks) and background obstacles (such as curbs and man-made structures) in this scene.

[0066] The 3D semantic occupancy prediction method based on scene visibility provided in this invention improves local and global feature fusion by designing a bilateral feature fusion module. It proposes a feature compression layer to compress height while preserving as much information as possible, thereby alleviating the computational burden caused by an excessive number of voxels and efficiently completing the features of occluded foreground and background obstacles. Furthermore, it proposes applying different loss functions to the prediction of voxels in visible and invisible regions. By allowing flexible inference in invisible regions, the network focuses more on visible regions, further enhancing performance in invisible areas.

[0067] This invention also provides a 3D semantic occupancy prediction system based on scene visibility, comprising: a processor; and a memory storing a computer-executable program, which, when executed by the processor, causes the processor to perform the aforementioned 3D semantic occupancy prediction method based on scene visibility.

[0068] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described 3D semantic occupancy prediction method based on scene visibility.

[0069] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A 3D semantic occupancy prediction method based on scene visibility, characterized in that, include: A bilateral feature fusion module is constructed, which includes parallel local perception units and long-distance dependency units to perform local enhancement and self-attention global enhancement on the input voxel features respectively before fusion output; The image feature encoding module, the 2D-3D viewpoint conversion module, the bilateral feature fusion module, the occupancy decoding module, and the semantic decoding module are connected in sequence to construct a 3D semantic occupancy prediction model; A visibility mask generation module is constructed to divide the input voxels into visible voxels and invisible voxels. A one-to-one cross-entropy supervised loss structure is constructed to supervise the prediction results of visible voxels, and a 3D neighborhood block distribution loss structure is constructed to supervise the prediction results of invisible voxels. The 3D semantic occupancy prediction model is trained by combining a sample set with the convergence of the one-to-one cross-entropy supervised loss structure and the 3D neighborhood block distribution loss structure as the goal. In the prediction phase, the target surround view image is directly input into the trained 3D semantic occupancy prediction model to obtain the semantic occupancy prediction result of the 3D scene. The prediction stage specifically includes: The image feature encoding module extracts 2D image features from the target panoramic image; the 2D-3D view conversion module converts the 2D image features to a 3D occupies the coordinate system to obtain initial voxel features; the bilateral feature fusion module enhances the initial voxel features. The occupancy decoding module sequentially decodes and binary classifies the enhanced voxel features to obtain an occupancy prediction map; the semantic decoding module sequentially performs semantic decoding and classification on the occupancy voxels in the occupancy prediction map to obtain a semantic prediction map. The occupancy prediction map and the semantic prediction map are combined to obtain the semantic occupancy prediction result of the 3D scene.

2. The 3D semantic occupancy prediction method based on scene visibility as described in claim 1, characterized in that, The local sensing unit includes: The convolutional subunit includes a first convolutional layer, a first batch of normalized layers, a first activation layer, a second convolutional layer, and a second batch of normalized layers connected in sequence. The downsampling subunit includes a third convolutional layer and a third batch normalization layer connected in sequence; The second activation layer is used to output the sum of the outputs of the convolutional sub-unit and the downsampling sub-unit.

3. The 3D semantic occupancy prediction method based on scene visibility as described in claim 1, characterized in that, The long-distance dependent unit includes: A feature compression layer is used to compress the voxel features within the same columnar body according to the intensity distribution of each voxel feature within the columnar body, thereby obtaining BEV features. A feature self-attention enhancement layer is used to perform attention calculations on the BEV features to obtain BEV enhanced features; A feature recombination layer is used to redistribute the BEV enhancement features according to the feature height distribution within the columnar body to recover the enhanced voxel features.

4. The 3D semantic occupancy prediction method based on scene visibility as described in claim 3, characterized in that, The feature recombination layer is specifically used to: perform matrix multiplication calculation on the BEV enhanced features according to the feature height distribution within the columnar body, so as to redistribute the features in the height dimension and obtain the enhanced voxel features.

5. The 3D semantic occupancy prediction method based on scene visibility as described in claim 1, characterized in that, The visibility mask generation module includes: The ray occlusion judgment unit is used to make a preliminary visibility judgment on the input voxel. The judgment rule is as follows: the input voxel is regarded as a multiple ray formed by connecting the starting voxel and multiple ending voxels respectively. For any ray, the voxel from the starting voxel is judged as the initial visible voxel along the ray direction until the occupying voxel is encountered. The voxel after the occupying voxel is judged as the initial invisible voxel. The camera view range filtering unit is used to project the initial visible voxels onto the image coordinate system. If the projected voxel coordinates are not within the preset image range, the corresponding initial visible voxels are corrected to invisible voxels to obtain the final judgment result of visible and invisible voxels.

6. The 3D semantic occupancy prediction method based on scene visibility as described in claim 1, characterized in that, The 3D neighborhood block distribution loss structure is expressed as follows: in, The loss is the distribution of 3D neighborhood blocks. The number of invisible voxels. For divergence; To use invisible voxels Set the number of voxels within the neighborhood window centered on the target; Invisible voxels truth labels, voxels truth labels, voxels The network prediction results; For a symbolic function, if it satisfies The internal condition is set to 1; otherwise, it is set to 0.

7. The 3D semantic occupancy prediction method based on scene visibility as described in claim 1, characterized in that, The 3D semantic occupancy prediction model has 1-4 bilateral feature fusion modules, which are connected sequentially.

8. A 3D semantic occupancy prediction system based on scene visibility, characterized in that, include: processor; A memory storing a computer-executable program, which, when executed by the processor, causes the processor to perform the 3D semantic occupancy prediction method based on scene visibility as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the 3D semantic occupancy prediction method based on scene visibility as described in any one of claims 1-7.

Images