Three-dimensional semantic scene completion method based on semantic segmentation prior

Abstract

According to the three-dimensional semantic scene completion model based on semantic segmentation prior, the model is combined with a deep learning method, rich semantic information contained in a semantic segmentation map is utilized, and the precision of three-dimensional scene completion is effectively improved. A model framework guided by aligned semantic segmentation features is designed, the framework is composed of a voxel generation branch and a semantic supplement branch, and a three-dimensional scene can be reconstructed from different information sources. Experiments show that the method can significantly improve the precision of three-dimensional scene completion, and especially has significant advantages in the aspect of fine-grained scene reconstruction. The model is verified on a SemanticKITTI data set, a three-dimensional voxel representation with a semantic tag is generated, and more accurate three-dimensional semantic scene completion is successfully realized.

Description

Technical Field

[0001] The present invention proposes a 3D semantic scene completion model based on semantic segmentation prior, which belongs to image data processing (G06T) and image enhancement (G06T5) field. Background Art

[0002] With the rise of fields such as autonomous driving and virtual reality, in-depth understanding of three-dimensional scenes in the real world has become particularly important. Therefore, researchers have gradually turned their attention to three-dimensional scene understanding, which has made it a new focus of research in the field of computer vision. However, research on three-dimensional semantic scene completion based on pure vision faces a key challenge, namely the lack of input information. Due to the dimensionality increase from two-dimensional information to three-dimensional information, visual data is inevitably affected by environmental factors such as occlusion and noise. At the same time, there are some areas that cannot be observed. The current pure vision method cannot achieve ideal results in accurately completing three-dimensional scenes.

[0003] To solve this problem, a series of 3D semantic scene completion methods have emerged. With the recent development of deep neural networks, many very effective methods have been proposed. Among them, 3D semantic scene completion based on camera 2D images is committed to inferring the geometry and semantic categories of objects in 3D scenes from multiple 2D images. SSCNet (https: / / ieeexplore.ieee.org / document / 8099511) took the lead in jointly predicting voxel occupancy and semantic categories in a single depth view. Early 3D semantic scene completion research focused on using 3D convolutional networks to infer object attributes in small-scale and uniformly dense indoor scenes. Due to the need for contextual information, such networks usually use the U-Net architecture (https: / / link.springer.com / chapter / 10.1007 / 978-3-319-24574-4_28), that is, an encoder-decoder structure with skip connections. The benefit of the U-Net structure is not only that it provides contextual information but also that it can extract meaningful coarse scene representations, which are used to output multi-scale predictions or to achieve coarse-to-fine feature refinement in 3D semantic scene completion (https: / / openaccess.thecvf.com / content_cvpr_2018 / CameraReady / 2835.pdf). However, 3D convolutional neural networks have two important limitations, namely the cubic growth of memory requirements and the expansion of sparse input manifolds due to convolution. To improve both situations, SparseConvNet (https: / / ieeexplore.ieee.org / document / 8579059) uses sparse 3D convolutions that can conveniently operate on input geometry, thereby achieving feature extraction at high grid resolution. DDRNet (https: / / openaccess.thecvf.com / content_CVPR_2019 / html / Li_RGBD_Based_Dimensional_Decomposition_Residual_Network_for_3D_Semantic_Scene_CVPR_2019_paper.html) proposed a dimensional decomposition residual block DDR for three-dimensional convolution. This structure replaces the original three-dimensional convolution with one-dimensional convolution along three different directions, solving the problem that the network parameters of three-dimensional convolution grow in a cubic form.AIC-Net (https: / / ieeexplore.ieee.org / document / 9156985) proposed a new module called anisotropic convolution. Compared with the standard 3D convolution limited to a fixed 3D receptive field, this module achieves anisotropic 3D receptive field by decomposing the 3D convolution into three consecutive 1D convolutions and dynamically and adaptively determining the convolution kernel size of each 1D convolution. PCANet (https: / / ieeexplore.ieee.org / document / 10007036) proposed the PCR block, which decomposes the traditional 3D convolution into three consecutive planar convolutions to maintain the planar features of the 3D scene. The above three structures reduce the computational overhead on the one hand, and on the other hand, they also incorporate the idea that different directions of a 3D scene are different.

[0004] In addition to using 3D convolutional networks, there are also works using view-volume networks, point networks, and hybrid networks to achieve 3D semantic scene completion tasks. In view-volume networks, in order to exploit the efficiency of 2D convolutional neural networks, a common strategy is to combine them with 3D convolutional neural networks, such as the literature (https: / / www.ijcai.org / Proceedings / 2018 / 101). Specifically, these methods use a 2D convolutional encoder to extract 2D features from 2D texture and geometric inputs (RGB, depth, etc.), which are then lifted to 3D and processed by 3D convolutions. To obtain richer feature representations, another scheme projects the 3D input data into 2D and then processes it with ordinary 2D convolutions that are much lighter than their 3D counterparts. The resulting 2D features are then lifted back to 3D and decoded using 3D convolutions. This scheme is more efficient in terms of computation and memory usage, but is more suitable for outdoor scenes because outdoor data shows major changes along two axes (i.e., longitudinal and lateral).

[0005] The SemanticKITTI dataset (https: / / semantic-kitti.org / ) is a large-scale semantic scene understanding dataset based on radar sequences. SemanticKITTI is based on the KITTI visual benchmark and provides dense annotations for each individual scan of the odometer benchmark sequence 00-10, which enables semantic scene interpretation such as semantic segmentation and semantic scene completion using multiple consecutive scans. The remaining sequences, i.e., sequences 11-21, are used as a test set, showing a variety of challenging traffic conditions and environment types. The official dataset does not provide labels for the test set, but uses an evaluation service to score submissions and provide test set results, which makes the test results more objective and accurate. Summary of the invention

[0006] The purpose of this invention is to study a 3D semantic scene completion model based on semantic segmentation prior based on the theory and method of deep learning, which can utilize the rich semantic prior information contained in the semantic segmentation map, and propose a model framework guided by aligned semantic segmentation features. Finally, the aligned semantic features are merged into low-resolution scene features through post-fusion to achieve finer-grained scene reconstruction, thereby effectively improving the performance of 3D semantic scene completion.

[0007] The present invention designs a model framework guided by aligned semantic segmentation features. The model framework consists of a voxel generation branch and a semantic supplementation branch, and uses different information sources to reconstruct a three-dimensional scene. First, a basic three-dimensional layout is established for large objects in the voxel generation branch, and then reliable visual features in the semantic segmentation map are used to guide the reconstruction of smaller elements and the refinement of larger objects. In the supplementation branch, the present invention proposes a class alignment module to adaptively learn the importance of different categories in a specific task dataset without the need for additional annotations or category mapping. Experiments show that this improvement can effectively understand objects of different scales in the scene and achieve more fine-grained three-dimensional scene reconstruction.

[0008] The present invention adopts the above-mentioned three-dimensional semantic scene completion model based on semantic segmentation prior. First, the input images and point cloud truth labels in the SemanticKITTI dataset are preprocessed to construct training data, and then the three-dimensional semantic scene completion model based on semantic segmentation prior is trained. Finally, the model is used to generate fine-grained scene reconstruction results, which are specifically in the form of scenes represented by three-dimensional voxels with semantic labels.

[0009] The present invention comprises the following steps:

[0010] Step 1: Data preparation:

[0011] The SemanticKITTI dataset provides dense annotations for each individual scan of the odometry benchmark sequences 00-10, which enables semantic scene interpretation such as semantic segmentation and semantic scene completion using multiple consecutive scans. The remaining sequences, i.e., sequences 11-21, are used as a test set, showing a variety of challenging traffic situations and environment types.

[0012] Step 2: Train the depth completion model:

[0013] A model for 3D semantic scene completion is constructed, and the network model is divided into two branches. 1) The first part is the voxel generation branch, which takes the RGB image as input. Through the temporal encoder module and the bird's-eye view cross-fusion module, the main branch effectively uses visual image features and deep features to generate low-resolution scene representations. 2) The second part is the semantic supplement branch, which uses a pre-trained semantic segmentation model to generate image segmentation results, which provide information about object instances and layout constraints as visual perception priors. The class alignment module adaptively learns intra-class and inter-class relationships to obtain reliable semantic prior information. The semantic voxel encoder uses prior semantic information to optimize the reconstruction of low-resolution 3D scenes through post-fusion. Finally, the fine scene features are upsampled and linearly projected to obtain high-resolution scene completion results.

[0014] Step 3: Set the loss function:

[0015] This part uses the weighted cross entropy loss L class The final scene completion result of the training can be calculated using the following formula.

[0016]

[0017] Where N v represents the number of voxels, c0-cM represents all semantic categories, c is the semantic category, c0 is the first category, ω c represents the category weight, and nc Represent the true value annotation and prediction results respectively. And this part also uses the scene class affinity loss L proposed in MonoScene (https: / / arxiv.org / abs / 2112.00726) scal At the same time, the depth estimation network is supervised by the depth map generated by MobileStereoNet (https: / / arxiv.org / abs / 2108.09770), using the binary cross entropy loss L depth Following the setting of BEVDepth (https: / / arxiv.org / abs / 2206.10092), the final loss is the sum of the above three losses, which can be expressed by the following formula.

[0018] L=L class +L scal +L depth

[0019] Step 4: Set training parameters:

[0020] For the voxel generation branch, in the BEV-CA module, this paper divides the voxels into 2 groups along the height to balance performance and computational efficiency. The semantic segmentation map to semantic voxels in the semantic supplement branch are all pre-processed and not trained. In addition, the depth map inferred by MobileStereoNet is used to supervise the depth distribution of DepthNet output. All models are trained for 20 iterations with a learning rate of 2×10^(-4).

[0021] The present invention utilizes the rich visual semantic priors in the two-dimensional semantic segmentation map to guide the refinement of three-dimensional semantic scene completion. A powerful and efficient three-dimensional semantic scene completion framework AssemblyScene is proposed. This framework is a novel dual-branch structure that uses a bird's-eye view cross-fusion module to effectively utilize depth estimation information and a temporal encoder to effectively utilize temporal information. The present invention also proposes a semantic voxel encoder that guides the refinement of scene voxels from low resolution to high resolution by utilizing object instances and layout constraint information provided by the semantic segmentation image. In addition, the class alignment module adaptively handles category importance, enhancing the universality of the framework on different datasets. This process is similar to optimizing and assembling specific details after completing the rough construction of the scene, which can effectively understand objects of different scales in the scene and achieve more fine-grained three-dimensional scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0023] Figure 2 A schematic diagram of the RGB image of the SemanticKITTI dataset used in the present invention;

[0024] Figure 3 This is an example diagram of the depth extraction result;

[0025] Figure 4 This is an example image of the semantic segmentation map extraction result;

[0026] Figure 5 Build a sample image of the result for the semantic segmentation point cloud;

[0027] Figure 6 Build result example graphs for semantic segmentation voxels;

[0028] Figure 7 It is a bird's-eye view cross-fusion module structure diagram;

[0029] Figure 8 Align module structure diagrams for classes;

[0030] Fig. 9 This is the structure diagram of the semantic voxel encoder;

[0031] Fig.10 This is a visual comparison of the qualitative results of the model on the SemanticKITTI validation set. Specific implementation plan

[0032] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the implementation modes and the accompanying drawings.

[0033] According to an embodiment of the present application, a 3D semantic scene completion model based on semantic segmentation prior is provided. The 3D semantic scene completion is implemented by, for example, a computer program, and when the computer program is executed by a processor, the 3D semantic scene completion model according to the embodiment of the present application is instantiated.

[0034] See also Figure 1 When the present invention is used for three-dimensional semantic scene completion, the specific processing steps are as follows:

[0035] 1. Data preparation.

[0036] For the task of 3D semantic scene completion, the SemanticKITTI dataset contains ground truth annotations based on outdoor LiDAR scans, represented by 256×256×32 number of 0.2m×0.2m×0.2m voxel grids, with RGB images of size 1226×370 as initial images. The dataset is annotated with 21 categories (19 semantic categories, 1 free category, and 1 unknown category), of which 19 semantic categories include highways, sidewalks, parking lots, other roads, buildings, cars, trucks, bicycles, motorcycles, other vehicles, vegetation, tree trunks, grass, people, cyclists, motorcyclists, fences, pillars, and traffic signs.

[0037] We preprocess the data as follows:

[0038] (1) Data annotation construction: This paper follows the setting of MonoScene and uses cam2 RGB images of size 1226×370 as input, such as Figure 2 As shown, the image is cropped from the left to 1220 × 370. In addition, this paper uses the point cloud annotations provided in the original dataset to generate voxel category annotations with a resolution of 1 / 1 256 × 256 × 32, voxel category annotations with a resolution of 1 / 2 128 × 128 × 16, and voxel category annotations with a resolution of 1 / 8 32 × 32 × 4 for subsequent training.

[0039] (2) Depth extraction: This paper follows the setting of VoxFormer (https: / / arxiv.org / abs / 2302.12251) and uses the pre-trained MobileStereoNet to extract the depth value of the RGB image, such as Figure 3 shown.

[0040] (3) Semantic segmentation map extraction: This paper uses Mask2Former (https: / / arxiv.org / abs / 2112.01527) pre-trained on the Cityscapes dataset (https: / / www.cityscapes-dataset.com / ) to extract the semantic segmentation map of the RGB image, such as Figure 4 shown.

[0041] (4) Semantic segmentation point cloud construction: This paper uses the camera parameters provided by the dataset and the predicted depth to back-project the categories of the semantic segmentation map into the 3D point cloud, such as Figure 5 shown.

[0042] (5) Semantic segmentation voxel construction: The present invention generates corresponding voxels through semantic point cloud, which are used as semantic priors input in the semantic supplement branch, such as Figure 6 shown.

[0043] 2. Build a network.

[0044] like Figure 1 As shown, the overall architecture of this model has three types: AssemblyScene-B / S / T. Each type of model architecture consists of multiple parts. For example, the AssemblyScene-B architecture only contains Figure 1 The gray modules in the upper part, while the AssemblyScene-T architecture consists of all the gray, green, and orange modules in the figure, and is also the most complete structure of the three architectures, such as Figure 1 shown in the upper part of .

[0045] (1) Voxel Generation Branch: The voxel generation branch is composed of multiple images I = {I t ,I t-1 ,…,I t-N+1} as input, t represents different moments, and extracts the features of each image Where C represents the number of channels of the image feature, N is the number of images, and h and w are the sizes of the image features. The features of multiple frames are then cross-fused using a two-dimensional temporal encoder (Temporal Encoder 2D, hereinafter referred to as TE2D) to enhance the features of visible voxels. In addition, the depth distribution predicted by DepthNet Calculate the global features of the entire space The features of occluded voxels are supplemented, such as Figure 1 As shown in Figure 1, DepthNet consists of three convolutional layers. DCA stands for deformable cross attention module, which mainly learns the features of the region of interest from the two-dimensional feature map.

[0046] Considering the significant occlusion in the perspective view, the present invention designs a bird's-eye view cross-fusion module (BEV-CA) module to aggregate visible features and global features from the perspective of the bird's-eye view (BEV), effectively solving the occlusion problem. Finally, the voxel generation branch uses a deformable self-attention (DSA) module (https: / / arxiv.org / abs / 2311.12091) to refine the voxel features to obtain a low-resolution feature representation of the scene. Where H, W, Z represent the length, width, and depth of the scene, respectively, and C represents the number of channels of the scene features. Mathematically speaking, the calculation process of the voxel generation branch can be expressed as the following formula.

[0047] F′ t =TE2D(F t ,f t-1 ,…,F t-N+1 )

[0048]

[0049] F visible =DCA(Q p ,F′ t ,F′ t )

[0050] F 3D =BEV-CA(F global ,F visible )

[0051] F coarse =DSA(F 3D )

[0052] Where Q p Represents the query suggestions obtained through preprocessing, Represents the outer product.

[0053] (1.1) Reprojection preprocessing. Both branches designed in the present invention utilize camera parameters and reprojection. Specifically, the intrinsic parameters of the camera and the predicted depth project the pixel values ​​from the pixel reference system to the camera reference system. Subsequently, the extrinsic parameters of the camera transform these coordinates from the camera reference system to the world reference system through rotation and translation. The two branches have different goals for utilizing the results of reprojection. The voxel generation branch focuses on using the voxel index containing the reprojected pixel value as the query input index, while the semantic supplement branch focuses on assigning semantic categories to these voxels corresponding to the pixels in the image.

[0054] (1.2) Two-dimensional temporal encoder. Figure 1 As shown in the lower right corner, the temporal encoder first processes each image feature F separatelyt , to obtain deep features Then, multi-head criss-cross attention (MCA) is used to concatenate the temporal information Combined with the single view features to model the temporal information. For the features of each time frame, the calculation process of the temporal encoder TE2D can be expressed by the following detailed formula.

[0055] C t =Conv2d(F t )

[0056] C=Concat(C1,…,C t-1 ,C t+1 ,…,C N )

[0057] F′ t =MCA(F t ,C,C)

[0058] where F t It represents the features of an image at a certain time. Time t-1 is the moment before time t. N represents the number of input images. MCA(Q, K, V) represents multi-head cross attention. Q, K, and V represent the query, key, and value in the attention mechanism, respectively.

[0059] (3) Bird’s-Eye View Cross-Fusion (BEV-CA) module. The detailed structure of the BEV-CA module is shown in Figure 7 As shown. For outdoor scenes, the distribution of objects at different heights varies greatly. For example, people and cars usually appear above the street. Therefore, in the BEV-CA module, in order to aggregate BEV features more finely, this part groups them by height. Then, windowed self-attention (WSA) is used to learn the contextual relationship within the same height group, and multi-head cross attention is adopted to aggregate visible features with global features. Obtain BEV characteristics It can be expressed by the following formula.

[0060]

[0061] Where [:,:,i,:] represents the index slice of the i-th channel of the feature, and Z and C represent the height and feature channel in the BEV feature, respectively.

[0062] On this basis, this part also directly aggregates uncompressed visible features and global features to reduce the loss of height information. The scene features are obtained by merging the visible features and global features introduced above. Here, the DDR block Figure 7 ( Figure 7The bird's-eye view cross-fusion module (shown in the right module of the structure diagram) is used to improve the computational efficiency. The performance and memory consumption under different grouping settings are also verified in subsequent experimental studies. Finally, the DSA module is used to obtain the fine low-resolution scene features F coarse .

[0063] (2) Semantic Supplement Branch: Previous methods uniformly process voxels in space, ignoring the need for different types of features for objects of different scales. As the network goes deeper, the features of small dynamic objects are often masked by the features of large objects. Therefore, this section proposes a semantic supplement branch to solve this problem. It uses prior visual features in a post-fusion manner to enhance the details of small-scale objects and capture changes in scene boundaries.

[0064] The semantic supplement branch first segments the input RGB image through the pre-trained segmentation model, and then back-projects the segmentation result to the point cloud and voxelizes the point cloud to obtain the semantic segmentation category. As input. This section designs a class alignment ( Figure 1 The Class Alignment Module in the CNN allows adaptive learning of intra-class and inter-class relationships. This module is conducive to improving the transferability of the model between different datasets and does not require specific annotations for voxelized semantic segmentation categories. Subsequently, the output of the class alignment module and the scene features of the voxel generation branch are jointly input into the semantic voxel encoder for final feature fusion. The computation process of the semantic complement branch can be expressed as follows.

[0065] V aligned =CAlign(V seg )

[0066] F fine =SegVoxel(V aligned ,F coarse )

[0067] (2.1) Class Alignment Module. Since the class mappings between different datasets are different, such as Figure 8 As shown at the bottom, this limits the direct use of voxelized semantic segmentation categories, but the relationships within and between objects are universal. Therefore, in order to enhance the versatility of the proposed framework, a class alignment module is designed to align the inherent attribute relationships between different categories. The specific structure of the class alignment module is shown in Figure 8 First, the input voxel V is convolved using 3D convolutions with kernel sizes of 1 and 3 respectively. seg Then, as Figure 8As shown in the right half of the upper part, the SESENet network (https: / / ieeexplore.ieee.org / document / 8578843) is constructed to jointly model the importance of category and position, thereby obtaining aligned prior visual features.

[0068] (2.2) Semantic voxel encoder. The low-resolution scene features F obtained by the voxel generation branch coarse It is inevitable to ignore or lose some fine features of the object. Semantic label V in the semantic segmentation map aligned Aligned by the class alignment module, the features that can complement the scene features output by the voxel generation branch are accurately captured. In order to prevent these reliable visual features from being attenuated during feature fusion, this part uses a post-fusion approach to utilize them after generating a low-resolution scene representation.

[0069] like Fig. 9 As shown in Figure 2, the semantic voxel encoder uses a multi-scale fusion network to perform multiple downsampling to obtain voxel features of different resolutions. Shallow features provide more detailed semantic information, while deep features supplement high-level context information. Through multi-resolution fusion, the prior visual features obtained by the class alignment module Guidance scene representation F fine refinement.

[0070] The semantic segmentation label-guided refinement helps to enhance and refine the low-resolution scene feature representation. Finally, after obtaining the refined feature F fine After that, it is upsampled and projected into the high-resolution output space. Finally, by using the prediction head SSC Head, the feature dimension is mapped to the semantic type dimension, and the reconstructed scene semantic category result is output to obtain Y p .

[0071] 3. Train the model.

[0072] For the model guided by the aligned semantic segmentation feature, regarding the voxel generation branch, in the BEV-CA module, this paper divides the voxels into 2 groups along the height to balance performance and computational efficiency. In addition, the depth map inferred by MobileStereoNet (https: / / arxiv.org / abs / 2108.09770) is used to supervise Figure 1 Depth distribution of DepthNet output in . All models were trained for 20 epochs with a learning rate of 2×10^(-4).

[0073] 4. Evaluate and compare the model of the present invention on the data set.

[0074] Two types of evaluation indicators are used for the quantitative evaluation of the model experimental results. The first type of indicators is used to evaluate the reconstruction performance of the model, including intersection over union (IoU), mean intersection over union (mIoU), accuracy (Precision) and recall (Recall); the second type of indicators is used to evaluate the computational complexity and computational overhead of the model, including video memory usage (Memory) during training, model parameters (Params) and floating point operations per second (FLOPs). This paper mainly uses IoU and mIoU to evaluate the performance of the model, which are the two most recognized indicators for the 3D semantic scene completion task. In addition, this paper uses other indicators for auxiliary analysis.

[0075] IoU and mIoU are the most critical performance evaluation indicators in the task of 3D semantic scene completion. IoU takes into account the voxel occupancy and focuses on the geometry of objects, which is crucial for obstacle avoidance in autonomous driving scenarios. On the other hand, mIoU emphasizes the semantic category of voxels, providing more meaningful insights for a comprehensive understanding of real-world scenes. For the calculation of IoU and mIoU, it is first necessary to calculate the number of voxels labeled as true and predicted as true (TP), the number of voxels labeled as false and predicted as true (FP), and the number of voxels labeled as true and predicted as false. The calculation of IoU can be expressed as the following formula. The calculation of mIoU can be obtained by averaging the IoU of all semantic classes.

[0076]

[0077] Precision and Recall can also be used to assist in evaluating the performance of the 3D semantic scene completion model. The calculation of Precision and Recall can be expressed by the following formulas respectively.

[0078]

[0079] The qualitative comparison between the method of the present invention and VoxFormer-T (VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion) on the SemanticKITTI validation set is as follows: Fig.10 (Visual comparison of qualitative results of the model on the SemanticKITTI validation set) is shown. It is obvious that the method of the present invention shows stronger performance in distinguishing changes in road layout and moving objects. For example, in the first two comparisons, our method generates better street scenes and outperforms VoxFormer in the transition of various large objects. In the last two visual examples, our method successfully completes the reconstruction of the person in front and the vehicle entering from the right, which is crucial for tasks such as autonomous driving.

[0080] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A 3D semantic scene completion method based on semantic segmentation prior, comprising the following steps: Step 1: Data preparation; Get the SemanticKITTI open source dataset, which is a large-scale semantic scene understanding dataset based on radar sequences; SemanticKITTI is based on the KITTI vision benchmark and provides dense annotations for each individual scan of the odometry benchmark sequences 00-10; the remaining sequences 11-21 are used as a test set; The SemanticKITTI dataset provides true value annotations based on outdoor LiDAR scans, represented by 256×256×32 voxel grids of 0.2m×0.2m×0.2m, and provides RGB images of size 1226×370; the RGB images of size 1226×370 are cropped to 1220×370; in addition, the point cloud annotations provided in the original dataset are used to generate voxel category annotations with a resolution of 256×256×32 for subsequent training; secondly, the pre-trained MobileStereoNet is used to extract the depth value of the RGB image, and the pre-trained Mask2Former is used to extract the semantic segmentation map of the RGB image. Then, the camera parameters provided by the dataset and the predicted depth are used to back-project the categories of the semantic segmentation map into the three-dimensional point cloud, where the corresponding voxels generated by the semantic point cloud will be used as the semantic prior of the model input; Step 2: Training the 3D semantic scene completion model; The cropped RGB image data of the training sample obtained in step 1 and the corresponding voxels generated by the semantic point cloud are used as the input of the 3D semantic scene completion model; The three-dimensional scene semantic completion model includes two branches: a voxel generation branch and a semantic supplement branch; The voxel generation branch takes the RGB image obtained in step 1 as input, and includes a ResNet feature extraction module, a two-dimensional temporal encoder module, and a bird's-eye view cross-fusion module; The ResNet feature extraction module is composed of multiple convolutional layers and residual connection blocks, the two-dimensional temporal encoder module is composed of multiple convolutional layers and cross-attention layers, and the bird's-eye view cross-fusion module is composed of window self-attention layers, cross-attention layers and multiple convolutional layers; the input of the semantic supplement branch is the voxelization result of the semantic segmentation map in step 1, which is composed of a class alignment module and a semantic voxel encoder; the class alignment module is composed of multiple convolutional layers and linear layers, and the semantic voxel encoder is composed of multiple convolutional layers; The voxel generation branch uses the current image and the previous N-1 images I = {I t ,I t-1 ,…,I t-N+1 } as input, and use ResNet to extract the features of each image Where N is the number of images, h and w are the sizes of image features; then the features of multiple frames are cross-fused by using a 2D Temporal Encoder (TE2D); in addition, the predicted depth distribution Calculate the global features of the entire space Among them, c, d, h, and w represent the number of channels, depth, height of the feature map, and width of the feature map respectively; the calculation process of the voxel generation branch can be expressed as: F t ′=TE2D(F t ,F t-1 ,…,F t-N+1 ) F visible =DCA(Q p ,F t ′,F t ′) F 3D =BEVCA(F global ,F visible ) F coarse =DSA(F 3D ) Where Q p Represents the query suggestions obtained through preprocessing, represents the outer product; (1) Reprojection preprocessing: Both the voxel generation branch and the semantic complementation branch utilize camera parameters and reprojection. The camera’s intrinsic parameters and predicted depth project pixel values ​​from the pixel reference system to the camera reference system. Subsequently, the camera’s extrinsic parameters transform these coordinates from the camera reference system to the world reference system through rotation and translation. The voxel generation branch then uses the voxel index containing the reprojected pixel value as the query input index, while the semantic complementation branch assigns semantic categories to these voxels corresponding to pixels in the image as input. (2) Two-dimensional temporal encoder: The temporal encoder first processes each image feature F separately t , to obtain deep features Then, multi-head criss-cross attention (MCA) is used to concatenate the temporal information Combined with the single view features; for the features of each time frame, the calculation process of the temporal encoder can be expressed as follows: C t =Conv2d(F t ) C=Concat(C1,…,C t-1 ,C t+1 ,…,C N ) F′ t =MCA(F t ,C,C) where F t represents the features of an image at a certain time, MCA(Q,K,V) represents multi-head cross attention, Q, K, V represent the query, key, and value in the attention mechanism respectively; (3) Bird’s-eye view cross-fusion module: Group them by height, then use windowed self-attention (WSA) to learn the semantic relationship within the same height group, and use multi-head cross-attention to aggregate visible features with global features; Obtain BEV characteristics It can be expressed by the following formula: Where [:,:,i,:] represents the index slice of the i-th channel of the feature, Z and C represent the height and feature channel in the BEV feature, respectively; H and W represent the height and width of the feature map, respectively; On this basis, the uncompressed visible features and global features are directly aggregated, and the scene features are obtained by merging these two features. Finally, the DSA module is used to obtain the refined low-resolution scene features F coarse ; The semantic supplement branch converts the voxelized semantic segmentation categories As input; then, the output of the class alignment module and the scene features of the voxel generation branch are jointly input into the semantic voxel encoder for final feature fusion; the calculation process of the semantic supplement branch can be expressed by the following formula: V aligned =CAlign(V seg ) F fine =SegVoxel(V aligned ,F coarse ) (1) Class Alignment Module: First, the input voxel V is aligned using 3D convolution with kernel sizes of 1 and 3 respectively. seg The category and position features of the image are modeled; then, the SE network is used to jointly model the importance of category and position to obtain aligned prior visual features. (2) Semantic voxel encoder: The semantic voxel encoder uses a multi-scale fusion network to perform multiple downsampling to obtain a refined scene representation F. fine ; Finally, after obtaining the refined feature F fine After that, it is upsampled and projected into the high-resolution output space to obtain Y p ; In the training of the 3D semantic scene completion model, the weighted cross entropy loss L is used. class The final scene completion result of the training is also used. The scene class affinity loss L proposed in MonoScene is also used. scal At the same time, the depth estimation network is supervised by the depth map generated by MobileStereoNet, using the binary cross entropy loss L depth Follow the settings of BEVDepth; the final loss is the sum of the above three losses; Step 3: Set training parameters; For the voxel generation branch, the input image N is 5. In the BEV-CA module, the voxels are divided into 2 groups along the height. The semantic segmentation map to semantic voxels in the semantic supplement branch are all preprocessed and no training is performed. The depth map inferred by MobileStereoNet is used to supervise the depth distribution of DepthNet output. All models are trained for 20 iterations with a learning rate of 2×10^(-4).

2. The method according to claim 1, wherein In the training of the depth completion model, the loss consists of three parts: Use weighted cross entropy loss L class The final scene completion result of the training can be calculated using the following formula: Where N v represents the number of voxels, c M represents the number of categories, ω c represents the category weight, and nc Represent the true value annotation and prediction results respectively; and this section also uses the scene class affinity loss L proposed in MonoScene scal At the same time, the depth estimation network is supervised by the depth map generated by MobileStereoNet, using the binary cross entropy loss L depth Following the setting of BEVDepth; the final loss is the sum of the above three losses, which can be expressed by the following formula: L=L class +L scal +L depth .

3. The method according to claim 2, wherein The second step also includes that all models are trained for 20 iterations with a learning rate of 2×10 -4 ; ResNet50 and FPN are used to extract image features; the depth map used in the query suggestion network comes from the pre-trained MobileStereoNet; the feature size is also set to d=128.

Images