A 3D semantic scene completion method based on hierarchical grouping and aggregation

Abstract

This invention proposes a 3D semantic scene completion method based on hierarchical grouping and aggregation. The model proposes a novel dual-branch structure. The explicit constraint branch starts with the overall 3D scene features, focusing on regional aspects of the scene and aggregating explicitly similar features into grouped regional features. The implicit diffusion branch starts with the visible voxel query suggestions output by the query suggestion network, focusing on the details of objects in the scene and learning the implicit fine-grained features of the objects. A hierarchical grouping module is also proposed, which divides the 3D space into multiple sub-regions and performs similarity calculation and aggregation on the voxel features within each region, thereby generating a compact regional feature representation. This grouping method can focus on the local regions of specific objects in the scene, avoiding mutual interference from features of independently distributed objects in the distance, thus extracting more detailed and accurate semantic features.

Description

Technical Field

[0001] This invention proposes a three-dimensional semantic scene completion method based on hierarchical grouping and aggregation, belonging to the technical fields of image data processing (G06T) and image enhancement (G06T5). Background Technology

[0002] 3D semantic scene completion is a key technology aimed at predicting the complete geometric structure and semantic category of a scene from incomplete 3D scene data. This technology has wide applications in various fields such as autonomous driving and virtual reality. However, due to the complexity of the scene and the sparsity of object distribution, accurately predicting the geometric structure and semantic category of each voxel from incomplete observations presents significant challenges.

[0003] To address this problem, a series of 3D semantic scene completion methods have emerged. 3D semantic scene completion was first proposed by Song et al. [1] The paper proposes combining geometric occupancy prediction with semantic category prediction to jointly infer geometric and semantic information. Subsequent research, such as that by Li et al., has... [2] and Garbade et al. [3] This further validates the close relationship between scene completion and semantic annotation tasks, and is mainly applied to indoor scenes. Silberman et al. [4] and Firman et al. [5] Early research on indoor environments laid the foundation for this. To address the broader outdoor scenarios, Roldao et al. [6] and Cheng et al. [7] Extending LiDAR-based methods, such as those by Yan et al. [8] and Xia et al. [9] More complex scene rendering has been achieved using LiDAR data. However, in large-scale deployments and cost-sensitive applications, vision cameras are more economically feasible than sensors like LiDAR, driving research into vision-based solutions. For example, Cao and Charette...

[10] The proposed MonoScene is a pioneering method for 3D semantic scene creation in outdoor scenes using monocular images. Subsequent research has modeled dense 3D scenes using bird's-eye view features, three-view features, and voxel occupancy features. These methods typically use uniformly generated features to represent all voxels and facilitate their interaction, leading to computational inefficiency and difficulty in effectively capturing fine-grained object features. Furthermore, features of voxels that are spatially distant are prone to interference, especially since the shapes and categories of distant objects are usually independent, while the features of nearby objects are more continuous and correlated. Summary of the Invention

[0004] The purpose of this invention is to improve the accuracy and efficiency of scene completion based on a hierarchical grouping and similarity aggregation method for 3D semantic scene completion. This method involves hierarchically dividing object regions in 3D space and combining feature aggregation within regions with feature interaction between regions. This invention is based on SemanticKITTI.

[11] and SSCBench-KITTI360

[12] It achieves state-of-the-art performance (SOTA) on the dataset with both single-image and multi-image inputs.

[0005] This invention proposes a 3D semantic scene completion model based on hierarchical grouping and aggregation. The model introduces a novel dual-branch structure. The explicit constraint branch starts with the overall 3D scene features, focusing on regional aspects of the scene and aggregating explicitly similar features into grouped regional features. The implicit diffusion branch starts with the visible voxel query suggestions output by the query suggestion network, focusing on the details of objects in the scene and learning the implicit fine-grained features of the objects. A hierarchical grouping module is also proposed, which divides the 3D space into multiple sub-regions and performs similarity calculation and aggregation on the voxel features within each region, thereby generating a compact regional feature representation (e.g., ...). Figure 1 This grouping method can focus on local areas of specific objects in the scene, avoiding mutual interference between features of independently distributed objects in the distance, thereby extracting more detailed and accurate semantic features.

[0006] This invention employs the aforementioned hierarchical grouping and aggregation-based 3D semantic scene completion model. First, data preparation is performed on the SemanticKITTI and SSCBench-KITTI360 datasets, extracting 2D features and voxel occupancy information from RGB images. Next, the model is trained. During training, a multi-scale loss function is used for supervision. Finally, the 3D semantic scene completion model, based on deep hierarchical grouping and aggregation, is used for 3D semantic scene completion.

[0007] This invention includes the following steps:

[0008] Step S1: Data preparation; Step S1 includes steps S11 and S12;

[0009] Step S11: Data labeling construction;

[0010] Step S12: Depth extraction;

[0011] Step S2: Calculate the characteristics of the explicit constraint branch module;

[0012] Step S21: Extract explicit voxel features;

[0013] Step S22: Balance redundant features using hierarchical grouping modules;

[0014] Step S3: Calculate the features of the implicit diffusion branch module;

[0015] Step S4: Calculate the fusion features;

[0016] Step S5: Calculate the multi-scale prediction results and set the loss function constraints.

[0017] The beneficial effects of this invention are as follows:

[0018] 1. Based on a thorough analysis of the shortcomings of existing 3D semantic scene completion models, this invention proposes an improved 3D semantic scene completion method from aspects such as data preparation, hierarchical grouping and aggregation model design, and training loss function setting.

[0019] 2. This invention compares the model with state-of-the-art models such as MonoScene, TPVFormer, OccFormer, and VoxFormer, and verifies the model's performance on the SemanticKITTI dataset and SSCBench-KITTI360, demonstrating its significant application value and practical significance. Attached Figure Description

[0020] Figure 1 This is the overall architecture of the hierarchical grouping and similarity aggregation model of the present invention;

[0021] Figure 2 This is a network structure diagram of the query suggestion proposed in this invention;

[0022] Figure 3 This is a structural diagram of the hierarchical grouping module proposed in this invention;

[0023] Figure 4 This is a visual comparison chart of the qualitative results of HGroupScene on the SemanticKITTI validation set of this invention. Detailed Implementation Plan

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0025] According to embodiments of this application, a three-dimensional semantic scene model based on hierarchical grouping aggregation is provided. The three-dimensional semantic scene completion is implemented by a computer program, and the three-dimensional semantic scene model according to embodiments of this application is instantiated when the processor runs the computer program.

[0026] When this invention is used for 3D semantic scene completion, the specific processing steps are as follows:

[0027] Step S1: Data preparation;

[0028] In this step, the present invention annotates and constructs the data and performs in-depth extraction of data features.

[0029] Step S11: Data labeling construction.

[0030] This invention first collects the SemanticKITTI dataset, using a 1226×370 RGB image from cam2 as input, following the MonoScene settings, and cropping the image from the left to 1220×370. Furthermore, this invention utilizes the point cloud annotations provided in the original dataset to generate voxel category annotations at 1 / 1 resolution (256×256×32), 1 / 2 resolution (128×128×16), and 1 / 8 resolution (32×32×4) for subsequent training.

[0031] Step S12: Depth extraction.

[0032] This invention follows the VoxFormer setup and uses a pre-trained MobileStereoNet to extract depth values ​​from RGB images, outputting two-dimensional image features F. 2D .

[0033] Step S2: Calculate the characteristics of the explicit constraint branch module;

[0034] In this step, the present invention extracts explicit voxel features and uses a hierarchical grouping module to balance redundant features.

[0035] Step S21: Extract explicit voxel features.

[0036] Explicitly constraining the branch from the two-dimensional image features F output in step S12 2D Explicit voxel features are extracted. This invention utilizes the Flosp module in Monoscene to project two-dimensional image features onto three-dimensional voxels. The Flosp method involves projecting the center of the three-dimensional voxels onto a multi-scale two-dimensional feature map and sampling the corresponding features. The 3D features F output by the Flosp module are aggregated through a hierarchical grouping module. Flosp .

[0037] F Flosp =Flosp(F 2D )

[0038] F region =HGM(F Flosp )

[0039] Step S22: Extract explicit voxel features.

[0040] The Hierarchical Grouping Module (HGM) is based on a hierarchical architecture, dividing the scene into regions and aggregating features within each relatively small region. The HGM consists of three parts: downsampling, grouping sub-blocks, and upsampling. In each downsampling stage, this invention employs a novel downsampling method based on voxel feature similarity calculated using the Gumbel Softmax attention algorithm, thereby compactly aggregating features. First, the current scene is uniformly divided into eight groups along three orthogonal directions, ensuring that the length, width, and height of each sub-region are half that of the previous one. Then, grouping sub-blocks are used to compress features within each sub-region, aggregating similar voxel features within that region. Finally...

[0041] DDR blocks are used for feature interaction between adjacent regions; in the grouping sub-block, the present invention divides the voxel features in each sub-region into voxel tags and aggregation tags proportionally, and the grouping sub-block (hereinafter referred to as GBlock) divides the segmented voxel tags {v l} and aggregation tag {a l} is used as input, where l represents the index of the downsampling layer. It aggregates similar features within sub-regions based on the similarity of two labels in the embedding space, thus forming more compact region features. This invention uses the Gumbel Softmax operation to calculate the similarity matrix A between voxel labels and aggregated labels. l It can be expressed by the following formula:

[0042]

[0043] Among them, a i and v i These are the query and the key, respectively; l is the module's layer number; W... q and W k M are learnable linear projection weights. l It is the sequence length, γ i The distribution matrix is ​​obtained by independent random sampling from the Gumbel(0,1) distribution, and then the assignment matrix is ​​calculated using the argmax operation. After merging similar features, more compact voxel features are obtained. The feature update process can be represented by the following formula:

[0044]

[0045] Among them, W o and W vThese are learnable linear projection weights; in the upsampling stage of the hierarchical grouping module, this invention designs the same number of upsampling layers as the downsampling layer Feature Diffusion Module, restores the region features to the input resolution, uses three-dimensional transposed convolution for upsampling, and uses DDR blocks for feature interaction between adjacent regions. This invention introduces skip connections between the corresponding upsampling and downsampling layers at the same resolution.

[0046] Step S3: Calculate the features of the implicit diffusion branch module;

[0047] In this step, the present invention realizes the implicit voxel features in the diffuse 2D image through implicit diffusion branching. A Query Proposals Network (QPN) is used to predefine a set of learnable query proposals from the depth estimation map and camera parameters. Based on this set of queries, deformable cross-attention (DCA) is used to learn the features of the region of interest from the 2D feature map. The calculation process of the implicit diffusion branching can be represented by the following formula, where Q... p The query suggestion (QP) is a predefined set of learnable query suggestions from the depth estimation map and camera parameters, and is the output information of the QPN network. DCA(Q, K, V) represents the deformable cross-attention operation, where (Q, K, V) are the parameters to be filled in. In this method, the corresponding input is Q. p F 2D F 2D ;

[0048] F visible =DCA(Q p F 2D F 2D )

[0049] F object =HGM(F visible +F mask )

[0050] Step S4: Calculate the fusion features;

[0051] In this step, the present invention accepts the region features F of the explicit constraint branch through the feature diffusion module. region Object features F of implicit diffusion branches object The Feature Diffusion Module (FDM) then aggregates these two types of features into a complete 3D scene feature F. 3D .

[0052] Step S5: Calculate the multi-scale prediction results and set the loss function constraints;

[0053] To generate 3D scene completion results using a multi-scale prediction head, firstly, the result of the explicit constraint branch (i.e., the output of step two) is input into the low-resolution prediction head to obtain... Here, H, W, and Z represent the original dimensions of the scene. Next, the results of the explicit constraint branch and the implicit diffusion branch are concatenated, and features are aggregated through the AIC and ASPP modules, i.e., the output of step S4, to enable feature interaction between grouped regions. Finally, the interaction results are input into the original resolution prediction head (i.e.,... Figure 1 In Head2), we get Because Y low Focusing on compact grouped region features, 1 / 2 resolution ground truth annotations are used to supervise Y. low Y low The predictions only occur during the training phase. Weighted cross-entropy loss L is used. CE The completion results for training two scenes can be calculated using the following formula:

[0054]

[0055] Where N v c represents the number of voxels. M ω represents the number of categories. c Indicates category weight, and y nc Representing the ground truth label and prediction result respectively, this invention uses the scene class affinity loss proposed in MonoScene during the training process. and The final loss is the sum of the above six losses, which can be expressed as the following formula, where λ is a custom coefficient of the low-resolution loss function.

[0056]

[0057] The comparison of experimental results is shown as follows: Figure 4 As shown.

[0058] This invention achieves superior performance by reducing the chaotic distribution of voxel categories while providing better control over the regional similarity and continuity of the scene. For example, as observed in the black rectangle, this framework reconstructs region objects with improved continuity, making them look more natural. Furthermore, even in locations where the camera provides sparse information, this invention preserves region features without predicting cluttered and disordered objects. This also opens up the possibility of exploring the possibility of merging predictions of multiple regions into a larger scene in the future.

[0059] The experimental data are shown in Tables 1 and 2. Table 1 shows the first comparison result of the present invention on the SemanticKITTI hiding test set; Table 2 shows the second comparison result of the present invention on the SemanticKITTI hiding test set.

[0060] This invention was compared with state-of-the-art SSC methods on the SemanticKITTI hidden test set, including MonoScene, TPVFormer, OccFormer, and VoxFormer. This invention achieves state-of-the-art performance in both geometric (IoU) and semantic (mloU) aspects. Particularly when using a single image as input, this framework achieves significant improvements over state-of-the-art techniques, surpassing single-image SOTA and multi-image SODA. This demonstrates the feasibility and effectiveness of the framework. The success of the single-image input version provides further development possibilities for this framework. Specifically, state-of-the-art performance is achieved in static object categories such as buildings and vegetation; in most dynamic sparse object categories, such as motorcycles (0.03% of the dataset) and other vehicles (0.20% of the dataset), the proposed method performs comparably to existing methods. These dynamically sparse object types constitute only a small fraction of the voxels in the scene, and their features become less prominent after undergoing multiple downsampling stages through sub-region processing, and they are susceptible to noise features in the scene, making the performance of these object categories only comparable to existing methods.

[0061] Table 1

[0062]

[0063] Table 2

[0064]

[0065] [1]Song S, Yu F, Zeng A, et al. Semantic scene completion from asingledepth image[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:1746-1754.

[0066] [2]Li J,Liu Y,Gong D,et al.Rgbd based dimensional decompositionresidual network for 3d semantic scene completion[C] / / Proceedings of theIEEE / CVF Conference on Computer Vision and Pattern Recognition.2019:7693-7702.

[0067] [3]Garbade M,Chen Y T,Sawatzky J,et al.Two stream 3d semantic scenecompletion[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition Workshops.2019:0-0.

[0068] [4]Silberman N,Hoiem D,Kohli P,et al.Indoor segmentation and supportinference from rgbd images[C] / / Computer Vision–ECCV 2012:12th EuropeanConference on Computer Vision,Florence,Italy,October 7-13,2012,Proceedings,Part V 12.Springer Berlin Heidelberg,2012:746-760.

[0069] [5]Firman M,Mac Aodha O,Julier S,et al.Structured prediction ofunobserved voxels from a single depth image[C] / / Proceedings of the IEEEConference on Computer Vision and Pattern Recognition.2016:5431-5440.

[0070] [6]Roldao L,de Charette R,Verroust-Blondet A.Lmscnet:Lightweightmultiscale 3d semantic completion[C] / / 2020 International Conference on 3DVision(3DV).IEEE,2020:111-119.

[0071] [7]Cheng R,Agia C,Ren Y,et al.S3cnet:A sparse semantic scenecompletion network for lidar point clouds[C] / / Conference on RobotLearning.PMLR,2021:2148-2161.

[0072] [8]Yan X,Gao J,Li J,et al.Sparse single sweep lidar point cloudsegmentation via learning contextual shape priors from scene completion[C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2021,35(4):3101-3109.

[0073] [9]Xia Z,Liu Y,Li X,et al.Scpnet:Semantic scene completion on pointcloud[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2023:17642-17651.

[0074]

[10] Cao A Q,De Charette R.Monoscene:Monocular 3d semantic scenecompletion[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition.2022:3991-4001.

[0075]

[11] Behley J,Garbade M,Milioto A,et al.Semantickitti:A dataset forsemantic scene understanding of lidar sequences[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2019:9297-9307.

[0076]

[12] Li Y,Li S,Liu X,et al.Sscbench:A large-scale 3d semantic scenecompletion benchmark for autonomous driving[J].arXiv preprint arXiv:2306.09001,2023.

Claims

1. A three-dimensional semantic scene completion method based on hierarchical grouping and aggregation, characterized in that, Includes the following steps: Step S1: Data preparation; Step S1 includes steps S11 and S12; Step S11: Data labeling construction; Step S12: Depth extraction; Step S2: Calculate the characteristics of the explicit constraint branch module; Step S21: Extract explicit voxel features; Step S22: Balance redundant features using hierarchical grouping modules; Step S3: Calculate the features of the implicit diffusion branch module; Step S4: Calculate the fusion features; Step S5: Calculate the multi-scale prediction results and set the loss function constraints; In step S11, the SemanticKITTI dataset is first collected. Following the MonoScene settings, an RGB image of size 1226×370 from cam2 is used as input, and the image is cropped from the left to 1220×370. Using the point cloud annotations provided in the original dataset, voxel class annotations of 1 / 1 resolution 256×256×32, voxel class annotations of 1 / 2 resolution 128×128×16, and voxel class annotations of 1 / 8 resolution 32×32×4 are generated for subsequent training. In step S12, following the settings of VoxFormer, the pre-trained MobileStereoNet is used to extract the depth values ​​of the RGB image, and the output two-dimensional image features F 2D ; In step S21, the explicit constraint branch is derived from the two-dimensional image features F output in step S12. 2D Explicit voxel features are extracted; the Flosp module in Monoscene is used to project the two-dimensional image features onto the three-dimensional voxels; the Flosp method involves projecting the center of the three-dimensional voxels onto a multi-scale two-dimensional feature map and sampling the corresponding features; The 3D features F output by the Flosp module are aggregated through a hierarchical grouping module. Flosp , F Flosp =Flosp(F 2D ) F region =HGM(F Flosp ); In step S22, the hierarchical grouping module HGM, based on a hierarchical architecture, divides the scene into region groups and aggregates features within each relatively small region. The hierarchical grouping module includes three parts: downsampling, grouping sub-blocks, and upsampling. In each downsampling stage, novel downsampling based on voxel feature similarity calculated using the Gumbel Softmax attention algorithm is employed to compactly aggregate features. First, the current scene is uniformly divided into eight groups along three orthogonal directions, ensuring that the length, width, and height of each sub-region are half that of the previous one. Then, grouping sub-blocks are used to compress features within each sub-region and aggregate similar voxel features within that region. Finally, DDR blocks are used for feature interaction between adjacent regions. Within the grouping sub-blocks, voxel features within each sub-region are proportionally divided into voxel labels and aggregation labels. The grouping sub-blocks then divide the segmented voxel labels {v l } and aggregation tag {a l } is used as input, where l represents the index of the downsampling layer. Based on the similarity of two labels in the embedding space, similar features in the sub-regions are aggregated to form more compact region features. The similarity matrix A between the voxel labels and the aggregated labels is calculated using the GumbelSoftmax operation. l It can be expressed by the following formula: Among them, a i and v j These are the query and the key, respectively; l is the module's layer number; W... q and W k M are learnable linear projection weights. l It is the sequence length, γ i The distribution matrix is ​​obtained by independent random sampling from the Gumbel(0,1) distribution, and then assigned values ​​using the argmax operation. After merging similar features, more compact voxel features are obtained. The feature update process is represented by the following formula: Among them, W o and W v It is a learnable linear projection weight; in the upsampling stage of the hierarchical grouping module, the same number of upsampling layers are designed to restore the regional features to the input resolution, upsampling is performed using 3D transposed convolution, and feature interaction between adjacent regions is performed using DDR blocks, and skip connections are introduced between the corresponding upsampling and downsampling layers at the same resolution. In step S3, the implicit diffusion branch realizes the implicit voxel features in the diffuse 2D image. A query proposal network predefines a set of learnable query proposals from the depth estimation map and camera parameters. Based on this set of queries, deformable cross-attention (DCA) is used to learn the features of the region of interest from the 2D feature map. The computation process of the implicit diffusion branch is represented by the following formula, where Q... p This indicates a query suggestion. DCA(Q,K,V) represents a deformable cross attention operation, where (Q,K,V) are the parameters that need to be filled in. F visible =DCA(Q p ,F 2D ,F 2D ) F object =HGM(F visible +F mask ); In step S4, the feature diffusion module accepts the region features F of the explicitly constrained branch. region Object features F of implicit diffusion branches object And through multi-scale feature diffusion, these two types of features are aggregated into a complete 3D scene feature F. 3D ; In step S5, a multi-scale prediction head is used to generate 3D scene completion results. First, the result of the explicit constraint branch, i.e., the output of step two, is input into the low-resolution prediction head to obtain... Here, H, W, and Z represent the original dimensions of the scene. Next, the results of the explicit constraint branch and the implicit diffusion branch are concatenated, and features are aggregated through the AIC and ASPP modules, i.e., the output of step S4, to facilitate feature interaction between grouped regions. Finally, the interaction results are input into the original resolution prediction head to obtain... Because Y low Focusing on compact grouped region features, 1 / 2 resolution ground truth annotations are used to supervise Y. low ;Y low Predictions only occur during the training phase; weighted cross-entropy loss L is used. CE The completion results for two scenes are trained and calculated using the following formula: Where N v c represents the number of voxels. M ω represents the number of categories. c Indicates category weight, and y nc These represent the ground truth labels and prediction results, respectively. The scene class affinity loss proposed in MonoScene was used during training. and The final loss is the sum of the six losses, expressed as follows, where λ is a user-defined coefficient of the low-resolution loss function.

Images