Three-dimensional semantic scene completion method based on hierarchical grouping aggregation

Abstract

The invention provides a three-dimensional semantic scene completion method based on hierarchical grouping aggregation. The model provides a novel double-branch structure. The explicit constraint branch starts from overall three-dimensional scene features, pays attention to regional aspects of the scene, and aggregates explicit similar features into grouped regional features. The implicit diffusion branch starts from visible voxel query suggestions output by the query suggestion network, focuses on details of objects in a scene, and learns implicit fine-grained features of the objects. The invention also provides a hierarchical grouping module which divides a three-dimensional space into a plurality of sub-regions and performs similarity calculation and aggregation on voxel features in each region so as to generate compact region feature representation. The grouping mode can focus on a local area of a specific object in a scene, mutual interference of features of objects independently distributed in the distance is avoided, and therefore more detailed and accurate semantic features are extracted.

Description

Technical Field

[0001] The present invention proposes a three-dimensional semantic scene completion method based on hierarchical grouping aggregation, which belongs to the technical field of image data processing (G06T) and image enhancement (G06T5). Background Art

[0002] 3D semantic scene completion is a key technology that aims to predict the complete scene geometry and semantic categories from incomplete 3D scene data. This technology has been widely used in many fields such as autonomous driving and virtual reality. However, due to the complexity of the scene and the sparse distribution of objects, it is significantly challenging to accurately predict the geometry and semantic category of each voxel from incomplete observations.

[0003] In order to solve this problem, a series of 3D semantic scene completion methods have emerged. 3D semantic scene completion was first proposed by Song et al. [1] proposed to jointly infer geometric and semantic information by combining geometric occupancy prediction with semantic category prediction. Subsequent studies such as Li et al. [2] and Garbade et al. [3] , further verifying the close connection between scene completion and semantic annotation tasks, mainly applied to indoor scenes. Silberman et al. [4] and Firman et al. [5] Early studies on indoor environments laid the foundation for this. In order to cope with a wider range of outdoor scenarios, Roldao et al. [6] and Cheng et al. [7] et al. extended the LiDAR-based method, such as Yan et al. [8] and Xia et al. [9] More complex scene completion is achieved on LiDAR data. However, in large-scale deployment and cost-sensitive application scenarios, visual cameras have higher economic feasibility than sensors such as LiDAR, which has promoted the research of vision-based solutions. For example, Cao and Charette

[10] The proposed MonoScene is a pioneering method for 3D semantic scene completion of outdoor scenes using monocular images. Subsequent studies modeled dense 3D scenes using bird's-eye view features, three-view features, and voxel occupancy features. These methods usually use uniformly generated features to represent all voxels and interact with each other, resulting in low computational efficiency and difficulty in effectively capturing fine-grained object features. In addition, voxel features with long spatial distances are prone to interfere with each other, especially the shapes and categories of distant objects are usually independent, while the features between close objects are more continuous and related. Summary of the invention

[0004] The purpose of the present invention is to improve the accuracy and efficiency of scene completion by hierarchical grouping and similarity aggregation based on a 3D semantic scene completion method, by dividing the object regions in the 3D space into layers and combining feature aggregation within the region and feature interaction between regions.

[11] and SSCBench-KITTI360

[12] The current state-of-the-art performance (SOTA) was achieved on the dataset with both single and multi-image inputs.

[0005] The present invention proposes a three-dimensional semantic scene completion model based on hierarchical grouping aggregation. The model proposes a new dual-branch structure. The explicit constraint branch starts from the overall three-dimensional scene features, focuses on the regional aspects of the scene, and aggregates explicitly similar features into grouped regional features. The implicit diffusion branch starts from the visible voxel query suggestions output by the query suggestion network, focuses on the details of the objects in the scene, and learns the implicit fine-grained features of the objects. A hierarchical grouping module is also proposed, which divides the three-dimensional space into multiple sub-regions, and calculates and aggregates the voxel features in each region to generate a compact regional feature representation (such as Figure 1 ). This grouping method can focus on the local area of ​​a specific object in the scene, avoiding the mutual interference of features of independently distributed objects in the distance, thereby extracting more detailed and accurate semantic features.

[0006] The present invention adopts the above-mentioned three-dimensional semantic scene completion model based on hierarchical grouping aggregation. First, data preparation is performed on the SemanticKITTI dataset and the SSCBench-KITTI360 dataset to extract two-dimensional features and voxel occupancy information in the RGB image. Next, the model is trained. During the model training process, a multi-scale loss function is used for supervision, and finally the three-dimensional semantic scene completion model based on deep hierarchical grouping aggregation is used to perform three-dimensional semantic scene completion.

[0007] The present invention comprises the following steps:

[0008] Step S1: data preparation; Step S1 includes step S11 and step S12;

[0009] Step S11: data annotation construction;

[0010] Step S12: depth extraction;

[0011] Step S2: Calculate explicit constraint branch module features;

[0012] Step S21: extracting explicit voxel features;

[0013] Step S22: balancing redundant features using a hierarchical grouping module;

[0014] Step S3: Calculate implicit diffusion branch module features;

[0015] Step S4: Calculate fusion features;

[0016] Step S5: Calculate the multi-scale prediction results and loss function constraint settings.

[0017] The beneficial effects of the present invention are:

[0018] 1. Based on a thorough analysis of the deficiencies of the three-dimensional semantic scene completion model, the present invention proposes an improved three-dimensional semantic scene completion method from the aspects of data preparation, hierarchical grouping aggregation model design, and training loss function setting.

[0019] 2. This paper compares the model with the most advanced models such as MonoScene, TPVFormer, OccFormer and VoxFormer, and verifies the effect of the model on the SemanticKITTI dataset and SSCBench-KITTI360, which has great application value and practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The overall architecture of the hierarchical grouping and similarity aggregation model of the present invention;

[0021] Figure 2 A query suggestion network structure diagram proposed by the present invention;

[0022] Figure 3 This is a structural diagram of the hierarchical grouping module proposed by the present invention;

[0023] Figure 4 This is a visualization comparison of the qualitative results of HGroupScene in the SemanticKITTI validation set of the present invention. Specific implementation plan

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the implementation modes and the accompanying drawings.

[0025] According to an embodiment of the present application, a three-dimensional semantic scene model for hierarchical grouping aggregation is provided. The three-dimensional semantic scene completion is implemented by a computer program, and when the computer program is executed by a processor, the three-dimensional semantic scene model according to the embodiment of the present application is instantiated.

[0026] When the present invention is used for three-dimensional semantic scene completion, the specific processing steps are as follows:

[0027] Step S1: data preparation;

[0028] In this step, the present invention constructs annotations on the data and deeply extracts the data features.

[0029] Step S11: Data annotation construction.

[0030] The present invention first collects the SemanticKITTI dataset, follows the setting of MonoScene, uses cam2's RGB image of size 1226×370 as input, and crops the image from the left to 1220×370. In addition, the present invention uses the point cloud annotations provided in the original dataset to generate voxel category annotations with a resolution of 1 / 1 256×256×32, voxel category annotations with a resolution of 1 / 2 128×128×16, and voxel category annotations with a resolution of 1 / 8 32×32×4 for subsequent training.

[0031] Step S12: Depth extraction.

[0032] The present invention follows the setting of VoxFormer and uses the pre-trained MobileStereoNet to extract the depth value of the RGB image, and the output two-dimensional image feature F 2D .

[0033] Step S2: Calculate explicit constraint branch module features;

[0034] In this step, the present invention extracts explicit voxel features and uses a hierarchical grouping module to balance redundant features.

[0035] Step S21: extracting explicit voxel features.

[0036] The explicit constraint branch is derived from the two-dimensional image feature F output from step S12. 2D The present invention uses the Flosp module in Monoscene to project the 2D image features into 3D voxels. The Flosp method involves projecting the center of the 3D voxel onto a multi-scale 2D feature map and sampling the corresponding features. The 3D features F output by the Flosp module are aggregated by a hierarchical grouping module. Flosp .

[0037] F Flosp =Flosp(F 2D )

[0038] F region =HGM(F Flosp )

[0039] Step S22: extracting explicit voxel features.

[0040] The Hierarchical Grouping Module (HGM) module is based on a hierarchical architecture, which divides the scene into regional groups and aggregates features in each relatively small region. The Hierarchical Grouping Module includes three parts: downsampling, grouping sub-blocks, and upsampling. In each downsampling stage, the present invention adopts a novel downsampling based on the voxel feature similarity calculated by the Gumbel Softmax attention algorithm to compactly aggregate features. First, the current scene is evenly divided into eight groups along three orthogonal directions, so that the length, width, and height of each sub-region are half of the previous one, and then the grouping sub-blocks are used to compress the features in each sub-region and aggregate similar voxel features in the region. Finally,

[0041] DDR blocks are used to perform feature interaction between adjacent regions. In the grouping sub-block, the present invention divides the voxel features in each sub-region into voxel labels and aggregate labels in proportion. The grouping sub-block (Grouping Block, hereinafter referred to as GBlock) divides the segmented voxel labels {v l} and the aggregation tag {a l} as input, where l represents the index of the downsampling layer. It aggregates similar features in the sub-region according to the similarity of the two labels in the embedding space, thereby forming a more compact regional feature. The present invention uses Gumbel Softmax operation to calculate the similarity matrix A between the voxel label and the aggregate label l , which can be expressed by the following formula:

[0042]

[0043] Among them, a i and v i are query and key respectively, l is the layer number of the module, and W q and W k is the learnable linear projection weight, M l is the sequence length, γ i is obtained by independent random sampling from the Gumbel (0, 1) distribution, and then the argmax operation is used to calculate the distribution matrix After merging similar features, a more compact voxel feature is obtained, and the feature update process can be expressed by the following formula:

[0044]

[0045] Among them, W o and W vis a learnable linear projection weight; in the upsampling stage of the hierarchical grouping module, the present invention designs the same number of upsampling layers as the downsampling layer Feature Diffusion Module, restores the regional features to the input resolution, uses three-dimensional transposed convolution for upsampling, and uses DDR blocks for feature interaction between adjacent regions. The present invention introduces jump connections between the corresponding upsampling layers and downsampling layers at the same resolution;

[0046] Step S3: Calculate implicit diffusion branch module features;

[0047] In this step, the present invention realizes the diffusion of implicit voxel features in the two-dimensional image through an implicit diffusion branch, and predefines a set of learnable query suggestions from the depth estimation map and camera parameters using the query proposal network (QPN). Based on this set of queries, the features of the region of interest are learned from the two-dimensional feature map using deformable cross attention (DCA). The calculation process of the implicit diffusion branch can be expressed by the following formula, where Q p represents query suggestions (the query suggestion network predefines a set of learnable query suggestions, i.e., QP, from the depth estimation map and camera parameters), is the output information of the QPN network, DCA(Q, K, V) represents the deformable cross attention operation, where (Q, K, V) are the parameters that need to be filled in, and the corresponding input of this method is Q p , F 2D , F 2D ;

[0048] F visible =DCA(Q p , F 2D , F 2D )

[0049] F object =HGM(F visible +F mask )

[0050] Step S4: Calculate fusion features;

[0051] In this step, the present invention accepts the regional features F of the explicit constraint branch through the feature diffusion module region and the object feature F of the implicit diffusion branch object , and aggregate these two types of features into a complete 3D scene feature F through the Feature Diffusion Module (FDM) 3D .

[0052] Step S5: Calculate the multi-scale prediction results and loss function constraint settings;

[0053] Use the multi-scale prediction head to generate the 3D scene completion result. First, the result of the explicit constraint branch, that is, the output of step 2, is input into the low-resolution prediction head to obtain Where H, W, and Z represent the original dimensions of the scene. Secondly, the results of the explicit constraint branch and the implicit diffusion branch are connected, and the features are aggregated through the AIC module and the ASPP module, that is, the output of step S4, so that feature interactions can be performed between grouped regions. Finally, the interaction results are input to the original resolution prediction head (i.e. Figure 1 In Head2), we get Because Y low Focusing on compact grouped region features, we use 1 / 2 resolution ground truth annotations to supervise Y low .Y low The prediction of occurs only during the training phase. Using the weighted cross entropy loss L CE The completion results of training two scenes can be calculated using the following formula:

[0054]

[0055] Where N v represents the number of voxels, c M represents the number of categories, ω c represents the category weight, and nc Represent the true value annotation and prediction result respectively. In the training process, the present invention uses the scene class affinity loss proposed in MonoScene and The final loss is the sum of the above six losses and can be expressed as the following formula, where λ is a custom coefficient of the low-resolution loss function.

[0056]

[0057] The comparison of experimental results is shown as follows Figure 4 shown.

[0058] The present invention achieves superior performance by providing better control over the regional similarity and continuity of the scene while reducing the chaotic distribution of voxel classes. For example, as observed in the black rectangle, the present framework reconstructs regional objects with improved continuity, making them appear more natural. In addition, even in locations where the camera provides sparse information, the present invention maintains regional features without predicting chaotic and disordered objects. This also opens the possibility of future exploration of merging predictions from multiple regions into larger scenes.

[0059] The experimental data are shown in Table 1 and Table 2. Table 1 is the comparison result 1 of the present invention on the SemanticKITTI hidden test set; Table 2 is the comparison result 2 of the present invention on the SemanticKITTI hidden test set.

[0060] The present invention is compared with the most advanced SSC methods on the SemanticKITTI hidden test set, including MonoScene, TPVFormer, OccFormer, and VoxFormer. The present invention achieves state-of-the-art performance in both geometry (IoU) and semantics (mloU). In particular, when using a single image as input, the present framework achieves significant improvements over the state-of-the-art techniques, surpassing single-image SOTA and multi-image SODA. This proves the feasibility and effectiveness of this framework. The success of the single-image input version provides more development possibilities for this framework. Specifically, the best performance is achieved in static object categories such as buildings and vegetation; in most dynamic sparse object categories, such as motorcycles (0.03% of the dataset) and other vehicles (0.20% of the dataset), the performance of this method is comparable to that of existing methods. These dynamic sparse types of objects only constitute a small part of the voxels in the scene, and after undergoing multiple downsampling stages through sub-region processing, their features become less prominent and are easily disturbed by noise features in the scene, which makes the performance of these object categories only comparable to existing methods.

[0061] Table 1

[0062]

[0063] Table 2

[0064]

[0065] [1]Song S, Yu F, Zeng A, et al. Semantic scene completion from asingledepth image[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:1746-1754.

[0066] [2]Li J,Liu Y,Gong D,et al.Rgbd based dimensional decompositionresidual network for 3d semantic scene completion[C] / / Proceedings of theIEEE / CVF Conference on Computer Vision and Pattern Recognition.2019:7693-7702.

[0067] [3]Garbade M,Chen Y T,Sawatzky J,et al.Two stream 3d semantic scenecompletion[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition Workshops.2019:0-0.

[0068] [4]Silberman N,Hoiem D,Kohli P,et al.Indoor segmentation and supportinference from rgbd images[C] / / Computer Vision–ECCV 2012:12th EuropeanConference on Computer Vision,Florence,Italy,October 7-13,2012,Proceedings,Part V 12.Springer Berlin Heidelberg,2012:746-760.

[0069] [5]Firman M,Mac Aodha O,Julier S,et al.Structured prediction ofunobserved voxels from a single depth image[C] / / Proceedings of the IEEEConference on Computer Vision and Pattern Recognition.2016:5431-5440.

[0070] [6]Roldao L,de Charette R,Verroust-Blondet A.Lmscnet:Lightweightmultiscale 3d semantic completion[C] / / 2020 International Conference on 3DVision(3DV).IEEE,2020:111-119.

[0071] [7]Cheng R,Agia C,Ren Y,et al.S3cnet:A sparse semantic scenecompletion network for lidar point clouds[C] / / Conference on RobotLearning.PMLR,2021:2148-2161.

[0072] [8]Yan X,Gao J,Li J,et al.Sparse single sweep lidar point cloudsegmentation via learning contextual shape priors from scene completion[C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2021,35(4):3101-3109.

[0073] [9]Xia Z,Liu Y,Li X,et al.Scpnet:Semantic scene completion on pointcloud[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2023:17642-17651.

[0074]

[10] Cao A Q,De Charette R.Monoscene:Monocular 3d semantic scenecompletion[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition.2022:3991-4001.

[0075]

[11] Behley J,Garbade M,Milioto A,et al.Semantickitti:A dataset forsemantic scene understanding of lidar sequences[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2019:9297-9307.

[0076]

[12] Li Y,Li S,Liu X,et al.Sscbench:A large-scale 3d semantic scenecompletion benchmark for autonomous driving[J].arXiv preprint arXiv:2306.09001,2023.

Claims

1. A three-dimensional semantic scene completion method based on hierarchical grouping aggregation, characterized in that: The following steps are involved: Step S1: data preparation; Step S1 includes step S11 and step S12; Step S11: data annotation construction; Step S12: depth extraction; Step S2: Calculate explicit constraint branch module features; Step S21: extracting explicit voxel features; Step S22: balancing redundant features using a hierarchical grouping module; Step S3: Calculate implicit diffusion branch module features; Step S4: Calculate fusion features; Step S5: Calculate the multi-scale prediction results and loss function constraint settings.

2. The method according to claim 1, characterized in that In step S11, the SemanticKITTI dataset is first collected. Following the setting of MonoScene, the RGB image of cam2 with a size of 1226×370 is used as input, and the image is cropped from the left to 1220×370. The point cloud annotations provided in the original dataset are used to generate voxel category annotations with a resolution of 1 / 1 256×256×32, voxel category annotations with a resolution of 1 / 2 128×128×16, and voxel category annotations with a resolution of 1 / 8 32×32×4 for subsequent training.

3. The method according to claim 2, characterized in that In step S12, the pre-trained MobileStereoNet is used to extract the depth value of the RGB image according to the setting of VoxFormer, and the output two-dimensional image feature F 2D .

4. The method according to claim 3, characterized in that In step S21, the explicit constraint branch is derived from the two-dimensional image feature F output from step S12. 2D Extract explicit voxel features from the image; Use the Flosp module in Monoscene to project the 2D image features into 3D voxels; Flosp's method involves projecting the centers of 3D voxels onto a multi-scale 2D feature map and sampling the corresponding features; The 3D features F output by the Flosp module are aggregated through the hierarchical grouping module Flosp , F Flosp = Flosp (F 2D ) F region =HGM(F Flosp )。 5. The method according to claim 4, characterized in that In step S22, the hierarchical grouping module (HGM) module divides the scene into regional groups based on a hierarchical architecture and aggregates features in each relatively small region. The hierarchical grouping module includes three parts: downsampling, grouping sub-blocks, and upsampling. In each downsampling stage, a novel downsampling based on the voxel feature similarity calculated by the Gumbel Softmax attention algorithm is used to compactly aggregate features. First, the current scene is evenly divided into eight groups along three orthogonal directions, so that the length, width, and height of each sub-region are half of the previous one, and then the grouping sub-blocks are used to compress the features in each sub-region and aggregate similar voxel features in the region. Finally, the DDR block is used to perform feature interaction between adjacent regions. In the grouping sub-blocks, the voxel features in each sub-region are divided into voxel labels and aggregate labels in proportion, and the grouping sub-blocks divide the segmented voxel labels {v l } and the aggregation tag {a l } as input, where l represents the index of the downsampling layer. Similar features in the sub-region are aggregated according to the similarity of the two labels in the embedding space to form a more compact regional feature. The similarity matrix A between the voxel label and the aggregate label is calculated using the Gumbel Softmax operation. l , which can be expressed by the following formula: Among them, a i and v i are query and key respectively, l is the layer number of the module, and W q and W k is the learnable linear projection weight, M l is the sequence length, γ i is obtained by independent random sampling from the Gumbel (0,1) distribution, and then the argmax operation is used to calculate the distribution matrix After merging similar features, a more compact voxel feature is obtained, and the feature update process can be expressed by the following formula: Among them, W o and W v are learnable linear projection weights; in the upsampling stage of the hierarchical grouping module, the same number of upsampling layers are designed to restore the regional features to the input resolution, three-dimensional transposed convolution is used for upsampling, and DDR blocks are used for feature interaction between adjacent regions, and jump connections are introduced between the corresponding upsampling layers and downsampling layers at the same resolution.

6. The method according to claim 5, characterized in that In step S3, the implicit diffusion branch realizes the diffusion of implicit voxel features in the two-dimensional image. A set of learnable query suggestions are predefined from the depth estimation map and camera parameters using the query proposal network. Based on this set of queries, the features of the region of interest are learned from the two-dimensional feature map using deformable cross attention (DCA). The calculation process of the implicit diffusion branch can be expressed by the following formula, where Q p represents query suggestion, DCA(Q,K,V) represents deformable cross attention operation, where (Q,K,V) are the parameters that need to be filled in; F visible =DCA(Q p ,F 2D ,F 2D ) F object =HGM(F visible +F mask )。 7. The method according to claim 6, characterized in that In step S4, the feature diffusion module accepts the region feature F of the explicit constraint branch region and the object feature F of the implicit diffusion branch object , and aggregate these two types of features into a complete three-dimensional scene feature F through multi-scale feature diffusion 3D .

8. The method according to claim 7, characterized in that In step S5, a multi-scale prediction head is used to generate a 3D scene completion result. First, the result of the explicit constraint branch, i.e., the output of step 2, is input into the low-resolution prediction head to obtain Where H, W, and Z represent the original dimensions of the scene. Secondly, the results of the explicit constraint branch and the implicit diffusion branch are connected, and the features are aggregated through the AIC module and the ASPP module, that is, the output of step S4, so that feature interaction can be performed between grouped regions; finally, the interaction result is input into the original resolution prediction head to obtain Because Y low Focusing on compact grouped region features, we use 1 / 2 resolution ground truth annotations to supervise Y low ; Y low The prediction of occurs only during the training phase; using the weighted cross entropy loss L CE The completion results of training two scenes can be calculated using the following formula: Where N v represents the number of voxels, c M represents the number of categories, ω c represents the category weight, and nc Represent the true value annotation and prediction results respectively. The scene class affinity loss proposed in MonoScene is used during training. and; the final loss is the sum of the above six losses, which can be expressed as the following formula, where λ is the custom coefficient of the low-resolution loss function,

Images