A three-dimensional semantic scene completion method, device, equipment and medium

Abstract

The present application discloses a three-dimensional semantic scene completion method, apparatus, device and medium, which relates to the field of three-dimensional scene completion. The method includes: using a depth estimation image corresponding to a two-dimensional image in a target RGB-D image to perform depth completion on the depth image in the target RGB-D image, and using a preset feature extractor to extract two-dimensional target features in the completed depth image; inputting the two-dimensional target features and two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features output by the preset projection layer; inputting the three-dimensional target features and three-dimensional semantic features into a 3D backbone network to obtain extracted features output by the 3D backbone network; performing feature fusion on the extracted features, three-dimensional target features and three-dimensional semantic features, and using the fused features to complete three-dimensional semantic scene completion. The present invention excavates depth completion priors and realizes the fusion with semantic priors to complete three-dimensional semantic scene completion.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional scene completion, and particularly to a three-dimensional semantic scene completion method, device, equipment and medium. Background Art

[0002] Semantic scene completion (SSC) refers to the task of completing the missing structures in a three-dimensional scene and simultaneously inferring the semantic labels of each voxel in the scene. Understanding the geometric structure and semantic information of a three-dimensional scene is a core challenge in computer vision research, which is crucial for mobile agents to interact with the real world and has a wide range of applications, such as augmented reality, robotic grasping and navigation.

[0003] SSC was initially used to jointly infer scene geometry and semantics from a single depth image, and the present invention classifies such methods as depth-only methods. Some research works have shown that using RGB and the corresponding depth, that is, RGB-D image pairs, can significantly improve the performance of the SSC task, and the present invention refers to such methods as RGB-D-based methods. RGB-D-based SSC techniques usually yield better results than depth-only techniques because the RGB part introduces additional color, texture and semantic information. In RGB-D-based methods, some existing technologies such as SATNet, TS3D, IMENet, etc. utilize the 2D (i.e., two-dimensional) dense semantics in the RGB image to help improve the 3D (i.e., three-dimensional) SSC accuracy, and the present invention defines them as semantics-based methods. However, the results of these semantics-based methods are still not satisfactory, suffering from problems such as semantic inconsistency and scene incompleteness.

[0004] As can be seen from the above, in the process of semantic scene completion, how to avoid the situation of semantic inconsistency and scene incompleteness in the completed scene results due to semantics-based methods is a problem to be solved in this field. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a three-dimensional semantic scene completion method, device, equipment and medium, which can deeply explore the depth prior of completion and explore the deep fusion with the semantic prior, so as to use the final fused features and make accurate predictions to complete the three-dimensional semantic scene completion. The specific solutions are as follows:

[0006] In a first aspect, the present application discloses a three-dimensional semantic scene completion method, including:

[0007] Using a depth estimation image corresponding to a two-dimensional image in a target RGB-D image containing a three-dimensional scene to perform depth completion on the depth image in the target RGB-D image to determine a completed depth image, and using a preset feature extractor to extract two-dimensional target features in the completed depth image;

[0008] Inputting the two-dimensional target features and two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features output by the preset projection layer;

[0009] Inputting the three-dimensional target features and the three-dimensional semantic features into a 3D backbone network to obtain extracted features output by the 3D backbone network;

[0010] Fusing the extracted features, the three-dimensional target features, and the three-dimensional semantic features, and using the fused features to complete three-dimensional semantic scene completion.

[0011] Optionally, the using a preset feature extractor to extract two-dimensional target features in the completed depth image includes:

[0012] Using a preset feature extractor composed of a preset two-dimensional convolutional layer and a preset number of cascaded dimensional decomposition residual modules to extract two-dimensional target features in the completed depth image.

[0013] Optionally, the inputting the two-dimensional target features and two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features output by the preset projection layer includes:

[0014] Inputting the two-dimensional target features, two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image, and the completed depth image into a preset projection layer, so as to determine a three-dimensional surface voxel corresponding to each two-dimensional pixel in the completed depth image based on a camera projection equation predefined in the preset projection layer, establish a three-dimensional space based on the three-dimensional surface voxels, discretize the three-dimensional space, determine a projection coefficient between the completed depth image and the discretized three-dimensional space, and determine three-dimensional target features and three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features based on the projection coefficient.

[0015] Optionally, the inputting the three-dimensional target features and the three-dimensional semantic features into a 3D backbone network to obtain extracted features output by the 3D backbone network includes:

[0016] Input the three-dimensional target feature and the three-dimensional semantic feature into the 3D SATNet-TNet to obtain the extracted feature output by the 3D SATNet-TNet.

[0017] Optionally, the feature fusion of the extracted feature, the three-dimensional target feature, and the three-dimensional semantic feature includes:

[0018] Use a preset feature fusion method to fuse the extracted feature and the three-dimensional target feature to obtain a target matrix; each element in the target matrix represents the probability that the corresponding voxel is occupied in three-dimensional space;

[0019] Determine a predicted rough feature based on the extracted feature;

[0020] Encode the three-dimensional semantic feature based on a preset encoding function to determine the encoded feature;

[0021] Add the predicted rough feature and the encoded feature, and perform pixel-level multiplication on the added feature and the target matrix, and use the result of the pixel-level multiplication as the fused feature.

[0022] Optionally, the use of a preset feature fusion method to fuse the extracted feature and the three-dimensional target feature to obtain a target matrix includes:

[0023] Expand the three-dimensional target feature to determine a first feature to be concatenated, and input the extracted feature into a preset conversion module to obtain a second feature to be concatenated output by the conversion module; the preset conversion module consists of a three-dimensional convolutional layer, a regularization layer, and an activation function layer;

[0024] Concatenate the first feature to be concatenated and the second feature to be concatenated to obtain a concatenated feature;

[0025] Input the concatenated feature into the conversion module, connect a preset convolutional layer after the conversion module, and perform a preset activation function after the convolutional layer to generate a target matrix.

[0026] Optionally, the three-dimensional semantic scene completion method further includes:

[0027] Construct a binary cross-entropy loss function as a first auxiliary loss function to optimize the target matrix;

[0028] Construct a first cross-entropy loss function as a second auxiliary loss function to optimize the predicted rough feature;

[0029] Construct a second cross-entropy loss function as a main loss function to optimize the fused feature;

[0030] Construct a total loss function by using the first auxiliary loss function, the second auxiliary loss function, and the main loss function.

[0031] In a second aspect, the present application discloses a three-dimensional semantic scene completion device, including:

[0032] A depth enhancement module, configured to use a depth estimation image corresponding to a two-dimensional image in a target RGB-D image including a three-dimensional scene to perform depth completion on the depth image in the target RGB-D image to determine a completed depth image, and use a preset feature extractor to extract two-dimensional target features in the completed depth image;

[0033] A feature projection module, configured to input the two-dimensional target features and two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features respectively output by the preset projection layer corresponding to the two-dimensional target features and the two-dimensional semantic features;

[0034] A feature extraction module, configured to input the three-dimensional target features and the three-dimensional semantic features into a 3D backbone network to obtain extracted features output by the 3D backbone network;

[0035] A feature fusion module, configured to perform feature fusion on the extracted features, the three-dimensional target features, and the three-dimensional semantic features, and use the fused features to complete three-dimensional semantic scene completion.

[0036] In a third aspect, the present application discloses an electronic device, including:

[0037] A memory, configured to store a computer program;

[0038] A processor, configured to execute the computer program to implement the foregoing three-dimensional semantic scene completion method.

[0039] In a fourth aspect, the present application discloses a computer storage medium, configured to store a computer program; wherein, when the computer program is executed by a processor, the steps of the foregoing disclosed three-dimensional semantic scene completion method are implemented.

[0040] This application uses the depth estimation image corresponding to the 2D image in the target RGB-D image containing a three-dimensional scene to complete the depth of the depth image in the target RGB-D image to determine the completed depth image, and uses a preset feature extractor to extract the two-dimensional target features in the completed depth image; inputs the two-dimensional target features and the two-dimensional semantic features obtained by performing semantic segmentation on the 2D image into a preset projection layer to obtain the three-dimensional target features and three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features output by the preset projection layer; inputs the three-dimensional target features and the three-dimensional semantic features into a 3D backbone network to obtain the extracted features output by the 3D backbone network; fuses the extracted features, the three-dimensional target features, and the three-dimensional semantic features, and uses the fused features to complete the three-dimensional semantic scene completion. In this way, an additional enhanced depth method is proposed in this embodiment to improve the SSC accuracy by filling holes and denoising the original depth, and complete the conversion of two-dimensional target features and two-dimensional semantic features to three-dimensional features through a 2D-3D preset projection layer, and then use the 3D backbone network to extract the extracted features in the three-dimensional features, and fuse the two-dimensional target features, two-dimensional semantic features, and the extracted features. In the whole process, the completed depth prior is deeply mined, and the depth fusion with the semantic prior is explored to use the final fused features and make accurate predictions to complete the three-dimensional semantic scene completion. Description of the Drawings

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0042] Figure 1 It is a flowchart of a three-dimensional semantic scene completion method provided by this application;

[0043] Figure 2 It is a schematic diagram of a depth enhancement process provided by this application;

[0044] Figure 3 It is a flowchart of a specific three-dimensional semantic scene completion method provided by this application;

[0045] Figure 4 It is a schematic diagram of the overall semantic and depth fusion provided by this application;

[0046] Figure 5 It is an overall framework diagram provided by this application;

[0047] Figure 6 Schematic structural diagram of a three-dimensional semantic scene completion device provided by this application;

[0048] Figure 7 Structural diagram of an electronic device provided by this application. Specific embodiments

[0049] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0050] Existing SSC technologies usually ignore the noise and holes in the original depth and fail to make good use of the two-dimensional dense semantic segmentation priors. These shortcomings lead to problems such as semantic inconsistency and structural incompleteness in the final SSC prediction. In this application, based on the RGB-D method, by enhancing the original depth through completion, and deeply coupling the semantic and structural branches, the mIoU (i.e., Mean Intersection over Union) accuracy of the three-dimensional semantic segmentation task is greatly improved.

[0051] An embodiment of the present invention discloses a three-dimensional semantic scene completion method. Refer to Figure 1 As described, this method includes:

[0052] Step S11: Use the depth estimation image corresponding to the two-dimensional image in the target RGB-D image containing a three-dimensional scene to perform depth completion on the depth image in the target RGB-D image to determine the depth-completed image, and use a preset feature extractor to extract two-dimensional target features in the depth-completed image.

[0053] In this step, the target RGB-D image containing a three-dimensional scene includes a two-dimensional image and a depth image. In this step, the process of performing depth completion and feature extraction on the depth image is completed. In the present invention, the depth value of the depth image is denoted as D raw , the depth value of the depth estimation image is denoted as D pre , the depth value of the depth-completed image is denoted as D aug , and the two-dimensional target features extracted from the depth-completed image using the preset feature extractor are denoted as F aug .

[0054] It should be noted that in this step, the depth estimation image is a predicted depth map generated after depth estimation of the two-dimensional image in the target RGB-D image. In the specific implementation, based on the DispNet network and using a pre-trained depth estimation model, monocular depth estimation can be performed to generate the predicted depth map. When obtaining the depth estimation image of the predicted depth map, the depth value D of the depth map is also obtained. pre .

[0055] In addition, while performing depth estimation on the two-dimensional image, semantic segmentation is also performed on the two-dimensional image. In the specific implementation, based on the IMENet and using a pre-trained segmentation model, advanced 2D semantic segmentation can be performed to finally obtain two-dimensional semantic features.

[0056] It should be noted that when performing the above depth estimation and semantic segmentation, during the training of the two models, the weights of the two networks are fixed and do not participate in the parameter update of the network.

[0057] In this embodiment, when using the depth estimation image corresponding to the two-dimensional image in the target RGB-D image containing a three-dimensional scene to perform depth completion on the depth image in the target RGB-D image, specifically, it can be based on the depth value D of the depth estimation image corresponding to the two-dimensional image in the target RGB-D image containing a three-dimensional scene pre and the depth value D of the depth image in the target RGB-D image raw , to perform depth completion on the depth image in the target RGB-D image.

[0058] As Figure 2 shown, the left dashed box is the depth completion process, and the right dashed box is the feature extraction process of the feature extractor. In the figure, D raw and D pre generate the enhanced depth map D through depth enhancement aug . In the specific implementation, D raw is generally a depth map with holes. In this embodiment, D pre can be used to replace the holes in D raw to achieve depth completion, and then the depth at the hole boundary is smoothed to obtain the completed depth image and the depth value D aug of the completed depth image, and then enter the feature extractor to extract the two-dimensional target feature F aug corresponding to the completed depth image.

[0059] In this embodiment, the extraction of two-dimensional target features from the completed depth image by using a preset feature extractor may include: using a preset feature extractor composed of a preset two-dimensional convolutional layer and a preset number of cascaded dimensional decomposition residual modules to extract two-dimensional target features from the completed depth image. As Figure 2 shown, the preset feature extractor includes a 2D convolutional layer and four serialized DDR modules (i.e., Dimensional Decomposition Residual modules) to extract higher-level enhanced depth features. In the figure, the parameters in the convolutional layer and DDR are shown as (kernel size, dilation, stride). Among them, the first 2D convolutional layer in the feature extractor increases the number of channels of the depth feature map, and the four 2D DDR modules can be used for residual learning and reduce the number of parameters.

[0060] Step S12: Input the two-dimensional target features and the two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features output by the preset projection layer.

[0061] In the present invention, the two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image are denoted as S pre .

[0062] In this embodiment, the preset projection layer can be understood as a 2D-3D projection layer for mapping the two-dimensional feature output in the previous step to the three-dimensional space. In the present invention, the three-dimensional semantic features generated after projecting the two-dimensional semantic features are denoted as S proj , and the three-dimensional target features generated after projecting the two-dimensional target features are denoted as F proj .

[0063] In this embodiment, the input of the two-dimensional target features and the two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features output by the preset projection layer includes: inputting the two-dimensional target features, the two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image, and the completed depth image into the preset projection layer, so as to determine the three-dimensional surface voxels corresponding to each two-dimensional pixel in the completed depth image based on the camera projection equation predefined in the preset projection layer, and establish a three-dimensional space based on the three-dimensional surface voxels. After discretizing the three-dimensional space, determine the projection coefficients between the completed depth image and the discretized three-dimensional space, and determine the three-dimensional target features and the three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features based on the projection coefficients.

[0064] In this embodiment, the predefined camera projection equation is specifically based on the intrinsic camera matrix K of the camera that captures the RGBD image 3x3 and the extrinsic camera matrix [R|t] 3x4 to determine. The camera projection equation can be: p uv = K 3×3 [R|t] 3×4 P XYZ . Each two-dimensional pixel point P uv ([u, v, 1] T , homogeneous coordinates) in the 2D image can be easily projected onto the corresponding 3D point P XYZ [X, Y, Z, 1] T , homogeneous coordinates).

[0065] When the projected 3D scene space is discretized into a volume with a specific voxel size (such as 0.02 m), it will generate an incomplete three-dimensional space, which will assign its corresponding two-dimensional feature vector to each three-dimensional surface voxel. For the voxels in the three-dimensional space not occupied by any depth value in the depth image D aug after completion, their feature vectors will be set to zero. During training, the projection coefficients between the two-dimensional features generated from the completed depth map and the three-dimensional space will be recorded in a table for gradient backpropagation.

[0066] Step S13: Input the three-dimensional target feature and the three-dimensional semantic feature into the 3D backbone network to obtain the extracted feature output by the 3D backbone network.

[0067] In this embodiment, the inputting the three-dimensional target feature and the three-dimensional semantic feature into the 3D backbone network to obtain the extracted feature output by the 3D backbone network includes: inputting the three-dimensional target feature and the three-dimensional semantic feature into the 3D SATNet-TNet to obtain the extracted feature output by the 3D SATNet-TNet. That is, the 3D backbone network in the present invention is preferably the 3D SATNet-TNet.

[0068] In a specific implementation manner, the extracted feature F feat ∈R C ' ×W×H×D , where C' is the number of channels, W, H, and D are the width, height, and depth of the three-dimensional space respectively, and F feat can be regarded as aggregating global geometric context information and semantic representation.

[0069] Step S14: Perform feature fusion on the extracted feature, the three-dimensional target feature, and the three-dimensional semantic feature, and use the fused feature to complete three-dimensional semantic scene completion.

[0070] In this step, the extracted feature F feat , the three-dimensional target feature F proj and the three-dimensional semantic feature S proj are feature-fused to achieve a better fusion of semantics and enhanced depth, obtaining the fused feature and making the final prediction.

[0071] In this embodiment, a depth estimation image corresponding to the two-dimensional image in the target RGB-D image containing a three-dimensional scene is used to perform depth completion on the depth image in the target RGB-D image to determine the depth-completed image, and a preset feature extractor is used to extract the two-dimensional target feature in the depth-completed image; the two-dimensional target feature and the two-dimensional semantic feature obtained by performing semantic segmentation on the two-dimensional image are input into a preset projection layer to obtain the three-dimensional target feature and the three-dimensional semantic feature respectively corresponding to the two-dimensional target feature and the two-dimensional semantic feature output by the preset projection layer; the three-dimensional target feature and the three-dimensional semantic feature are input into a 3D backbone network to obtain the extracted feature output by the 3D backbone network; the extracted feature, the three-dimensional target feature, and the three-dimensional semantic feature are feature-fused, and the three-dimensional semantic scene completion is completed using the fused feature. In this way, an additional enhanced depth method is proposed in this embodiment to improve the SSC accuracy by filling holes and denoising the original depth, and the conversion of the two-dimensional target feature and the two-dimensional semantic feature to the three-dimensional feature is completed through the 2D-3D preset projection layer, and then the 3D backbone network is used to extract the extracted feature in the three-dimensional feature, and the two-dimensional target feature, the two-dimensional semantic feature, and the extracted feature are feature-fused. In the whole process, the completed depth prior is deeply mined, and the depth fusion with the semantic prior is explored to use the final fused feature and make accurate predictions to complete the three-dimensional semantic scene completion.

[0072] Figure 3 This is a flowchart of a specific three-dimensional semantic scene completion method provided by an embodiment of the present application. Refer to Figure 3 As shown, this method includes:

[0073] Step S21: Use a depth estimation image corresponding to the two-dimensional image in the target RGB-D image containing a three-dimensional scene to perform depth completion on the depth image in the target RGB-D image to determine the depth-completed image, and use a preset feature extractor to extract the two-dimensional target feature in the depth-completed image.

[0074] Among them, for the more specific processing process of step S21, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be elaborated here.

[0075] Step S22: Input the two-dimensional target feature and the two-dimensional semantic feature obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer, so as to obtain a three-dimensional target feature and a three-dimensional semantic feature respectively corresponding to the two-dimensional target feature and the two-dimensional semantic feature output by the preset projection layer.

[0076] Among them, for a more specific processing procedure of step S22, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be elaborated herein.

[0077] Step S23: Input the three-dimensional target feature and the three-dimensional semantic feature into a 3D backbone network to obtain an extracted feature output by the 3D backbone network.

[0078] Among them, for a more specific processing procedure of step S23, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be elaborated herein.

[0079] Step S24: Use a preset feature fusion method to fuse the extracted feature and the three-dimensional target feature to obtain a target matrix.

[0080] In this embodiment, the use of a preset feature fusion method to fuse the extracted feature and the three-dimensional target feature to obtain a target matrix may include: expanding the three-dimensional target feature to determine a first feature to be concatenated, and inputting the extracted feature into a preset conversion module to obtain a second feature to be concatenated output by the conversion module; the preset conversion module is composed of a three-dimensional convolutional layer, a regularization layer, and an activation function layer; concatenating the first feature to be concatenated and the second feature to be concatenated to obtain a concatenated feature; inputting the concatenated feature into the conversion module, connecting a preset convolutional layer after the conversion module, and performing a preset activation function after the convolutional layer to generate a target matrix. In the present invention, the target matrix is denoted as A 3d 。

[0081] As Figure 4 shown is an overall schematic diagram of semantic and depth fusion proposed by the present invention. In the figure, the upper box is the semantic branch, and the lower box is the structural branch. This step refers to the structural branch process below. First, expand the three-dimensional target feature F proj obtained in the previous step to F 3d '∈R C ' ×W×H×D . At the same time, apply a conversion module to the extracted feature F feat to generate F 3d . Then, F 3d ’ and F 3dThey are connected in series, and the concatenated features are input into the conversion module again. A preset convolutional layer is added after the conversion module, and finally, a sigmoid activation function is executed after the convolutional layer to generate the target matrix A 3d . Among them, each value in the target matrix A 3d belongs to [0, 1], which represents the probability of occupancy of each voxel space. Therefore, A 3d can also be understood as a spatial occupancy rate matrix. The A 3d can be specifically expressed as: A 3d = sigmoid(C 1×1×1 CBR([D 3d , F 3d ))), where [.,.] represents the concatenation operation, and the channel C’ is 128. It should be noted that Figure 4 the 3D convolution parameters in are (input channel, output channel, kernel size).

[0082] In this embodiment, the conversion module can also be called the CBR module, which is specifically expressed as Conv3d - BatchNorm3d - ReLU, that is, a module composed of a three - dimensional convolutional layer, a regularization layer, and an activation function layer.

[0083] Step S25: Determine the predicted rough features based on the extracted features, and encode the three - dimensional semantic features based on a preset encoding function to determine the encoded features. Then add the predicted rough features and the encoded features, and perform pixel - level multiplication of the added features with the target matrix. The result of the pixel - level multiplication is used as the fused features.

[0084] This step can refer to Figure 4 the following structural branch process. First, the extracted feature F feat passes through a 3D convolutional layer with a convolution kernel of 1x1x1 (denoted as C 1x1x1 ) to generate F pre . Then, the softmax activation function is used to generate the rough SSC prediction feature F coarse , that is, the above - mentioned predicted rough feature. The specific formula is: F coarse = softmax(C 1×1×1 (F feat )); For the projected three - dimensional semantic features, it is first encoded by the one - hot function to obtain the ROI (i.e., region of interest) region of a specific category. In the present invention, the 3D semantic S proj is encoded into 12 channels to obtain the encoded feature F one-hotAmong them, in each channel, the voxel values within the ROI regions corresponding to each category are set to 1; otherwise, they are set to zero. In this step, the one-hot encoding introduces spatial boundary constraints into the network for each category, thus improving the prediction of 3D semantics.

[0085] In this embodiment, after obtaining the target matrix A 3d , the predicted rough feature F coarse , and the encoded feature F one-hot , we first add F coarse and F one-hot , and then perform pixel-wise multiplication with A 3d to generate the final SSC result F fine , that is, F fine = A 3D * (F coarse + F one-hot ).

[0086] It should be noted that the 3D backbone network in the present invention can be replaced by 3D convolutions of any popular semantic scene completion model, except for removing the last 12-channel prediction layer and keeping the number of channels of F feat . However, generally, the 3D SATNet-TNet is used as the 3D backbone network.

[0087] Step S26: Complete three-dimensional semantic scene completion using the fused feature.

[0088] In this embodiment, the three-dimensional semantic scene completion method is characterized in that it may further include: constructing a binary cross-entropy loss function as the first auxiliary loss function to optimize the target matrix; constructing a first cross-entropy loss function as the second auxiliary loss function to optimize the predicted rough feature; constructing a second cross-entropy loss function as the main loss function to optimize the fused feature; and constructing a total loss function using the first auxiliary loss function, the second auxiliary loss function, and the main loss function.

[0089] To improve performance and make the 3D network easier to optimize, the present invention uses multiple loss functions for supervision. In a specific implementation, we use the main loss function L fine to supervise the improved prediction F fine , and add two specific auxiliary loss functions, namely the first auxiliary loss function L binary and the second auxiliary loss function L coarse functions to supervise A 3d and F coarse , where L fine and L coarse are cross-entropy loss functions, as shown in the following formula, and L binary is a binary cross-entropy loss function.

[0090] The specific formula is as follows:

[0091] Among them, y nc is a one-hot true value vector, that is, if the nth voxel belongs to category c, then y nc = 1, otherwise y nc = 0. Secondly, C and N are the total numbers of categories and voxels respectively. w c is the category weight.

[0092] Finally, we define the total loss function L total using the main loss function and the auxiliary loss function: L total = L fine + L binary + L coarse .

[0093] In this embodiment, the specific process of the 3D semantic and depth fusion module is proposed, including a structural branch and a semantic branch. The structural branch can clearly predict the scene occupancy, and the semantic branch includes a one-hot encoding process and a rough prediction process. The results of the two branches are fused to complete a more accurate SSC prediction. Finally, the Loss is reasonably designed to display the supervised space occupancy and SSC prediction. Ultimately, a three-dimensional semantic scene completion method based on semantic and depth enhancement proposed by the present invention can provide a more comprehensive understanding of the three-dimensional scene by enhancing the original two-dimensional depth input with holes and noise and combining two-dimensional dense semantics. In actual implementation, by complementing and enhancing the original depth and deeply coupling the semantic and structural branches, the mIoU accuracy of the three-dimensional semantic segmentation task can be greater than 60%, and the IoU of the three-dimensional scene completion task exceeds 83%.

[0094] The present invention can be divided into a preprocessing module, a depth enhancement module, and a 3D semantic and depth fusion module according to the processing flow. As Figure 5 shown in the overall framework diagram proposed by the present invention, a in the figure is the preprocessing module, which includes a 2D semantic segmentation sub-network and a 2D monocular depth estimation sub-network. These two sub-networks respectively predict semantic features and estimate depth from a single RGB; b in the figure is the depth enhancement module, which uses the predicted depth to enhance the incomplete original depth and combines convolutional blocks to find the enhanced depth feature representation; then uses a 2D-3D projection layer to map the 2D features to their corresponding 3D space; finally passes through the 3D semantic and depth fusion module c, which searches for deeper semantic-structure fusion through the structural and semantic branches to achieve high-level semantic scene completion.

[0095] Referring to Figure 6 shown, an embodiment of the present application discloses a three-dimensional semantic scene completion device, which may specifically include:

[0096] A depth enhancement module 11 is configured to use a depth estimation image corresponding to a two-dimensional image in a target RGB-D image including a three-dimensional scene to perform depth completion on the depth image in the target RGB-D image to determine a completed depth image, and use a preset feature extractor to extract two-dimensional target features in the completed depth image;

[0097] A feature projection module 12 is configured to input the two-dimensional target features and two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features output by the preset projection layer;

[0098] A feature extraction module 13 is configured to input the three-dimensional target features and the three-dimensional semantic features into a 3D backbone network to obtain extracted features output by the 3D backbone network;

[0099] A feature fusion module 14 is configured to perform feature fusion on the extracted features, the three-dimensional target features, and the three-dimensional semantic features, and use the fused features to complete three-dimensional semantic scene completion.

[0100] The present invention uses a depth estimation image corresponding to a two-dimensional image in a target RGB-D image including a three-dimensional scene to perform depth completion on the depth image in the target RGB-D image to determine a completed depth image, and uses a preset feature extractor to extract two-dimensional target features in the completed depth image; inputs the two-dimensional target features and two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features output by the preset projection layer; inputs the three-dimensional target features and the three-dimensional semantic features into a 3D backbone network to obtain extracted features output by the 3D backbone network; performs feature fusion on the extracted features, the three-dimensional target features, and the three-dimensional semantic features, and uses the fused features to complete three-dimensional semantic scene completion. In this way, an additional enhanced depth method is introduced in this embodiment to improve the SSC accuracy by completing holes and denoising the original depth, and complete the conversion of two-dimensional target features and two-dimensional semantic features to three-dimensional features through a 2D-3D preset projection layer, then use the 3D backbone network to extract the extracted features in the three-dimensional features, and perform feature fusion using the two-dimensional target features, two-dimensional semantic features, and the extracted features. During the whole process, the completed depth prior is deeply mined, and the depth fusion with the semantic prior is explored to use the final fused features and perform accurate prediction to complete three-dimensional semantic scene completion.

[0101] Furthermore, an embodiment of the present application also discloses an electronic deviceFigure 7 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure should not be regarded as any limitation on the scope of use of this application.

[0102] Figure 7 This is a schematic structural diagram of an electronic device 20 provided by an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a display screen 24, an input / output interface 25, a communication interface 26, and a communication bus 27. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the three-dimensional semantic scene completion method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0103] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 26 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of this application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.

[0104] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon may include an operating system 221, a computer program 222, and virtual machine data 223, etc. The virtual machine data 223 may include various kinds of data. The storage method may be temporary storage or permanent storage.

[0105] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, and it may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the three-dimensional semantic scene completion method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0106] Further, the present application also discloses a computer-readable storage medium, which includes a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, magnetic disk, optical disk, or any other form of storage medium known in the technical field. Among them, when the computer program is executed by a processor, it implements the three-dimensional semantic scene completion method disclosed above. For the specific steps of this method, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be elaborated here.

[0107] The various embodiments in this specification are described in a progressive manner. The key point of each embodiment is the difference from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and reference can be made to the method part for the relevant part. Professionals can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0108] The steps of the method or algorithm described in combination with the embodiments disclosed in this article can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the technical field.

[0109] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0110] The three-dimensional semantic scene completion method, device, equipment, and storage medium provided by the present invention have been introduced in detail above. Specific examples are used in this text to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A three-dimensional semantic scene completion method, characterized in that, Including: Using a depth estimation image corresponding to a two-dimensional image in a target RGB-D image including a three-dimensional scene to perform depth completion on the depth image in the target RGB-D image to determine a completed depth image, and using a preset feature extractor to extract two-dimensional target features in the completed depth image; Inputting the two-dimensional target features, two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image, and the completed depth image into a preset projection layer, so as to determine a three-dimensional surface voxel corresponding to each two-dimensional pixel in the completed depth image based on a camera projection equation predefined in the preset projection layer, and establishing a three-dimensional space based on the three-dimensional surface voxel. After discretizing the three-dimensional space, determining a projection coefficient between the completed depth image and the discretized three-dimensional space, and determining three-dimensional target features and three-dimensional semantic features corresponding to the two-dimensional target features and the two-dimensional semantic features respectively based on the projection coefficient; Inputting the three-dimensional target features and the three-dimensional semantic features into a 3D backbone network to obtain extracted features output by the 3D backbone network; Using a preset feature fusion method to fuse the extracted features and the three-dimensional target features to obtain a target matrix; each element in the target matrix represents the probability that the corresponding voxel is occupied in the three-dimensional space; Determining predicted rough features based on the extracted features; Encoding the three-dimensional semantic features based on a preset encoding function to determine encoded features; Adding the predicted rough features and the encoded features, performing pixel-level multiplication on the added features and the target matrix, and using the result of the pixel-level multiplication as fused features; Completing three-dimensional semantic scene completion using the fused features.

2. The three-dimensional semantic scene completion method according to claim 1, characterized in that The step of using a preset feature extractor to extract two-dimensional target features in the completed depth image includes: Using a preset feature extractor composed of a preset two-dimensional convolutional layer and a preset number of cascaded dimensional decomposition residual modules to extract two-dimensional target features in the completed depth image.

3. The three-dimensional semantic scene completion method according to claim 1, characterized in that The step of inputting the three-dimensional target features and the three-dimensional semantic features into a 3D backbone network to obtain extracted features output by the 3D backbone network includes: Inputting the three-dimensional target features and the three-dimensional semantic features into 3D SATNet-TNet to obtain extracted features output by 3DSATNet-TNet.

4. The three-dimensional semantic scene completion method according to claim 1, characterized in that The step of using a preset feature fusion method to fuse the extracted features and the three-dimensional target features to obtain a target matrix includes: Expanding the three-dimensional target features to determine a first feature to be concatenated, and inputting the extracted features into a preset conversion module to obtain a second feature to be concatenated output by the conversion module; the preset conversion module is composed of a three-dimensional convolutional layer, a regularization layer, and an activation function layer; Concatenating the first feature to be concatenated and the second feature to be concatenated to obtain a concatenated feature; Inputting the concatenated feature into the conversion module, connecting a preset convolutional layer after the conversion module, and performing a preset activation function after the convolutional layer to generate a target matrix.

5. The three-dimensional semantic scene completion method according to claim 4, wherein It also includes: Constructing a binary cross-entropy loss function as a first auxiliary loss function to optimize the target matrix; Constructing a first cross-entropy loss function as a second auxiliary loss function to optimize the predicted rough features; Constructing a second cross-entropy loss function as a main loss function to optimize the fused features; Constructing a total loss function by using the first auxiliary loss function, the second auxiliary loss function, and the main loss function.

6. A three-dimensional semantic scene completion device, characterized in that It includes: A depth enhancement module for using a depth estimation image corresponding to a two-dimensional image in a target RGB-D image including a three-dimensional scene to perform depth completion on the depth image in the target RGB-D image to determine a completed depth image, and using a preset feature extractor to extract two-dimensional target features in the completed depth image; A feature projection module for inputting the two-dimensional target features and two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features respectively output by the preset projection layer corresponding to the two-dimensional target features and the two-dimensional semantic features; wherein, inputting the two-dimensional target features and two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image into a preset projection layer to obtain three-dimensional target features and three-dimensional semantic features respectively output by the preset projection layer corresponding to the two-dimensional target features and the two-dimensional semantic features includes: inputting the two-dimensional target features, two-dimensional semantic features obtained by performing semantic segmentation on the two-dimensional image, and the completed depth image into the preset projection layer, so as to determine a three-dimensional surface voxel corresponding to each two-dimensional pixel in the completed depth image based on a pre-defined camera projection equation in the preset projection layer, establishing a three-dimensional space based on the three-dimensional surface voxel, discretizing the three-dimensional space, determining a projection coefficient between the completed depth image and the discretized three-dimensional space, and determining three-dimensional target features and three-dimensional semantic features respectively corresponding to the two-dimensional target features and the two-dimensional semantic features based on the projection coefficient; A feature extraction module for inputting the three-dimensional target features and the three-dimensional semantic features into a 3D backbone network to obtain extracted features output by the 3D backbone network; A feature fusion module for performing feature fusion on the extracted features, the three-dimensional target features, and the three-dimensional semantic features, and using the fused features to complete three-dimensional semantic scene completion; wherein, performing feature fusion on the extracted features, the three-dimensional target features, and the three-dimensional semantic features includes: performing feature fusion on the extracted features and the three-dimensional target features by using a preset feature fusion method to obtain a target matrix; each element in the target matrix represents the probability that the corresponding voxel is occupied in the three-dimensional space; determining predicted rough features based on the extracted features; encoding the three-dimensional semantic features by using a preset encoding function to determine encoded features; adding the predicted rough features and the encoded features, and performing pixel-level multiplication on the added features and the target matrix, and using the result of the pixel-level multiplication as the fused features.

7. An electronic device, characterized in that, It includes a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the three-dimensional semantic scene completion method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It is used to store a computer program; wherein, when the computer program is executed by a processor, it implements the three-dimensional semantic scene completion method according to any one of claims 1 to 5.

Images