Three-dimensional semantic scene completion method and device, storage medium and computer equipment

Abstract

The invention relates to a three-dimensional semantic scene completion method and device, a storage medium and computer equipment, and the method comprises the steps: inputting a two-dimensional image and a depth image into a three-dimensional semantic scene completion model, carrying out the feature extraction through a feature extraction module, extracting a two-dimensional semantic feature in the two-dimensional image, and extracting a depth feature in the depth image; carrying out feature fusion on the two-dimensional semantic features and the depth features to obtain a fused feature map; inputting the fused feature map into a feature enhancement module for feature enhancement to obtain an enhanced fused feature map; inputting the enhanced fusion feature map into a down-sampling module for down-sampling to obtain a down-sampled feature map; inputting the down-sampling feature map into an attention enhancement module for attention enhancement processing, and outputting a three-dimensional image of the scene to be complemented; the adaptive capacity to complex scenes is enhanced, and the accuracy of three-dimensional semantic scene completion is improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a three-dimensional semantic scene completion method, device, storage medium and computer equipment. Background Art

[0002] With the rapid development of computer vision and artificial intelligence, 3D semantic scene completion, as a core technology, is gradually becoming a key driving force for innovation and progress in fields such as robot navigation, autonomous driving, and virtual / augmented reality (VR / AR). 3D semantic scene completion reconstructs the geometric details of the scene based on sparse 3D point clouds, depth maps or partial scan data. However, the existing 3D semantic scene completion methods still face many challenges in processing complex scenes, coping with data scarcity, and fusing multimodal information. They are inefficient during use, have poor overall consistency, and inconsistent prediction results, which still have a gap with the application requirements of actual scenes. Summary of the invention

[0003] The main purpose of the present invention is to provide a three-dimensional semantic scene completion method, device, storage medium and computer equipment to solve the problem of insufficient accuracy of three-dimensional semantic scene completion in the prior art.

[0004] To achieve the above object, the present invention provides a three-dimensional semantic scene completion method in a first aspect, comprising:

[0005] Inputting a two-dimensional image and a corresponding depth image of the scene to be completed into a pre-trained three-dimensional semantic scene completion model, performing feature extraction through a feature extraction module of the three-dimensional semantic scene completion model, extracting two-dimensional semantic features from the two-dimensional image, and extracting depth features from the depth image;

[0006] Performing feature fusion on the two-dimensional semantic features and the depth features to obtain a fused feature map;

[0007] Inputting the fused feature map into the feature enhancement module of the three-dimensional semantic scene completion model for feature enhancement to obtain an enhanced fused feature map;

[0008] Inputting the enhanced fusion feature map into the downsampling module of the three-dimensional semantic scene completion model for downsampling to obtain a downsampled feature map;

[0009] The downsampled feature map is input into the attention enhancement module of the three-dimensional semantic scene completion model for attention enhancement processing, and a three-dimensional image of the scene to be completed is output.

[0010] Furthermore, the step of fusing the two-dimensional semantic features with the depth features to obtain a fused feature map includes:

[0011] The two-dimensional semantic features and the depth features are fused element by element to obtain a fused feature map.

[0012] Furthermore, the step of inputting the fused feature map into the feature enhancement module of the three-dimensional semantic scene completion model for feature enhancement to obtain the enhanced fused feature map includes:

[0013] Randomly rotating the fused feature map at a predefined angle in three-dimensional space to generate a feature missing region after the random rotation;

[0014] The image features of the feature-missing area are interpolated and calculated using the nearest neighbor interpolation method to complete the features of the feature-missing area.

[0015] Further, the random rotation is expressed as:

[0016] F rotated =R(F,θ);

[0017] Among them, F is the original feature voxel, θ is the angle of random rotation, and R is the rotation operation;

[0018] The range of θ is -5° to 5°.

[0019] Furthermore, the training process of the three-dimensional semantic scene completion model includes:

[0020] Acquire a training data set, wherein the training data set includes a plurality of sets of associated corresponding two-dimensional training images, depth training images, and three-dimensional training images;

[0021] The training objective function of the neural network model is set, wherein the training objective function includes cross entropy loss and category entropy loss, and the training objective function is expressed as:

[0022] L=L ce +λ*L E ;

[0023] Among them, L ce is the cross entropy loss, L E is the category entropy loss, and λ is the weight factor of the category entropy loss.

[0024] Inputting the training data set into a preset neural network model, setting model parameters, and outputting a predicted three-dimensional semantic scene through a forward propagation process of the neural network;

[0025] Calculating a loss value between the predicted three-dimensional semantic scene and the three-dimensional training image;

[0026] The gradient information of the loss function is propagated backward layer by layer through the back-propagation algorithm, and the model parameters are updated using the gradient descent optimization algorithm. Through iterative training until the loss value is less than a preset threshold, the trained neural network model is used as the three-dimensional semantic scene completion model.

[0027] Furthermore, the category entropy loss is:

[0028]

[0029] Among them, N c is the number of voxels of class c in the ground truth, p ci is the predicted probability of voxel i in category c in the ground truth, i∈S c And S c is the set of voxels labeled as class c, is the average probability vector of category c, and

[0030] Furthermore, the step of inputting the downsampled feature map into the attention enhancement module of the three-dimensional semantic scene completion model for attention enhancement processing and outputting the three-dimensional image of the scene to be completed includes:

[0031] Splitting all voxel features of the downsampled feature map into a plurality of patch maps, each patch map containing a plurality of voxel features;

[0032] Performing local self-attention processing in each of the patch images, encoding voxel features in each patch image, and generating corresponding query vectors, key vectors, and value vectors;

[0033] By calculating the similarity between the query vector and the key vector, the degree of attention of each voxel feature to other voxel features in the patch map where it is located is determined, and the value vector of each voxel feature in the patch map where it is located is updated accordingly to obtain a locally enhanced patch map;

[0034] Each locally enhanced patch map is used as part of the global feature map, and global self-attention processing is performed on the global feature map, so that each patch map pays attention to and aggregates the global information from other patch maps, and a feature map after global attention processing is obtained;

[0035] The locally enhanced patch map is fused with the feature map after global attention processing to obtain a final attention-enhanced feature map as the three-dimensional image of the scene to be completed.

[0036] A second aspect of the present invention provides a three-dimensional semantic scene completion device, the device comprising:

[0037] A feature extraction module, used to input the two-dimensional image and the corresponding depth image of the scene to be completed into a pre-trained three-dimensional semantic scene completion model, perform feature extraction through the feature extraction module of the three-dimensional semantic scene completion model, extract the two-dimensional semantic features in the two-dimensional image, and extract the depth features in the depth image;

[0038] A feature fusion module, used for fusing the two-dimensional semantic features with the depth features to obtain a fused feature map;

[0039] A feature enhancement module, used for inputting the fused feature map into the feature enhancement module of the three-dimensional semantic scene completion model for feature enhancement to obtain an enhanced fused feature map;

[0040] A downsampling module, used for inputting the enhanced fusion feature map into the downsampling module of the three-dimensional semantic scene completion model for downsampling to obtain a downsampled feature map;

[0041] The attention enhancement module is used to input the downsampled feature map into the attention enhancement module of the three-dimensional semantic scene completion model for attention enhancement processing, and output the three-dimensional image of the scene to be completed.

[0042] A third aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned three-dimensional semantic scene completion method when executing the computer program.

[0043] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned three-dimensional semantic scene completion method.

[0044] The three-dimensional semantic scene completion method, device, storage medium and computer equipment provided by the present invention input a two-dimensional image and a corresponding depth image of a scene to be completed into a pre-trained three-dimensional semantic scene completion model, perform feature extraction through a feature extraction module of the three-dimensional semantic scene completion model, extract two-dimensional semantic features in the two-dimensional image, and extract depth features in the depth image; perform feature fusion on the two-dimensional semantic features and the depth features to obtain a fused feature map; input the fused feature map into a feature enhancement module of the three-dimensional semantic scene completion model to perform feature enhancement to obtain an enhanced fused feature map; input the enhanced fused feature map into a lower layer of the three-dimensional semantic scene completion model The sampling module performs downsampling to obtain a downsampled feature map; the downsampled feature map is input into the attention enhancement module of the three-dimensional semantic scene completion model for attention enhancement processing, and the three-dimensional image of the scene to be completed is output; the model structure is optimized, and the feature diversity is enriched and the overfitting phenomenon is reduced through the feature enhancement module, and the generalization ability of the model under limited three-dimensional semantic labels is enhanced; and by introducing the attention enhancement module, the fused feature map is subjected to attention enhancement processing for efficiently capturing long-distance contextual information, so that the network can fully understand the 3D scene and pay more attention to the important information and details in the scene, thereby enhancing the adaptability to complex scenes and improving the accuracy of three-dimensional semantic scene completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a three-dimensional semantic scene completion method in one embodiment of the present invention;

[0046] Figure 2 is a schematic block diagram of the structure of a three-dimensional semantic scene completion device in one embodiment of the present invention;

[0047] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0048] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0050] Reference Figure 1 , an embodiment of the present invention provides a three-dimensional semantic scene completion method, comprising:

[0051] S1, inputting a two-dimensional image and a corresponding depth image of a scene to be completed into a pre-trained three-dimensional semantic scene completion model, performing feature extraction through a feature extraction module of the three-dimensional semantic scene completion model, extracting two-dimensional semantic features from the two-dimensional image, and extracting depth features from the depth image;

[0052] S2, fusing the two-dimensional semantic features and the depth features to obtain a fused feature map;

[0053] S3, inputting the fused feature map into the feature enhancement module of the 3D semantic scene completion model for feature enhancement to obtain an enhanced fused feature map;

[0054] S4, inputting the enhanced fusion feature map into the downsampling module of the 3D semantic scene completion model for downsampling to obtain a downsampled feature map;

[0055] S5. Input the downsampled feature map into the attention enhancement module of the three-dimensional semantic scene completion model for attention enhancement processing, and output the three-dimensional image of the scene to be completed.

[0056] In this embodiment, in the above step S1, the above two-dimensional image and the depth image correspond to the same two-dimensional image. The two-dimensional image of the scene to be completed can be obtained by shooting with a camera, and the depth image of the scene to be completed can be obtained by measuring with a depth sensor or processing with a computer vision algorithm. Specifically, a depth sensor (such as a lidar, a structured light sensor or a stereo camera) is used to scan the target scene to obtain the depth information of the scene. Alternatively, the two-dimensional image is processed by a computer vision algorithm (such as stereo matching, optical flow method, etc.), the depth information of the scene is estimated, and a depth image is generated.

[0057] Specifically, the input of the feature extraction module is a two-dimensional image and a depth image, wherein the two-dimensional image contains the semantic content of the scene to be completed, and the depth image provides the geometric structure information of the scene to be completed.

[0058] The step of extracting two-dimensional semantic features from the two-dimensional image comprises:

[0059] Preprocess the two-dimensional image, the preprocessing steps include image scaling, normalization and denoising; use a pre-trained convolutional neural network (CNN) or semantic segmentation network to extract features from the preprocessed two-dimensional image; extract two-dimensional semantic features from the output of the convolutional neural network or the semantic segmentation network.

[0060] The step of extracting the depth features in the depth image comprises:

[0061] Preprocess the depth image, including image scaling, normalization, denoising and interpolation; perform feature extraction on the preprocessed depth image through a deep feature extraction network (such as a convolutional neural network); extract deep features from the output of the deep feature extraction network.

[0062] In the above step S2, feature fusion is performed to combine information from different modalities, including semantic information and depth information, and can be performed using methods such as feature concatenation and feature weighted sum.

[0063] Specifically, the step S2 of fusing the two-dimensional semantic features and the depth features to obtain a fused feature map includes:

[0064] S201, performing element-by-element feature fusion on the two-dimensional semantic features and the depth features to obtain a fused feature map.

[0065] In this embodiment, the two-dimensional semantic features and the depth features are aligned, and feature fusion is performed by element-by-element addition. Element-by-element addition refers to adding the element values ​​of corresponding positions to fuse the information in the two feature maps. In this way, the complementary information in the two-dimensional semantic features and the depth features is effectively fused together to generate a fused feature map containing richer information.

[0066] In the above step S3, the feature enhancement module is used to enrich the diversity of features and reduce overfitting, thereby enhancing the generalization ability of the 3D semantic scene completion model. The feature enhancement module includes data enhancement techniques such as random rotation to introduce changes in 3D shapes and improve the robustness of the 3D semantic scene completion model to complex and distorted geometric structures.

[0067] Specifically, the step S3 of inputting the fused feature map into the feature enhancement module of the three-dimensional semantic scene completion model for feature enhancement to obtain the enhanced fused feature map includes:

[0068] S301, randomly rotating the fused feature map at a predefined angle in three-dimensional space, and generating a feature missing region after the random rotation;

[0069] S302: performing interpolation calculation on the image features of the feature-missing region by using a nearest neighbor interpolation method to complete the features of the feature-missing region.

[0070] Wherein, the random rotation is expressed as:

[0071] F rotated =R(F,θ);

[0072] Among them, F is the original feature voxel, θ is the angle of random rotation, and R is the rotation operation;

[0073] The range of θ is -5° to 5°, excluding 0°.

[0074] In this embodiment, random rotation can occur in any direction of the three axes x, y, and z of the spatial coordinate system, thereby enhancing the fusion feature map's ability to understand and express complex scenes.

[0075] In the above step S301, the fused feature map is randomly rotated in three-dimensional space within a predefined angle range, i.e., -5° to 5°, to ensure that the rotated fused feature map can still maintain certain original feature information, while introducing certain variability to enhance the generalization ability of the model.

[0076] The feature voxels in the fusion feature map rotate accordingly in three-dimensional space, resulting in feature missing areas. These missing areas are caused by the change in the position of some feature voxels due to the rotation, resulting in vacancies in the original positions.

[0077] In the above step S302, the nearest neighbor interpolation method is used to complete the features of the missing area to maintain the integrity and coherence of the feature map. The nearest neighbor interpolation method estimates the value of the missing area based on the value of the feature voxel of the nearest neighbor around the missing area. Specifically, for each position in the missing area, the feature voxel of the nearest neighbor around it is found, and the value of the feature voxel is assigned to the missing position, thereby completing the feature completion.

[0078] This embodiment realizes feature enhancement of the fused feature map, which not only improves the model's ability to understand complex scenes, but also enhances the robustness and generalization ability of the model by introducing randomness and feature completion mechanisms.

[0079] In the above step S4, in the downsampling module, one or more downsampling algorithms are applied to process the enhanced fused feature map, such as pooling, convolution operation, etc., to reduce the resolution of the feature map.

[0080] The pooling operation defines a pooling window on the feature map according to the selected pooling type (maximum pooling or average pooling) and performs the corresponding pooling operation within the window. Maximum pooling selects the maximum value in the window as the output, while average pooling calculates the average of all values ​​in the window as the output.

[0081] The convolution operation uses, for example, a convolution kernel with a step size of 2 to perform a convolution operation on the enhanced fusion feature map, thereby achieving feature extraction and downsampling at the same time.

[0082] Output downsampled feature map: After being processed by the downsampling algorithm, a downsampled feature map with reduced resolution but retained key information is obtained.

[0083] By processing the enhanced fused feature map through a downsampling module, the dimension and resolution of the data can be reduced while retaining key information.

[0084] In the above step S5, the downsampled feature map is input into the attention enhancement module to efficiently capture long-distance context information, so that the network can fully understand the 3D scene. Using a kernel function instead of the traditional Softmax function can rearrange the multiplication order of the query, key, and value (QKV) matrices, which helps to reduce the computational complexity from O(N^2) to O(N), where N represents the sequence length. This efficiency improvement is crucial for processing large-scale data sets and achieving real-time processing of complex multimodal data, such as data encountered in 3D vision tasks.

[0085] Specifically, a linear attention enhancement mechanism is adopted, and the voxel feature sequence in the downsampled feature map is taken as the input sequence x.

[0086] The input sequence x is mapped to the query vector (Query), key vector (Key), and value vector (Value), denoted as Q, K, and V, respectively, and denoted as W Q ∈R N×C , W K ∈R N×C and W V ∈R N×C . Subsequently, the attention mechanism A calculates the weighted average features of all elements in the sequence, where the weights are determined by the similarity between elements, which is measured by the kernel function φ. Specifically, the linear attention mechanism can be expressed as: Q = xW Q , K = xW K , V = xW V ,

[0087] A(x)=(φ(Q)φ(K) T )V=φ(Q)(φ(K) T V);

[0088] In a specific embodiment, the step S5 of inputting the downsampled feature map into the attention enhancement module of the three-dimensional semantic scene completion model for attention enhancement processing and outputting the three-dimensional image of the scene to be completed includes:

[0089] S501, dividing all voxel features of the downsampled feature map into a plurality of patch maps, each patch map containing a plurality of voxel features;

[0090] S502, performing local self-attention processing in each patch image, encoding voxel features in each patch image, and generating corresponding query vectors, key vectors, and value vectors;

[0091] S503, by calculating the similarity between the query vector and the key vector, determining the degree of attention of each voxel feature to other voxel features in the patch map where it is located, and updating the value vector of each voxel feature in the patch map where it is located accordingly, to obtain a locally enhanced patch map;

[0092] S504, taking each locally enhanced patch map as part of the global feature map, performing global self-attention processing on the global feature map, so that each patch map pays attention to and aggregates global information from other patch maps, and obtaining a feature map after global attention processing;

[0093] S505: Fusing the locally enhanced patch map with the feature map after global attention processing to obtain a final attention-enhanced feature map as the three-dimensional image of the scene to be completed.

[0094] In this embodiment, in the above step S501, the downsampled feature map is segmented uniformly or non-uniformly to divide it into multiple small blocks or patch maps. Each patch map contains a certain number of voxel features, which represent local information in the three-dimensional space. The segmentation method can be determined according to the specific three-dimensional scene and application requirements to ensure that each patch map can capture enough local information while maintaining computational efficiency.

[0095] In the above step S502, for each patch image, the voxel features inside it are first encoded to generate the corresponding query vector (Query), key vector (Key) and value vector (Value). The generation of the query vector, key vector and value vector can be achieved through linear transformation to ensure that they can accurately represent the relationship between voxel features.

[0096] Local self-attention processing is performed on the voxel features in each patch map. Specifically, by calculating the similarity between the query vector and the key vector, the degree of attention of each voxel feature to other voxel features in the patch map where it is located is determined. According to the degree of attention, the value vector of each voxel feature is updated to obtain a locally enhanced patch map. This process enables each voxel feature to pay more attention to the local information around it, thereby improving the accuracy of 3D semantic scene completion.

[0097] In the above step S503, in the local self-attention processing, the similarity can be calculated by methods such as dot product, cosine similarity or bilinear function. By calculating the similarity between the query vector and the key vector, an attention weight matrix can be obtained, which represents the degree of attention of each voxel feature to other voxel features. According to the attention weight matrix, the value vector is weighted and summed to obtain an updated value vector. This process realizes the local enhancement of each voxel feature.

[0098] In the above step S504, the patch images after local enhancement are spliced ​​together to form a global feature map. Global self-attention processing is performed on the global feature map so that each patch image can pay attention to and aggregate global information from other patch images. This is achieved by calculating the similarity between different patch images in the global feature map. The similarity can also be calculated by methods such as dot product, cosine similarity or bilinear function. According to the similarity, the value vector of each patch image is weighted and summed to obtain a feature map after global attention processing. This process enables each patch image to contain global information, thereby improving the robustness and accuracy of three-dimensional semantic scene completion.

[0099] In the above step S505, the fusion method can adopt weighted summation, splicing or convolution operations. Through fusion, the final attention-enhanced feature map contains both local information and global information, so that it can better represent the semantic information in the three-dimensional scene. A complete three-dimensional semantic scene is generated through modules such as a decoder. The generated three-dimensional scene not only contains the geometric shape and position information of the object, but also contains information such as the semantic label of the object, thereby realizing the three-dimensional semantic completion of complex scenes.

[0100] This embodiment not only improves the accuracy of three-dimensional semantic scene completion, but also enhances the robustness and generalization ability of the model.

[0101] In a specific embodiment, the training process of the 3D semantic scene completion model includes:

[0102] S01, obtaining a training data set, wherein the training data set includes a plurality of groups of associated corresponding two-dimensional training images, depth training images and three-dimensional training images;

[0103] S02. Setting a training objective function of the neural network model, wherein the training objective function includes a cross entropy loss and a category entropy loss, and the training objective function is expressed as:

[0104] L=L ce +λ*L E ;

[0105] Among them, L ce is the cross entropy loss, L E is the category entropy loss, and λ is the weight factor of the category entropy loss.

[0106] S03, inputting the training data set into a preset neural network model, setting model parameters, and outputting a predicted three-dimensional semantic scene through a forward propagation process of the neural network;

[0107] S04, calculating a loss value between the predicted three-dimensional semantic scene and the three-dimensional training image;

[0108] S05. The gradient information of the loss function is propagated backward layer by layer through a back-propagation algorithm, and the model parameters are updated using a gradient descent optimization algorithm. The training is iterated until the loss value is less than a preset threshold, and the trained neural network model is used as the three-dimensional semantic scene completion model.

[0109] In this embodiment, in the above step S01, the training data set is collected by various means (such as sensor acquisition, manual annotation, etc.), including multiple sets of associated corresponding two-dimensional training images, depth training images and three-dimensional training images. The two-dimensional training images provide visual information of the scene, the depth training images provide depth information of the scene, and the three-dimensional training images contain the complete three-dimensional structure and semantic information of the scene, which constitute the input and labels required for the training model.

[0110] In the above step S02, the training objective function combines the cross entropy loss and the class entropy loss. By adjusting the weight between the two, the difference between the predicted result and the actual label is minimized, the consistency of the prediction is improved, and the training process of the model is optimized. By adjusting the value of λ, the importance of the two loss functions can be balanced to achieve the best performance.

[0111] The cross entropy loss is used to measure the difference between the pixel category predicted by the model and the true category; the category entropy loss is used to supervise and minimize the prediction inconsistency between different categories, and to enhance the prediction consistency of each instance by reducing the randomness of the predicted probability distribution.

[0112] Among them, the category entropy loss is:

[0113]

[0114] Among them, N c is the number of voxels of class c in the ground truth, p ci is the predicted probability of voxel i in category c in the ground truth, i∈S c And S c is the set of voxels labeled as class c, is the average probability vector of category c, and

[0115] The class entropy loss is only applied when the following two conditions are met: first, class c must contain at least one voxel in the ground truth; second, the average probability vector must have its highest probability at location c, indicating a reasonable prediction for that class. In this way, we hope that the average probability is concentrated in the right location and as large as possible.

[0116] In the above step S03, the training data set is input into a preset neural network model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). The model parameters are initialized before the training begins, and then through the forward propagation process, the three-dimensional structure and semantic information of the scene are gradually inferred based on the input two-dimensional and depth image information, as well as the parameters such as weights and biases inside the neural network model.

[0117] In the above step S04, the loss value reflects the degree of difference between the model prediction result and the actual situation. The loss value can be calculated by, for example, mean square error (MSE), mean absolute error (MAE), etc., depending on the requirements of the task and the characteristics of the data.

[0118] In the above step S05, the model parameters are optimized by the back propagation algorithm and the gradient descent optimization algorithm. In each iteration, the model parameters are adjusted according to the gradient information to reduce the loss value and improve the prediction accuracy of the model. The iterative training process will continue until the loss value is less than the preset threshold. At this point, the trained neural network model is regarded as a three-dimensional semantic scene completion model and can be used in subsequent scene completion tasks.

[0119] Reference Figure 2 The embodiment of the present invention further provides a three-dimensional semantic scene completion device, the device comprising:

[0120] A feature extraction module 10 is used to input the two-dimensional image and the corresponding depth image of the scene to be completed into a pre-trained three-dimensional semantic scene completion model, perform feature extraction through the feature extraction module of the three-dimensional semantic scene completion model, extract the two-dimensional semantic features in the two-dimensional image, and extract the depth features in the depth image;

[0121] A feature fusion module 20 is used to fuse the two-dimensional semantic features and the depth features to obtain a fused feature map;

[0122] A feature enhancement module 30, used for inputting the fused feature map into the feature enhancement module of the 3D semantic scene completion model for feature enhancement to obtain an enhanced fused feature map;

[0123] A downsampling module 40 is used to input the enhanced fusion feature map into the downsampling module of the three-dimensional semantic scene completion model for downsampling to obtain a downsampled feature map;

[0124] The attention enhancement module 50 is used to input the downsampled feature map into the attention enhancement module of the three-dimensional semantic scene completion model to perform attention enhancement processing, and output the three-dimensional image of the scene to be completed.

[0125] In a specific embodiment, the feature fusion module 20 includes:

[0126] The feature fusion unit is used to perform element-by-element feature fusion of the two-dimensional semantic features and the depth features to obtain a fused feature map.

[0127] In a specific embodiment, the feature enhancement module 30 includes:

[0128] A rotation unit, used for randomly rotating the fused feature map at a predefined angle in a three-dimensional space to generate a feature missing region after the random rotation;

[0129] The interpolation unit is used to perform interpolation calculation on the image features of the feature-missing area by using the nearest neighbor interpolation method to complete the features of the feature-missing area.

[0130] In a specific embodiment, the random rotation is expressed as:

[0131] F rotated =R(F,θ);

[0132] Among them, F is the original feature voxel, θ is the angle of random rotation, and R is the rotation operation;

[0133] The range of θ is -5° to 5°.

[0134] In a specific embodiment, the training process of the 3D semantic scene completion model includes:

[0135] Acquire a training data set, wherein the training data set includes a plurality of sets of associated corresponding two-dimensional training images, depth training images, and three-dimensional training images;

[0136] The training objective function of the neural network model is set, wherein the training objective function includes cross entropy loss and category entropy loss, and the training objective function is expressed as:

[0137] L=L ce +λ*L E ;

[0138] Among them, L ce is the cross entropy loss, L E is the category entropy loss, and λ is the weight factor of the category entropy loss.

[0139] Inputting the training data set into a preset neural network model, setting model parameters, and outputting a predicted three-dimensional semantic scene through a forward propagation process of the neural network;

[0140] Calculating a loss value between the predicted three-dimensional semantic scene and the three-dimensional training image;

[0141] The gradient information of the loss function is propagated backward layer by layer through the back-propagation algorithm, and the model parameters are updated using the gradient descent optimization algorithm. Through iterative training until the loss value is less than a preset threshold, the trained neural network model is used as the three-dimensional semantic scene completion model.

[0142] In a specific embodiment, the category entropy loss is:

[0143]

[0144] Among them, N c is the number of voxels of class c in the ground truth, p ci is the predicted probability of voxel i in category c in the ground truth, i∈S c And S c is the set of voxels labeled as class c, is the average probability vector of category c, and

[0145] In a specific embodiment, the attention enhancement module 50 includes:

[0146] A segmentation unit, used for segmenting all voxel features of the downsampled feature map into a plurality of patch maps, each patch map containing a plurality of voxel features;

[0147] A first local unit is used to perform local self-attention processing in each patch image, encode voxel features in each patch image, and generate corresponding query vectors, key vectors, and value vectors;

[0148] The second local unit is used to determine the degree of attention of each voxel feature to other voxel features in the patch map where it is located by calculating the similarity between the query vector and the key vector, and update the value vector of each voxel feature in the patch map where it is located accordingly to obtain a locally enhanced patch map;

[0149] A global unit is used to treat each locally enhanced patch map as part of the global feature map, perform global self-attention processing on the global feature map, so that each patch map pays attention to and aggregates the global information from other patch maps, and obtains a feature map after global attention processing;

[0150] The output unit is used to fuse the locally enhanced patch map with the feature map after global attention processing to obtain a final attention-enhanced feature map as the three-dimensional image of the scene to be completed.

[0151] For the specific implementation of each module in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0152] Reference Figure 3In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected via a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above-mentioned three-dimensional semantic scene completion method is implemented.

[0153] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0154] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0156] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0157] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A three-dimensional semantic scene completion method, characterized in that: include: Inputting a two-dimensional image and a corresponding depth image of the scene to be completed into a pre-trained three-dimensional semantic scene completion model, performing feature extraction through a feature extraction module of the three-dimensional semantic scene completion model, extracting two-dimensional semantic features from the two-dimensional image, and extracting depth features from the depth image; Performing feature fusion on the two-dimensional semantic features and the depth features to obtain a fused feature map; Inputting the fused feature map into the feature enhancement module of the three-dimensional semantic scene completion model for feature enhancement to obtain an enhanced fused feature map; Inputting the enhanced fusion feature map into the downsampling module of the three-dimensional semantic scene completion model for downsampling to obtain a downsampled feature map; The downsampled feature map is input into the attention enhancement module of the three-dimensional semantic scene completion model for attention enhancement processing, and a three-dimensional image of the scene to be completed is output.

2. The three-dimensional semantic scene completion method according to claim 1, characterized in that: The step of fusing the two-dimensional semantic features and the depth features to obtain a fused feature map comprises: The two-dimensional semantic features and the depth features are fused element by element to obtain a fused feature map.

3. The three-dimensional semantic scene completion method according to claim 1, characterized in that: The step of inputting the fused feature map into the feature enhancement module of the three-dimensional semantic scene completion model for feature enhancement to obtain the enhanced fused feature map comprises: Randomly rotating the fused feature map at a predefined angle in three-dimensional space to generate a feature missing region after the random rotation; The image features of the feature-missing area are interpolated and calculated using the nearest neighbor interpolation method to complete the features of the feature-missing area.

4. The three-dimensional semantic scene completion method according to claim 3, characterized in that: The random rotation is expressed as: F rotated =R(F,θ); Among them, F is the original feature voxel, θ is the angle of random rotation, and R is the rotation operation; The range of θ is -5° to 5°.

5. The three-dimensional semantic scene completion method according to claim 1, characterized in that: The training process of the three-dimensional semantic scene completion model includes: Acquire a training data set, wherein the training data set includes a plurality of sets of associated corresponding two-dimensional training images, depth training images, and three-dimensional training images; The training objective function of the neural network model is set, wherein the training objective function includes cross entropy loss and category entropy loss, and the training objective function is expressed as: L=L ce +λ*L E ; Among them, L ce is the cross entropy loss, L E is the category entropy loss, and λ is the weight factor of the category entropy loss. Inputting the training data set into a preset neural network model, setting model parameters, and outputting a predicted three-dimensional semantic scene through a forward propagation process of the neural network; Calculating a loss value between the predicted three-dimensional semantic scene and the three-dimensional training image; The gradient information of the loss function is propagated backward layer by layer through the back-propagation algorithm, and the model parameters are updated using the gradient descent optimization algorithm. Through iterative training until the loss value is less than a preset threshold, the trained neural network model is used as the three-dimensional semantic scene completion model.

6. The three-dimensional semantic scene completion method according to claim 5, characterized in that: The class entropy loss is: Among them, N c is the number of voxels of class c in the ground truth, p ci is the predicted probability of voxel i in category c in the ground truth, i∈S c And S c is the set of voxels labeled as class c, is the average probability vector of category c, and 7. The three-dimensional semantic scene completion method according to claim 1, characterized in that: The step of inputting the downsampled feature map into the attention enhancement module of the three-dimensional semantic scene completion model for attention enhancement processing and outputting the three-dimensional image of the scene to be completed includes: Splitting all voxel features of the downsampled feature map into a plurality of patch maps, each patch map containing a plurality of voxel features; Performing local self-attention processing in each of the patch images, encoding voxel features in each patch image, and generating corresponding query vectors, key vectors, and value vectors; By calculating the similarity between the query vector and the key vector, the degree of attention of each voxel feature to other voxel features in the patch map where it is located is determined, and the value vector of each voxel feature in the patch map where it is located is updated accordingly to obtain a locally enhanced patch map; Each locally enhanced patch map is used as part of the global feature map, and global self-attention processing is performed on the global feature map, so that each patch map pays attention to and aggregates the global information from other patch maps, and a feature map after global attention processing is obtained; The locally enhanced patch map is fused with the feature map after global attention processing to obtain a final attention-enhanced feature map as the three-dimensional image of the scene to be completed.

8. A three-dimensional semantic scene completion device, characterized in that: The device comprises: A feature extraction module, used to input the two-dimensional image and the corresponding depth image of the scene to be completed into a pre-trained three-dimensional semantic scene completion model, perform feature extraction through the feature extraction module of the three-dimensional semantic scene completion model, extract the two-dimensional semantic features in the two-dimensional image, and extract the depth features in the depth image; A feature fusion module, used for fusing the two-dimensional semantic features with the depth features to obtain a fused feature map; A feature enhancement module, used for inputting the fused feature map into the feature enhancement module of the three-dimensional semantic scene completion model for feature enhancement to obtain an enhanced fused feature map; A downsampling module, used for inputting the enhanced fusion feature map into the downsampling module of the three-dimensional semantic scene completion model for downsampling to obtain a downsampled feature map; The attention enhancement module is used to input the downsampled feature map into the attention enhancement module of the three-dimensional semantic scene completion model for attention enhancement processing, and output the three-dimensional image of the scene to be completed.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the three-dimensional semantic scene completion method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional semantic scene completion method described in any one of claims 1 to 7 are implemented.

Images