Three-dimensional semantic scene completion method based on camera enhancement, medium and equipment

Abstract

The invention discloses a three-dimensional semantic scene completion method based on camera enhancement, a medium and equipment, and the method comprises the steps: obtaining a depth map and an optical flow graph through a left image and a right image, extracting a two-dimensional feature from the left image, and extracting a two-dimensional feature from the depth map; mapping the left image to obtain context features; performing depth optimization based on the two-dimensional features, the two-dimensional features and the depth map to obtain depth estimation distribution; obtaining and generating an expanded three-dimensional feature map through context feature operation expansion, and sequentially executing three-dimensional deformable cross attention and deformable self-attention operations on the three-dimensional feature map to output updated three-dimensional features; and geometric semantic enhancement is carried out on the updated three-dimensional features, and finally three-dimensional semantic scene completion is completed. The depth prediction precision is improved, depth estimation optimization is carried out, finally the geometric structure and detail prediction capability of the model is enhanced, and three-dimensional semantic scene completion is completed.

Description

Technical Field

[0001] This invention relates to the field of 3D scene completion, and in particular to a method, medium, and device for 3D semantic scene completion based on camera enhancement. Background Technology

[0002] Semantic scene completion (SSC) is the task of completing missing structures in a 3D scene and simultaneously inferring the semantic label of each voxel in the scene. Understanding the geometric structure and semantic information of 3D scenes is a core challenge in computer vision research, crucial for mobile intelligent agents to interact with the real world, and has wide applications, such as autonomous driving and navigation.

[0003] Currently, semantic scene completion technology has become an emerging and crucial core task in the field of autonomous driving due to its ability to provide a more refined description of the 3D world. However, some challenging issues still hinder its progress: 1) Depending on the input, semantic scene completion technology can be broadly categorized into LiDAR-based methods and camera-based methods. Although LiDAR-based methods have made significant progress with accurate depth information, LiDAR sensors are expensive, and LiDAR scans are typically sparse and incomplete, limiting their widespread application in real-world scenarios. 2) To complete 3D scenes, 2D features are often upscaled to 3D space, such as the line-of-sight feature projection (FLoSP) method used in existing MonoScene. However, due to its lack of a depth perception mechanism, this upscaling operation may map 2D features to incorrect spatial regions (such as blank or semantically overlapping areas). Summary of the Invention

[0004] In view of this, this application discloses a camera-enhanced 3D semantic scene completion method, medium, and device. It uses optical flow-guided depth networks to improve depth prediction accuracy, then optimizes depth estimation, and finally enhances the model's geometric structure and detail prediction capabilities to complete the 3D semantic scene completion.

[0005] To achieve the above objectives, the disclosed technical solution is as follows: A camera-enhanced 3D semantic scene completion method includes: Depth maps and optical flow maps are obtained from the left and right images. Two-dimensional features are extracted from the left image and the depth map. Map the left-hand image to obtain contextual features; The depth estimation distribution is obtained by using two-dimensional features and a depth map and performing depth optimization. The extended 3D feature map is obtained by expanding the context feature operation. The 3D feature map is then subjected to 3D deformable cross attention and deformable self attention operations in sequence to output the updated 3D features. The updated 3D features are then subjected to geometric semantic enhancement to ultimately complete the 3D semantic scene.

[0006] As a further improvement, the left-side image is mapped to obtain contextual features, wherein the mapping is based on a context network; The context network input to output sequentially passes through a multilayer perceptron, a compressed excitation attention layer, and a two-dimensional convolutional network.

[0007] As a further improvement, the depth estimation distribution is obtained by performing depth optimization based on two-dimensional features and a depth map. The depth optimization is performed using an optical flow-guided depth network, and includes... Initial optical flow estimation is performed based on two-dimensional features, two-dimensional features, and depth map to obtain initial depth features, optical flow features, and initial sampling features; Based on the initial sampled features, deformation processing is performed to obtain the modulated depth features; Based on the initial depth features and the modulated depth features, a dual-fusion interactive optimization is performed to obtain the refined depth estimation distribution.

[0008] As a further improvement, the initial optical flow estimation based on two-dimensional features, two-dimensional features and depth map to obtain initial depth features, optical flow features and initial sampling features includes: performing SE operation on the two-dimensional features and the two-dimensional features respectively, and then processing them by a basic module, a holed spatial pyramid pooling and a deformable convolutional network to obtain the initial depth features and optical flow features, and performing downsampling and one-hot encoding on the depth map to obtain the initial sampling features. The method of obtaining modulated depth features by performing deformation processing based on initial sampling features includes: firstly, deforming the initial sampling features to obtain depth features; secondly, fusing optical flow features with splicing to obtain fused features; and then using an adaptive integration function to output modulated depth features from the fused features. The method described above, which performs dual-fusion interactive optimization based on initial depth features and modulated depth features to obtain a refined depth estimation distribution, includes: firstly, fusing local and global features through a cross-local-global attention module, and secondly, optimizing the fused features through a self-local-global attention module to obtain the depth estimation distribution.

[0009] As a further improvement, an expanded 3D feature map is obtained through context feature operations. The 3D feature map is then subjected to 3D deformable cross attention and deformable self attention operations to output updated 3D features, including: First, the initial sampled features and context features are multiplied by an outer product to generate an extended 3D feature map. Then, the depth estimation distribution and context features are multiplied by a dot product and converted into voxel features through coordinate transformation, post-processing, and voxel pooling. A pre-defined initial voxel query is then used and combined with the voxel features to generate new query features. Seed query features are extracted by generating a priori mask index from the depth map. Secondly, the seed query feature is used as the query vector, and the expanded 3D feature map is used as the key vector and value vector. The result is input into the 3D deformable cross attention module to obtain the updated query vector, and the query vector is input into the deformable self attention module to obtain the updated 3D features.

[0010] As a further improvement, the process of generating a priori mask index from the depth map to extract seed query features includes: back-projecting the depth map into a point cloud and voxelizing it to obtain a binary query proposal; analyzing the binary query proposal to determine the unmasked and masked index positions, and combining them with query features to extract seed query features.

[0011] As a further improvement, the updated 3D features are subjected to geometric semantic enhancement using a geometric semantic enhancement module, which includes a lightweight residual voxel network and a sparse UNet network.

[0012] As a further improvement, the lightweight residual voxel network includes several residual voxel blocks, each of which contains two three-dimensional convolutional layers with a stride of 2, and each three-dimensional convolutional layer is followed by a normalization layer and a ReLU activation function. The sparse UNet network includes a downsampling module, an adaptive hierarchical aggregator, an upsampling module, and an interpolation part.

[0013] The present invention also provides a storage medium comprising stored instructions, wherein, when the instructions are executed, the device on which the storage medium resides executes the camera-enhanced 3D semantic scene completion method.

[0014] The present invention also provides an electronic device, including a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors, the camera-enhanced 3D semantic scene completion method described herein.

[0015] The beneficial effects of this application are: This application employs 3D deformable attention to achieve efficient feature updating, thereby mitigating depth ambiguity and enhancing features. A dual-fusion cross-attention mechanism is introduced to optimize depth distribution. An optical flow modulation module is used to analyze the strong correlation between optical flow abrupt changes and depth variations to optimize the depth map. A residual voxel network is introduced after the deformable self-attention module to enhance the model's geometric structure and detail prediction capabilities. A more efficient sparse UNet structure is used to replace traditional 3D convolution or deconvolution, and cross-scale semantic coherent reasoning is achieved through an adaptive hierarchical aggregator (AHA). Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a camera-based 3D semantic scene completion method disclosed in an embodiment of this application. Figure 2 This is a schematic diagram of the optical flow-guided deep network structure disclosed in an embodiment of this application; Figure 3 For generating updated 3D features as disclosed in the embodiments of this application And a logical diagram illustrating geometric semantic enhancement through alignment; Figure 4 This is a schematic diagram of the residual voxel block structure disclosed in an embodiment of this application; Figure 5 This is a schematic diagram of the sparse UNet network structure disclosed in the embodiments of this application; Figure 6 The quantitative comparison results between our method and existing camera-based semantic scene completion methods are reported on the SemanticKITTI validation set. Figure 7 The quantitative comparison results between our method and existing camera-based semantic scene completion methods are reported on the SemanticKITTI test set. Figure 8 This is a qualitative comparison chart of our method with existing methods, including Monoscene, VoxFormer, OccFormer, and Symphonies. Figure 9 The results of quantitative analysis of this method on the SSCBench-KITTI360 dataset are also shown in the figure. Figure 10This is a schematic diagram of the structure of the electronic device disclosed in the embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0020] Existing semantic scene completion techniques are mostly based on LiDAR input. However, LiDAR sensors are expensive, and LiDAR scans are typically sparse and incomplete, limiting their widespread application. This application proposes a 3D semantic scene completion method based on camera sampling input.

[0021] As one application scenario, this application can be used in the field of intelligent assisted driving, where vehicles identify their surroundings while driving. This application significantly improves depth prediction accuracy by fusing depth maps generated by an offline estimator with optical flow images. Subsequently, a 3D deformable cross-attention mechanism is employed to achieve efficient feature interaction in the 3D pixel space. Furthermore, prior mask indexing is used to guide the update process of the initial voxel query. Finally, through the synergistic effect of the residual voxel network and the sparse UNet, geometric details are accurately captured and the consistency of multi-scale semantic reasoning is ensured, achieving 3D semantic scene completion. The specific implementation is described in detail in the following embodiments.

[0022] It is understood that before using the technical solutions disclosed in the embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0023] Furthermore, this application can also be applied to other fields in other situations, such as navigation and augmented reality.

[0024] refer to Figure 1The diagram shown is a flowchart illustrating a camera-enhanced 3D semantic scene completion method disclosed in this application. The 3D semantic scene completion method mainly includes the following steps: S10, obtain depth map and optical flow map.

[0025] The left and right images are acquired using the left and right cameras on the vehicle, respectively, and are denoted as follows: and Both the left and right images are RGB images, which are then fused with the depth map generated by the offline estimator and the optical flow map. For ease of description, the depth map is denoted as... Optical flow diagram is recorded as .

[0026] The fusion offline estimator in this application is based on the existing pre-trained deep estimation models MobileStereoNet and GMFlow. Input and Obtain depth map Input several consecutive numbers Obtain optical flow map Optical flow diagram When retrieving the images, you can input, for example, 10 consecutive images from the left side.

[0027] In this application, a depth map is first obtained through a pre-trained model. and optical flow graph This can improve the accuracy of subsequent depth prediction.

[0028] S20, extract two-dimensional features.

[0029] Input the image on the left and optical flow graph Based on the pre-trained EfficientNet-B7 model, respectively from and Extract two-dimensional features from them, denoted as follows: and .

[0030] S30, through the context network and optical flow, guides the deep network to obtain the corresponding context features and depth estimation distributions, respectively.

[0031] The left image is mapped onto the context network to obtain context features. Specifically, in the context network construction, we combine a multilayer perceptron (MLP), a compressed activation (SE) layer, and a two-dimensional convolutional network (CNN) to process camera parameters and adjust feature maps. Specifically, in the context network, the input image is the image on the left. After passing through a multilayer perceptron, a compressed excitation attention layer (SE), and a two-dimensional convolutional network in sequence, the output contextual features are denoted as... .

[0032] Based on two-dimensional features Two-dimensional features and depth map The depth estimation distribution is obtained through an optical flow-guided depth network (OFGDepthNet). Specifically, this includes... S3010 performs initial optical flow estimation to obtain initial depth features, optical flow features, and sampling features.

[0033] Through the The SE operation is performed, followed by processing through a convolutional sequence consisting of a base module, a dilated spatial pyramid pooling (ASPP) network, and a deformable convolutional network (DCN) to obtain the initial depth features, denoted as . .

[0034] Similarly, through the The optical flow features are obtained by performing SE operations and then processing them through a convolutional sequence consisting of basic modules, spatial pyramid pooling (ASPP), and deformable convolutional networks (DCN), denoted as .

[0035] At the same time, for depth maps Downsampling and one-hot encoding are performed to obtain initial sampled features with consistent dimensions, denoted as... .

[0036] This step involves initial optical flow estimation to obtain initial depth features, optical flow features, and sampling features, which significantly improves the depth prediction accuracy of OFG DepthNet and prepares for the optimization of the depth estimation distribution.

[0037] S3020 is subjected to deformation processing to obtain modulated depth features.

[0038] For depth maps predicted by depth estimation models The problem of insufficient accuracy in areas such as the edges of moving objects is addressed here. Continue to Optical flow modulation is applied to improve the precision of object edges.

[0039] See Figure 2 As shown, during optical flow modulation, the optical flow is first modulated by a two-dimensional convolutional layer according to... and The spatial difference is used to predict the coordinate offset, and then the calculated optical flow field Ω is used to... Perform deformation processing.

[0040] The deformation processing involves several steps: first, creating a mesh grid; second, normalizing the optical flow to a smaller range to prevent excessive deformation; third, adjusting the dynamic factor (Calcu.adju.factor); and finally, performing mesh clipping and bilinear sampling (Grid clamp. & bilinear.samp.). The resulting optical flow field Ω is then used to... Deformation processing is performed to obtain the depth features after deformation processing, denoted as... .

[0041] Next, and After concatenation, the features are input into a 3×3 two-dimensional convolutional network for feature fusion to obtain fused features, denoted as . Subsequently, an adaptive ensemble function is used to output the modulated deep features, denoted as... This adaptive function primarily generates an attention weight map through global average pooling and 1×1 two-dimensional convolution, which is then combined with the fused features. Multiplication is used to achieve adaptive feature enhancement.

[0042] S3030 performs dual-fusion interactive optimization to obtain the refined depth estimation distribution. .

[0043] To effectively fuse initial deep features With modulated depth features For these two different forms of depth features, we employ a dual-fusion cross-attention module to improve depth estimation accuracy.

[0044] See Figure 2 As shown, the core mechanism of this module is to achieve cross-transmission of query vectors during the interaction of two deep features. Figure 2 In this context, q, v, and k correspond to the query vector, value vector, and numerical vector of the attention mechanism, respectively.

[0045]

[0046]

[0047] It should be noted that the widely used cross-attention mechanism was not employed during the interaction process. Based on a comprehensive consideration of efficiency and effectiveness, this application uses a cross-local-global attention module (Cross-LG-Attention) consisting of neighborhood attention and detail neighborhood attention, where the expansion coefficient... The values ​​are set to 1 and 2 respectively to fuse local and global features. The fused features are then optimized and enhanced using a self-LG-Attention module, ultimately yielding a refined depth estimation distribution. .

[0048] In this step, this application optimizes the depth distribution by introducing a dual-fusion cross-attention mechanism in the Optical Flow Guided Deep Network (OFG DepthNet). Considering insufficient geometric constraints, the pre-trained depth estimation model... Abrupt regions (such as vehicle boundaries) are prone to prediction bias. This application combines an optical flow modulation module to optimize the depth map by analyzing the strong correlation between optical flow abrupt changes and depth variations, thus obtaining a refined depth estimation distribution. .

[0049] S40: Based on contextual features, perform computational expansion to obtain an expanded 3D feature map. Then, sequentially perform 3D deformable cross attention and deformable self attention operations on the 3D feature map to output the updated 3D features.

[0050] By initial sampling features With context features Perform the outer product operation to generate an extended 3D feature map, denoted as . Simultaneously, depth estimation distribution and context features After dot product operation, coordinate transformation is performed. Post-processing and voxel pooling operations are performed to convert the data into voxel features. Here This represents the transformation from the camera coordinate system to the vehicle coordinate system. To facilitate subsequent feature queries and updates, a pre-defined initial voxel query is used. and with Combine to generate new query features .

[0051] To further improve computational efficiency, a pre-trained depth estimation model is used to generate a priori mask index. Specifically, the depth map... Back-projection yields a point cloud, which is then voxelized to obtain a binary query proposal. By analyzing binary query proposals Determine the index positions of the unmasked and masked elements, and match them with the query features. Based on this, Extracting seed query features .

[0052] Subsequently, a 3D deformable cross-attention approach is employed to achieve depth-ambiguity-free feature enhancement, effectively addressing the inherent depth-ambiguity problem of projection methods. Specifically, we use seed query features... The expanded 3D feature map will be used as the query vector. As key vectors and value vectors.

[0053] For those located at position (x, y, z) in the autonomous driving coordinate system, query vector Its corresponding three-dimensional pixel coordinate system reference point It can be calculated using the following formula:

[0054]

[0055] in , , , This is the camera intrinsic parameter. Subsequently, a 3D deformable cross-attention operation is performed in the pixel coordinate system, and the updated query vector is obtained using the following formula. :

[0056] Where K represents the number of sampling points, W represents the learnable weights, and A represents the attention weights. R refers to the sampling offset, T rili (·) indicates that in The bilinear interpolation operation is performed in the middle.

[0057] To further enhance the overall scenario's granularity, the query vector will be... Rephrased as voxel features We employ a deformable self-attention mechanism for voxel features. After performing diffusion processing, the updated 3D features are finally obtained. :

[0058] Where R represents the reference point.

[0059] This application elevates two-dimensional features to three-dimensional space using this method, thereby significantly enhancing the model's geometric perception capability in scene reconstruction.

[0060] S50 inputs the updated 3D features into the geometric semantic enhancement module for geometric semantic enhancement, and finally completes the 3D semantic scene completion.

[0061] Since the core of Semantic Scene Completion (SSC) lies in predicting complete geometric details and accurate semantic segmentation results, this application achieves this dual goal through a geometric semantic enhancement module, rather than using simple convolution operations or direct interpolation methods for prediction.

[0062] like Figure 4 and Figure 5 As shown, specifically, the geometric semantic enhancement module includes a lightweight residual voxel network and a sparse UNet network.

[0063] To enhance geometric detail, this application designs a lightweight residual voxel network based on residual voxel blocks as the basic unit. Each residual voxel block contains two 3D convolutional layers with a stride of 2, followed by a normalization (BN) layer and a ReLU activation function. To improve computational efficiency and save memory, a checkpointing strategy can be optionally employed in the processing of each layer. As a specific implementation, the network is constructed using three consecutive residual voxel blocks (denoted as m=3).

[0064] Based on the effectiveness of UNet in semantic segmentation, a sparse UNet was customized to achieve accurate semantic perception for scenes containing multi-scale objects and regions. This network includes a downsampling module, an adaptive hierarchical aggregator (AHA), an upsampling module, and an interpolation part. The downsampling module uses sparse convolution to achieve feature dimensionality reduction; AHA adapts to the scale variations of different objects and backgrounds in the 3D scene through multi-scale adaptive relational convolution (AR convolution), adaptive aggregator, and submanifold convolution. In the sparse UNet, small-scale objects such as pedestrians and vehicles require smaller receptive fields to capture fine semantic information, while large areas such as roads and buildings require larger receptive fields to ensure consistency in semantic reasoning; the upsampling module includes linear operations and normalization operations; and the interpolation part adjusts the results to the target size using trilinear interpolation.

[0065] Specifically, the updated 3D features First, a lightweight residual voxel network is input for geometric enhancement to improve the model's geometric structure and detail prediction capabilities. Then, a sparse UNet is input for semantic enhancement, ultimately completing the 3D semantic scene completion.

[0066] This application introduces a residual voxel network after the deformable self-attention module to enhance the model's ability to predict geometric structure and details. It also adopts a more efficient sparse UNet structure to replace traditional 3D convolution or deconvolution, which achieves cross-scale semantic coherent reasoning (such as semantic association between vehicles and roads) through an adaptive hierarchical aggregator (AHA).

[0067] Finally, regarding Figure 2-5 middle, This indicates element-wise addition. It indicates a concatenation. The sigmoid function is represented by the sigmoid function. This represents the Hadamard product operation.

[0068] like Figure 6 As shown, we report quantitative comparison results between our method and existing camera-based semantic scene completion methods on the SemanticKITTI validation set. Our method not only achieves significant performance improvements in both geometric completion and semantic segmentation—verified by improvements in IoU and mIoU metrics—but also excels in key categories related to autonomous driving safety. Specifically, our method outperforms Symphonies in multiple categories: a 10.33 improvement in IoU for the road category, a 5.74 improvement for the car category, and a 4.12 improvement for the motorcycle category. Compared to contemporaneous work CGFormer, our method achieves improvements of 0.32 and 0.33 in overall IoU and mIoU metrics, respectively. Notably, our method demonstrates a significant advantage in the car and motorcycle categories. Furthermore, our method also maintains excellent performance on the SemanticKITTI test set; complete category comparison data can be found here. Figure 7 .

[0069] Figure 8 The qualitative results in the table show a qualitative comparison between our method and existing methods, including Monoscene, VoxFormer, OccFormer, and Symphonies. Observations reveal that our method has the following advantages over other methods: 1) It effectively avoids semantic confusion issues that occur in VoxFormer and Symphonies, such as misclassification between buildings and terrain; 2) It generates clearer boundaries in the car category, avoiding trailing artifacts (as shown in the first row of results); 3) The generated spatial layout closely matches the ground truth (as shown in the third row of road examples); 4) It successfully detects pedestrians in front of the car (as shown in the second row), while other methods fail in this task.

[0070] Quantitative analysis was also performed on the SSCBench-KITTI360 dataset, and the experimental results are as follows: Figure 9As shown in the figure, this method demonstrates superior overall performance compared to eight existing camera-based semantic scene completion methods in the field. Specifically, compared to the Symphonies method, the mIoU and IoU metrics of this application are improved by 1.16 and 4.49, respectively. Notably, in the car and motorcycle categories, the mIoU of this method is improved by 1.38 and 2.03, respectively, further validating its ability to enhance the perception of dynamic objects. Compared to the LOMA method, the mIoU and IoU of this application are also improved by 1.46 and 2.26, respectively. In addition, significant performance gains are also observed in the road and sidewalk categories. This method effectively alleviates the blurring problem generated during the 3D to 2D depth projection process.

[0071] This application also provides a storage medium, characterized in that the storage medium includes stored instructions, wherein, when the instructions are executed, the device where the storage medium is located is controlled to execute the camera-based 3D semantic scene completion method.

[0072] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 10 As shown, it specifically includes a memory 801 and one or more instructions 802, wherein one or more instructions 802 are stored in the memory 801 and are configured to be executed by one or more processors 803 to perform the above-mentioned camera-based 3D semantic scene completion method.

[0073] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0074] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to in the description of the method embodiments. The steps in the methods of each embodiment of this application can be adjusted, combined, or deleted according to actual needs.

[0075] Finally, it should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0076] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0077] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A camera-enhanced 3D semantic scene completion method, characterized in that, include: Depth maps and optical flow maps are obtained from the left and right images. Two-dimensional features are extracted from the left image and the depth map. Map the left-hand image to obtain contextual features; The depth estimation distribution is obtained by using two-dimensional features and a depth map and performing depth optimization. The extended 3D feature map is obtained by expanding the context feature operation. The 3D feature map is then subjected to 3D deformable cross attention and deformable self attention operations in sequence to output the updated 3D features. The updated 3D features are then subjected to geometric semantic enhancement to ultimately complete the 3D semantic scene.

2. The camera-enhanced 3D semantic scene completion method according to claim 1, characterized in that: The method described above involves mapping the left-side image to obtain contextual features, wherein the mapping is based on a context network. The context network input to output sequentially passes through a multilayer perceptron, a compressed excitation attention layer, and a two-dimensional convolutional network.

3. The camera-enhanced 3D semantic scene completion method according to claim 1, characterized in that: The method described above obtains a depth estimation distribution based on two-dimensional features, a depth map, and depth optimization. The depth optimization is performed using an optical flow-guided depth network and includes... Initial optical flow estimation is performed based on two-dimensional features, two-dimensional features, and depth map to obtain initial depth features, optical flow features, and initial sampling features; Based on the initial sampled features, deformation processing is performed to obtain the modulated depth features; Based on the initial depth features and the modulated depth features, a dual-fusion interactive optimization is performed to obtain the refined depth estimation distribution.

4. The camera-enhanced 3D semantic scene completion method according to claim 3, characterized in that: The method of performing initial optical flow estimation based on two-dimensional features, two-dimensional features and depth map to obtain initial depth features, optical flow features and initial sampling features includes: performing SE operation on two-dimensional features and two-dimensional features respectively, and then processing them by a basic module, a holed spatial pyramid pooling and a deformable convolutional network to obtain initial depth features and optical flow features, and performing downsampling and one-hot encoding on the depth map to obtain initial sampling features. The method of obtaining modulated depth features by performing deformation processing based on initial sampling features includes: firstly, deforming the initial sampling features to obtain depth features; secondly, fusing optical flow features with splicing to obtain fused features; and then using an adaptive integration function to output modulated depth features from the fused features. The method described above, which performs dual-fusion interactive optimization based on initial depth features and modulated depth features to obtain a refined depth estimation distribution, includes: firstly, fusing local and global features through a cross-local-global attention module, and secondly, optimizing the fused features through a self-local-global attention module to obtain the depth estimation distribution.

5. The camera-enhanced 3D semantic scene completion method according to claim 3, characterized in that: An expanded 3D feature map is obtained through contextual feature operations. The 3D feature map is then subjected to 3D deformable cross-attention and deformable self-attention operations to output updated 3D features, including: First, an outer product operation is performed between the initial sampled features and the context features to generate an expanded 3D feature map; then, the depth estimation distribution and the context features are multiplied and transformed using coordinates. Post-processing and voxel pooling operations are performed to convert voxel features into voxel features. Voxel queries are pre-defined and initialized, and combined with voxel features to generate new query features. Seed query features are extracted by generating a priori mask index from the depth map. Secondly, the seed query feature is used as the query vector, and the expanded 3D feature map is used as the key vector and value vector. The result is input into the 3D deformable cross attention module to obtain the updated query vector, and the query vector is input into the deformable self attention module to obtain the updated 3D features.

6. The camera-enhanced 3D semantic scene completion method according to claim 5, characterized in that: The process of generating a priori mask index from a depth map to extract seed query features includes: back-projecting the depth map into a point cloud and voxelizing it to obtain a binary query proposal; analyzing the binary query proposal to determine the unmasked and masked index positions, and combining them with query features to extract seed query features.

7. The camera-enhanced 3D semantic scene completion method according to any one of claims 1-6, characterized in that: The updated 3D features are subjected to geometric semantic enhancement using a geometric semantic enhancement module, which includes a lightweight residual voxel network and a sparse UNet network.

8. The camera-enhanced 3D semantic scene completion method according to claim 7, characterized in that: The lightweight residual voxel network includes several residual voxel blocks, each of which contains two three-dimensional convolutional layers with a stride of 2. Each three-dimensional convolutional layer is followed by a normalization layer and a ReLU activation function. The sparse UNet network includes a downsampling module, an adaptive hierarchical aggregator, an upsampling module, and an interpolation part.

9. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device containing the storage medium is controlled to perform the camera-enhanced 3D semantic scene completion method as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, It includes a memory, and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any one of claims 1 to 8.

Images