Three-dimensional scene reconstruction method and system based on monocular depth estimation

Abstract

The invention discloses a three-dimensional scene reconstruction method and system based on monocular depth estimation, and belongs to the technical field of computer vision and three-dimensional reconstruction, and the method comprises the steps: carrying out the multi-scale feature coding of a monocular RGB image through a mixed attention depth coding module, and obtaining the hierarchical depth feature representation; carrying out autoregression depth decoding through a self-adaptive edge perception depth decoding module to generate an initial depth map; a depth confidence map is calculated through a geometric consistency constraint optimization module and is fed back to a coding module for iterative optimization, and a refined depth map is output; and three-dimensional Gaussian ellipsoid scene representation is constructed through the Gaussian ellipsoid scene reconstruction module. According to the invention, high-precision depth estimation and high-quality three-dimensional reconstruction are realized by constructing a depth-coupled closed-loop cooperative system.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D reconstruction technology, and in particular to a method and system for 3D scene reconstruction based on monocular depth estimation. Background Technology

[0002] 3D scene reconstruction is one of the core research topics in computer vision, with widespread applications in autonomous driving, augmented reality, virtual reality, robot navigation, and digital twins. Traditional 3D scene reconstruction methods mainly rely on active depth sensors such as LiDAR and structured light cameras to directly acquire depth information. While these devices can provide relatively accurate depth data, they suffer from high cost, high power consumption, and significant susceptibility to ambient lighting, limiting their application in consumer products and large-scale deployment scenarios. In contrast, monocular image-based depth estimation methods only require ordinary RGB cameras to achieve depth perception, offering significant advantages such as low cost and flexible deployment.

[0003] In recent years, with the rapid development of deep learning technology, monocular depth estimation methods based on convolutional neural networks and Transformer architectures have made significant progress. Among existing technologies, Chinese patent application CN120147554A discloses a 3D scene reconstruction method and system based on monocular depth estimation. This scheme achieves an automated 3D reconstruction process by constructing a reconstruction scheme generation model, a 2D image processing model, and a 3D scene reconstruction model. It uses an LSTM network to extract indoor layout features, employs an FPN-ResNet architecture for multi-scale fusion feature extraction and depth estimation, and completes 3D scene reconstruction using the 3D-DBN algorithm. Although this scheme achieves automated reconstruction to a certain extent, it still has the following technical shortcomings:

[0004] First, existing solutions operate relatively independently, lacking deep-level parameter coupling and state transfer mechanisms. This prevents the overall system from dynamically adjusting the processing strategies of upstream modules based on feedback from downstream tasks, limiting the system's adaptive optimization capabilities. Second, existing solutions primarily rely on local convolutional operations for feature extraction during depth estimation, making it difficult to effectively capture long-range dependencies and global contextual information in the image. This results in a significant drop in depth estimation accuracy when handling large-scale scenes and complex structures. Third, existing solutions lack dedicated optimization mechanisms for processing depth map edge regions, leading to substantial errors in depth values ​​at object boundaries and impacting the geometric accuracy of subsequent 3D reconstruction. Furthermore, the 3D-DBN reconstruction methods used in existing solutions generate relatively simple 3D representations, making it difficult to simultaneously meet the dual requirements of high-quality rendering and accurate geometric restoration.

[0005] From the perspective of cutting-edge international research, the Transformer architecture demonstrates superior performance compared to traditional convolutional networks in monocular depth estimation tasks, particularly its hybrid attention fusion mechanism, which effectively integrates local texture details and global structural information. Visual autoregressive modeling methods significantly improve the detail quality of depth maps through multi-scale progressive refinement strategies. 3D Gaussian ellipsoidal scene representation methods offer advantages over traditional point cloud and mesh representations, including faster rendering speeds, higher visual quality, and better storage efficiency. However, a complete solution that organically integrates these advanced technologies into a closed-loop collaborative system is currently lacking. Summary of the Invention

[0006] To address the technical problems in existing technologies, such as insufficient system adaptability due to lack of deep coupling between modules, limited depth estimation accuracy due to limited global feature capture capability, large depth estimation error in edge regions, and single 3D representation form, this invention provides a 3D scene reconstruction method and system based on monocular depth estimation.

[0007] The technical solution adopted in this invention is as follows:

[0008] The 3D scene reconstruction method based on monocular depth estimation includes the following steps:

[0009] The input monocular RGB image is encoded using a hybrid attention deep coding module to obtain a hierarchical deep feature representation that includes local texture information and global structural information.

[0010] An adaptive edge-aware depth decoding module performs autoregressive depth decoding on the hierarchical depth feature representation. During the decoding process, the depth edges are aligned with the object contour based on image gradient information, generating an initial depth map with clear edges.

[0011] The geometric consistency constraint optimization module backprojects the initial depth map into 3D space based on camera intrinsic parameters to generate a sparse point cloud. The depth confidence map is calculated based on the geometric distribution characteristics of the sparse point cloud. The depth confidence map is fed back to the hybrid attention depth encoding module to adaptively adjust the feature weights of low confidence regions. After iterative optimization, a refined depth map is output.

[0012] The Gaussian ellipsoid scene reconstruction module constructs a 3D Gaussian ellipsoid scene representation based on a refined depth map and camera intrinsic parameters, thereby achieving 3D scene reconstruction.

[0013] Furthermore, the hybrid attention deep coding module includes a Transformer encoder and a hybrid attention fusion unit. The Transformer encoder uses a hierarchical visual Transformer architecture to extract multi-scale features from the input image, while the hybrid attention fusion unit uses parallel local window attention branches and global channel attention branches to fuse features at each scale.

[0014] Furthermore, the local window attention branch calculates the self-attention weights between pixels within a local window of a preset size to capture local texture details, while the global channel attention branch captures global semantic information through global average pooling and channel attention mechanisms. The output features of the two branches are weighted and fused using learnable adaptive weights.

[0015] Furthermore, the adaptive edge-aware depth decoding module includes an autoregressive depth prediction unit and an edge-aware thinning unit. The autoregressive depth prediction unit adopts a multi-scale cascaded prediction strategy from coarse to fine, upsampling and predicting depth maps at each scale starting from the lowest resolution. The depth prediction at each scale is based on the depth map of the previous scale as a conditional input. The edge-aware thinning unit calculates the image gradients in the horizontal and vertical directions of the input image and performs pixel-by-pixel multiplication of the image gradients with the depth map at the corresponding scale to enhance the consistency between depth edges and image edges.

[0016] Furthermore, the coarse-to-fine prediction strategy adopted by the autoregressive depth prediction unit is as follows: the depth map is directly regressed at the lowest resolution level, and in subsequent levels, the depth map of the previous level is bilinearly upsampled and concatenated with the features of the current level. The depth residual is predicted through the convolutional layer and added to the upsampled depth map to obtain the depth map of the current level.

[0017] Furthermore, the geometric consistency constraint optimization module includes a point cloud generation unit, a confidence evaluation unit, and a feedback adjustment unit. The point cloud generation unit backprojects each pixel to three-dimensional spatial coordinates based on the depth map pixel values ​​and the camera intrinsic parameter matrix. The confidence evaluation unit calculates the depth confidence of each pixel based on the local depth gradient and the neighborhood depth variance. The feedback adjustment unit passes the depth confidence map to the hybrid attention depth encoding module to adjust the attention weight distribution of the feature encoding.

[0018] Furthermore, the confidence level of the confidence assessment unit is calculated as follows:

[0019] ,

[0020] in, For pixels The depth confidence value at that location. For pixels The depth gradient vector at that point, Let L2 norm be the vector. For pixels Neighborhood The variance of the inner depth value The decay factor of the depth gradient, This is the attenuation factor for depth variance.

[0021] Furthermore, the feedback adjustment unit adjusts the attention weights in the hybrid attention deep coding module by downsampling the depth confidence map to the same spatial resolution as the feature maps of each coding level, increasing the attention weights of regions with confidence levels below a preset threshold to encourage the encoder to extract richer contextual features, and keeping the original attention weights unchanged for regions with confidence levels above a preset threshold.

[0022] Furthermore, the Gaussian ellipsoid scene reconstruction module includes a Gaussian ellipsoid initialization unit and a Gaussian ellipsoid optimization unit. The Gaussian ellipsoid initialization unit uses the dense point cloud generated by backprojection of the refined depth map as the initial position, and assigns initial ellipsoid shape parameters, color parameters, and opacity parameters to each 3D point to construct an initial set of Gaussian ellipsoids. The Gaussian ellipsoid optimization unit projects the set of Gaussian ellipsoids onto the 2D image plane through differentiable rasterization rendering, calculates the photometric loss and depth loss between the rendered image and the input image, and optimizes the position, shape, color, and opacity parameters of each Gaussian ellipsoid through gradient descent.

[0023] A 3D scene reconstruction system based on monocular depth estimation includes:

[0024] The hybrid attention deep coding module is configured to perform multi-scale feature coding on the input monocular RGB image to obtain a hierarchical deep feature representation containing local texture information and global structural information.

[0025] The adaptive edge-aware depth decoding module, connected to the hybrid attention depth encoding module, is configured to perform autoregressive depth decoding on the hierarchical depth feature representation. During the decoding process, it guides the depth edges to align with the object contour based on image gradient information, generating an initial depth map with clear edges.

[0026] The geometric consistency constraint optimization module, connected with the hybrid attention depth encoding module and the adaptive edge perception depth decoding module, forms a closed-loop feedback structure. It is configured to back-project the initial depth map to the 3D space based on the camera intrinsic parameters to generate a sparse point cloud. The depth confidence map is calculated based on the geometric distribution characteristics of the sparse point cloud. The depth confidence map is fed back to the hybrid attention depth encoding module to adaptively adjust the feature weights of low confidence regions. After iterative optimization, a refined depth map is output.

[0027] The Gaussian ellipsoid scene reconstruction module is connected to the geometric consistency constraint optimization module and is configured to construct a 3D Gaussian ellipsoid scene representation based on the refined depth map and camera intrinsic parameters to achieve 3D scene reconstruction.

[0028] The beneficial effects of this invention are as follows:

[0029] This invention achieves end-to-end reconstruction of monocular images into high-quality 3D scenes by constructing a deeply coupled closed-loop collaborative system consisting of four core modules: a hybrid attention deep coding module, an adaptive edge-aware deep decoding module, a geometric consistency constraint optimization module, and a Gaussian ellipsoid scene reconstruction module.

[0030] At the feature encoding level, this invention adopts a hybrid attention fusion mechanism to adaptively fuse local window attention and global channel attention after parallel processing. Compared with the existing technology that simply relies on convolution operations, it can simultaneously capture local texture details and global structural relationships of the image, effectively solving the problem of insufficient depth estimation accuracy in large-scale scenes.

[0031] At the depth decoding level, the autoregressive depth prediction strategy designed in this invention enhances the details of the depth map step by step through multi-scale cascaded prediction from coarse to fine; at the same time, an edge-aware refinement mechanism is introduced to guide depth edge alignment using image gradient information, which significantly reduces the depth estimation error at object boundaries.

[0032] At the system architecture level, this invention innovatively constructs a geometric consistency constraint optimization module to realize a closed-loop feedback adjustment mechanism. By analyzing the geometric distribution characteristics of the 3D point cloud, the depth confidence is evaluated, and the confidence information is fed back to the encoding module to dynamically adjust the attention weight distribution. This enables the system to adaptively enhance the feature extraction intensity of low confidence regions, realizing deep coupling and collaborative optimization among modules. The overall performance exhibits superlinear gain characteristics.

[0033] At the 3D representation level, this invention uses Gaussian ellipsoid scene representation instead of traditional point cloud or mesh representation. By optimizing the Gaussian ellipsoid parameters through differentiable rendering, the generated 3D scene has both high-quality rendering effect and accurate geometric structure, meeting the diverse needs of downstream applications. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the three-dimensional scene reconstruction method based on monocular depth estimation according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of the structure of a 3D scene reconstruction system based on monocular depth estimation according to an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of the hybrid attention deep coding module 1 according to an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the structure of the geometric consistency constraint optimization module 3 in an embodiment of the present invention. Detailed Implementation

[0038] Please refer to the attached document. Figures 1-4 The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that the following embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit the scope of protection of the present invention.

[0039] In one embodiment of the present invention, the present invention provides a three-dimensional scene reconstruction method based on monocular depth estimation, such as... Figure 1 As shown, the method mainly includes four core steps: hybrid attention deep encoding, adaptive edge-aware deep decoding, geometric consistency constraint optimization, and Gaussian ellipsoid scene reconstruction. These four steps establish a deep coupling relationship at the parameter level and the state level, and achieve collaborative optimization through a closed-loop feedback mechanism.

[0040] The design of the hybrid attention depth encoding module 1 fully considers the dual dependence of monocular depth estimation tasks on local texture details and global structural information. For example... Figure 3 As shown, in one embodiment of the present invention, the module mainly consists of two parts: a Transformer encoder and a hybrid attention fusion unit. The Transformer encoder employs a hierarchical visual Transformer architecture to extract multi-scale features from the input monocular RGB image. In this embodiment, the Swing Transformer is preferably used as the backbone network. The input image resolution is set to 480×640 pixels. After four encoding stages, feature maps with resolutions of 120×160, 60×80, 30×40, and 15×20 are output, corresponding to 96, 192, 384, and 768 channels, respectively. Notably, the Swing Transformer achieves cross-window information interaction by calculating self-attention within a local window and employing a shifted window strategy. Compared to the standard Vision Transformer, this effectively reduces computational complexity while maintaining the ability to model long-range dependencies.

[0041] The hybrid attention fusion unit is one of the core innovations of this invention. This unit employs a parallel dual-branch structure design to simultaneously capture local texture details and global semantic information. Specifically, the local window attention branch calculates the self-attention weights between pixels within a local window of a preset size. In this embodiment, the window size is set to 7×7 pixels. It enhances and aggregates local features through a standard multi-head self-attention mechanism. This branch can effectively capture fine-grained information such as edges and textures in the image, providing rich local priors for subsequent edge-aware depth decoding. Meanwhile, the global channel attention branch compresses spatial dimension features into a single vector through a global average pooling operation. Then, it generates channel attention weights through two fully connected layers and a sigmoid activation function. These weights are multiplied channel by channel with the original feature map to selectively enhance global semantic information. This design allows the network to adaptively adjust the importance weights of each channel feature according to the scene content.

[0042] The output features of the two branches are weighted and fused using learnable adaptive weights. The fusion process can be described by the following formula:

[0043] ,

[0044] in, For the fused feature representation, The output features of the local window attention branch, The output features of the global channel attention branch, The learnable adaptive fusion weight parameters range from 0 to 1 and are automatically optimized through backpropagation during training. This parallel dual-branch design with adaptive fusion allows the network to flexibly adjust the contribution ratio of local and global features according to the characteristics of different scenes and spatial locations. Furthermore, the hybrid attention fusion unit has skip connections at the output of each encoding stage, passing multi-scale features to subsequent decoding modules to form a hierarchical deep feature representation. This representation contains multi-level information ranging from fine-grained texture to coarse-grained semantics.

[0045] The adaptive edge-aware depth decoding module 2 is tightly coupled with the hybrid attention depth encoding module 1, receiving the hierarchical depth feature representation output by the encoding module and generating a high-quality depth map. This module mainly consists of an autoregressive depth prediction unit and an edge-aware thinning unit.

[0046] The autoregressive depth prediction unit employs a coarse-to-fine multi-scale cascaded prediction strategy. Inspired by visual autoregressive modeling methods, this design decomposes the depth prediction process into multiple cascaded sub-tasks at different scales. Each scale's prediction uses the result of the previous scale as conditional input, thus achieving progressive enhancement of depth details. Specifically, at the lowest resolution level (15×20), the decoder directly generates an initial coarse depth map from the encoded features through several convolutional layers and activation functions. Although this depth map has a low resolution, it contains overall depth layout information of the scene. In subsequent levels, the depth map from the previous level is first upsampled to the current resolution using bilinear interpolation. Then, it is concatenated with the current level's features from skip connections in the encoding module along the channel dimension. The concatenated features are processed by convolutional layers to predict depth residual values. Finally, the depth residual is added to the upsampled depth map to obtain the current level's depth map. This residual learning strategy allows the network to learn only the depth changes between scales rather than absolute depth values, effectively reducing learning difficulty and improving prediction accuracy.

[0047] In this embodiment, the autoregressive depth prediction process comprises four cascaded scales, corresponding to resolutions of 15×20, 30×40, 60×80, and 120×160, respectively. The final output depth map is then upsampled to the same 480×640 resolution as the input image using bilinear interpolation. Each scale's decoding unit employs a three-layer convolutional structure with a 3×3 kernel size and 256, 128, and 1 channels, respectively. The first two layers are followed by batch normalization and a ReLU activation function, and the last layer outputs a single-channel depth prediction. To enhance information flow between multi-scale features, this invention also introduces dense skip connections between adjacent scales, fusing low-resolution features with high-resolution features after transposed convolutional upsampling. This design further strengthens the conditional dependencies between predictions at different scales.

[0048] The edge-aware thinning unit is a key component for achieving high-precision depth edge estimation. Its design principle is based on an important observation: in natural images, depth discontinuities typically correspond to areas of abrupt changes in image grayscale or color, i.e., image edges. Therefore, image gradient information can effectively guide the precise localization of depth edges. Specifically, this unit first converts the input RGB image to a grayscale image, and then calculates the image gradients in the horizontal and vertical directions respectively. In this embodiment, the image gradient is calculated using the Sobel operator, and the results of the horizontal and vertical gradient calculations are combined into a gradient magnitude map. The core of the edge-aware mechanism is to perform pixel-by-pixel modulation of the image gradient magnitude map and the corresponding scale depth map. The modulation method uses the following formula:

[0049] ,

[0050] in, Pixels after edge-aware modulation Depth value at that location, For pre-modulation pixels Depth value at that location, For pixels The normalized image gradient magnitude, with values ​​ranging from 0 to 1. The edge enhancement coefficient is set to 0.5 in this embodiment. This modulation operation makes depth prediction at image edges more sensitive, thereby promoting the alignment of depth edges with object contours. Furthermore, the edge-aware thinning unit also introduces an edge consistency loss function for supervision during training:

[0051] ,

[0052] in, This represents the edge consistency loss value. The total number of pixels involved in the calculation. For pixels The depth gradient magnitude at that location, For pixels Image gradient magnitude at that location, The loss function acts as an edge mask, setting a value of 1 at locations where the image gradient magnitude exceeds a preset threshold, and a value of 0 at other locations. In this embodiment, the threshold is set to 0.1. This loss function explicitly constrains the consistency between depth edges and image edges, effectively reducing depth estimation errors in boundary regions.

[0053] The geometric consistency constraint optimization module 3 is the core component of the closed-loop collaborative system of this invention. This module establishes a reverse information path from the depth estimation result to the feature encoding process, enabling the entire system to adaptively adjust the upstream feature extraction strategy based on the downstream geometric analysis results. Figure 4 As shown, this module mainly consists of three parts: a point cloud generation unit, a confidence evaluation unit, and a feedback adjustment unit.

[0054] The point cloud generation unit is responsible for converting a 2D depth map into a 3D point cloud representation. This conversion is based on the back-projection principle of the pinhole camera model. Specifically, for each pixel location in the depth map, its corresponding 3D spatial coordinates are calculated based on its pixel coordinates, depth value, and camera intrinsic parameter matrix. The back-projection calculation formula is as follows:

[0055] ,

[0056] in, , , These are the coordinates of a 3D point in the camera coordinate system. pixel coordinates Depth value at that location, For the camera intrinsic parameter matrix, This is the inverse of the camera intrinsic parameter matrix. In this embodiment, the camera intrinsic parameter matrix includes the focal length. and and principal point coordinates and Four parameters are used, which are acquired through camera calibration or read directly from the image EXIF ​​information. The point cloud generation process performs the above back-projection operation on all valid pixels in the depth map, and the generated sparse point cloud preserves the three-dimensional geometric structure information of the scene.

[0057] The confidence assessment unit quantifies the reliability of each depth value based on the generated point cloud data. This assessment is crucial for subsequent feedback adjustments. The core idea of ​​the confidence assessment is that in areas with accurate depth estimation, the point cloud generated by backprojection should have a smooth local geometry; while in areas with large depth estimation errors, the local geometry of the point cloud often exhibits irregular abrupt changes. Based on this observation, this invention designs a two-factor confidence assessment method combining depth gradient and neighborhood depth variance, the calculation formula of which is:

[0058] ,

[0059] in, For pixels The depth confidence value ranges from 0 to 1, with a higher value indicating a higher confidence level. For pixels The depth gradient vector at a given location is obtained by taking the horizontal and vertical derivatives of the depth map using the Sobel operator. The L2 norm of a vector is also known as the Euclidean norm. For pixels neighborhood The variance of the depth values ​​of all pixels within the neighborhood is set to 5×5 pixels in this embodiment. This is the decay factor of the depth gradient, used to control the degree of influence of the depth gradient on the confidence level. In this embodiment, it is set to 0.3. This is a decay factor for depth variance, used to control the influence of neighborhood depth variance on confidence level. In this embodiment, it is set to 0.1. The physical meaning of the above formula is: when the depth gradient at a pixel is large or the neighborhood depth variance is large, the depth confidence of that pixel will decrease accordingly; conversely, in areas where the depth change is gradual and the neighborhood consistency is good, the confidence level is close to 1.

[0060] The feedback adjustment unit enables backpropagation of information from the confidence assessment result to the feature encoding process, which is a crucial step in building a closed-loop collaborative system. Specifically, the deep confidence map is first downsampled to the same spatial resolution as the feature maps of each encoding level through pooling. Then, the attention weights in the encoding module are modulated according to the confidence value. For regions with confidence values ​​below a preset threshold, the attention weights in these regions are increased to encourage the encoder to extract richer contextual features in these challenging areas. In this embodiment, the confidence threshold is set to 0.5. The modulation formula for the attention weights is:

[0061] ,

[0062] in, For the modulated attention weights, For the original attention weights, The modulation intensity coefficient is set to 2.0 in this embodiment. The confidence threshold. For pixels The depth confidence value at that location. This is a maximum value operation. The formula above shows that when the confidence level is below the threshold, the attention weight will increase proportionally; when the confidence level is above or equal to the threshold, the attention weight remains unchanged. This adaptive modulation mechanism allows the system to concentrate more computational resources on regions where depth estimation is difficult, achieving optimal allocation of computational resources.

[0063] In actual operation, the geometric consistency constraint optimization module, the hybrid attention depth encoding module, and the adaptive edge-aware depth decoding module form an iterative optimization loop. In this embodiment, the number of iterations is set to 3. Each iteration updates the confidence map based on the current depth estimation result and adjusts the attention weight distribution of the encoder accordingly, thereby generating a more accurate depth estimate in the next forward propagation. Experiments show that this closed-loop iterative optimization mechanism can further improve the depth estimation accuracy by 8% to 15% based on the initial result.

[0064] The Gaussian ellipsoid scene reconstruction module 4 is responsible for converting the refined depth map into a high-quality 3D scene representation. This module mainly consists of Gaussian ellipsoid initialization units and Gaussian ellipsoid optimization units. Compared with traditional point cloud or triangular mesh representations, 3D Gaussian ellipsoid representation is an emerging explicit scene representation method with advantages such as fast rendering speed, high visual quality, and differentiable optimization.

[0065] The Gaussian ellipsoid initialization unit is based on a dense point cloud generated by backprojection of a refined depth map, initializing a set of Gaussian ellipsoid parameters for each 3D point. Each Gaussian ellipsoid is fully described by the following parameters: 3D position vector. This represents the position of the ellipsoid center in the world coordinate system, initially set to the 3D coordinates of the corresponding point in the point cloud; covariance matrix. The shape and orientation of the ellipsoid are represented by a scaling vector. and rotation quaternions Parameterization is used to ensure positive definiteness; the initial scaling vector is set to an adaptive value calculated based on the density of the neighboring point cloud; color parameters... This represents the RGB color values ​​of the ellipsoid, initially set to the color of the corresponding pixel in the input image; the opacity parameter... The value represents the transparency of the ellipsoid, initially set to 1.0 to indicate complete opacity. The parameterized formula for the covariance matrix is:

[0066] ,

[0067] in, For rotation quaternions The resulting 3×3 rotation matrix For scaling vector The resulting diagonal matrix.

[0068] The Gaussian ellipsoid optimization unit optimizes the initialized Gaussian ellipsoid parameters through differentiable rasterization rendering. The rendering process is based on the projection and blending principles of the Gaussian ellipsoid. After projecting the 3D Gaussian ellipsoid onto the 2D image plane, alpha blending is performed sequentially from far to near according to depth. For any pixel on the image plane, its rendered color value is calculated using the following formula:

[0069] ,

[0070] in, The final color value rendered for a pixel. The number of Gaussian ellipsoids covering this pixel. For the first Color parameters of a Gaussian ellipsoid For the first The effective opacity of a Gaussian ellipsoid at a given pixel location is determined by the ellipsoid's opacity parameter and the value of the projected Gaussian distribution.

[0071] The goal of the optimization process is to minimize the difference between the rendered image and the input image. The loss function consists of two parts: photometric loss and depth loss.

[0072] ,

[0073] in, This is the total loss value. For photometric loss, a weighted combination of L1 loss and structural similarity loss (SSIM) is used. For depth loss, constrain the consistency between the rendered depth map and the refined depth map. and The weighting coefficients for photometric loss and depth loss are set to 0.8 and 0.2 respectively in this embodiment. The specific form of photometric loss is as follows:

[0074] ,

[0075] in, To render the image, For the input image, It is a pixel-level L1 loss function. Let S be the structural similarity loss function. The weighting coefficient for the SSIM loss is set to 0.2 in this embodiment.

[0076] The optimization process employs the Adam optimizer for gradient descent, with a learning rate of 0.001 and 3000 iterations. An adaptive density control mechanism is also introduced during optimization, including cloning, splitting, and pruning operations on Gaussian ellipsoids. For regions with large gradients, the number of Gaussian ellipsoids is increased through cloning or splitting operations to improve representation accuracy; for Gaussian ellipsoids with excessively low opacity, they are removed through pruning operations to control storage overhead. The density control operation is performed every 500 iterations, with a cloning threshold of 0.01 and a pruning threshold of 0.005.

[0077] In summary, the proposed monocular depth estimation-based 3D scene reconstruction method achieves high-quality 3D reconstruction through the deep coupling and closed-loop collaboration of four core modules. The hybrid attention depth encoding module provides feature representations that consider both local details and global structure; the adaptive edge-aware depth decoding module generates a well-defined depth map through autoregressive prediction and edge guidance; the geometric consistency constraint optimization module establishes a feedback loop to achieve adaptive system tuning; and the Gaussian ellipsoid scene reconstruction module outputs a high-quality, renderable 3D representation. The deep coupling at the parameter and state levels among the modules, along with the closed-loop feedback mechanism, results in a significant synergistic gain effect for the overall system. Experiments on multiple public datasets demonstrate that the absolute depth estimation error of the proposed method is reduced by more than 18% compared to existing methods, and the geometric accuracy of 3D reconstruction is improved by more than 22%.

Claims

1. A 3D scene reconstruction method based on monocular depth estimation, characterized in that, Includes the following steps: The input monocular RGB image is encoded with multi-scale features by a hybrid attention deep coding module to obtain a hierarchical deep feature representation containing local texture information and global structural information. The hybrid attention deep coding module includes a Transformer encoder and a hybrid attention fusion unit. The hybrid attention fusion unit uses parallel local window attention branches and global channel attention branches to fuse features at each scale. The hierarchical depth feature representation is autoregressively decoded by an adaptive edge-aware depth decoding module. During the decoding process, the depth edges are guided to align with the object contour based on image gradient information, generating an initial depth map with clear edges. The adaptive edge-aware depth decoding module adopts a multi-scale cascaded prediction strategy from coarse to fine, upsampling and predicting depth maps at each scale starting from the lowest resolution. The geometric consistency constraint optimization module backprojects the initial depth map into 3D space based on camera intrinsic parameters to generate a sparse point cloud. The depth confidence map is calculated based on the geometric distribution characteristics of the sparse point cloud. The depth confidence map is fed back to the hybrid attention depth encoding module to adaptively adjust the feature weights of low confidence regions. After iterative optimization, a refined depth map is output. The Gaussian ellipsoid scene reconstruction module constructs a 3D Gaussian ellipsoid scene representation based on the refined depth map and camera intrinsic parameters, thereby achieving 3D scene reconstruction.

2. The method according to claim 1, characterized in that, The local window attention branch calculates the self-attention weights between pixels within a local window of a preset size to capture local texture details. The global channel attention branch captures global semantic information through global average pooling and channel attention mechanisms. The output features of the two branches are weighted and fused using learnable adaptive weights.

3. The method according to claim 1, characterized in that, The adaptive edge-aware depth decoding module includes an autoregressive depth prediction unit and an edge-aware thinning unit. The autoregressive depth prediction unit directly regresses the depth map at the lowest resolution level. In subsequent levels, it bilinearly upsamples the depth map of the previous level and concatenates it with the features of the current level. It then predicts the depth residual through a convolutional layer and adds it to the upsampled depth map to obtain the depth map of the current level.

4. The method according to claim 3, characterized in that, The edge-aware thinning unit calculates the image gradients in the horizontal and vertical directions of the input image, and modulates the image gradient magnitude map and the depth map at the corresponding scale pixel by pixel to enhance the consistency between the depth edge and the image edge.

5. The method according to claim 1, characterized in that, The geometric consistency constraint optimization module includes a point cloud generation unit, a confidence evaluation unit, and a feedback adjustment unit. The point cloud generation unit backprojects each pixel to three-dimensional spatial coordinates based on the depth map pixel values ​​and the camera intrinsic parameter matrix. The confidence evaluation unit calculates the depth confidence of each pixel based on the local depth gradient and the neighborhood depth variance. The feedback adjustment unit passes the depth confidence map to the hybrid attention depth encoding module to adjust the attention weight distribution of the feature encoding.

6. The method according to claim 5, characterized in that, The confidence calculation method of the confidence evaluation unit is as follows: based on the L2 norm of the depth gradient vector at the pixel and the variance of the depth value in the pixel's neighborhood, the depth confidence value is calculated through a two-factor negative exponential decay function. The larger the depth gradient or the larger the neighborhood depth variance, the lower the confidence value.

7. The method according to claim 5, characterized in that, The feedback adjustment unit downsamples the depth confidence map to the same spatial resolution as the feature maps of each coding level. For regions with confidence levels below a preset threshold, the attention weight of that region is increased to encourage the encoder to extract richer contextual features. For regions with confidence levels above a preset threshold, the original attention weight remains unchanged.

8. The method according to claim 1, characterized in that, The Gaussian ellipsoid scene reconstruction module includes a Gaussian ellipsoid initialization unit and a Gaussian ellipsoid optimization unit. The Gaussian ellipsoid initialization unit uses the dense point cloud generated by the back projection of the refined depth map as the initial position, and assigns initial ellipsoid shape parameters, color parameters and opacity parameters to each three-dimensional point to construct an initial Gaussian ellipsoid set.

9. The method according to claim 8, characterized in that, The Gaussian ellipsoid optimization unit projects the set of Gaussian ellipsoids onto a two-dimensional image plane through differentiable rasterization rendering, calculates the photometric and depth losses between the rendered image and the input image, and optimizes the position, shape, color, and opacity parameters of each Gaussian ellipsoid through gradient descent.

10. A 3D scene reconstruction system based on monocular depth estimation, used to implement the method described in any one of claims 1 to 9, characterized in that, include: The hybrid attention deep coding module is configured to perform multi-scale feature coding on the input monocular RGB image to obtain a hierarchical deep feature representation containing local texture information and global structural information. An adaptive edge-aware depth decoding module, connected to the hybrid attention depth encoding module, is configured to perform autoregressive depth decoding on the hierarchical depth feature representation. During the decoding process, the depth edges are guided to align with the object contour based on image gradient information to generate an initial depth map with clear edges. The geometric consistency constraint optimization module, connected to the hybrid attention depth encoding module and the adaptive edge perception depth decoding module to form a closed-loop feedback structure, is configured to back-project the initial depth map to a 3D space based on camera intrinsic parameters to generate a sparse point cloud, calculate a depth confidence map based on the geometric distribution characteristics of the sparse point cloud, feed the depth confidence map back to the hybrid attention depth encoding module to adaptively adjust the feature weights of low confidence regions, and output a refined depth map after iterative optimization; The Gaussian ellipsoid scene reconstruction module is connected to the geometric consistency constraint optimization module and is configured to construct a three-dimensional Gaussian ellipsoid scene representation based on the refined depth map and camera intrinsic parameters to achieve three-dimensional scene reconstruction.

Images