Depth-guided three-dimensional Gaussian reconstruction method and system suitable for sparse view angle image

Abstract

The invention discloses a depth-guided three-dimensional Gaussian reconstruction method and system suitable for a sparse view angle image, and belongs to the technical field of computer vision and three-dimensional reconstruction, and the method comprises the steps: carrying out the wavelet transformation super-resolution processing of a sparse multi-view angle image; outputting depth prior through a pre-trained monocular depth model, and extracting multi-view image features; constructing cross-view depth candidates by adopting a planar scanning stereo method, and generating initial depth distribution through feature similarity calculation; a self-attention-cross attention structure and deformable sampling are adopted to realize coarse-to-fine depth matching optimization; using an improved UNet network to fuse multi-scale features, and optimizing a depth estimation result; predicting three-dimensional Gaussian primitive parameters; and constructing a Gaussian field to generate a new visual angle image. According to the method, the problems of low accuracy, integrity and efficiency of existing sparse view angle three-dimensional reconstruction are solved. According to the invention, the precision and the detail fidelity of the depth map are improved, and high-fidelity three-dimensional reconstruction under the sparse visual angle condition is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, specifically to a depth-guided 3D Gaussian reconstruction method and system suitable for sparse viewpoint images. Background Technology

[0002] In recent years, with the rapid development of technologies such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), 3D reconstruction based on sparse viewpoint images has shown broad application prospects in various fields such as virtual reality (VR), augmented reality (AR), cultural heritage protection, and autonomous driving. Compared with traditional high-cost reconstruction methods such as structured light and laser scanning, sparse multi-viewpoint image input combined with neural rendering technology can achieve a more flexible and efficient 3D reconstruction process, greatly reducing data acquisition and modeling costs.

[0003] NeRF achieves high-quality novel perspective synthesis by learning implicit radiative field representations, but it heavily relies on intensive supervised training and optimization for each scene, resulting in low reconstruction efficiency and high deployment barriers. To alleviate this problem, 3D Gaussian splashing technology proposes rendering explicit Gaussian primitives in a differentiable rasterization manner, significantly improving rendering efficiency while maintaining visual quality, and has become a rapidly developing 3D representation framework in recent years. Especially in sparse perspectives, 3DGS has advantages over NeRF in training speed and inference efficiency, and has become an important direction for general 3D reconstruction methods.

[0004] Despite this, current generalizable 3D Gaussian Splatting (G-3DGS) methods still face numerous challenges when dealing with real-world sparse scenes. On one hand, existing methods heavily rely on accurate cross-view feature matching. However, in the presence of occlusion, missing textures, or structural repetition, traditional multi-view matching methods based on pixel alignment or cost volume often struggle to provide robust consistency constraints, leading to inaccurate reconstruction results or missing local structures. On the other hand, most methods still employ a scene-by-scene optimization process, requiring multiple rounds of iterative training for each new scene, which limits their practical deployment capabilities in large-scale applications.

[0005] Recent forward-inference-based G-3DGS methods attempt to construct direct reconstruction schemes without scene optimization by incorporating the prior power of large-scale training data. These methods can complete 3D structure reconstruction in a single forward propagation, exhibiting good cross-scene generalization ability and efficient inference characteristics. However, due to the sparse number of input images and limited camera view coverage, the geometric overlap between images is small, further weakening the stability of feature alignment-based depth estimation. In areas where features are not significant or where there is significant scene occlusion, reconstruction still struggles to guarantee consistency and completeness.

[0006] Furthermore, existing cost volume-based learning-based multi-view stereo (MVS) methods, while capable of fusing image information from multiple viewpoints to predict dense depth maps, often rely on fully sampled input images to construct a complete cost volume, resulting in a significant performance drop in sparse viewpoint settings. To enhance the robustness of depth estimation, some methods attempt to introduce monocular depth priors, but these methods struggle with multi-view fusion and cross-view... Figure 1 There are still shortcomings in consistency modeling.

[0007] How to achieve high-quality, general-purpose 3D reconstruction without scene-by-scene optimization under sparse viewpoint settings, through effective feature modeling mechanisms and structure-aware strategies, remains a crucial research problem in the field of computer vision. There is an urgent need for a novel framework that can integrate frequency domain information, multi-scale context, monocular depth priors, and cross-view semantic guidance to improve the accuracy, completeness, and efficiency of sparse viewpoint 3D reconstruction. Summary of the Invention

[0008] The purpose of this invention is to provide a depth-guided 3D Gaussian reconstruction method and system suitable for sparse viewpoint images. While improving the quality and detail fidelity of 3D reconstruction from a sparse viewpoint, it also has strong versatility and robustness, thus solving the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A depth-guided 3D Gaussian reconstruction method suitable for sparse viewpoint images includes:

[0011] Wavelet transform super-resolution processing is performed on sparse multi-view images to enhance high-frequency details;

[0012] Specifically, sparse multi-view images are processed by convolutional neural networks to extract preliminary features. The input image is decomposed into low-frequency sub-bands and high-frequency sub-bands in three directions by two-dimensional discrete wavelet transform. Lightweight CNN sub-modules are inserted before and after each layer of two-dimensional discrete wavelet transform to capture contextual information across sub-bands. High-resolution image features are recovered by inverse wavelet transform. Feature maps of shrinking and expanding paths are fused using element-wise addition to achieve multi-scale information integration.

[0013] By performing wavelet transform super-resolution processing on the input sparse multi-view RGB image, and combining two-dimensional discrete wavelet transform with a lightweight convolutional neural network, cross-subband context information modeling and high-frequency detail enhancement are achieved. The image is decomposed and reconstructed in multiple layers in the frequency domain, and high-resolution feature maps are generated while maintaining the integrity of multi-frequency band features using inverse wavelet transform, thereby effectively improving the detail representation capability of the input data.

[0014] The depth prior is output by a pre-trained monocular depth model, and multi-view image features are extracted based on multi-view images and camera parameters.

[0015] Among them, camera parameters are injected through the SE module, image features are further extracted using CNN-Transformer, and depth features and relative depth maps are obtained by using a pre-trained monocular depth network, providing a reliable geometric reference for subsequent depth optimization.

[0016] Coarse matching generates an initial depth map, a planar scanning stereo method is used to construct cross-view depth candidates, and an initial depth distribution is generated by calculating feature similarity.

[0017] Among them, depth candidate values ​​are constructed based on the planar scanning stereo method, and projection sampling and feature similarity calculation are performed in the feature space to obtain a coarse depth map;

[0018] Coarse-to-fine matching is used to optimize the depth map, and a self-attention-cross-attention structure and deformable sampling are employed to achieve coarse-to-fine depth matching optimization.

[0019] Specifically, the deep spatial geometric features are constructed, coarse depth, depth features and camera parameters are fused, deformable sampling features are obtained by applying a deformable sampling module, coarse depth map is processed by a self-attention module, correlation between local regions is modeled, depth context information is aggregated, depth residual is calculated by a cross-attention module, key region features are enhanced, and a fine depth map is output.

[0020] To further enhance the understanding of scene geometry, an improved UNet deep optimization network was designed. This network extracts semantic and geometric features through a multi-scale encoder, constructs multi-level depth maps through staged pooling and feature concatenation, fuses geometric features at different scales, and combines skip connections to decode and fuse multi-scale features to form a detailed scene representation. At the tail of UNet, a feature enhancement module with large and small convolutional kernels is integrated. The large convolutional kernel captures global structural relationships, while the small convolutional kernel strengthens local feature aggregation. A variable weight aggregation mechanism is used to achieve collaborative modeling of information at different scales, thereby enhancing the global consistency and detail fidelity of depth prediction.

[0021] We utilize an improved UNet network to fuse multi-scale features and optimize depth estimation results;

[0022] Specifically, the image semantic and geometric features are extracted by a multi-scale encoder, and the multi-scale features of the encoder are fused with the depth map to fuse depth information at different scales. Decoding is performed through skip connections to form a detailed geometric representation of the scene. A feature enhancement module with large and small convolutional kernels is added at the end of UNet. The large convolutional kernel captures global structural relationships, while the small convolutional kernel focuses on highly related regions to achieve efficient feature integration and further refine the depth map.

[0023] Predict 3D Gaussian parameters based on the optimized depth map, including center position, opacity, color, and covariance matrix;

[0024] Among them, the optimized depth map is combined with camera parameters for back projection to obtain the Gaussian center position, the opacity is regarded as the depth confidence, the multilayer perceptron is used for prediction, the spherical harmonic coefficient is used to predict the color, and the covariance matrix is ​​represented by the scaling matrix and the rotation matrix.

[0025] A Gaussian field is constructed based on three-dimensional Gaussian primitive parameters, and a differentiable renderer is used to render images from a new perspective, achieving high-quality three-dimensional reconstruction under sparse perspective and generating images from a new perspective.

[0026] Specifically, a three-dimensional Gaussian field is constructed based on the predicted three-dimensional Gaussian primitive parameters, and a differentiable renderer is used to generate images from new perspectives.

[0027] Preferably, wavelet transform super-resolution processing is performed on sparse multi-view images, and the following operations are performed:

[0028] The system receives sparse multi-view images as input and performs preliminary feature extraction through a convolutional neural network. Each convolutional layer in the convolutional neural network uses a 3×3 convolutional kernel with a stride of 1 and is equipped with a ReLU activation function to enhance the non-linear expressive power of the model.

[0029] A two-dimensional discrete wavelet transform is applied to the initial feature map to decompose it into one low-frequency sub-band and three high-frequency sub-bands. Before and after each wavelet transform layer, a lightweight CNN sub-module is embedded. The lightweight CNN sub-module consists of three convolutional layers. The first and second layers use 3×3 convolutional kernels with a stride of 1 and ReLU activation function to extract local context information. The third layer uses 3×3 convolutional kernels with a stride of 1 and no activation function to linearly fuse the feature information output by the first two layers.

[0030] The generated sub-band images are recursively decomposed, including two-dimensional discrete wavelet transform decomposition. A lightweight CNN sub-module is introduced to process the obtained sub-bands, and the output of this module is used as the input of the next level of recursion. The decomposition and lightweight CNN sub-module processing flow are repeatedly executed. The context information of the sub-bands is modeled through the synergistic effect of wavelet transform and lightweight CNN sub-module.

[0031] High-resolution feature maps are reconstructed layer by layer using inverse wavelet transform. Multi-scale information integration is achieved by fusing the corresponding feature maps in the shrinking and expanding paths through element-wise addition, thereby enhancing the image's ability to express details.

[0032] Preferably, the depth prior is output through a pre-trained monocular depth model, and multi-view image features are extracted based on multi-view images and camera parameters. The following operations are then performed:

[0033] An SE module is introduced into the feature block of the convolutional neural network. The camera intrinsic and extrinsic parameters are processed by a multilayer perceptron (MLP), and the processed parameters are injected into the SE module so that the image features contain global spatial location information.

[0034] Multi-view image features are extracted using a CNN-Transformer hybrid network architecture. ,in , Indicates the channel dimension;

[0035] Convolutional neural networks extract preliminary, dense, low- to mid-level visual features. The features extracted by the convolutional neural network are input into the Transformer network, and the features are modeled and enhanced globally through self-attention and cross-attention mechanisms.

[0036] Obtain monocular depth prior information using a pre-trained monocular depth estimation model; extract depth feature maps based on the monocular depth prior information. ,in And simultaneously generate relative depth output. ,in .

[0037] Preferably, coarse matching generates an initial depth map, a planar scanning stereo method is used to construct cross-view depth candidates, and an initial depth distribution is generated through feature similarity calculation, including:

[0038] Based on the planar scanning stereo method, a cross-view depth candidate set is constructed in the feature space. For each pixel in the scene, multiple candidate depth values ​​are generated according to camera parameters and depth dimension.

[0039] For each generated candidate depth value, a feature similarity score is calculated by feature projection sampling and dot product similarity measurement, and an initial coarse depth map is generated based on the feature similarity score.

[0040] When there are more than two input viewpoints, a pixel-level averaging fusion operation is performed on the initial coarse depth maps calculated from all viewpoints to generate the final coarse depth map.

[0041] Preferably, coarse-to-fine matching is used for depth map optimization. A self-attention-cross-attention structure and deformable sampling are employed to achieve coarse-to-fine depth matching optimization, and the following operations are performed:

[0042] By fusing coarse depth maps, depth prior features, and camera parameters, the fused information is input into a depth spatial geometry network consisting of a multilayer perceptron (MLP) and a SE layer, and the output is depth spatial geometry features.

[0043] Based on the generated deep spatial geometric features, deformable sampling techniques are used to aggregate spatial feature information of local neighborhoods to enhance the robustness of matching.

[0044] The depth spatial geometric features are input into the self-attention layer to perform contextual aggregation of local depth information. The output features of the self-attention layer are input into the cross-attention layer to calculate the depth residual. The coarse depth is added to the calculated depth residual to obtain the optimized fine depth map.

[0045] Preferably, by utilizing the improved UNet network to fuse multi-scale features and optimize the depth estimation results, the following operations are performed:

[0046] The multi-scale encoder constructs a multi-level depth map through staged pooling and feature concatenation, fusing geometric features at different scales. Specifically, it integrates geometric features of different sizes. The input image, through a The convolutional layer performs preliminary processing and downsampling, using two layers with a stride of 1. Convolutional layers extract local features and generate a size of depth feature map The generated deep feature map The input image, after undergoing average pooling, is concatenated channel by channel. The concatenated features are then applied with a stride of 2. The convolutional layer performs downsampling to generate a size of depth feature map ,Will Channel concatenation is performed with the average pooled input image at the corresponding scale, using a stride of 2. The convolutional layer performs downsampling to generate a size of depth feature map The generated deep feature map , and The feature maps at the corresponding scales of the UNet encoder are fused to construct feature representations that incorporate multi-level depth information.

[0047] During the decoding stage, the UNet decoder uses skip connections to progressively upsample the fused feature map at the corresponding scale in the encoding stage. By fusing feature information from multiple scales, the decoder reconstructs a feature map with progressively improved spatial resolution, generating a detailed and richly layered geometric representation of the scene, comprehensively preserving both local details and global structural information.

[0048] A feature enhancement module is integrated at the end of the UNet encoder to fuse features extracted from different receptive fields, enhancing the aggregation capability of multi-scale contextual information. This is achieved given the input feature map. , apply one Convolutional layers reduce the channel dimension to For the dimensionality-reduced feature map, apply a kernel size of Depthwise separable convolution operations are used to extract spatial context information covering a large neighborhood. Two methods are then applied sequentially to the extracted large receptive field features. Convolutional layers generate a spatially and channel-adaptive three-dimensional weight tensor. ,in As the weight dimension, the original input feature map Divide the channel evenly into 8 groups, each group containing Each channel represents a spatial location in the feature map. The generated location Corresponding weight vector Reconstructed as a tensor Where 8 corresponds to the number of channel groups. Represents the small convolution kernel size, using the reconstructed weight tensor. As a grouped convolution kernel, at position local neighborhood Above, the 8 groups of features were divided and executed respectively. Convolution operations enable feature aggregation, and channels within a group share the same convolution kernel weights.

[0049] Preferably, the prediction of 3D Gaussian meta-parameters based on the optimized depth map includes:

[0050] Using the optimized depth map and combined with camera intrinsic and extrinsic parameters, each pixel is mapped to the world coordinate system through back projection calculation to determine its corresponding three-dimensional spatial coordinates, which serve as the center position of the Gaussian unit.

[0051] The opacity parameter is regarded as a confidence index for depth estimation. The optimized depth map is encoded and mapped by a multilayer perceptron (MLP) to predict the opacity value of each Gaussian point.

[0052] Color information is modeled using spherical harmonic function coefficients to represent viewpoint-dependent appearance changes and color distribution under complex lighting conditions;

[0053] A covariance matrix is ​​constructed using a combination of rotation and scaling matrices to accurately describe the geometry and distribution characteristics of each Gaussian element in three-dimensional space.

[0054] According to another aspect of the present invention, a depth-guided 3D Gaussian reconstruction system for sparse viewpoint images is provided, for implementing the depth-guided 3D Gaussian reconstruction method for sparse viewpoint images as described above, comprising:

[0055] The wavelet transform super-resolution module is used to perform high-frequency enhancement on the input sparse multi-view images;

[0056] The feature extraction module is used to extract multi-view image features and monocular depth priors;

[0057] The coarse matching module is used to generate coarse depth maps based on the planar scanning stereo method.

[0058] The coarse-to-fine matching module is used to generate fine depth maps based on attention mechanisms and deformable sampling;

[0059] The depth optimization module is used to optimize depth estimation results based on the improved UNet network.

[0060] The Gaussian parameter prediction module is used to predict the three-dimensional Gaussian meta-parameters corresponding to each pixel;

[0061] The rendering module is used to render new perspective images based on predicted 3D Gaussian meta-parameters, achieving high-quality 3D reconstruction under sparse perspectives and generating new perspective images.

[0062] Compared with the prior art, the beneficial effects of the present invention are:

[0063] This invention significantly improves the quality of high-frequency details by introducing wavelet transform super-resolution processing in the sparse multi-view RGB image input stage, providing richer texture information for subsequent feature extraction. By combining a CNN-Transformer network that fuses camera parameters with depth prior guidance, depth estimation becomes more robust in challenging scenarios such as low-texture and repetitive textures. By employing a depth optimization strategy with deformable sampling and attention mechanisms, the accuracy and detail fidelity of the depth map are significantly improved. Through 3D Gaussian primitive modeling and differentiable rendering, high-fidelity 3D reconstruction under sparse view conditions is achieved. Moreover, the entire process is end-to-end trained, relying only on RGB images as supervision signals, avoiding dependence on expensive depth sensors or additional 3D geometric data. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the depth-guided 3D Gaussian reconstruction method of the present invention;

[0065] Figure 2 This is a schematic diagram of the wavelet transform super-resolution module of the present invention;

[0066] Figure 3 This is a schematic diagram of the improved UNet network structure of the present invention;

[0067] Figure 4 This is a schematic diagram of the feature enhancement module for fusion of large and small convolutional kernels according to the present invention;

[0068] Figure 5 This is a schematic diagram of the depth-guided 3D Gaussian reconstruction method of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] To address the issue that existing 3D Gaussian reconstruction methods cannot integrate frequency domain information, multi-scale context, monocular depth priors, and cross-view semantic guidance, thus reducing the accuracy, completeness, and efficiency of sparse-view 3D reconstruction, please refer to [link to relevant documentation]. Figures 1-5 This embodiment provides the following technical solution:

[0071] Example 1

[0072] A depth-guided 3D Gaussian reconstruction method suitable for sparse viewpoint images includes:

[0073] Wavelet transform super-resolution processing is performed on sparse multi-view images to enhance high-frequency details;

[0074] Specifically, wavelet transform super-resolution is based on two-dimensional wavelet packet transform and lightweight CNN, specifically including: convolution operation, used to capture cross-subband contextual information in the wavelet domain, while reducing the number of feature channels;

[0075] Wavelet transform is used for multi-frequency decomposition. Multi-frequency feature representation is particularly important for detail preservation and texture enhancement. At the same time, inverse wavelet transform can be used to achieve accurate reconstruction of high-resolution feature maps, ensuring that no intermediate feature information is lost.

[0076] It should be noted that the lightweight CNN submodule includes a three-layer convolutional structure used to model the contextual relationships between multi-band features in the wavelet transform domain, including:

[0077] The first and second layers are 3×3 convolutional layers with ReLU activation function, used to extract local context information and enhance non-linear expressive power;

[0078] The third layer is a 3×3 convolutional layer without an activation function, used to linearly fuse the feature information output from the first two layers.

[0079] In this embodiment, wavelet transform super-resolution processing is performed on sparse multi-view images, and the following operations are performed:

[0080] Preliminary feature extraction: Receive sparse multi-view images as input and perform preliminary feature extraction through a convolutional neural network. Each convolutional layer in the convolutional neural network uses a 3×3 convolutional kernel with a stride of 1 and is equipped with a ReLU activation function to enhance the non-linear expressive power of the model.

[0081] Wavelet decomposition and feature modeling: Two-dimensional discrete wavelet transform is applied to the preliminary feature map to decompose it into one low-frequency sub-band and three high-frequency sub-bands. Before and after each wavelet transform layer, a lightweight CNN sub-module is embedded. The lightweight CNN sub-module consists of three convolutional layers. The first and second layers use 3×3 convolutional kernels with a stride of 1 and ReLU activation function to extract local context information. The third layer uses 3×3 convolutional kernels with a stride of 1 and no activation function to linearly fuse the feature information output by the first two layers.

[0082] Recursive decomposition: The generated sub-band images are recursively decomposed, including two-dimensional discrete wavelet transform decomposition. A lightweight CNN sub-module is introduced to process the obtained sub-bands, and the output of this module is used as the input of the next level of recursion. The decomposition and lightweight CNN sub-module processing flow are repeatedly executed. The context information across sub-bands is modeled through the synergistic effect of wavelet transform and lightweight CNN sub-module, while reducing the number of feature channels to reduce computational complexity.

[0083] Inverse wavelet transform and multi-scale feature fusion: High-resolution feature maps are reconstructed layer by layer using inverse wavelet transform to ensure the integrity of multi-band features. Feature maps corresponding to the shrinking and expanding paths are fused by element-wise addition to achieve multi-scale information integration and further enhance the image's detail expression capabilities.

[0084] The depth prior is output by a pre-trained monocular depth model, and multi-view image features are extracted based on multi-view images and camera parameters.

[0085] Features are extracted from multi-view images using a standard convolutional neural network (CNN) and Transformer architecture, achieved through the following steps:

[0086] Camera parameter injection: A Squeeze-and-Excitation (SE) module is introduced into the feature block of a convolutional neural network (CNN). The camera intrinsic and extrinsic parameters are processed by a multilayer perceptron (MLP), and the processed parameters are injected into the SE module, so that the image features contain global spatial location information.

[0087] Multi-view feature extraction: A CNN-Transformer hybrid network architecture is used to extract multi-view image features. (in , (Indicates channel dimension);

[0088] Convolutional neural networks extract preliminary, dense, low- to mid-level visual features. The features extracted by the convolutional neural network are input into the Transformer network, which performs global context modeling and enhancement of the features through its self-attention mechanism and cross-attention mechanism.

[0089] Depth information extraction: Obtain monocular depth prior information using a pre-trained monocular depth estimation model (such as DepthAnythingV2 or its equivalent); extract depth feature maps based on this information. (in ), and simultaneously generate relative depth output. (in ).

[0090] Coarse matching generates an initial depth map, a planar scanning stereo method is used to construct cross-view depth candidates, and an initial depth distribution is generated by calculating feature similarity.

[0091] After completing image feature extraction, it is necessary to construct an initial depth distribution. The coarse matching module achieves this through the following steps:

[0092] Depth candidate construction: Based on the planar scanning stereo method, a cross-view depth candidate set is constructed in the feature space. For each pixel in the scene, multiple candidate depth values ​​are generated according to camera parameters and depth dimension.

[0093] Feature similarity calculation and initial depth map generation: For each generated candidate depth value, a feature similarity score is calculated by feature projection sampling and dot product similarity measurement; based on the feature similarity score calculated above, an initial coarse depth map is generated; when the number of input viewpoints is greater than two, a pixel-level averaging fusion operation is performed on the initial coarse depth maps calculated from all viewpoints to generate the final coarse depth map.

[0094] Among them, the source view and target view (in Taking a 3D model as an example, we first use a planar scanning 3D method to construct depth candidates. Then, sample the first Features of a view :

[0095]

[0096] in, This indicates a sampling operation, while and Indicates from the first The first view and the first Camera projection matrix from one view to the world coordinate system Indicates depth dimension;

[0097] Subsequently, calculation and Dot product between to obtain coarse depth :

[0098]

[0099] in It represents element-wise product, and the result is summed along the channel dimension. When more than two views are provided as input, the coarse matching module is applied to the other views, and the pixel-level average of all views is calculated to obtain the final coarse depth.

[0100] Coarse-to-fine matching is used to optimize the depth map, and a self-attention-cross-attention structure and deformable sampling are employed to achieve coarse-to-fine depth matching optimization.

[0101] Specifically, the coarse-to-fine matching stage based on deep optimization of the attention mechanism includes:

[0102] The system constructs deep spatial geometric features, integrates coarse depth maps, depth prior features, and camera parameters; a self-attention layer is used to model local depth context relevance and aggregate depth spatial consistency information; a cross-attention layer is used to calculate depth residuals; and a fine depth map is generated using deformable sampling and attention guidance.

[0103] Since coarse matching mainly relies on pixel-level feature alignment, its matching accuracy may be limited in low-texture regions or scenes with repetitive structures, thus affecting the accuracy of depth estimation. To guide the network to focus on more reliable depth candidate regions, a coarse-to-fine matching module is introduced for optimization. This module performs the following operations:

[0104] Construction of deep spatial geometric features: Fusing coarse depth map, depth prior features and camera parameters; inputting the fused information into a deep spatial geometric network consisting of multilayer perceptron (MLP) and Squeeze-and-Excitation (SE) layers, and outputting deep spatial geometric features;

[0105] Deformable sampling and attention-driven depth optimization: Based on the generated depth spatial geometric features, deformable sampling technology is used to aggregate spatial feature information of local neighborhoods to enhance the robustness of matching; the depth spatial geometric features are input into the self-attention layer to perform contextual aggregation of local depth information, the output features of the self-attention layer are input into the cross-attention layer to calculate the depth residual, and the obtained coarse depth is added to the calculated depth residual to obtain the optimized fine depth map;

[0106] Similar to coarse matching, the source view is considered first. and target view In order to construct deep spatial geometric features (d.s_g. features) Coarse depth map Depth features and camera parameters Fusion via deep spatial geometric networks:

[0107]

[0108] in, This represents a multilayer perceptron (MLP) layer, while Indicates the SE layer;

[0109] Apply deformable sampling techniques to obtain the first Deformable sampling features of the view :

[0110]

[0111] Where DS represents deformable sampling, and Represents the image-level sampling offset, where It is the number of deformable points;

[0112] Further deep optimization is performed using attention mechanisms, including self-attention layers and cross-attention layers, to enhance deep prediction;

[0113] rough depth The input is fed into a self-attention layer to aggregate local depth information, and the depth residual is calculated in a cross-attention layer. :

[0114]

[0115] in, This represents element-wise multiplication, and the results are added along the channel dimension. The depth-space geometric attention weights are represented by the sampling offset and attention weights, which are passed through the offset layer. and attention layer Make a prediction:

[0116]

[0117] Fineness depth is calculated as follows:

[0118]

[0119] When there are more than two views as input, each can be obtained separately. The fine depth of each view.

[0120] By leveraging the improved UNet network to fuse multi-scale features, the geometric representation capability is enhanced, and the depth estimation results are further optimized.

[0121] Specifically, the improved UNet structure includes: a multi-scale depth map encoder that stitches and fuses with pooled images stage by stage to construct multi-level depth maps; a decoder stage that fuses multi-scale features through skip connections; and a feature enhancement module that uses large and small convolutional kernels at the end of UNet, where the large convolutional kernel is used to capture wide-area contextual relationships and the small convolutional kernel is used to enhance local feature aggregation, introducing a variable weight aggregation mechanism to achieve collaborative modeling of heterogeneous scale information.

[0122] It should be noted that the feature enhancement module with fusion of large and small convolutional kernels includes: channel dimensionality reduction of the input feature map to reduce computational overhead, extraction of long-distance spatial context using large kernel depthwise convolution (kernel size of 7×7), generation of dynamic weights and application to small convolutional kernels (kernel size of 3×3) aggregation process, realizing collaborative modeling of heteroscale information, thereby achieving feature enhancement.

[0123] To enhance the model's ability to understand scene geometry and further optimize depth estimation results, a monocular depth estimation network is introduced. The improved UNet architecture is implemented through the following steps:

[0124] Multi-scale encoding and feature fusion: The multi-scale encoder constructs multi-level depth maps through stage-wise pooling and feature concatenation, fusing geometric features at different scales. The specific process is as follows:

[0125] The size is The input image, through a The convolutional layer performs preliminary processing and downsampling, using two layers with a stride of 1. Convolutional layers extract local features and generate a size of depth feature map The generated deep feature map The input image, after undergoing average pooling, is concatenated channel by channel. The concatenated features are then applied with a stride of 2. The convolutional layer performs downsampling to generate a size of depth feature map ,Will Channel concatenation is performed with the average pooled input image at the corresponding scale. A stride of 2 is applied. The convolutional layer performs downsampling to generate a size of depth feature map The generated deep feature map , , The feature maps are fused with the feature maps of the corresponding scales of the UNet encoder to construct feature representations that incorporate multi-level depth information.

[0126] Feature Decoding and Upsampling: In the decoding stage, the UNet decoder uses skip connections to progressively upsample the fused feature map at the corresponding scale in the encoding stage. By fusing feature information from multiple scales, the decoder reconstructs a feature map with progressively improved spatial resolution. Finally, it generates a detailed and richly layered geometric representation of the scene, comprehensively preserving local details and global structural information.

[0127] Feature Enhancement Module: To further improve feature representation capabilities, a feature enhancement module is integrated at the end of the UNet encoder (i.e., before the highest semantic level feature output). This module fuses features extracted from different receptive fields (large and small convolutional kernels) to enhance the aggregation capability of multi-scale contextual information. For example... Figure 4 As shown, the specific process is as follows: Given an input feature map First apply a Convolutional layers reduce the channel dimension to To reduce computational complexity, a kernel size of [value missing] is applied to the dimensionality-reduced feature map. The depthwise separable convolution operation extracts spatial context information covering a large neighborhood. Two methods are then applied sequentially to the extracted large receptive field features. Convolutional layer; generates a spatially and channel-adaptive 3D weight tensor. ,in As the weight dimension, the original input feature map Divide the channel evenly into 8 groups, each group containing Each channel; for each spatial location in the feature map The generated location Corresponding weight vector Reshape into a tensor Where 8 corresponds to the number of channel groups. Represents the small convolution kernel size, using the reconstructed weight tensor. As a grouped convolution kernel, at position local neighborhood Above, the 8 groups of features were divided and executed respectively. Convolution operations enable feature aggregation, with channels within a group sharing the same convolution kernel weights.

[0128] Through the above mechanism, this module collaboratively integrates the wide-area structural information captured by the large convolutional kernel and the local fine information captured by the adaptive small convolutional kernel. This multi-scale feature fusion method based on dynamically generated weights significantly improves the discriminative ability of features and its ability to express the consistency of scene geometric structure.

[0129] Predict 3D Gaussian parameters based on the optimized depth map, including center position, opacity, color, and covariance matrix;

[0130] Specifically, predicting 3D Gaussian primitive parameters includes: back-projecting pixels into 3D space using an optimized depth map combined with camera parameters to determine the Gaussian center position; treating opacity as depth confidence and predicting opacity values ​​using an optimized depth input multilayer perceptron; representing color information using spherical harmonic coefficients; and representing the covariance matrix using scaling and rotation matrices.

[0131] To construct a high-quality 3D scene representation, based on the optimized depth map, various Gaussian primitive parameters for 3D Gaussian Splatting rendering are predicted. The Gaussian parameter prediction module is implemented through the following steps:

[0132] Gaussian center position prediction: Using the optimized depth map and combined with camera intrinsic and extrinsic parameters, each pixel is mapped to the world coordinate system through back projection calculation to determine its corresponding three-dimensional spatial coordinates, which are used as the center position of the Gaussian element.

[0133] Opacity prediction: The opacity parameter is regarded as a confidence index for depth estimation. The optimized depth map is encoded and mapped by a multilayer perceptron (MLP) to predict the opacity value of each Gaussian point.

[0134] Color prediction: Spherical harmonic coefficients are used to model color information to effectively represent view-dependent appearance changes and color distribution under complex lighting conditions;

[0135] Covariance matrix prediction: A covariance matrix is ​​constructed using a combination of rotation and scaling matrices to accurately describe the geometry and distribution characteristics of each Gaussian element in three-dimensional space.

[0136] A Gaussian field is constructed based on three-dimensional Gaussian primitive parameters, and a differentiable renderer is used to render images from a new perspective, achieving high-quality three-dimensional reconstruction under sparse perspective and generating images from a new perspective.

[0137] All predicted 3D Gaussian primitives are combined into a structured, renderable 3D Gaussian scene representation. Based on this representation, a differentiable renderer is used to generate images from any viewpoint according to the target camera parameters. During the rendering process, the Gaussian primitives are depth-sorted and color-blended with transparency weights according to the relationship between the camera viewpoint and geometric projection. Finally, a new viewpoint image with high visual consistency and rich texture details is synthesized.

[0138] Example 2

[0139] A depth-guided 3D Gaussian reconstruction system for sparse viewpoint images is provided to implement the aforementioned depth-guided 3D Gaussian reconstruction method for sparse viewpoint images. Through the collaborative work of various modules, it processes sparse multi-viewpoint images efficiently and accurately, generating high-quality 3D reconstruction results, including:

[0140] The data acquisition and preprocessing module is responsible for acquiring sparse multi-view RGB images and corresponding camera parameters (including intrinsic and extrinsic parameters) to construct the initial image and parameter dataset. Furthermore, it includes necessary preprocessing steps, such as image correction and noise reduction, to ensure the quality of the input data. Through accurate camera parameters and multi-view image input, it ensures accurate spatial information for feature extraction and depth estimation in subsequent processing. This module provides the system's input interface, supporting multi-view images from different sources and types.

[0141] The wavelet transform super-resolution module is used to enhance the high frequency of the input sparse multi-view image. Based on the wavelet transform structure, it performs wavelet transform super-resolution processing to enhance the high frequency details of the image. It receives sparse multi-view RGB images as input, performs preliminary feature extraction through a convolutional neural network (CNN), and improves the image clarity and detail representation by combining recursive decomposition and inverse wavelet transform with a lightweight CNN submodule.

[0142] The feature extraction module is used to extract multi-view image features and monocular depth priors. It combines the standard convolutional neural network (CNN) and Transformer architecture to extract features from multi-view images and introduces a camera parameter injection mechanism and a pre-trained monocular depth estimation model to output image features and depth prior features that fuse spatial location information, providing high-quality features for subsequent depth estimation and Gaussian modeling.

[0143] The coarse matching module is used to generate coarse depth maps based on the planar scanning stereo method.

[0144] The coarse-to-fine matching module is used to generate fine depth maps based on attention mechanisms and deformable sampling;

[0145] The depth optimization module is used to optimize depth estimation results based on the improved UNet network.

[0146] The coarse matching module constructs depth candidates and calculates feature similarity based on the planar scanning stereo method to generate an initial depth distribution; the coarse-to-fine matching module integrates coarse depth, depth prior and camera parameters, and realizes depth residual calculation and fine depth map generation through deformable sampling and attention mechanism; the depth optimization module further refines the depth matching results by fusing depth information and feature enhancement modules with large and small convolutional kernels, thereby obtaining a high-quality fine depth map.

[0147] The Gaussian parameter prediction module is used to predict the three-dimensional Gaussian meta-parameters corresponding to each pixel;

[0148] Based on the optimized depth map, parameters of the 3D Gaussian primitives are predicted, including center position, opacity, color, and covariance matrix. Each pixel is mapped to the world coordinate system through backprojection to determine its 3D spatial coordinates as the center position of the Gaussian primitive. Simultaneously, a multilayer perceptron (MLP) is used to encode the optimized depth map to predict the opacity parameter. Color information is modeled using spherical harmonics coefficients, and the construction of the covariance matrix depends on a combination of rotation and scaling matrices.

[0149] The rendering module is used to render new perspective images based on predicted 3D Gaussian meta-parameters, achieving high-quality 3D reconstruction under sparse perspectives and generating new perspective images.

[0150] All predicted 3D Gaussian primitives are integrated into a coherent and renderable 3D Gaussian field. Then, using advanced differentiable rendering technology, a new perspective image with high visual consistency and rich texture details is generated based on the constructed Gaussian field and the target camera parameters.

[0151] Through the collaborative work of the above modules, end-to-end processing from sparse input to high-quality 3D reconstruction and new perspective synthesis is achieved. It can generate high-quality 3D reconstruction results based on sparse perspective images and provide accurate geometric representations for further applications such as virtual reality and augmented reality, thereby improving the accuracy and visual effects of the reconstruction results.

[0152] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0153] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A depth-guided 3D Gaussian reconstruction method suitable for sparse viewpoint images, characterized in that, include: Wavelet transform super-resolution processing is performed on sparse multi-view images to enhance high-frequency details; The depth prior is output by a pre-trained monocular depth model, and multi-view image features are extracted based on multi-view images and camera parameters. Coarse matching generates an initial depth map, a planar scanning stereo method is used to construct cross-view depth candidates, and an initial depth distribution is generated by calculating feature similarity. Coarse-to-fine matching is used to optimize the depth map, and a self-attention-cross-attention structure and deformable sampling are employed to achieve coarse-to-fine depth matching optimization. We utilize an improved UNet network to fuse multi-scale features and optimize depth estimation results; Predict 3D Gaussian parameters based on the optimized depth map, including center position, opacity, color, and covariance matrix; A Gaussian field is constructed based on three-dimensional Gaussian primitive parameters, and a differentiable renderer is used to render images from a new perspective, achieving high-quality three-dimensional reconstruction under sparse perspective and generating images from a new perspective.

2. The depth-guided 3D Gaussian reconstruction method for sparse viewpoint images according to claim 1, characterized in that, Perform wavelet transform super-resolution processing on sparse multi-view images, and perform the following operations: It receives sparse multi-view images as input and performs preliminary feature extraction through a convolutional neural network. Each convolutional layer in the convolutional neural network uses a 3×3 convolutional kernel with a stride of 1 and is equipped with a ReLU activation function to enhance the non-linear expressive power of the model. Two-dimensional discrete wavelet transform is applied to the initial feature map, decomposing it into one low-frequency sub-band and three high-frequency sub-bands. Before and after each wavelet transform layer, a lightweight CNN sub-module is embedded. The lightweight CNN sub-module consists of three convolutional layers. The first and second layers use 3×3 convolutional kernels with a stride of 1 and ReLU activation function to extract local context information. The third layer uses 3×3 convolutional kernels with a stride of 1 and no activation function to linearly fuse the feature information output by the first two layers. The generated sub-band images are recursively decomposed, including two-dimensional discrete wavelet transform decomposition. A lightweight CNN sub-module is introduced to process the obtained sub-bands, and the output of the lightweight CNN sub-module is used as the input of the next level of recursion. The decomposition and lightweight CNN sub-module processing flow are repeatedly executed. The context information of the sub-bands is modeled through the synergistic effect of wavelet transform and lightweight CNN sub-module. High-resolution feature maps are reconstructed layer by layer using inverse wavelet transform, and multi-scale information integration is achieved by fusing the corresponding feature maps in the shrinking and expanding paths through element-wise addition.

3. The depth-guided 3D Gaussian reconstruction method for sparse viewpoint images according to claim 2, characterized in that, The depth prior is output through a pre-trained monocular depth model. Multi-view image features are extracted based on multi-view images and camera parameters. The following operations are performed: An SE module is introduced into the feature block of the convolutional neural network. The camera intrinsic and extrinsic parameters are processed by a multilayer perceptron (MLP), and the processed parameters are injected into the SE module so that the image features contain global spatial location information. Multi-view image features are extracted using a CNN-Transformer hybrid network architecture. ,in , Indicates the channel dimension; Convolutional neural networks extract preliminary, dense, low- to mid-level visual features. The features extracted by the convolutional neural network are input into the Transformer network, and the features are modeled and enhanced globally through self-attention and cross-attention mechanisms. Obtain monocular depth prior information using a pre-trained monocular depth estimation model; extract depth feature maps based on the monocular depth prior information. ,in And simultaneously generate relative depth output. ,in .

4. The depth-guided 3D Gaussian reconstruction method for sparse viewpoint images according to claim 3, characterized in that, Coarse matching generates an initial depth map, a planar scanning stereo method is used to construct cross-view depth candidates, and an initial depth distribution is generated through feature similarity calculation, including: Based on the planar scanning stereo method, a cross-view depth candidate set is constructed in the feature space. For each pixel in the scene, multiple candidate depth values ​​are generated according to camera parameters and depth dimension. For each generated candidate depth value, a feature similarity score is calculated by feature projection sampling and dot product similarity measurement, and an initial coarse depth map is generated based on the feature similarity score. When there are more than two input viewpoints, a pixel-level averaging fusion operation is performed on the initial coarse depth maps calculated from all viewpoints to generate the final coarse depth map.

5. The depth-guided 3D Gaussian reconstruction method for sparse viewpoint images according to claim 4, characterized in that, Coarse-to-fine matching is used for depth map optimization. A self-attention-cross-attention structure and deformable sampling are employed to achieve coarse-to-fine depth matching optimization. The following operations are performed: By fusing coarse depth maps, depth prior features, and camera parameters, the fused information is input into a depth spatial geometry network consisting of a multilayer perceptron (MLP) and a SE layer, and the output is depth spatial geometry features. Based on the generated deep spatial geometric features, deformable sampling techniques are used to aggregate spatial feature information of local neighborhoods. The depth spatial geometric features are input into the self-attention layer to perform contextual aggregation of local depth information. The output features of the self-attention layer are input into the cross-attention layer to calculate the depth residual. The coarse depth is added to the calculated depth residual to obtain the optimized fine depth map.

6. The depth-guided 3D Gaussian reconstruction method for sparse viewpoint images according to claim 5, characterized in that, By leveraging an improved UNet network to fuse multi-scale features and optimizing depth estimation results, the following operations are performed: The multi-scale encoder constructs a multi-level depth map through staged pooling and feature concatenation, fusing geometric features at different scales. Specifically, it integrates geometric features of different sizes. The input image, through a The convolutional layer performs preliminary processing and downsampling, using two layers with a stride of 1. Convolutional layers extract local features and generate a size of depth feature map The generated deep feature map The input image, after undergoing average pooling, is concatenated channel by channel. The concatenated features are then applied with a stride of 2. The convolutional layer performs downsampling to generate a size of depth feature map ,Will Channel concatenation is performed with the average pooled input image at the corresponding scale, using a stride of 2. The convolutional layer performs downsampling to generate a size of depth feature map The generated deep feature map , and The feature maps at the corresponding scales of the UNet encoder are fused to construct feature representations that incorporate multi-level depth information. During the decoding stage, the UNet decoder uses skip connections to progressively upsample the fused feature map at the corresponding scale in the encoding stage. By fusing feature information from multiple scales, the decoder reconstructs a feature map with progressively improved spatial resolution, generating a detailed and richly layered geometric representation of the scene, and comprehensively preserving local details and global structural information. The UNet encoder integrates a feature enhancement module at its end, fusing features extracted from different receptive fields, where, given the input feature map... , apply one Convolutional layers reduce the channel dimension to For the dimensionality-reduced feature map, apply a kernel size of Depthwise separable convolution operations are used to extract spatial context information covering a large neighborhood. Two methods are then applied sequentially to the extracted large receptive field features. Convolutional layers generate a spatially and channel-adaptive three-dimensional weight tensor. ,in As the weight dimension, the original input feature map Divide the channel evenly into 8 groups, each group containing Each channel represents a spatial location in the feature map. The generated location Corresponding weight vector Reconstructed as a tensor Where 8 corresponds to the number of channel groups. Represents the small convolution kernel size, using the reconstructed weight tensor. As a grouped convolution kernel, at position local neighborhood Above, the 8 groups of features were divided and executed respectively. Convolution operations enable feature aggregation, and channels within a group share the same convolution kernel weights.

7. The depth-guided 3D Gaussian reconstruction method for sparse viewpoint images according to claim 6, characterized in that, Predict 3D Gaussian meta-parameters based on the optimized depth map, including: Using the optimized depth map and combined with camera intrinsic and extrinsic parameters, each pixel is mapped to the world coordinate system through back projection calculation to determine its corresponding three-dimensional spatial coordinates, which serve as the center position of the Gaussian unit. The opacity parameter is regarded as a confidence index for depth estimation. The optimized depth map is encoded and mapped by a multilayer perceptron (MLP) to predict the opacity value of each Gaussian point. Color information is modeled using spherical harmonic function coefficients to represent viewpoint-dependent appearance changes and color distribution under complex lighting conditions; A covariance matrix is ​​constructed using a combination of rotation and scaling matrices to accurately describe the geometry and distribution characteristics of each Gaussian element in three-dimensional space.

8. A depth-guided 3D Gaussian reconstruction system for sparse viewpoint images, used to implement the depth-guided 3D Gaussian reconstruction method for sparse viewpoint images as described in claim 7, characterized in that, include: The wavelet transform super-resolution module is used to perform high-frequency enhancement on the input sparse multi-view images; The feature extraction module is used to extract multi-view image features and monocular depth priors; The coarse matching module is used to generate coarse depth maps based on the planar scanning stereo method. The coarse-to-fine matching module is used to generate fine depth maps based on attention mechanisms and deformable sampling; The depth optimization module is used to optimize depth estimation results based on the improved UNet network. The Gaussian parameter prediction module is used to predict the three-dimensional Gaussian meta-parameters corresponding to each pixel; The rendering module is used to render new perspective images based on predicted 3D Gaussian meta-parameters, achieving high-quality 3D reconstruction under sparse perspectives and generating new perspective images.

Images