A three-dimensional reconstruction method based on guided filtering and Mamba geometric feature fusion

Abstract

The application provides a three-dimensional reconstruction method based on guided filtering and Mamba geometric feature fusion in the technical field of three-dimensional reconstruction, and comprises the following steps: S1, collecting multi-view RGB images, calibrating the camera to obtain a projection matrix and an intrinsic parameter matrix, and obtaining a reference image; S2, extracting image features from each RGB image; S3, sampling a depth hypothesis plane on the reference image, performing homography transformation on each image feature, and aggregating the depth hypothesis plane to obtain a cost volume; S4, sequentially performing filtering, regularization and exponential normalization on the cost volume to obtain a probability volume, constructing a depth map and a confidence map based on the probability volume; S5, performing frequency domain filtering and upsampling on the depth map and the confidence map, and then performing feature fusion to obtain a view depth; and S6, performing a three-dimensional reconstruction operation based on the view depth, the projection matrix and the intrinsic parameter matrix. The application has the advantages that the accuracy and the robustness of three-dimensional reconstruction are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, and in particular to a 3D reconstruction method based on guided filtering and Mamba geometric feature fusion. Background Technology

[0002] 3D reconstruction refers to the technical process of recovering the three-dimensional geometric structure and appearance information of an object or scene from a two-dimensional image. In traditional 3D reconstruction methods, multi-view stereo (MVS) networks are widely used, with the cost volume serving as its core data structure to encode the cost or uncertainty of pixel matching between images from different viewpoints. However, existing MVS-based schemes still have the following key problems:

[0003] 1. Missing global context information in cost volume regularization:

[0004] In the initial stage of the cascaded MVS process, a cost volume needs to be constructed to represent the matching cost under different depth assumptions, and then optimized using a regularization network. Existing methods generally employ three-dimensional convolutional neural networks (3D CNNs) to perform this regularization process. However, the receptive field of 3D CNNs is limited by locality; when processing the cost volume, it can only perceive information from neighboring regions, making it difficult to effectively capture the global geometric context and long-range dependencies of the entire scene, thus restricting the consistency and accuracy of depth estimation.

[0005] 2. Error accumulation and amplification in cross-stage feature fusion:

[0006] Cascaded MVS architectures typically utilize depth prediction results from a coarser stage to guide depth estimation in subsequent finer stages. In this process, existing methods often fuse geometric information from the previous stage (such as upsampled depth maps) with image features from the current stage through simple feature concatenation or addition. This fusion strategy lacks a mechanism to evaluate the reliability of the depth information from the previous stage. If estimation deviations occur in error-prone areas such as weak textures, the erroneous information will be forcibly propagated to the next stage, causing errors to accumulate and even amplify during the cascading process, severely reducing the final reconstruction accuracy.

[0007] 3. Noise interference and structural degradation of cost bodies in cascade transmission:

[0008] In the cascaded processing of MVS, the cost volume generated at each stage often contains noise, especially in challenging regions such as low-texture, occluded, or non-Lambertian surfaces. This noise leads to erroneous low cost values, interferes with the depth probability distribution, and consequently causes noise and artifacts in the depth map. Existing techniques lack a direct cost volume optimization mechanism across stages, failing to effectively "clean up" the cost volume itself. Noise and structural uncertainties are propagated to subsequent stages through the depth estimation process, affecting the setting of depth assumption intervals and the construction of new cost volumes, thus causing error propagation. Although general filtering methods can be introduced to suppress noise, they often indiscriminately smooth all information, resulting in the loss of key geometric details such as edges and thin-walled structures, leading to structural degradation.

[0009] Therefore, how to provide a 3D reconstruction method based on guided filtering and Mamba geometric feature fusion to improve the accuracy and robustness of 3D reconstruction has become an urgent technical problem to be solved. Summary of the Invention

[0010] The technical problem to be solved by this invention is to provide a three-dimensional reconstruction method based on guided filtering and Mamba geometric feature fusion, so as to improve the accuracy and robustness of three-dimensional reconstruction.

[0011] This invention is implemented as follows: a 3D reconstruction method based on guided filtering and Mamba geometric feature fusion, comprising the following steps:

[0012] Step S1: Acquire RGB images from multiple perspectives, calibrate the intrinsic and extrinsic parameters of the camera that acquired the RGB images to obtain the projection matrix and intrinsic parameter matrix, and obtain a reference image for 3D reconstruction.

[0013] Step S2: Input each of the RGB images into the feature extraction network for feature extraction to obtain image features;

[0014] Step S3: On the reference image, a number of depth hypothesis planes are obtained by uniform sampling inverse depth. Homography transformation is performed on each of the image features and they are aggregated with the depth hypothesis planes to obtain the cost volume.

[0015] Step S4: Filter the cost body using an adaptive guided filtering network;

[0016] Step S5: Perform a regularization operation on the cost body after filtering using a regularization network;

[0017] Step S6: Perform exponential normalization on the channel dimension of the cost volume after regularization to obtain the probability volume, and construct a depth map and a confidence map based on the probability volume;

[0018] Step S7: After performing frequency domain filtering and upsampling on the depth map and confidence map, input them into the adaptive geometric feature fusion network for feature fusion to obtain the view depth;

[0019] Step S8: Perform a 3D reconstruction operation based on the stated view depth, projection matrix, and intrinsic parameter matrix to obtain a dense point cloud.

[0020] Furthermore, in step S4, the filtering process of the adaptive guided filter network is specifically as follows:

[0021] In stage 0, the reference map is used as the guide map; in subsequent stages, the depth map generated in the previous stage is used as the guide map.

[0022] Calculate the horizontal and vertical gradients of the guide graph:

[0023] ;

[0024] ;

[0025] in, Represents the horizontal gradient; Represents the vertical gradient; This represents a guide diagram; This represents the Sobel convolution kernel in the horizontal direction; This represents the Sobel convolution kernel in the vertical direction;

[0026] The edge preservation weights are calculated based on the horizontal and vertical gradients:

[0027] ;

[0028] in, This indicates that the weights are preserved at the edges; Represents the natural exponential function; This indicates the hyperparameter for edge preservation strength control;

[0029] An enhanced guided representation is constructed based on the edge-preserving weights:

[0030] ;

[0031] in, This indicates enhanced guidance;

[0032] For each pixel of the cost body slice of the cost body Build an adaptive window:

[0033] ;

[0034] in, This indicates an adaptive window, i.e., using... The central window; Represents the pixel coordinates of an RGB image; Indicating a guide map in The grayscale value at that location; Indicating a guide map in The grayscale value at that location; Indicates the continuity threshold; Indicates the maximum window radius; This indicates taking the maximum value; Indicates taking x and The maximum horizontal distance between them, its y and The maximum vertical distance between them;

[0035] Calculate the linear regression coefficients within the adaptive window:

[0036] ;

[0037] in, as well as All represent linear regression coefficients; Represents the covariance function; Represents the variance function; This represents the k-th adaptive window; Indicating a guide map in Extraction within; Indicates the cost body slice p in Extraction within; This represents the regularization parameter to prevent the denominator from being zero; Indicates the cost body slice p in The mean within; Indicating a guide map in The mean within;

[0038] Based on the enhanced guided representation, each cost body is filtered slice by slice along the depth dimension:

[0039] ;

[0040] in, This represents a slice of the cost volume after filtering. This indicates an adaptive guided filtering operation based on linear regression coefficients; This represents the cost volume slice corresponding to the d-th depth hypothesis;

[0041] The filtered cost volume slices are then weighted and fused:

[0042] ;

[0043] in, This represents a slice of the cost body after weighted fusion; This represents the adaptive mitigation factor.

[0044] Furthermore, in step S5, the regularization network is constructed based on a global attention regularization module and a three-dimensional U-shaped module; the global attention regularization module is used to perform regularization operations on the cost body of stage 0, and the three-dimensional U-shaped module is used to perform regularization operations on the cost bodies of the remaining stages.

[0045] Furthermore, the regularization process of the global attention regularization module is as follows:

[0046] A three-dimensional coordinate grid matching the dimension of the cost volume is generated and normalized. Sine position coding is applied independently to each coordinate axis of the normalized three-dimensional coordinate grid to obtain encoded features. The encoded features are projected through a convolutional layer to obtain projected features. The projected features are added to the cost volume to obtain the projected cost volume. The projected cost volume is input into a 3D convolutional layer with stride for downsampling to obtain a compact cost volume.

[0047] The compact cost volume is flattened along the spatial and depth dimensions to obtain a one-dimensional sequence. The one-dimensional sequence is then input into several Transformer encoders employing a global attention mechanism to calculate the global attention value.

[0048] The global attention value is reshaped into a three-dimensional feature volume, and then the three-dimensional feature volume is upsampled through a 3D transposed convolutional layer to restore the original depth and spatial resolution, resulting in a regularized cost volume.

[0049] Furthermore, the formula for the adaptive geometric feature fusion network is:

[0050] ;

[0051] in, The depth map is represented by the first... The viewing depth corresponding to each pixel; Indicates a Mamba network; This represents the visual features extracted from the reference image; Indicates a splicing operation; Indicates reliability weight; It represents the Hadamardi (or Hadama) stack; Represents a geometric branching network; Indicates reference image; This represents the depth map obtained from the previous upsampling stage.

[0052] The advantages of this invention are:

[0053] 1. By acquiring RGB images from multiple perspectives, the intrinsic and extrinsic parameters of the camera acquiring the RGB images are calibrated to obtain the projection matrix and intrinsic parameter matrix, and a reference image for 3D reconstruction is obtained. Then, each RGB image is input into a feature extraction network for feature extraction to obtain image features. Several depth hypothesis planes are obtained by uniformly sampling inverse depth on the reference image. Homography transformation is performed on each image feature and aggregated with the depth hypothesis planes to obtain the cost body. Next, an adaptive guided filtering network is used to filter the cost body, and a regularization network is used to regularize the filtered cost body. The channel dimensions of the regularized cost body are exponentially normalized to obtain a probability body. A depth map and a confidence map are constructed based on the probability body. The depth map... After frequency domain filtering and upsampling of the confidence map, it is input into an adaptive geometric feature fusion network to obtain the view depth. Finally, a 3D reconstruction operation is performed based on each view depth, projection matrix, and intrinsic parameter matrix to obtain a dense point cloud. That is, the adaptive guided filtering network suppresses the propagation of cost volume noise and maintains the geometric edge structure, the global attention regularization module captures long-distance dependencies to enhance the context consistency of the cost volume, and the reliability weight mechanism in the adaptive geometric feature fusion network dynamically evaluates the confidence of cross-stage depth information. Combined with the Mamba network, high-confidence geometric features are selectively fused, thereby effectively blocking error accumulation in complex scenes such as weak texture and occlusion, which greatly improves the accuracy and robustness of 3D reconstruction.

[0054] 2. By introducing an adaptive guided filter network, the accuracy and stability of filtering are effectively improved by dynamically adjusting the guide map (using the reference map in the initial stage and the depth map of the previous stage in subsequent stages) and combining edge-preserving weights and an adaptive window. The edge-preserving weights are based on the gradient calculated by the Sobel operator and control the edge intensity through an exponential function, protecting texture details during the filtering process, reducing edge blurring in reconstruction, and improving point cloud quality. The window size is adaptively adjusted according to pixel grayscale continuity and spatial distance. Compared with fixed window filtering, it can better handle lighting changes and occlusion problems in the scene and enhance robustness to complex environments. In subsequent stages, the depth map of the previous stage is used as the guide map, realizing adaptive iterative optimization, gradually refining the depth estimation, reducing the impact of noise, improving reconstruction consistency, and ultimately solving the problem of traditional guided filtering easily losing details and over-smoothing in 3D reconstruction, thus improving generalization ability and practicality.

[0055] 3. By setting up a regularization network that combines a global attention regularization module and a 3D U-shaped module, the feature representation of the cost volume is optimized. The global attention regularization module uses 3D coordinate grid encoding and Transformer encoder to process one-dimensional sequences, capturing global contextual dependencies (such as the correlation between depth and spatial dimensions), improving feature consistency, and reducing holes or distortions in reconstruction. In the subsequent stage, the 3D U-shaped module is used, combined with downsampling and upsampling operations, to effectively handle local details and multi-scale features, improving the resolution recovery capability of the cost volume. The module design avoids redundant computation (such as downsampling to compact the cost volume), and combines convolution and transposed convolution to reduce computational overhead while maintaining accuracy, making it suitable for real-time 3D reconstruction applications. This dual-module architecture integrates global and local regularization, overcoming the shortcomings of single regularization methods (such as pure CNN) in balancing global consistency and local details, and improving the density and accuracy of reconstruction.

[0056] 4. The adaptive geometric feature fusion network utilizes the Mamba network to combine visual and geometric features, achieving efficient feature fusion. As a state-space model, Mamba, compared to traditional RNNs or Transformers, has linear computational complexity and long sequence processing capabilities, enabling efficient fusion of multi-view features and improving the estimation efficiency of depth maps. By combining concatenation operations (⊕) and Hadamard product (⊙) with reliability weights, the geometric branch output is dynamically weighted, enhancing robustness to unreliable regions (such as occlusion or low texture) and reducing reconstruction errors. Frequency domain filtering and upsampling are performed before fusion to preprocess the depth map, ensuring the quality of input features and further improving the accuracy of view depth. This fusion mechanism solves the problem of insufficient fusion of visual and geometric features in traditional methods. Combined with the advanced architecture of Mamba, it significantly improves the accuracy and speed of depth estimation.

[0057] 5. A complete end-to-end process is formed from image acquisition to point cloud generation, with key steps employing iterative design. It integrates feature extraction, cost volume construction, filtering, regularization, and fusion, reducing manual intervention and achieving a high degree of automation, making it suitable for large-scale 3D reconstruction scenarios. Multiple stages (such as filtering and fusion) use the output of the previous stage (such as depth maps as guide maps) to achieve progressive optimization, gradually improving the quality of depth maps and enhancing the method's adaptability to noise and viewpoint changes. By exponentially normalizing the cost volume to obtain the probability volume, combined with the confidence map, reliable depth uncertainty estimation is provided, improving point cloud density and reliability. This process design enhances the robustness and scalability of the method, reduces the risk of error accumulation compared to staged independent algorithms, and is easy to integrate into existing systems.

[0058] 6. Based on standard RGB images and camera parameters (intrinsic / extrinsic parameters), no special hardware (such as depth sensors) is required, and it has strong compatibility; inverse depth uniform sampling and homography transformation simplify the cost volume construction and are easy to implement; through adaptive filtering and Mamba fusion, it may surpass the traditional multi-view stereo (MVS) method in terms of accuracy (edge ​​preservation) and speed (Mamba efficiency), and is suitable for high dynamic scenes.

[0059] 7. By innovatively integrating an adaptive guided filtering mechanism, an efficient regularization network, and the Mamba geometric feature fusion framework, and by dynamically adjusting the guide map (reference map → depth map iterative optimization) combined with edge preservation weights and adaptive windows, the ability to preserve reconstruction edge details and scene adaptability are significantly improved. By utilizing the dual-path design of the global attention regularization module and the 3D U-shaped module, the global consistency features and local detail recovery are balanced, reducing computational overhead. By leveraging the efficient long sequence processing capability of the Mamba network, visual features and weighted geometric information are fused to effectively solve the depth estimation problem in occluded and low-texture areas, ultimately achieving end-to-end high-precision and robust 3D reconstruction from multi-view RGB images to dense point clouds. Attached Figure Description

[0060] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0061] Figure 1 This is a flowchart of a three-dimensional reconstruction method based on guided filtering and Mamba geometric feature fusion according to the present invention.

[0062] Figure 2 This is a flowchart illustrating the present invention.

[0063] Figure 3 This is a schematic diagram of the adaptive guided filtering network of the present invention.

[0064] Figure 4 This is a schematic diagram of the global attention regularization module of the present invention.

[0065] Figure 5 This is a schematic diagram of the adaptive geometric feature fusion network of the present invention.

[0066] Figure 6 This is a schematic diagram comparing the filtering process before and after the present invention.

[0067] Figure 7 This is a schematic diagram comparing the present invention with traditional methods on the DTU dataset.

[0068] Figure 8 This is a schematic diagram of the 3D reconstruction of the Tanks & Temples dataset according to the present invention. Detailed Implementation

[0069] The technical solution in this application embodiment has the following general idea: the cost volume noise propagation is suppressed and the geometric edge structure is maintained by an adaptive guided filtering network, the long-distance dependency is captured by a global attention regularization module to enhance the contextual consistency of the cost volume, and the confidence of cross-stage depth information is dynamically evaluated by the reliability weight mechanism in the adaptive geometric feature fusion network. The high-confidence geometric features are selectively fused by the Mamba network, thereby effectively blocking error accumulation in complex scenes such as weak texture and occlusion, and thus improving the accuracy and robustness of 3D reconstruction.

[0070] Please refer to Figures 1 to 8 As shown, a preferred embodiment of the 3D reconstruction method based on guided filtering and Mamba geometric feature fusion of the present invention includes the following steps:

[0071] Step S1: Acquire RGB images from multiple perspectives, calibrate the intrinsic and extrinsic parameters of the camera that acquired the RGB images to obtain the projection matrix and intrinsic parameter matrix, and obtain a reference image for 3D reconstruction.

[0072] Step S2: Input each of the RGB images into the feature extraction network for feature extraction to obtain image features; during the feature extraction process, the feature pyramid network is used in stage 0 to obtain coarse features, and in subsequent stages, a confidence-guided adaptive geometric feature fusion network is used to fuse geometric information with the visual features of the current stage to obtain fine features.

[0073] Step S3: On the reference image, a number of depth hypothesis planes are obtained by uniform sampling inverse depth. Homography transformation is performed on each of the image features and they are aggregated with the depth hypothesis planes to obtain the cost volume.

[0074] Step S4: Filter the cost volume using an Adaptive Guided Filtering Network (AGF) and lightly smooth the cost volume along each depth assumption plane with a mild coefficient to suppress noise and preserve edges;

[0075] Step S5: Perform a regularization operation on the cost body after filtering using a regularization network (AttentionCostReg);

[0076] Step S6: Perform exponential normalization on the channel dimensions of the cost volume after regularization to obtain the probability volume. Construct a depth map and a confidence map based on the probability volume. The formula for calculating the probability volume is:

[0077] ;

[0078] in, Represents a probability volume; This represents the cost volume after the regularization operation;

[0079] Step S7: After performing frequency domain filtering and upsampling on the depth map and confidence map, input them into the adaptive geometric feature fusion network (GeoMambaFusion) for feature fusion to obtain the view depth; and use the depth map from the previous stage to adaptively guide the filtering of the cost volume.

[0080] Step S8: Perform a 3D reconstruction operation based on the aforementioned viewpoint depth, projection matrix, and intrinsic parameter matrix to obtain a dense point cloud (fused using the open3d library).

[0081] This invention employs a multi-stage cascaded MVS architecture to progressively refine depth estimation at different resolutions. In the initial stage, a global attention regularization module is used to replace the traditional 3D CNN to better capture global contextual information. In all stages, an adaptive guided filter network is introduced to refine the cost volume.

[0082] In step S4, the filtering process of the adaptive guided filter network is specifically as follows:

[0083] In stage 0, the reference map is used as the guide map; in subsequent stages, the depth map generated in the previous stage is used as the guide map; in the initial stage, due to the lack of reliable depth priors, the reference map is used as the guide map; in subsequent stages, the more reliable depth map generated in the previous stage is used as the guide map.

[0084] Calculate the horizontal and vertical gradients of the guide graph:

[0085] ;

[0086] ;

[0087] in, Represents the horizontal gradient; Represents the vertical gradient; This represents a guide diagram. R is a real number, B is the batch size, H represents the number of guide map channels, H represents the guide map height, and W represents the guide map width. This represents the Sobel convolution kernel in the horizontal direction; This represents the Sobel convolution kernel in the vertical direction;

[0088] The edge preservation weights are calculated based on the horizontal and vertical gradients:

[0089] ;

[0090] in, This indicates that the weights are preserved at the edges; Represents the natural exponential function; This indicates the hyperparameter for edge preservation strength control;

[0091] An enhanced guided representation is constructed based on the edge-preserving weights:

[0092] ;

[0093] in, This indicates enhanced guidance;

[0094] For each pixel of the cost volume slice (a two-dimensional plane segmented along the depth dimension, each slice corresponding to a matching cost distribution of a specific depth assumption) of the cost volume. Build an adaptive window:

[0095] ;

[0096] in, This indicates an adaptive window, i.e., using... The central window; Represents the pixel coordinates of an RGB image; Indicating a guide map in The grayscale value at that location; Indicating a guide map in The grayscale value at that location; This represents a continuity threshold (such as depth or color difference), which means dynamically adjusting the filter window based on the continuity of local depth (or color). Indicates the maximum window radius; This indicates taking the maximum value; Indicates taking x and The maximum horizontal distance between them, its y and The maximum vertical distance between them;

[0097] Calculate the linear regression coefficients within the adaptive window:

[0098] ;

[0099] in, as well as All represent linear regression coefficients; Represents the covariance function; Represents the variance function; This represents the k-th adaptive window; Indicating a guide map in Extraction within; Indicates the cost body slice p in Extraction within; This represents the regularization parameter to prevent the denominator from being zero; Indicates the cost body slice p in The mean within; Indicating a guide map in The mean within;

[0100] Based on the enhanced guided representation, each cost body is filtered slice by slice along the depth dimension:

[0101] ;

[0102] in, This represents a slice of the cost volume after filtering. This indicates an adaptive guided filtering operation based on linear regression coefficients; This represents the cost volume slice corresponding to the d-th depth hypothesis; the depth hypothesis is the D candidate depths obtained by discretely sampling the scene depth at each stage, which are used to construct the cost volume and perform matching and regression, taking the 1-D candidate planes.

[0103] The filtered cost volume slices are then weighted and fused:

[0104] ;

[0105] in, This represents a slice of the cost body after weighted fusion; Represents the adaptive mitigation factor. , This represents the basic mitigation factor.

[0106] In step S5, the regularization network is constructed based on a global attention regularization module and a three-dimensional U-shaped module (3D U-Net) based on probabilistic volume geometric embedding; the global attention regularization module is used to perform regularization operations on the cost volume of stage 0, and the three-dimensional U-shaped module is used to perform regularization operations on the cost volume of the remaining stages.

[0107] The adaptive guided filtering network refines the cost volume directly between cascaded stages. It uses a reference map or the depth map of the previous stage as a guide to adaptively suppress noise in smooth regions while maintaining the sharpness of geometric details at edges and thin-walled structures. This achieves "purification" of the cost volume without sacrificing the integrity of key structures, thereby providing higher quality input for subsequent stages and breaking the error propagation chain.

[0108] The adaptive guided filtering network ensures that more details of the original cost volume are preserved at geometric edges, while a stronger smoothing filter is applied to flat regions; achieving the goal of effectively filtering out cost volume noise while maintaining its geometric structure. Compared with traditional fixed kernel filtering, it can adaptively adjust the filtering intensity according to local geometric properties, thereby providing effective denoising in weakly textured regions while maintaining depth discontinuities at object boundaries, providing a higher quality cost volume for subsequent stages, and effectively suppressing error propagation.

[0109] The regularization process of the global attention regularization module is as follows:

[0110] A three-dimensional coordinate grid matching the dimension of the cost volume is generated and normalized. Sine position coding is applied independently to each coordinate axis of the normalized three-dimensional coordinate grid to obtain encoded features. The encoded features are projected through a convolutional layer to obtain projected features. The projected features are added to the cost volume to obtain the projected cost volume. The projected cost volume is input into a 3D convolutional layer with stride for downsampling to obtain a compact cost volume.

[0111] ;

[0112] in, Represents a compact cost body; Represents the original cost body; Represents feature projection; Represents the sine wave position code; Represents a three-dimensional coordinate grid;

[0113] The compact cost volume is flattened along the spatial and depth dimensions to obtain a one-dimensional sequence. The one-dimensional sequence is then input into several Transformer encoders employing a global attention mechanism to calculate the global attention value.

[0114] ;

[0115] in, Q represents the global attention value; K represents the query matrix; V represents the key matrix; and V represents the value matrix. The key vector dimension is represented by T; T represents the transpose.

[0116] This design enables each location in the cost volume to directly interact with all other locations, thereby comprehensively capturing long-distance dependencies and modeling the geometric consistency of the entire 3D scene.

[0117] The global attention value is reshaped into a three-dimensional feature volume, and then the three-dimensional feature volume is upsampled through a 3D transposed convolutional layer to restore the original depth and spatial resolution, resulting in a regularized cost volume.

[0118] To enable the Transformer encoder, which lacks position awareness, to understand the three-dimensional spatial relationships within the cost volume, a three-dimensional position encoding mechanism is introduced. This mechanism encodes the normalized index coordinates of the cost volume to enhance the model's generalization ability.

[0119] In the initial stage of the cascaded network, the global attention regularization module uses an attention mechanism with 3D position encoding to process the cost volume, thereby effectively capturing long-distance dependencies in the entire 3D scene, overcoming the limitations of the local receptive field of traditional 3D CNNs, and fundamentally improving the global consistency and accuracy of the initial cost volume estimation.

[0120] The formula for the adaptive geometric feature fusion network is:

[0121] ;

[0122] in, The depth map is represented by the first... The viewing depth corresponding to each pixel; Indicates a Mamba network; This represents the visual features extracted from the reference image; Indicates a splicing operation; Indicates reliability weight; It represents the Hadamardi (or Hadama) stack; Represents a geometric branching network; Indicates reference image; This represents the depth map obtained from the previous upsampling stage.

[0123] The adaptive geometric feature fusion network can dynamically evaluate the reliability of geometric priors based on the confidence level of the prediction results in the previous stage. According to the reliability of the prior information, it adaptively adjusts its weight in feature fusion, ensuring that only reliable geometric information is used to guide the next stage. This effectively blocks the transmission path of erroneous information and prevents the propagation of erroneous or noisy geometric data to the next stage. This not only suppresses the accumulation and amplification of errors in the cascade process, but also promotes the network to generate a feature representation that is more perceptive of geometric structure and more robust, providing a solid foundation for the subsequent cost matching process.

[0124] The fusion steps of the adaptive geometric feature fusion network are as follows:

[0125] (1) Geometric Prior Reliability Assessment and Weight Generation: The core idea is that not all geometric priors from the previous coarse stage are equally reliable. Therefore, the confidence map is derived using the probability volume generated in the previous stage (stage ℓ) and used as an indicator of prediction quality. Specifically, for any pixel z in the current refinement stage (stage ℓ+1), the upsampled confidence map is input into a small convolutional network g and a sigmoid activation function to generate a reliability weight in the range [0, 1]. The process is represented by the following formula:

[0126] ;

[0127] The reliability weight quantitatively represents the degree to which the geometric prior information at the corresponding location should be trusted. A value close to 1 indicates high confidence and reliable prior information, while a value close to 0 indicates low confidence and unreliable prior information.

[0128] (2) Dual-branch feature extraction, parallel processing of visual appearance information and geometric structure information: Visual feature branch, through a feature pyramid network (FPN) backbone, extracts basic visual features from the reference image; Geometric feature branch (geometric branch network), the input is the reference image and the coarse depth image obtained by upsampling the depth image in the previous stage, used to extract depth-related geometric structure features.

[0129] (3) Gated adaptive feature fusion and enhancement: The core is a gated fusion mechanism controlled by reliability weights. This mechanism uses the reliability weights generated in the first step to dynamically modulate the output of the geometric feature branch through Hadamard product operation. Then, the modulated geometric features and visual features are added element by element and fused. Finally, the fused complete feature representation is sent to the Mamba layer FM of depth perception for processing to capture long-distance dependencies.

[0130] In summary, the advantages of this invention are as follows:

[0131] 1. By acquiring RGB images from multiple perspectives, the intrinsic and extrinsic parameters of the camera acquiring the RGB images are calibrated to obtain the projection matrix and intrinsic parameter matrix, and a reference image for 3D reconstruction is obtained. Then, each RGB image is input into a feature extraction network for feature extraction to obtain image features. Several depth hypothesis planes are obtained by uniformly sampling inverse depth on the reference image. Homography transformation is performed on each image feature and aggregated with the depth hypothesis planes to obtain the cost body. Next, an adaptive guided filtering network is used to filter the cost body, and a regularization network is used to regularize the filtered cost body. The channel dimensions of the regularized cost body are exponentially normalized to obtain a probability body. A depth map and a confidence map are constructed based on the probability body. The depth map... After frequency domain filtering and upsampling of the confidence map, it is input into an adaptive geometric feature fusion network to obtain the view depth. Finally, a 3D reconstruction operation is performed based on each view depth, projection matrix, and intrinsic parameter matrix to obtain a dense point cloud. That is, the adaptive guided filtering network suppresses the propagation of cost volume noise and maintains the geometric edge structure, the global attention regularization module captures long-distance dependencies to enhance the context consistency of the cost volume, and the reliability weight mechanism in the adaptive geometric feature fusion network dynamically evaluates the confidence of cross-stage depth information. Combined with the Mamba network, high-confidence geometric features are selectively fused, thereby effectively blocking error accumulation in complex scenes such as weak texture and occlusion, which greatly improves the accuracy and robustness of 3D reconstruction.

[0132] 2. By introducing an adaptive guided filter network, the accuracy and stability of filtering are effectively improved by dynamically adjusting the guide map (using the reference map in the initial stage and the depth map of the previous stage in subsequent stages) and combining edge-preserving weights and an adaptive window. The edge-preserving weights are based on the gradient calculated by the Sobel operator and control the edge intensity through an exponential function, protecting texture details during the filtering process, reducing edge blurring in reconstruction, and improving point cloud quality. The window size is adaptively adjusted according to pixel grayscale continuity and spatial distance. Compared with fixed window filtering, it can better handle lighting changes and occlusion problems in the scene and enhance robustness to complex environments. In subsequent stages, the depth map of the previous stage is used as the guide map, realizing adaptive iterative optimization, gradually refining the depth estimation, reducing the impact of noise, improving reconstruction consistency, and ultimately solving the problem of traditional guided filtering easily losing details and over-smoothing in 3D reconstruction, thus improving generalization ability and practicality.

[0133] 3. By setting up a regularization network that combines a global attention regularization module and a 3D U-shaped module, the feature representation of the cost volume is optimized. The global attention regularization module uses 3D coordinate grid encoding and Transformer encoder to process one-dimensional sequences, capturing global contextual dependencies (such as the correlation between depth and spatial dimensions), improving feature consistency, and reducing holes or distortions in reconstruction. In the subsequent stage, the 3D U-shaped module is used, combined with downsampling and upsampling operations, to effectively handle local details and multi-scale features, improving the resolution recovery capability of the cost volume. The module design avoids redundant computation (such as downsampling to compact the cost volume), and combines convolution and transposed convolution to reduce computational overhead while maintaining accuracy, making it suitable for real-time 3D reconstruction applications. This dual-module architecture integrates global and local regularization, overcoming the shortcomings of single regularization methods (such as pure CNN) in balancing global consistency and local details, and improving the density and accuracy of reconstruction.

[0134] 4. The adaptive geometric feature fusion network utilizes the Mamba network to combine visual and geometric features, achieving efficient feature fusion. As a state-space model, Mamba, compared to traditional RNNs or Transformers, has linear computational complexity and long sequence processing capabilities, enabling efficient fusion of multi-view features and improving the estimation efficiency of depth maps. By combining concatenation operations (⊕) and Hadamard product (⊙) with reliability weights, the geometric branch output is dynamically weighted, enhancing robustness to unreliable regions (such as occlusion or low texture) and reducing reconstruction errors. Frequency domain filtering and upsampling are performed before fusion to preprocess the depth map, ensuring the quality of input features and further improving the accuracy of view depth. This fusion mechanism solves the problem of insufficient fusion of visual and geometric features in traditional methods. Combined with the advanced architecture of Mamba, it significantly improves the accuracy and speed of depth estimation.

[0135] 5. A complete end-to-end process is formed from image acquisition to point cloud generation, with key steps employing iterative design. It integrates feature extraction, cost volume construction, filtering, regularization, and fusion, reducing manual intervention and achieving a high degree of automation, making it suitable for large-scale 3D reconstruction scenarios. Multiple stages (such as filtering and fusion) use the output of the previous stage (such as depth maps as guide maps) to achieve progressive optimization, gradually improving the quality of depth maps and enhancing the method's adaptability to noise and viewpoint changes. By exponentially normalizing the cost volume to obtain the probability volume, combined with the confidence map, reliable depth uncertainty estimation is provided, improving point cloud density and reliability. This process design enhances the robustness and scalability of the method, reduces the risk of error accumulation compared to staged independent algorithms, and is easy to integrate into existing systems.

[0136] 6. Based on standard RGB images and camera parameters (intrinsic / extrinsic parameters), no special hardware (such as depth sensors) is required, and it has strong compatibility; inverse depth uniform sampling and homography transformation simplify the cost volume construction and are easy to implement; through adaptive filtering and Mamba fusion, it may surpass the traditional multi-view stereo (MVS) method in terms of accuracy (edge ​​preservation) and speed (Mamba efficiency), and is suitable for high dynamic scenes.

[0137] 7. By innovatively integrating an adaptive guided filtering mechanism, an efficient regularization network, and the Mamba geometric feature fusion framework, and by dynamically adjusting the guide map (reference map → depth map iterative optimization) combined with edge preservation weights and adaptive windows, the ability to preserve reconstruction edge details and scene adaptability are significantly improved. By utilizing the dual-path design of the global attention regularization module and the 3D U-shaped module, the global consistency features and local detail recovery are balanced, reducing computational overhead. By leveraging the efficient long sequence processing capability of the Mamba network, visual features and weighted geometric information are fused to effectively solve the depth estimation problem in occluded and low-texture areas, ultimately achieving end-to-end high-precision and robust 3D reconstruction from multi-view RGB images to dense point clouds.

[0138] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for three-dimensional reconstruction based on guided filtering and Mamba geometric feature fusion, characterized in that: The method comprises the following steps: Step S1, acquiring multi-view RGB images, calibrating the intrinsic and extrinsic parameters of a camera for acquiring the RGB images to obtain a projection matrix and an intrinsic matrix, and obtaining a reference graph for three-dimensional reconstruction; Step S2, inputting each RGB image into a feature extraction network for feature extraction to obtain image features; Step S3, obtaining a plurality of depth hypothesis planes by inversely and uniformly sampling the reference graph, performing homography transformation on each image feature, and aggregating the depth hypothesis planes to obtain a cost volume; Step S4, performing a filtering operation on the cost volume through an adaptive guided filtering network; the filtering process of the adaptive guided filtering network is specifically as follows: In the 0th stage, the reference graph is used as a guide graph; in subsequent stages, a depth map generated in a previous stage is used as a guide graph; horizontal and vertical gradients of the guide graph are calculated; an edge preservation weight is calculated based on the horizontal and vertical gradients; an enhanced guide representation is constructed based on the edge preservation weight; for each pixel of a cost volume slice of the cost volume, an adaptive window is constructed; a linear regression coefficient is calculated in the adaptive window; each cost volume is filtered along the depth dimension based on the enhanced guide representation; the filtered cost volume slices are fused by weighting; Step S5, performing a regularization operation on the cost volume after the filtering operation through a regularization network; Step S6, performing exponential normalization on the channel dimension of the cost volume after the regularization operation to obtain a probability volume, and constructing a depth map and a confidence map based on the probability volume; Step S7, after performing frequency domain filtering and upsampling on the depth map and the confidence map, inputting the depth map and the confidence map into an adaptive geometric feature fusion network for feature fusion to obtain a view depth; the formula of the adaptive geometric feature fusion network is as follows: ； wherein, represents the depth of a pixel in the depth map; represents the depth of a pixel in the depth map; represents the Mamba network; represents the visual features extracted from the reference image; represents the concatenation operation; represents the reliability weight; represents the Hadamard product; represents the geometry branch network; represents the reference image; represents the depth map obtained by upsampling the previous stage. Step S8, performing a three-dimensional reconstruction operation based on each view depth, the projection matrix, and the intrinsic matrix to obtain a dense point cloud.

2. The method of claim 1, wherein the method is a method of 3D reconstruction based on fusion of guided filtering and Mamba geometric features. In the step S5, the regularization network is constructed based on a global attention regularization module and a three-dimensional U-shaped module; the global attention regularization module is used to perform a regularization operation on the cost volume in the 0th stage, and the three-dimensional U-shaped module is used to perform a regularization operation on the cost volume in the remaining stages.

3. The method of claim 2, wherein the method is based on fusing guided filtering and Mamba geometric features for 3D reconstruction. The regularization process of the global attention regularization module is specifically as follows: a three-dimensional coordinate grid matching the dimension of the cost volume is generated and normalized, an encoding feature is obtained by independently applying sinusoidal position encoding to each coordinate axis of the normalized three-dimensional coordinate grid, a projection feature is obtained by projecting the encoding feature through a convolution layer, the projection feature is added to the cost volume to obtain a projected cost volume, and the projected cost volume is input into a 3D convolution layer with a stride to downsample the projected cost volume to obtain a compact cost volume; the compact cost volume is flattened along the spatial dimension and the depth dimension to obtain a one-dimensional sequence, and the one-dimensional sequence is input into a plurality of Transformer encoders adopting a global attention mechanism to calculate a global attention value; The global attention value is reshaped into a three-dimensional feature body, and a 3D transposed convolution layer is used to up-sample the three-dimensional feature body to restore the original depth and spatial resolution, so as to obtain a regularized cost volume.

Images