Degradation guidance and Mama enhancement-based depth map super-resolution reconstruction method

Abstract

The invention discloses a depth map super-resolution reconstruction method based on degradation guidance and Mama enhancement, and aims to solve the problems that a long-range pixel dependency relationship is difficult to capture when a large-scale zoom factor is processed in an existing depth super-resolution method, and artifacts are caused by noise and structure degradation interference in a real scene. A guide state space module and a degradation guide fusion mechanism are introduced. The GSS module captures long-range dependence by fusing RGB and depth features, realizes robust reconstruction of a missing region and inhibits noise; the DGF module selectively extracts geometric features highly related to depth from the RGB image by dynamically evaluating a degradation mode of the input depth map, and inhibits texture copy artifacts. According to the method, global geometric consistency recovery under large-scale zooming and robust feature integration under complex degradation are realized, and finally, high-quality and high-precision depth map reconstruction is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision and depth image processing technology, specifically to a depth map super-resolution reconstruction method based on degradation guidance and Mamba enhancement. Background Technology

[0002] Depth super-resolution (DSR) technology, a core task in computer vision, aims to reconstruct high-resolution (HR) depth maps with high-frequency structure and precise edges using algorithms from low-resolution (LR) depth maps and their simultaneously acquired color guide images (RGB). This technology plays a crucial role in fields such as 3D reconstruction, robotics, and mixed reality.

[0003] However, in current engineering practice and academic research, existing DSR technologies still face the following irreconcilable technical bottlenecks when dealing with complex real-world scenarios:

[0004] (1) The contradiction between local receptive field and large scaling in long-range modeling: Most mainstream DSR frameworks are based on convolutional neural networks (CNNs). The natural property of convolutional kernels is local connectivity, which means that the model can only capture geometric features within the local neighborhood. However, when scaling up by a large scale (such as ×32 or ×64), pixels that are adjacent in the LR space will be pushed to a very large distance in the HR space. Since CNNs lack an effective global pixel interaction mechanism, the reconstruction results often perform poorly in terms of global consistency, resulting in distortion or severe blurring of large-scale geometric contours. Although Transformer-based architectures have been introduced in recent years to try to alleviate this problem by using global self-attention, the computational cost of Transformers increases quadratically with the length of the input sequence. When processing the high-definition feature maps required for DSR tasks, it will generate extremely high memory pressure and inference latency, making it difficult to meet the real-time requirements of edge computing devices.

[0005] (2) Real-world scene degradation interferes with and disrupts cross-modal guidance: In non-ideal experimental environments, LR depth maps acquired by depth sensors (such as ToF or structured light cameras) are often accompanied by complex non-stationary degradation, including sensor thermal noise, structural voids caused by ambient light interference, and motion blur during acquisition. Existing multimodal fusion schemes typically assume that RGB images and depth maps are structurally strictly aligned and consistent. However, this assumption no longer holds when the depth map suffers severe degradation. If the algorithm blindly transfers gradient information from RGB images to the depth reconstruction process, information unrelated to geometric depth, such as color textures and light and shadow edges in the RGB images, will be incorrectly introduced into the depth domain, producing so-called "texture copying" artifacts. This not only reduces the accuracy of the depth map but also leads to misjudgments in downstream 3D perception tasks (such as obstacle avoidance).

[0006] (3) Disconnect between degradation information perception and feature enhancement: Existing studies often treat degradation processing and feature reconstruction as two independent stages, or simply use a general denoising module. This approach fails to delve into the "regional reliability" information implicit in the degradation pattern itself. For example, noisy regions should reduce their reliance on LR depth maps and increase their careful reference to RGB structural cues, while structurally intact regions should retain their original geometric details. Due to the lack of a closed-loop mechanism that can dynamically assess the degree of degradation and guide feature fusion in real time, it is difficult for the model to balance robustness and reconstruction quality.

[0007] To address the aforementioned challenges, structured state-space models (SSMs, such as Mamba) have demonstrated linear efficiency in capturing global long-range dependencies. However, effectively combining them with the degradation features unique to depth maps to construct a super-resolution framework that can both guarantee computational efficiency and accurately suppress texture artifacts is a technological hurdle that urgently needs to be overcome in the field of DSR. Summary of the Invention

[0008] The purpose of this invention is to propose a depth map super-resolution reconstruction method based on degradation-guided fusion and Mamba enhancement. Addressing the issues of long-range pixel dependency loss due to local receptive field limitations when handling large scaling factors, and edge blurring and artifacts caused by RGB texture interference in realistic degradation environments, this invention introduces a degradation-guided fusion (DGF) module and a guided state space (GSS) module. This achieves global geometric consistency restoration under large-scale scaling and robust feature integration under complex degradation, ultimately realizing high-quality, high-precision depth map reconstruction.

[0009] The technical solution for implementing this invention is a depth map super-resolution reconstruction method based on degradation guidance and Mamba enhancement, comprising the following steps:

[0010] Step 1: Obtain a low-resolution depth map and the corresponding color RGB image ; through bicubic interpolation pairs Pre-upsampling was performed to obtain ;

[0011] Step 2, Input the degradation learning module to generate implicit degradation representations. It is used to capture structural damage and noise priors in low-resolution depth maps;

[0012] Step 3: Construct a degradation-guided fusion module to... As a gating guide signal, features consistent with the depth geometry are dynamically evaluated and extracted from RGB image features. ;

[0013] Step 4: Construct a guided state space module to extend depth features to 2D space. Employ a multi-directional selective scanning mechanism to capture global long-range pixel dependencies, and combine this with the results obtained in Step 3. To refine and enhance the structure;

[0014] Step 5: Iterate through steps 3 and 4 together. Next, the quality of deep features is gradually improved by using degraded priors and RGB information to obtain the final enhanced features;

[0015] Step 6: Process the final enhanced depth features through a reconstruction layer to output a high-resolution depth map. .

[0016] Furthermore, in step 1, a dual-branch feature extraction path is constructed for the low-resolution depth map and the color guide map. The color path is used to provide high-frequency structural information, while the depth path is used to preserve the geometric topology.

[0017] Furthermore, the implicit degenerate representation predicted in step 2 With characteristics Figure 1 The spatial dimension is used to guide the adaptive allocation of multimodal features at the pixel level and suppress erroneous texture migration caused by degenerate regions.

[0018] Furthermore, step 3, the specific process of degradation-guided feature fusion, includes: converting the implicit degradation representation... With input RGB features Perform cascading and extract descriptors through pooling. ;Will Input weights to generate branches and calculate weight coefficients. and Calculate the weighted fusion features according to the following formula. :

[0019] .

[0020] Furthermore, step 3 also includes generating deep correlation features using a degenerate gating mechanism: generating deep correlation features using the following degenerate gating formula. :

[0021] ;

[0022] in For gated functions, For weighted fusion features; utilizing As a gating signal, it regulates the information flow and suppresses non-depth-related texture interference.

[0023] Furthermore, the multi-directional selective scanning mechanism described in step 4 specifically involves: flattening the two-dimensional feature map along four directions: horizontal left and right, horizontal right and left, vertical up and down, and vertical down and up, to construct four independent one-dimensional sequences; performing selective scanning on the four sequences in parallel to capture the long-range pixel dependencies of the entire image.

[0024] Furthermore, the linear complexity modeling process described in step 4 is as follows: The sequence is modeled using discretized state-space parameters, and the discretization calculation code is as follows:

[0025] ; .

[0026] in, and These are the discretized state transition matrix and input matrix, used to update the latent state sequence. ΔA is the product of the discretization step size and the state transition matrix, and ΔB is the product of the discretization step size and the input matrix.

[0027] Furthermore, the feature refinement process in step 4 includes: utilizing... Computational Spatial Attention Map The following attention-guided fusion formula is used to enhance features. To merge: ;in, It is a two-dimensional convolutional layer.

[0028] Furthermore, the reconstruction process in step 6 is as follows: the enhanced depth features after the final iteration are input into the reconstruction convolutional layer and mapped back to the original image space; using... The loss function calculates the absolute error between the predicted depth map and the true high-resolution depth map, and the parameters of the entire network are optimized through an end-to-end backpropagation algorithm.

[0029] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0030] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the above-described method.

[0031] A computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0032] Compared with the prior art, the significant advantages of this invention are: (1) This invention solves the texture duplication artifact problem in RGB guidance by using the Degenerate Guided Fusion (DGF) module: Compared with the traditional method of blindly fusing RGB features, this invention uses the learned implicit degenerate representation as a "filter" to dynamically select features that are highly related to the geometry of the depth map, thereby effectively suppressing structural errors caused by modality mismatch while utilizing the high-frequency information of RGB. (2) This invention breaks through the receptive field bottleneck of convolutional neural networks by using the Guided State Space (GSS) module: Compared with CNN, the GSS module achieves direct pixel interaction across the entire map range by using a multi-scan mechanism while maintaining linear computational complexity. Even at a maximum scaling factor of ×64, it can accurately recover missing large geometric structures through long-range dependencies, greatly improving the reconstruction accuracy of the model in complex scenes.

[0033] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the process of the present invention.

[0035] Figure 2 This is a diagram of the overall network framework of DegMamba in this invention (where the lower left corner is the Degenerate Bootstrap Fusion (DGF) module and the lower right corner is the Bootstrap State Space (GSS) module).

[0036] Figure 3 This is a schematic diagram illustrating the principle of the four-way selective scanning mechanism in this invention.

[0037] Figure 4 This is a visual comparison of the reconstruction results of this invention on the NYU-v2 dataset.

[0038] Figure 5This is a visual comparison of the reconstruction results of this invention on a real-world RGB-DD scene dataset.

[0039] Figure 6 This is a visual comparison of the reconstruction results of this invention on the Lu benchmark dataset. Detailed Implementation

[0040] This invention proposes a Degradation-Guided and Mamba-Enhanced Depth Map Super-Resolution Reconstruction Method (DegMamba). Addressing the shortcomings of existing Depth Super-Resolution (DSR) methods in capturing long-range pixel dependencies under large scaling factors and their susceptibility to noise and structural degradation artifacts in real-world scenarios, this invention introduces a Guided State Space (GSS) module and a Degradation-Guided Fusion (DGF) mechanism. The GSS module captures long-range dependencies by fusing RGB and depth features, achieving robust reconstruction of missing regions and suppressing noise. The DGF module selectively extracts depth-related geometric features from the RGB image by dynamically evaluating the degradation mode of the input depth map, suppressing texture duplication artifacts. Experiments on the NYUv2 dataset show that this invention reduces the RMSE by 21.0% compared to suboptimal methods at ×64 scaling.

[0041] As shown in Figures 1 and 2, a depth map super-resolution reconstruction method based on degradation guidance and Mamba enhancement achieves the repair and structural enhancement of degraded regions in the depth map by constructing a degradation-aware path and modeling the global state space. Given a low-resolution depth map... With color guide image First, implicit degenerate representations are predicted through pre-upsampling and a degradation learning branch. ; Process the input signals of the Degradation Guided Fusion (DGF) module, utilizing Dynamically generate weighted RGB information to extract depth-related features. ; Process the input sequence of the Guided State Space (GSS) module, capturing long-range pixel dependencies through four-way selective scanning; Utilize Spatial attention weights are constructed, residuals are refined on the state space features, and finally a high-resolution depth map is output through the reconstruction layer. The specific steps are as follows:

[0042] Step 1, Data Preprocessing and Branch Construction: (e.g.) Figure 2 As shown, for a given low-resolution depth map First, a bicubic interpolation algorithm is used for pre-upsampling to obtain a preliminary prediction map. This ensures that its spatial dimensions match those of the target high-resolution image. Simultaneously, it provides a color guide image. Construct a feature extraction branch and use a convolutional neural network to extract multi-scale RGB features. This provides high-frequency detail references for subsequent geometric repair.

[0043] Step 2, Degenerate Representation Learning: As shown in Figure 2, ... Input the degradation learning branch. This branch uses a cascaded residual block structure, trained in a supervised or self-supervised manner, and outputs an implicit degradation representation. . As a pixel-level map, it can accurately quantify the degree of degradation (such as noise intensity or hole distribution) at each location in the input depth map. This degradation representation will serve as a guiding signal for all subsequent fusion modules.

[0044] Step 3, Degeneracy-Guided Feature Fusion (DGF): such as Figure 2 The bottom left corner shows the internal logic of the DGF module. and The descriptors are obtained by cascading along the channel dimension and passing through a global average pooling layer. . use Calculate depth prior weights With RGB feature weights The final output is depth-related features. The calculation equation is defined as follows:

[0045]

[0046] in This is a gating function that uses a degenerate prior to dynamically suppress RGB texture copying artifacts.

[0047] Step 4, Guided State-Space Modeling (GSS): For example... Figure 2 As shown in the bottom right corner, the GSS module receives deep features from the previous layer. To capture global dependencies, it employs... Figure 3 The illustrated four-way scanning mechanism flattens the two-dimensional features along four directions. Each sequence is selectively scanned using Mamba's core S6 operator, and the state equation and output equation are defined as follows:

[0048]

[0049] The discretization parameter matrix and It is expressed as follows:

[0050]

[0051] in For the parameter matrix, , , Both are parameter matrices, and Is using , , Based on the discrete parameter matrix obtained using the zero-order preservation technique, ΔA is the product of the discretization step size and the state transition matrix, and ΔB is the product of the discretization step size and the input matrix. This process achieves long-range interaction of all pixels in the image with linear computational complexity.

[0052] Step 5, Feature Refinement and Attention Enhancement: Utilize the features obtained in Step 3. A spatial weight map is generated after the activation function layer. Combine w with the state-space augmentation features output from step 4. Element-wise product operations are performed to recalibrate the features. Finally, the output of the i-th layer encoder is obtained through residual connections. :

[0053]

[0054] in This represents a two-dimensional convolution operation used for feature transformation. This represents the depth feature at the (i-1)th iteration. This formula ensures that the model can effectively suppress structural discontinuities caused by degradation while enhancing edge details.

[0055] Step 6: Reconstruction and Optimization: The final enhanced features after multiple iterations are input into the reconstruction layer, and feature mapping is performed through sub-pixel convolution. During the training phase, the following steps are taken: The loss function constrains the predicted values ​​and the true values:

[0056]

[0057] in This represents the predicted depth map. Represents the true depth map.

[0058] By optimizing network parameters through end-to-end training, the model learns the mapping relationship from degraded low-resolution depth maps to high-quality depth maps.

[0059] This embodiment verifies the effectiveness of the proposed DegMamba framework through quantitative and qualitative experiments on multiple publicly available benchmark datasets. We first evaluate performance at normal scaling ratios, and then conduct extreme scaling tests to address the challenge of extremely low-resolution inputs. Notably, throughout all tests, this invention employs a uniform network configuration, demonstrating strong technical consistency.

[0060] (1) Evaluation of quantitative results under multiple datasets

[0061] As shown in Table 1 below, this invention compares several existing state-of-the-art depth map super-resolution methods, including DJF, CUNet, FDKN, RSAG, and the latest DORNet. We evaluated the RMSE metric on four core benchmark datasets: NYU-v2, RGB-DD, Lu, and Middlebury.

[0062] Table 1. Comparison of RMSE (×4 / ×8 / ×16) of different methods on multiple datasets.

[0063]

[0064] Overall, this invention ranks first across all test datasets and scaling factors. For example, on the ×16 task of the NYU-v2 dataset, the RMSE score of this invention is approximately 7.5% lower than the latest comparable method DORNet in 2025. Figures 4-6 The presentation shows a visual comparison of different methods on real and synthetic datasets, demonstrating that the object edges and contours restored by the present invention are more accurate and the content is clearer.

[0065] (2) Performance advantages under large scaling factors

[0066] In addition to the conventional scaling factors mentioned above, to verify the model's ability to recover extremely missing information, we compared our invention with previous state-of-the-art methods at large scaling factors (×32, ×64). Table 2 reports the numerical results.

[0067] Table 2. Comparison of RMSE for different methods under large scaling factors.

[0068]

[0069] At extremely challenging scaling ratios, this invention achieves an error of only 15.53 on NYU-v2, representing a 21.0% performance improvement over the suboptimal method DORNet (19.66). It also maintains best performance on the RGB-DD real-world degradation dataset. This is attributed to the superior long-range modeling capabilities of the Guided State Space (GSS) module and the effective suppression of cross-modal interference by the Degradation Guided Fusion (DGF) module.

[0070] This invention utilizes a guided state space module and a degenerate guided fusion mechanism to improve feature extraction efficiency while effectively solving the problems of geometric distortion and texture artifacts under large scaling ratios, thus significantly improving the performance of depth map super-resolution reconstruction.

[0071] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A depth map super-resolution reconstruction method based on degradation guidance and Mamba enhancement, characterized in that, Includes the following steps: Step 1: Obtain a low-resolution depth map and the corresponding color RGB image ; through bicubic interpolation pairs Pre-upsampling was performed to obtain ; Step 2, Input the degradation learning module to generate implicit degradation representations. It is used to capture structural damage and noise priors in low-resolution depth maps; Step 3: Construct a degradation-guided fusion module to... As a gating guide signal, features consistent with the depth geometry are dynamically evaluated and extracted from RGB image features. ; Step 4: Construct a guided state space module to extend depth features to 2D space. Employ a multi-directional selective scanning mechanism to capture global long-range pixel dependencies, and combine this with the results obtained in Step 3. To refine and enhance the structure; Step 5: Iterate through steps 3 and 4 together. Next, the quality of deep features is gradually improved by using degraded priors and RGB information to obtain the final enhanced features; Step 6: Process the final enhanced depth features through a reconstruction layer to output a high-resolution depth map. .

2. The method according to claim 1, characterized in that, Step 3, the specific process of degradation-guided feature fusion includes: converting the implicit degradation representation... With input RGB features Perform cascading and extract descriptors through pooling. ;Will Input weights to generate branches and calculate weight coefficients. and Calculate the weighted fusion features according to the following formula. : 。 3. The method according to claim 2, characterized in that, Step 3 also includes generating deep correlation features using a degenerate gating mechanism: generating deep correlation features using the following degenerate gating formula. : ； use As a gating signal, it regulates the information flow and suppresses non-depth-related texture interference.

4. The method according to claim 1, characterized in that, Step 4 The multi-directional selective scanning mechanism is as follows: flatten the two-dimensional feature map along four directions: horizontal left and right, horizontal right and left, vertical up and down, and vertical down and up, to construct four independent one-dimensional sequences; perform selective scanning on the four sequences in parallel to capture the long-range pixel dependencies of the whole image.

5. The method according to claim 4, characterized in that, Step 4 describes the linear complexity modeling process as follows: The sequence is modeled using discretized state-space parameters, and the discretization calculation code is as follows: ； ； in, and These are the discretized state transition matrix and input matrix, used to update the latent state sequence. ΔA is the product of the discretization step size and the state transition matrix, and ΔB is the product of the discretization step size and the input matrix.

6. The method according to claim 1, characterized in that, Step 4, the feature refinement process, includes: utilizing... Computational Spatial Attention Map The following attention-guided fusion formula is used to enhance features. To merge: ;in, It is a two-dimensional convolutional layer.

7. The depth map super-resolution reconstruction method based on degradation guidance and Mamba enhancement according to claim 1, characterized in that, Step 6 describes the reconstruction process as follows: The enhanced depth features from the final iteration are input into the reconstruction convolutional layer and mapped back to the original image space; using... The loss function calculates the absolute error between the predicted depth map and the true high-resolution depth map, and the parameters of the entire network are optimized through an end-to-end backpropagation algorithm.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the depth map super-resolution reconstruction method based on degradation bootstrapping and Mamba enhancement as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the depth map super-resolution reconstruction method based on degradation bootstrapping and Mamba enhancement as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the depth map super-resolution reconstruction method based on degradation guidance and Mamba enhancement as described in any one of claims 1-7.

Images