Depth Map Super-Resolution Methods and Systems Based on Mamba and Variants

Abstract

This invention proposes a depth map super-resolution method and system based on Mamba and its variants. The method includes: processing a high-resolution image dataset to obtain a high-resolution texture image and a low-resolution depth image; extracting shallow features from the high-resolution texture image and the low-resolution depth image to obtain shallow features of the texture image and shallow features of the depth image; extracting deep features from the shallow features of the texture image and the shallow features of the depth image to obtain deep guided features and deep depth features; fusing the deep guided features and deep depth features to obtain high-resolution guided features and low-resolution depth features; and processing the high-resolution guided features and low-resolution depth features to obtain a super-resolution reconstructed image. This invention effectively achieves cross-modal information interaction when facing the requirement of image token and depth token fusion.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and in particular to a depth map super-resolution method and system based on Mamba and its variants. Background Technology

[0002] Depth maps, as key data encoding the geometry of 3D scenes, along with RGB-D data and point clouds formed by 2D color images, are widely used in fields such as autonomous driving, 3D human pose estimation, and immersive virtual reality. However, depth maps acquired by commercial sensors generally suffer from low resolution (LR) and noise issues, showing a significant quality gap compared to high resolution (HR) color images. Therefore, recovering HR depth maps from LR depth maps has become a critical requirement. Guided Depth Map Super-Resolution (GDSR), with its advantage of "utilizing the semantic correlation of HR texture images to assist depth recovery," has become the mainstream technical approach for this task.

[0003] With the rapid development of GDSR technology driven by deep learning, Convolutional Neural Networks (CNNs) have achieved significant breakthroughs in this task due to their hierarchical feature representation capabilities. Related research has optimized performance through multi-scale fusion, residual learning, batch normalization, and the introduction of traditional joint filtering priors. However, CNNs have fundamental limitations: content-independent convolutional kernels can only capture local receptive fields, making it difficult to model long-range dependencies in depth maps. Furthermore, simply stacking layers cannot efficiently expand the effective receptive field, limiting the global structure reconstruction effect of HR depth maps. Inspired by natural language processing, Transformers provide a global receptive field for GDSR through content-aware attention mechanisms and can achieve cross-modal feature alignment through cross-attention. Related research has further optimized performance through local Transformers and neural implicit representations. However, the computational complexity of Transformers is quadratic with the number of tokens (O(N²), which is insufficient for the dense pixel modeling scenarios that GDSR needs to handle; while local Transformers reduce complexity, they sacrifice the ability to model non-local dependencies.

[0004] In recent years, state-space models (SSMs) have attracted attention in the field of computer vision due to their combination of global receptive field and linear computational complexity (O(N)), providing a new direction for balancing "efficiency and global modeling" in GDSR tasks. As the latest variant of SSM, Mamba has shown potential in high-level vision tasks, and some studies have extended it to the field of image restoration. However, the application of Mamba in GDSR has significant limitations: (1) it only supports single input sequence processing and cannot efficiently fuse cross-modal guidance information of depth and color (a core requirement of GDSR); (2) due to the high computational cost of large token dimensions, it only uses spatial dimension scanning and ignores inter-channel interaction, resulting in channel redundancy. Summary of the Invention

[0005] In view of the above, the main objective of this invention is to propose a depth map super-resolution method based on Mamba and its variants to solve the aforementioned technical problems.

[0006] This invention proposes a depth map super-resolution method based on Mamba and its variants, the method comprising the following steps:

[0007] Step 1: Input the high-resolution image dataset into the reconstruction model, and process the high-resolution image dataset through the image processing module to obtain high-resolution texture images and low-resolution depth images;

[0008] Step 2: Input the high-resolution texture image and the low-resolution depth image into the encoder module to extract shallow features, so as to obtain shallow features of the texture image and shallow features of the depth image.

[0009] Step 3: Use the encoder module to extract deep features from the shallow features of the texture image and the shallow features of the depth image to obtain deep guiding features and deep depth features;

[0010] Step 4: Use the encoder module to fuse the deep guidance features and deep depth features to obtain high-resolution guidance features and low-resolution depth features.

[0011] Step 5: Input the high-resolution guiding features and low-resolution depth features into the decoder module for processing to obtain the super-resolution reconstructed image.

[0012] This invention also proposes a depth map super-resolution system based on Mamba and its variants, the system comprising:

[0013] The data processing module is used for:

[0014] High-resolution image datasets are processed to obtain high-resolution texture images and low-resolution depth images;

[0015] Encoder module, used for:

[0016] Shallow feature extraction is performed on high-resolution texture images and low-resolution depth images to obtain shallow features of texture images and shallow features of depth images;

[0017] Deep features are extracted from shallow features of texture images and shallow features of depth images to obtain deep guiding features and deep depth features;

[0018] Deep guidance features and deep depth features are fused to obtain high-resolution guidance features and low-resolution depth features;

[0019] Decoder module, used for:

[0020] High-resolution guiding features and low-resolution depth features are processed to obtain super-resolution reconstructed images.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] 1. This invention combines the explicit attention mechanism of Transformer with the implicit state propagation of Mamba, which not only avoids the quadratic complexity of Transformer by leveraging the linear computational complexity of Mamba, but also makes up for the lack of local modeling in Mamba by using Transformer, thus realizing global dependency capture and fine-grained local interaction.

[0023] 2. This invention extends the scanning mode of Mamba from "spatial only" to "channel-spatial dual-dimensional" through channel Mamba, solving the problem that traditional Mamba only scans in space and ignores the interaction between channels;

[0024] 3. This invention overcomes the shortcomings of traditional Mamba, which only processes a single input sequence, by using cross-Mamba. When faced with the need for image token and depth token fusion, it can effectively realize cross-modal information interaction. Attached Figure Description

[0025] Figure 1 This is a flowchart of the depth map super-resolution method based on Mamba and its variants proposed in this invention;

[0026] Figure 2 This is a schematic diagram of the overall framework of the depth map super-resolution method based on Mamba and its variants proposed in this invention.

[0027] Figure 3 This is a schematic diagram of the cross-Mamba module of the depth map super-resolution method based on Mamba and its variants proposed in this invention;

[0028] Figure 4 This is a schematic diagram of the channel Mamba module of the depth map super-resolution method based on Mamba and its variants proposed in this invention;

[0029] Figure 5 This is a schematic diagram of the framework of the depth map super-resolution system based on Mamba and its variants proposed in this invention. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0031] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0032] Please see Figure 1 This invention proposes a depth map super-resolution method based on Mamba and its variants, which includes the following steps:

[0033] Step 1: Input the high-resolution image dataset into the reconstruction model, and process the high-resolution image dataset through the image processing module to obtain high-resolution texture images and low-resolution depth images;

[0034] In step 1, the high-resolution image dataset is processed by the image processing module to obtain a high-resolution texture image and a low-resolution depth image. The specific steps are as follows:

[0035] The input high-resolution image data is divided into training, validation, and test sets.

[0036] The image processing module crops the high-resolution image dataset into pairs of image blocks according to the preset image size, and then performs random flipping and rotation data enhancement processing to obtain pairs of high-resolution texture images and high-resolution depth real images.

[0037] A low-resolution depth image is obtained by bicubic interpolation of a high-resolution real depth image.

[0038] Furthermore, in this step, the scale of the low-resolution depth image is x4, x8, x16.

[0039] Step 2: Input the high-resolution texture image and the low-resolution depth image into the encoder module to extract shallow features, so as to obtain shallow features of the texture image and shallow features of the depth image.

[0040] Please see Figure 2 In step 2, the high-resolution texture image and the low-resolution depth image are input into the encoder module for shallow feature extraction to obtain shallow features of the texture image and shallow features of the depth image. The specific steps are as follows:

[0041] The encoder module consists of a guided feature extraction branch, a deep super-resolution branch, and a bidirectional fusion module. The guided feature extraction branch and the deep super-resolution branch have the same structure, each consisting of a convolutional layer and four deep feature extraction modules (STM modules).

[0042] The high-resolution texture image is input into the guided feature extraction branch and processed by a convolutional layer with a kernel size of 3×3 to obtain shallow features of the texture image.

[0043] The low-resolution depth image is input into the depth super-resolution branch and processed by a convolutional layer with a kernel size of 3×3 to obtain shallow features of the depth image.

[0044] Step 3: Use the encoder module to extract deep features from the shallow features of the texture image and the shallow features of the depth image to obtain deep guiding features and deep depth features;

[0045] In step 3, the encoder module is used to extract deep features from the shallow features of the texture image and the shallow features of the depth image to obtain deep guiding features and deep depth features. The specific steps are as follows:

[0046] The deep feature extraction module in the encoder module contains a Transformer group and a VMamba group; the Transformer group contains Swing Transformer and Restormer; the VMamba group contains VMamba and channel VMamba.

[0047] The shallow features of the texture image are processed sequentially by the Swing Transformer and the Restormer, and then residually connected with the shallow features of the texture image to obtain the output of the Transformer group.

[0048] The output of the Transformer group is processed by VMamba and then by channel VMamba in sequence. Finally, it is residually concatenated with the output of the Transformer group to obtain the output of the VMamba group. The output of the VMamba group is then used as the output of the deep feature extraction module.

[0049] In the guided feature extraction branch, the output of the deep feature extraction module is used as the deep guided feature;

[0050] The deep guidance features of the previous layer are used as the input of the deep guidance features of the next layer. The features are processed by four deep feature extraction modules to obtain the first, second, third and fourth deep guidance features.

[0051] In the depth super-resolution branch, the shallow features of the depth image are processed in the same way as the shallow features of the texture image to obtain the first, second, third and fourth deep depth features.

[0052] After processing the shallow features of the texture image sequentially through the Swing Transformer and Restormer, and then performing a residual connection with the shallow features of the texture image to obtain the output of the Transformer group, the following relationship exists:

[0053] ;

[0054] in, This represents the output of the Transformer group. This indicates that it will be processed through Restormer. This indicates that it is processed using a Swing Transformer; This represents the input to the Transformer group, which in this step consists of shallow features of the texture image and shallow features of the depth image;

[0055] After processing the output of the Transformer group sequentially through VMamba and then through channel VMamba, and finally performing a residual join with the output of the Transformer group to obtain the output of the VMamba group, the following relationship exists:

[0056] ;

[0057] in, This indicates the output of the VMamba group. This indicates processing via VMamba channel. This indicates that it is processed via VMamba.

[0058] Furthermore, in this step, the Swin Transformer adopts a Transformer structure with a shift window. This structure can perform effective self-attention calculation within a local window and expand the receptive field through shift operations, thereby better capturing local feature information in the image. The Restormer has powerful global feature extraction capabilities, which can further enrich the feature representation.

[0059] The combination of Swin Transformer and Restormer performs preliminary processing of input features through residual connections, providing more representative features for subsequent VMamba groups.

[0060] For further details, please refer to Figure 4In this step, the channel VMamba includes a channel Mamba module, which receives intermediate features in this process, through a variable-size... The space is divided into windows, each window is regarded as a "super token", and the original features are reshaped through a reshaping operation. Convert to , among them For the new channel dimension, The new sequence length; where, The height of the intermediate feature. The width of the intermediate feature. The number of channels for intermediate features;

[0061] Next, the reshaped "super token" features are flattened into a one-dimensional sequence, and the traditional spatial dimension scan of Mamba is changed to a channel dimension scan, according to... The sequence length is processed element-wise to model channel dependencies, while also considering different... The generated sequence is then aligned to the specified length using an MLP layer. The length is adjusted to accommodate multi-scale processing needs;

[0062] Finally, the one-dimensional sequence after channel scanning is reversed and reshaped into a "super token" spatial structure, and then... The window rule is broken down into spatial pixels, and then restored using a state-space model and its inverse. The feature map is then used to perform residual connection with the input to achieve channel redundancy elimination and multi-scale representation enhancement.

[0063] Step 4: Use the encoder module to fuse the deep guidance features and deep depth features to obtain high-resolution guidance features and low-resolution depth features.

[0064] In step 4, the encoder module is used to fuse the deep guidance features and deep depth features to obtain high-resolution guidance features and low-resolution depth features. The specific steps are as follows:

[0065] The bidirectional fusion module consists of a downward cross-Tramba module (DCTM) and an upward cross-Tramba module (UCTM); both the downward cross-Tramba module and the upward cross-Tramba module are built based on the CTM module.

[0066] S401. Input the first layer deep guiding features and the first layer deep depth features into the downward crossover Tramba module and perform feature fusion processing to obtain the updated first layer deep depth features.

[0067] S402. Input the updated deep features of the first layer into the deep feature extraction module in the depth super-resolution branch for processing to obtain the features of the first depth super-resolution branch.

[0068] S403. Input the first deep super-resolution branch features and the first deep guided features into the upward crossover Tramba module for processing to obtain the updated deep guided features of the first layer.

[0069] S404. The updated deep guiding features of the first layer are processed by the deep feature extraction module in the guiding feature selection branch to obtain the first guiding feature selection branch features.

[0070] S405. Using the second deep layer guiding feature and the second deep layer depth feature as input, repeat steps S401 to S404 to obtain the second depth super-resolution branch feature and the second guiding feature selection branch feature.

[0071] S406. Using the third deep layer guiding feature and the third deep layer depth feature as input, repeat steps S401 to S404 to obtain the third deep super-resolution branch feature and the third guiding feature selection branch feature.

[0072] S407. Using the fourth deep layer guiding feature and the fourth deep layer depth feature as input, repeat steps S401 to S404 to obtain the fourth deep super-resolution branch feature and the fourth guiding feature selection branch feature.

[0073] S408. Channel splicing is performed on the first depth super-resolution branch feature, the second depth super-resolution branch feature, the third depth super-resolution branch feature and the fourth depth super-resolution branch feature to obtain low-resolution depth features.

[0074] S409. Channel splicing is performed on the first guide feature selection branch feature, the second guide feature selection branch feature, the third guide feature selection branch feature, and the fourth guide feature selection branch feature to obtain a high-resolution guide feature;

[0075] In the step of fusing deep guidance features and deep depth features using the encoder module to obtain high-resolution guidance features and low-resolution depth features, the CTM module is defined as follows:

[0076] ;

[0077] in, This indicates a feature that guides the selection process. This indicates processing via a linear layer. This indicates that the process has undergone channel splicing. This indicates that upsampling and downsampling are performed. Indicates guiding characteristics, Representing deep source features, This indicates that merging is performed via cross-Mamba. This indicates the fusion characteristics of the cross-Mamba outputs.

[0078] Furthermore, in this step, the bidirectional fusion module consists of an upward-crossing Tramba module and a downward-crossing Tramba module. Both are designed based on the CTM module and perform feature filtering and fusion on deep guiding features and deep deep features, respectively.

[0079] Specifically, the CTM module structure is similar to the deep feature extraction module, containing a Transformer group and a VMamba group; the Transformer group contains swin Transformer and Restormer, and the VMamba group consists of cross VMamba and cross channel VMamba.

[0080] Unlike previous deep feature extraction modules, this module is designed to adapt to deep source features. and guiding features For bimodal input, the swin Transformer and Restormer in the Transformer group replace self-attention with cross-attention, while the cross VMamba and cross channel VMamba in the VMamba group introduce cross Mamba to achieve cross-modal global dependency modeling and channel-level interaction.

[0081] Specifically, the downward crossover Tramba module uses guided feature extraction branch features as... Deep super-resolution branch features as Branch features are extracted from the guiding features. Perform downsampling and alignment of depth super-resolution branch features The size of the feature is determined to obtain the updated depth super-resolution branch features.

[0082] Specifically, the upward crossover Tramba module uses deep super-resolution branch features as... Guided feature extraction branch features as And through deep super-resolution branch features Upsampling alignment guided feature extraction branch features The size of the branch features is determined to obtain the updated guiding features.

[0083] For further details, please refer to Figure 3In this step, to address the problem of inefficient multimodal information exchange in traditional Mamba single-sequence processing and to achieve effective information exchange between two branches, the proposed cross-Mamba redefines the core parameter of Mamba (discretization step size). Input matrix of state-space model The introduction of the "Guidance Selection Module" upgrades Mamba from "single sequence processing" to "multimodal dynamic fusion," enabling "real-time integration of guidance information into sequence updates."

[0084] Traditional Mamba's discretization parameters rely only on a single input sequence and cannot correlate multimodal information. Cross-Mamba, based on Mamba's gating mechanism, […]. With the discretized input matrix Redefining it as a function guiding selection allows multimodal information to directly participate in hidden state updates, by... The filtered guiding features are reinterpreted and directly fused with the input features at the current time step by element-wise multiplication to achieve local token interaction between the guiding and the source.

[0085] Furthermore, in order to generate suitable guidance selection features, the input color guidance features are first upsampled / downsampled to match their spatial size with the depth source features. Then, feature embedding and fusion are performed, and the scaled guidance features and depth source features are respectively input into the blocking MLP layer to enhance feature representation.

[0086] By dynamically stitching together, the association information between the guidance and the source is initially fused. Then, a 3×3 convolution is used to capture the local spatial consistency between the guidance and the source. Finally, the MLP layer further filters and generates guidance selection features with the same size and channels as the deep source features.

[0087] Step 5: Input the high-resolution guiding features and low-resolution depth features into the decoder module for processing to obtain the super-resolution reconstructed image;

[0088] In step 5, the high-resolution guiding features and low-resolution depth features are input into the decoder module for processing to obtain the super-resolution reconstructed image. The specific steps are as follows:

[0089] High-resolution guiding features are processed by convolutional layers to obtain high-resolution deep features for the guiding feature extraction branch;

[0090] The low-resolution depth features are processed sequentially by convolutional layers and pixel shuffle upsampling layers to obtain high-resolution depth features for the depth super-resolution branch.

[0091] After the high-resolution depth features of the guided feature extraction branch and the high-resolution depth features of the depth super-resolution branch are stitched together in the channel dimension, the image is then processed by channel attention and convolutional layers to obtain the stitched image.

[0092] Bicubic interpolation upsampling is performed on the low-resolution depth image to obtain a low-frequency reference depth image;

[0093] The stitched image is then stitched together with the low-frequency reference depth image via channels to obtain a super-resolution reconstructed image.

[0094] Specifically, this invention also designs a loss function for training the reconstruction model, and the specific steps are as follows:

[0095] The high-resolution depth features of the guiding feature extraction branch are processed through a convolutional layer and then concatenated with the low-frequency reference depth image to obtain the high-resolution depth image of the image guiding branch.

[0096] The high-resolution depth features of the depth super-resolution branch are processed through a convolutional layer and then concatenated with the low-frequency baseline depth image to obtain the high-resolution depth image of the depth super-resolution branch.

[0097] A loss function is constructed using high-resolution depth images, super-resolution reconstructed images, low-frequency baseline depth images, and high-resolution ground truth depth images. Its expression is:

[0098] ;

[0099] in, Represents the loss function; This represents the images obtained from different branches, when hour, Represents a high-resolution depth image of the image guide branch, when hour, Indicates super-resolution reconstructed image, when hour, A high-resolution depth image representing the depth super-resolution branch; Represents a low-frequency reference depth image. Represents a high-resolution, true-to-depth image. This indicates L1 norm processing.

[0100] Specifically, the present invention uses the training set, validation set and test set obtained in step 1, combined with the loss function, to train the reconstruction model.

[0101] Please see Figure 5 This invention also provides a depth map super-resolution system based on Mamba and its variants, the system comprising:

[0102] The data processing module is used for:

[0103] High-resolution image datasets are processed to obtain high-resolution texture images and low-resolution depth images;

[0104] Encoder module, used for:

[0105] Shallow feature extraction is performed on high-resolution texture images and low-resolution depth images to obtain shallow features of texture images and shallow features of depth images;

[0106] Deep features are extracted from shallow features of texture images and shallow features of depth images to obtain deep guiding features and deep depth features;

[0107] Deep guidance features and deep depth features are fused to obtain high-resolution guidance features and low-resolution depth features;

[0108] Decoder module, used for:

[0109] High-resolution guiding features and low-resolution depth features are processed to obtain super-resolution reconstructed images.

[0110] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0111] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A depth map super-resolution method based on Mamba and its variants, characterized in that, The method is implemented through a reconstruction model, which includes an image processing module, an encoder module, and a decoder module. The specific steps for generating a super-resolution reconstructed image using the reconstruction model include: Step 1: Input the high-resolution image dataset into the reconstruction model, and process the high-resolution image dataset through the image processing module to obtain high-resolution texture images and low-resolution depth images; Step 2: Input the high-resolution texture image and the low-resolution depth image into the encoder module to extract shallow features, so as to obtain shallow features of the texture image and shallow features of the depth image. Step 3: Use the encoder module to extract deep features from the shallow features of the texture image and the shallow features of the depth image to obtain deep guiding features and deep depth features. The specific steps are as follows: The deep feature extraction module in the encoder module contains a Transformer group and a VMamba group; the Transformer group contains Swing Transformer and Restormer; the VMamba group contains VMamba and channel VMamba. The shallow features of the texture image are processed sequentially by the Swing Transformer and the Restormer, and then residually connected with the shallow features of the texture image to obtain the output of the Transformer group. The output of the Transformer group is processed by VMamba and then by channel VMamba in sequence. Finally, it is residually concatenated with the output of the Transformer group to obtain the output of the VMamba group. The output of the VMamba group is then used as the output of the deep feature extraction module. In the guided feature extraction branch, the output of the deep feature extraction module is used as the deep guided feature; The deep guidance features of the previous layer are used as the input of the deep guidance features of the next layer. The features are processed by four deep feature extraction modules to obtain the first, second, third and fourth deep guidance features. In the depth super-resolution branch, the shallow features of the depth image are processed in the same way as the shallow features of the texture image to obtain the first, second, third and fourth deep depth features. The VMamba channel includes the Mamba channel module. The Mamba channel module receives intermediate features, divides the spatial dimensions using variable-sized windows, treats each window as a super token, and reshapes the intermediate features through a refactoring operation. Convert to ,in For the new channel dimension, The new sequence length; where, The height of the intermediate feature. The width of the intermediate feature. The number of channels for the intermediate features; the reshaped super token features are flattened into a one-dimensional sequence, and the traditional spatial dimension scan of Mamba is changed to a channel dimension scan, according to... The sequence length is processed element-wise to model channel dependencies, while also considering different... The generated sequence is then aligned to the specified length using an MLP layer. The length; the one-dimensional sequence after channel scanning is inversely reshaped into a super token space structure, and then reordered according to the original... The window rule is broken down into spatial pixels, and then restored using a state-space model and its inverse. Feature map; Step 4: Use the encoder module to fuse the deep guidance features and deep depth features to obtain high-resolution guidance features and low-resolution depth features. Step 5: Input the high-resolution guiding features and low-resolution depth features into the decoder module for processing to obtain the super-resolution reconstructed image.

2. The depth map super-resolution method based on Mamba and its variants according to claim 1, characterized in that, In step 1, the high-resolution image dataset is processed by the image processing module to obtain a high-resolution texture image and a low-resolution depth image. The specific steps are as follows: The image processing module crops the high-resolution image dataset into pairs of image blocks according to the preset image size, and then performs random flipping and rotation data enhancement processing to obtain pairs of high-resolution texture images and high-resolution depth real images. A low-resolution depth image is obtained by bicubic interpolation of a high-resolution real depth image.

3. The depth map super-resolution method based on Mamba and its variants according to claim 2, characterized in that, In step 2, the high-resolution texture image and the low-resolution depth image are input into the encoder module for shallow feature extraction to obtain shallow features of the texture image and shallow features of the depth image. The specific steps are as follows: The encoder module consists of a guided feature extraction branch, a deep super-resolution branch, and a bidirectional fusion module. The guided feature extraction branch and the deep super-resolution branch have the same structure, each consisting of one convolutional layer and four deep feature extraction modules. The high-resolution texture image is input into the guided feature extraction branch and processed through a convolutional layer to obtain shallow features of the texture image. The low-resolution depth image is input into the depth super-resolution branch and processed through convolutional layers to obtain shallow features of the depth image.

4. The depth map super-resolution method based on Mamba and its variants according to claim 3, characterized in that, After processing the shallow features of the texture image sequentially through the Swing Transformer and Restormer, and then performing a residual connection with the shallow features of the texture image to obtain the output of the Transformer group, the following relationship exists: ； in, This represents the output of the Transformer group. This indicates that it will be processed through Restormer. This indicates that it is processed through a Swing Transformer. This represents the input to the Transformer group.

5. The depth map super-resolution method based on Mamba and its variants according to claim 4, characterized in that, After processing the output of the Transformer group sequentially through VMamba and then through channel VMamba, and finally performing a residual join with the output of the Transformer group to obtain the output of the VMamba group, the following relationship exists: ； in, This indicates the output of the VMamba group. This indicates processing via VMamba channel. This indicates that it is processed via VMamba.

6. The depth map super-resolution method based on Mamba and its variants according to claim 5, characterized in that, In step 4, the encoder module is used to fuse the deep guidance features and deep depth features to obtain high-resolution guidance features and low-resolution depth features. The specific steps are as follows: The bidirectional fusion module consists of a downward cross-Trumpa module and an upward cross-Trumpa module; both the downward cross-Trumpa module and the upward cross-Trumpa module are built based on the CTM module. S401. Input the first layer deep guiding features and the first layer deep depth features into the downward crossover Tramba module and perform feature fusion processing to obtain the updated first layer deep depth features. S402. Input the updated deep features of the first layer into the deep feature extraction module in the depth super-resolution branch for processing to obtain the features of the first depth super-resolution branch. S403. Input the first deep super-resolution branch features and the first deep guided features into the upward crossover Tramba module for processing to obtain the updated deep guided features of the first layer. S404. The updated deep guiding features of the first layer are processed by the deep feature extraction module in the guiding feature selection branch to obtain the first guiding feature selection branch features. S405. Using the second deep layer guiding feature and the second deep layer depth feature as input, repeat steps S401 to S404 to obtain the second depth super-resolution branch feature and the second guiding feature selection branch feature. S406. Using the third deep layer guiding feature and the third deep layer depth feature as input, repeat steps S401 to S404 to obtain the third deep super-resolution branch feature and the third guiding feature selection branch feature. S407. Using the fourth deep layer guiding feature and the fourth deep layer depth feature as input, repeat steps S401 to S404 to obtain the fourth deep super-resolution branch feature and the fourth guiding feature selection branch feature. S408. Channel splicing is performed on the first depth super-resolution branch feature, the second depth super-resolution branch feature, the third depth super-resolution branch feature and the fourth depth super-resolution branch feature to obtain low-resolution depth features. S409. Channel splicing is performed on the first guide feature selection branch feature, the second guide feature selection branch feature, the third guide feature selection branch feature, and the fourth guide feature selection branch feature to obtain a high-resolution guide feature; The CTM module structure contains a Transformer group and a VMamba group; the Transformer group contains swinTransformer and Restormer, and the VMamba group consists of cross-VMamba and cross-channel VMamba.

7. The depth map super-resolution method based on Mamba and its variants according to claim 6, characterized in that, In step 5, the high-resolution guiding features and low-resolution depth features are input into the decoder module for processing to obtain the super-resolution reconstructed image. The specific steps are as follows: High-resolution guiding features are processed by convolutional layers to obtain high-resolution deep features for the guiding feature extraction branch; The low-resolution depth features are processed sequentially by convolutional layers and pixel shuffle upsampling layers to obtain high-resolution depth features for the depth super-resolution branch. After the high-resolution depth features of the guided feature extraction branch and the high-resolution depth features of the depth super-resolution branch are stitched together in the channel dimension, the image is then processed by channel attention and convolutional layers to obtain the stitched image. Bicubic interpolation upsampling is performed on the low-resolution depth image to obtain a low-frequency reference depth image; The stitched image is then stitched together with the low-frequency reference depth image via channels to obtain a super-resolution reconstructed image.

8. A depth map super-resolution system based on Mamba and its variants, characterized in that, The system applies the depth map super-resolution method based on Mamba and its variants as described in any one of claims 1 to 7, and the system comprises: The data processing module is used for: High-resolution image datasets are processed to obtain high-resolution texture images and low-resolution depth images; Encoder module, used for: Shallow feature extraction is performed on high-resolution texture images and low-resolution depth images to obtain shallow features of texture images and shallow features of depth images; Deep features are extracted from shallow features of texture images and shallow features of depth images to obtain deep guiding features and deep depth features; Deep guidance features and deep depth features are fused to obtain high-resolution guidance features and low-resolution depth features; Decoder module, used for: High-resolution guiding features and low-resolution depth features are processed to obtain super-resolution reconstructed images.

Images