A color-guided depth map super-resolution reconstruction method

Abstract

The present invention relates to a color-guided depth map super-resolution reconstruction method, which comprises the following steps: 1. Extracting multi-level color features and depth features from LR depth maps and HR color maps through a feature extraction module; 2. Implementing color-guided depth feature learning and detail recovery in collaboration with an attention-based feature projection module, a low-level detail embedding module, and a high-level semantic guidance module; and finally obtaining a super-resolution reconstructed depth map with accurate accuracy and rich details. The present invention realizes the divide-and-conquer guidance of color information on depth super-resolution from both low-level and high-level features, suppresses worthless redundancy, and improves reconstruction performance. Examples show that the method of the present invention achieves excellent visual effects in many challenging scenes, with high accuracy at object boundaries and details of tiny objects.

Description

Technical Field

[0001] The invention relates to a color-guided depth map super-resolution reconstruction method. Background Art

[0002] Depth maps describe the distance relationship of a scene, including occlusion and overlap of objects, and are crucial for 3D understanding tasks such as autonomous driving, 3D object detection, 3D reconstruction, salient object detection, and many other fields. However, due to the limitations of existing depth acquisition equipment, the resolution of the acquired depth maps is relatively low, especially the low-power depth sensors equipped on smartphones. Low-resolution (LR) depth maps cannot match the resolution of high-resolution (HR) color images, which hinders the further expansion of depth-oriented applications. Therefore, depth map super-resolution reconstruction (DSR) technology has emerged, which has practical research value and industrial application value. Depth map super-resolution is an ill-posed inverse problem because there is no one-to-one correspondence between LR depth maps and HR depth maps, and it is difficult to recover details such as clear boundaries when the upsampling factor is large. Therefore, considering that HR color images are relatively easy to obtain and the brightness boundaries of color images have obvious structural similarities with the depth boundaries of depth images, many existing DSR models use HR color images as auxiliary input to provide more comprehensive additional guidance information. Wen et al. used a coarse-to-fine DCNN network to learn different filters with different convolution kernel sizes, allowing data-driven training to replace hand-designed filters. Huang et al. proposed a deep dense residual network with a pyramid structure, which used multi-scale features to predict high-frequency residuals through dense connections and residual learning. Guo et al. designed a residual UNet structure for deep reconstruction tasks and introduced hierarchical feature-driven residual learning. Zuo et al. proposed a data-driven super-resolution network based on global and local residual learning. Sun et al. proposed a progressive multi-branch aggregation network that used multi-scale information and high-frequency features to reconstruct depth maps step by step. They also showed that low-level color information is only suitable for early feature fusion and is not very helpful for DSR in the ×2 and ×4 cases.

[0003] In the process of implementing the present invention, the inventors found that the prior art has at least the following shortcomings and deficiencies:

[0004] The structural similarity between the brightness boundary of color images and the depth boundary of depth images does not always exist. Color images contain not only object boundaries but also texture boundaries inside objects, while depth maps only have object boundaries, so color images contain much more boundary information than depth maps, which may introduce additional noise while providing guidance for DSR. Existing methods have designed some patterns and strategies in the form of guidance, such as using low-level color features as detail guidance, or treating color features of different levels equally as guidance. However, these methods do not take into account the role and diversity of different color information in the guidance process, so making full use of color guidance information requires further exploration. Summary of the invention

[0005] In view of the defects existing in the prior art, the present invention aims to fully explore the cross-modal information guidance of color on depth and reduce the redundancy and additional noise impact in color information, and designs a hierarchical color guided projection network for depth map super-resolution reconstruction. The network models different color guidance roles according to different levels to obtain better reconstruction performance.

[0006] This paper explores a color-guided depth map super-resolution reconstruction method, namely a hierarchical color-guided projection network, which realizes the divide-and-conquer guidance of color information on depth super-resolution from both low-level and high-level features. The specific implementation steps are as follows:

[0007] A deep image super-resolution reconstruction network, comprising: a feature extraction module, an attention-based feature projection module, a low-level detail embedding module and a high-level semantic guidance module;

[0008] The feature extraction module consists of five progressive convolution blocks, each of which consists of two 3×3 convolutional layers and one 1×1 convolutional layer. The attention-based feature projection module consists of a multi-scale content enhancement block and four cascaded adaptive attention projection blocks;

[0009] The feature extraction module is used to extract multi-level color features and depth features from the LR depth map and the HR color map;

[0010] The multi-scale content enhancement block is used to enhance the depth features before projection, and uses different receptive fields to restore as many detail features at different scales as possible, thereby improving the reconstruction effect of areas with more severe degradation (such as depth boundaries and fine objects); the adaptive attention projection block is used to project effective information into the attention domain to ensure the effectiveness and compactness of the projected features. The four cascaded adaptive attention projection blocks gradually restore the depth features in a coarse-to-fine manner.

[0011] The low-level detail embedding module is used to introduce the color features of the lower levels (i.e., the first two layers) of the HR color map. It learns a residual mask to highlight the spatial position where the color features are most consistent with the depth features, and uses high-frequency color guidance in the form of residual attention to supplement the depth features, thereby better restoring the details.

[0012] The high-level semantic guidance module is used to generate a semantic mask that encodes global semantic guidance information using the semantic information of the top layer of the HR color map, and further perform semantic correction on the initial deep reconstruction features.

[0013] The multi-scale content enhancement block consists of a stack of four convolutional layers with different dilation rates to capture details at different receptive field scales.

[0014] A color-guided depth map super-resolution reconstruction method, using the above-mentioned DSR projection network, comprises the following steps:

[0015] Step 1: Extract multi-level color features and depth features from the LR depth map and the HR color map through the feature extraction module;

[0016] Step 2: Under the collaboration of the attention-based feature projection module, the low-level detail embedding module, and the high-level semantic guidance module, color-guided deep feature learning and detail restoration are achieved;

[0017] Step 3: Divide the image into multiple reconstruction levels according to features at different levels, aggregate the restored features of different reconstruction levels, and gradually reconstruct the depth map, so as to finally obtain a super-resolution reconstructed depth map (SR depth map) with accurate accuracy and rich details.

[0018] Based on the above scheme, the color features and depth features are expressed as and i = {1, 2, 3, 4, 5}; at different reconstruction levels, the input features of the attention-based feature projection module are different, specifically expressed as:

[0019]

[0020] Among them, Concat represents the concatenation operation along the channel dimension. represents the deep features of the i-th layer, is the low-level detail feature generated by the i-th layer low-level detail embedding module, represents the transmission characteristics after the reconstruction of the kth layer is completed, which is calculated by the following steps:

[0021]

[0022] in, is the output feature of the kth high-level semantic guidance module, ↓ is the downsampling operation, k = {i+1, i+2,…, 5}.

[0023] Then, the top layer color features and deep features Send it to the high-level semantic guidance module to correct the initial deep reconstruction features Generate final reconstruction features Finally, and Perform sub-pixel convolution operations to obtain the final upsampled super-resolution reconstructed depth map.

[0024] At the same time, the DSR projection network constructs the training loss function by calculating the L1 norm between the super-resolution reconstructed depth map and the real depth map:

[0025] Loss=||D SR -D HR ||1 (3)

[0026] Among them, D SR , D HR are the super-resolution reconstructed depth map and the real depth map respectively, and ||·||1 is the L1 norm function.

[0027] On the basis of the above scheme,

[0028]

[0029] Where MD represents the multi-scale dilated convolution operation with dilation rates of 1, 2, 3, and 4. is the input of the i-th layer attention-based feature projection module, is the output of each multi-scale dilated convolution. Finally, all the multi-scale dilated features Cascade fusion through 1×1 convolutional layers:

[0030]

[0031] Among them, Conv 1×1 represents a convolutional layer with a convolution kernel size of 1×1, is the output of the multi-scale content enhancement block.

[0032] Based on the above scheme, the input of the adaptive attention projection block is uniformly expressed as:

[0033]

[0034] Where DeConv represents the upsampling operation performed by the deconvolution layer, is the jth HR output of the i-th layer adaptive attention projection block. As can be seen from formula (6), the input of the first adaptive attention projection block is the upsampled output feature of the multi-scale content enhancement block, while the input of other adaptive attention projection blocks is the output of the previous adaptive attention projection block.

[0035] The calculation formula of the projected attention map is:

[0036]

[0037] Among them, ReLU is a linear rectifier unit, Conv is a convolutional layer with down-projection, and DeConv is a deconvolution layer.

[0038] With the projected attention map, the output of the current adaptive attention projection block is described as:

[0039]

[0040] in, Represents element-wise multiplication.

[0041] Using 4 consecutive adaptive attention projection blocks, 4 HR reconstruction features are generated from coarse to fine. The final output features of the attention-based feature projection module are obtained by combining:

[0042]

[0043] in, Represents the initial deep reconstruction features.

[0044] Based on the above scheme, the color features and depth features are mapped to the same domain, and then they are subtracted to obtain the redundant part of the color information relative to the depth information. Then the residual mask is generated by normalizing and negating it:

[0045]

[0046] Among them, RM i represents the residual mask, W c and W d Represents the mapping matrix of color features and depth features. Sigmoid is a normalization operation, and i is 1 or 2.

[0047] The final output feature of the low-level detail embedding module is represented as:

[0048]

[0049] in, represents element-wise multiplication, Low-level detail features generated for the low-level detail embedding module.

[0050] Based on the above scheme, spatial attention is applied to the top-level color features This generates enhanced color features that highlight important locations Concatenate the enhanced color features with the initial deep reconstruction features to generate a semantic mask:

[0051]

[0052] Where PReLU is a parameterized linear rectifier unit, Conv 3×3 It is a convolution layer with a convolution kernel size of 3×3.

[0053] Using semantic masks, the initial deep reconstruction features can be obtained by: To refine:

[0054]

[0055] in, The final reconstructed features for each reconstruction level.

[0056] Beneficial effects of the present invention: The present invention explores a color-guided depth map super-resolution reconstruction method and proposes a hierarchical color-guided projection network, which realizes the divide-and-conquer guidance of color information for depth super-resolution from both low-level and high-level features. To this end, a low-level detail embedding (LDE) module and a high-level semantic guidance (HSG) module are designed to achieve hierarchical color guidance. In addition, the present invention also proposes an adaptive attention projection (AAP) block for strengthening the key recovery area in the attention domain and suppressing redundant information. Examples show that the method of the present invention achieves excellent visual effects in many challenging scenes, with high accuracy at object boundaries and details of tiny objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention has the following accompanying drawings:

[0058] Figure 1 Schematic diagram of the overall architecture of the DSR projection network of the present invention.

[0059] Figure 2 Schematic diagram of a visualization example of the present invention at 8x super resolution. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0061] The present invention rethinks the role of color guidance and proposes a hierarchical color-guided DSR projection network. We believe that color information at different levels is instructive for DSR tasks, but the roles they play and how they are used should be distinguished. To this end, the color branch of the present invention implements a divide-and-conquer guidance of the deep super-resolution branch in terms of both low-level and high-level features in the network: 1) a low-level detail embedding (LDE) module that supplements deep features with high-frequency color guidance in a residual attention manner to restore details such as boundaries; 2) a high-level semantic guidance (HSG) module that corrects the initial reconstructed features by using a semantic mask that encodes global semantic guidance information. In addition, an attention-based feature projection (AFP) module is designed, including a multi-scale content enhancement block that obtains more comprehensive features through multi-scale receptive fields, and an adaptive attention projection block that projects valuable information from the LR domain to the HR domain in an attentional manner.

[0062] Figure 1 The overall architecture diagram of the DSR projection network. First, the LR depth map and HR color image are embedded in the feature extraction module to extract multi-level features. Then, the AFP module, LDE module and HSG module work together to gradually restore the details in the low-resolution depth features and generate the HR depth map. The use of color information is manifested in two aspects: on the one hand, the low-level color features are used in the low-level reconstruction stage to restore the details through the LDE module; on the other hand, the high-level semantic features are used at the end of the AFP module to provide semantic guidance through the HSG module.

[0063] 1. Technical route

[0064] The DSR projection network designed by the present invention is summarized as follows: Figure 1 As shown in Figure 1, this is a two-stream progressive reconstruction architecture. Given an LR depth map and a corresponding HR color image as input, the goal of the network is to reconstruct a SR (super-resolution reconstruction) depth map with the same resolution as the color image.

[0065] The network first extracts multi-level color features and depth features through 5 progressive convolution blocks. Each convolution block includes two 3×3 convolution layers and one 1×1 convolution layer. The obtained color features and depth features are expressed as and (i = {1, 2, 3, 4, 5}). Then, with the cooperation of the attention-based feature projection (AFP) module, the low-level detail embedding (LDE) module and the high-level semantic guidance (HSG) module, color-guided deep feature learning and detail recovery are achieved. It is worth noting that three inputs (if any) are sent to the AFP module: (1) the deep features of the corresponding level (2) Low-level detail features generated by the LDE module Used for detail recovery in the low-level reconstruction stage; (3) Transfer features of all completed reconstruction layers At different reconstruction levels, the input features of the AFP module are different, which can be specifically expressed as follows:

[0066]

[0067] Among them, Concat represents the concatenation operation along the channel dimension. represents the deep features of the i-th layer, The low-level detail features generated by the i-th layer LDE module, It represents the transmission characteristics after the reconstruction of the kth layer is completed, which can be calculated by the following steps:

[0068]

[0069] in, is the output feature of the kth HSG module, ↓ is the downsampling operation, k = {i+1, i+2, ..., 5}. It should be noted that the input of the LDE module includes the depth feature of the corresponding layer and color features

[0070] Then, the top layer color features and deep features Send to HSG module to correct the output characteristics of AFP module Generate Reconstruction Features Finally, and Perform sub-pixel convolution operation to obtain the final upsampled SR depth map.

[0071] At the same time, the DSR projection network of the present invention constructs a training loss function by calculating the L1 norm between the super-resolution reconstructed depth map and the real depth map:

[0072] Loss=||D SR -D HR ||1 (3)

[0073] Among them, D SR , D HR are the super-resolution reconstructed depth map and the real depth map respectively, and ||·||1 is the L1 norm function.

[0074] 2. Attention-based feature projection module

[0075] In order to achieve super-resolution of the depth map, it is necessary to map the low-resolution features to the required high-resolution reconstructed features. Specifically, there are two issues that need to be noted: (1) In order to restore severely degraded local details (such as depth boundaries and fine objects), simply increasing the network depth is not enough and unwise. Therefore, the present invention introduces a multi-scale content enhancement block to enhance the depth features before projection, and uses different receptive fields to restore as many detail features at different scales as possible. (2) The information between the LR and HR domains is not absolutely one-to-one corresponding during the projection process. The interference of too much redundant information can easily introduce additional errors, thereby affecting the reconstruction accuracy. To this end, the present invention proposes an adaptive attention projection (AAP) block to project valid information into the attention domain to ensure the effectiveness and compactness of the projected features. It should be noted that four cascaded AAP blocks are used in the AFP module to obtain better performance.

[0076] 1) Multi-scale content enhancement (MCE) block. Multi-scale information can effectively perceive and model different details, which is of great significance for detail recovery in DSR. The MCE module contains the superposition of 4 convolutional layers with different dilation rates to capture more details of different receptive field scales. In addition, dense connections are used to obtain complete information from all layers. The whole process can be expressed as:

[0077]

[0078] Where MD represents the multi-scale dilated convolution operation with dilation rates of 1, 2, 3, and 4. is the input of the i-th layer AFP module, is the output of each multi-scale dilated convolution. Finally, all the multi-scale dilated features Cascade fusion through 1×1 convolutional layers:

[0079]

[0080] Among them, Conv 1×1 represents a convolutional layer with a convolution kernel size of 1×1, is the output of the MCE block, which perceives content information at different scales.

[0081] 2) Adaptive Attention Projection (AAP) block. Super-resolution processing of depth maps needs to bridge the huge gap between the LR domain and the HR domain. In fact, the focus of the DSR task is not to generate content from scratch, but to supplement, improve and enhance details such as boundaries. From this perspective, blindly and indiscriminately performing super-resolution reconstruction on all areas is a suboptimal approach, and it is difficult to achieve the purpose of optimizing important areas with more severe degradation. That is, in the process of restoring information from the LR domain to the HR domain (also known as the projection process), redundant interference may be introduced without filtering, thereby introducing additional errors and affecting the reconstruction accuracy. Therefore, the present invention designs an AAP block to pay attention to strengthening the key recovery area, thereby suppressing interference and improving reconstruction performance.

[0082] Specifically, two methods, down-projection and up-projection, are used to simulate the DSR process, thereby obtaining a reconstructed HR feature map under poor conditions. Then, the reconstructed HR features are subtracted from the original HR features to generate residual features, which encode the content information that needs to be restored during the reconstruction process. Finally, the residual map is activated as a projected attention map and used to adaptively refine the original HR features. The projected attention map can correct errors in the reconstruction process and avoid degradation caused by feature projection between the LR and HR domains.

[0083] To further improve the restoration performance, we stack four AAP blocks together and integrate the output of each AAP block through 1×1 convolution.

[0084] Therefore, the input of the AAP block can be uniformly expressed as:

[0085]

[0086] Where DeConv represents the upsampling operation performed by the deconvolution layer, is the jth HR output of the i-th AAP block (further derivation will be given below). As can be seen from equation (6), the input of the first AAP block is the upsampled output feature of the MCE block, while the input of other AAP blocks is the output of the previous AAP block. The calculation formula of the projected attention map is:

[0087]

[0088] Among them, ReLU is a linear rectifier unit, Conv is a convolutional layer with down-projection, and DeConv is a deconvolution layer.

[0089] With the projected attention map, the output of the current AAP block can be described as:

[0090]

[0091] in, Represents element-wise multiplication.

[0092] Using 4 consecutive AAP blocks, 4 HR reconstruction features are generated from coarse to fine. Combining them, the final output features of the AFP module can be obtained:

[0093]

[0094] in, Represents the initial deep reconstruction features.

[0095] 3. Low-level detail embedding module

[0096] As we all know, high-resolution color images are easily available and contain a lot of useful information, such as boundaries, textures, semantics, etc. Therefore, introducing color guidance in DSR models has become the mainstream idea in this field. However, there is no complete consensus on which color information to use and how to use it. Considering the different roles of color features at different levels, this paper proposes a differentiated color guidance strategy solution. Specifically, a low-level detail embedding (LDE) module is designed in the low-level reconstruction stage, and low-level color features are used to supplement the high-frequency guidance of deep features to restore details such as boundaries. In addition, the present invention also designs a high-level semantic guidance (HSG) module, which uses high-level color features to perform semantic correction on the original reconstructed features to prevent semantic deviation in the deep reconstruction process. This section will introduce the LDE module, and the next section will introduce the HSG module in detail.

[0097] For super-resolution of depth maps, accurate and clear boundary reconstruction has always been the focus of researchers' unremitting efforts. It happens that the bottom layer of the color branch learns detailed information such as texture and boundaries. Therefore, the present invention introduces the color features of the lower level (i.e., the first two layers) of the HR color branch through the LDE module, and outputs it as one of the inputs of the AFP module. However, the depth boundary is not completely consistent with the color boundary. In fact, the boundaries in the depth map are mainly object boundaries, while the color image includes rich texture boundaries inside the object in addition to the object boundaries. Obviously, texture boundaries are redundant or even interfering for DSR. Then it is urgent to choose appropriate color information as a guide. Therefore, the network learns a residual mask to highlight the spatial position where the color features are most consistent with the depth features, and uses the residual mask to guide the information transfer from the color features to the depth features.

[0098] First, the color features and depth features are mapped to the same domain, and then they are subtracted to obtain the redundant part of the color information relative to the depth information, which is the part that the network needs to suppress. The residual mask can then be generated by normalizing and negating it:

[0099]

[0100] Among them, RM i represents the residual mask, W c and W d Represents the mapping matrix of color features and depth features. Sigmoid is a normalization operation, and i is 1 or 2.

[0101] In this way, the residual mask highlights the most relevant parts of the color and depth information, so it is multiplied with the initial color feature to obtain an effective color feature that can be used for depth reconstruction guidance. Therefore, the final output of the LDE module can be expressed as:

[0102]

[0103] in, represents element-wise multiplication, Low-level detail features generated for the LDE module.

[0104] 4. High-level semantic guidance module

[0105] As analyzed above, existing methods mainly supplement the details of depth reconstruction by extracting color features, just like the function implemented by the LDE module. However, is this detail guidance strategy sufficient? In fact, high-level color features are very important for many tasks, and they can provide rich semantic information. In the DSR task, existing methods ignore a problem, that is, the global semantic preservation ability of reconstructed features. As the reconstruction process proceeds, semantic information may be offset or blurred, which is very unfavorable for subsequent depth-oriented application tasks. This is mainly due to the lack of semantic guidance in the reconstruction process. Inspired by this, the present invention designs an HSG module to maintain semantic attributes in the depth reconstruction process, which is configured after each AFP module. Specifically, the semantic information of the top layer of the HR color map is used to generate a semantic mask that encodes global semantic guidance information, and is further used to correct the initial depth reconstruction features. (i.e., the output features of the AFP module).

[0106] First, spatial attention is applied to the top-level color features This generates enhanced color features that highlight important locations Considering the auxiliary role of semantic features, the network is still dominated by deep reconstruction features in the guidance process. Therefore, the enhanced color features are concatenated with the initial deep reconstruction features to generate a semantic mask:

[0107]

[0108] Where PReLU is a parameterized linear rectifier unit, Conv 3×3 It is a convolution layer with a convolution kernel size of 3×3.

[0109] Using semantic masks, the initial deep reconstruction features can be obtained by: To refine:

[0110]

[0111] in, is the output feature of the corresponding HSG module. It should be noted that is the final reconstruction feature of each reconstruction level, the last layer The reconstructed features of will be directly used to generate the upsampled SR depth map, while the reconstructed features of other layers will achieve progressive learning of the entire network through densely connected transmission modules.

[0112] Attached Figure 2 The visualization example of the present invention at 8x super resolution is given. The first column is the HR real depth map and color image, the second column is the real depth image block, and the third column is the SR depth image block reconstructed for the present invention. From the results, it can be seen that the method of the present invention achieves excellent visual effects in many challenging scenes, with high accuracy at the object boundaries and details of tiny objects.

[0113] The depth map describes the distance relationship of the scene, and constitutes two modal information with the color image. Color-guided depth map super-resolution reconstruction is to reconstruct a high-resolution depth map by supplementing the low-resolution depth map with color information.

[0114] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the essence and scope of the present invention. Therefore, all equivalent technical solutions also belong to the protection scope of the present invention.

[0115] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. A color-guided depth map super-resolution reconstruction method, characterized in that: Using the depth map super-resolution reconstruction network, the following steps are included: Step 1: Extract multi-level color features and depth features from the LR depth map and the HR color map through the feature extraction module; Step 2: Under the collaboration of the attention-based feature projection module, the low-level detail embedding module, and the high-level semantic guidance module, color-guided deep feature learning and detail restoration are achieved; Step 3: Divide the image into multiple reconstruction levels according to the features at different levels, aggregate the restored features at different reconstruction levels, and gradually reconstruct the depth map, so as to finally obtain a super-resolution reconstructed depth map with accurate accuracy and rich details; The color feature and depth feature are respectively expressed as and i={1,2,3,4,5}; at different reconstruction levels, the input features of the attention-based feature projection module are different, specifically expressed as: Among them, Concat represents the concatenation operation along the channel dimension. represents the deep features of the i-th layer, is the low-level detail feature generated by the i-th layer low-level detail embedding module, represents the transmission characteristics after the k-th layer is reconstructed, which is calculated by the following steps: in, is the output feature of the k-th high-level semantic guidance module, ↓ is the downsampling operation, k = {i+1, i+2, …, 5}; Then, the top layer color features and deep features Send it to the high-level semantic guidance module to correct the initial deep reconstruction features Generate final reconstruction features Finally, and Perform sub-pixel convolution operations to obtain the final upsampled super-resolution reconstructed depth map.

2. The color-guided depth map super-resolution reconstruction method according to claim 1, characterized in that: The depth map super-resolution reconstruction network constructs the training loss function by calculating the L1 norm between the super-resolution reconstructed depth map and the true depth map: Loss = ‖D SR -D HR ‖1 (3) Where D SR , D HR are the super-resolution reconstructed depth map and the real depth map respectively, and ‖·‖1 is the L1 norm function.

3. The color-guided depth map super-resolution reconstruction method according to claim 1, characterized in that: Where MD represents the multi-scale dilated convolution operation with dilation rates of 1, 2, 3, and 4. is the input feature of the i-th layer attention-based feature projection module, is the output of each multi-scale dilated convolution; finally, all the multi-scale dilated features Cascade fusion through 1×1 convolutional layers: Among them, Conv 1×1 represents a convolutional layer with a convolution kernel size of 1×1, is the output of the multi-scale content enhancement block.

4. The color-guided depth map super-resolution reconstruction method according to claim 3, characterized in that: The input of the adaptive attention projection block is uniformly expressed as: Where DeConv represents the upsampling operation performed by the deconvolution layer, is the jth HR output of the i-th layer adaptive attention projection block; as can be seen from formula (6), the input of the first adaptive attention projection block is the upsampled output feature of the multi-scale content enhancement block, while the input of other adaptive attention projection blocks is the output of the previous adaptive attention projection block; The calculation formula of the projected attention map is: Among them, ReLU is a linear rectifier unit, Conv is a convolutional layer with down-projection, and DeConv is a deconvolution layer; With the projected attention map, the output of the current adaptive attention projection block is described as: in, Represents element-wise multiplication; Using 4 consecutive adaptive attention projection blocks, 4 HR reconstruction features are generated from coarse to fine; the final output features of the attention-based feature projection module are obtained by combining them: in, Represents the initial deep reconstruction features.

5. The color-guided depth map super-resolution reconstruction method according to claim 1, characterized in that: The color features and depth features are mapped to the same domain, and then subtracted to obtain the redundant part of the color information relative to the depth information. The residual mask is then generated by normalizing and negating it: Among them, RM i represents the residual mask, W c and W d Represents the mapping matrix of color features and depth features. Sigmoid is a normalization operation, and i takes 1 or 2; The final output feature of the low-level detail embedding module is represented as: in, represents element-wise multiplication, Low-level detail features generated for the low-level detail embedding module.

6. The color-guided depth map super-resolution reconstruction method according to claim 4, characterized in that: Apply spatial attention to top-level color features This generates enhanced color features that highlight important locations Concatenate the enhanced color features with the initial deep reconstruction features to generate a semantic mask: Where PReLU is a parameterized linear rectifier unit, Conv 3×3 It is a convolution layer with a convolution kernel size of 3×3; Using semantic masks, the initial deep reconstruction features are reconstructed by the following method To refine: in, The final reconstructed features for each reconstruction level.

7. The color-guided depth map super-resolution reconstruction method according to claim 1, characterized in that: The deep image super-resolution reconstruction network includes: feature extraction module, attention-based feature projection module, low-level detail embedding module and high-level semantic guidance module; The feature extraction module consists of five progressive convolution blocks, each of which consists of two 3×3 convolutional layers and one 1×1 convolutional layer; the attention-based feature projection module consists of a multi-scale content enhancement block and four cascaded adaptive attention projection blocks; The feature extraction module is used to extract multi-level color features and depth features from the LR depth map and the HR color map; The multi-scale content enhancement block is used to enhance the depth features before projection, and uses different receptive fields to restore as many detail features at different scales as possible, thereby improving the reconstruction effect of the more severely degraded areas; the adaptive attention projection block is used to project effective information into the attention domain to ensure the effectiveness and compactness of the projected features. The four cascaded adaptive attention projection blocks gradually restore the depth features in a coarse-to-fine manner; The low-level detail embedding module is used to introduce the color features of the first two layers of the HR color image. By learning a residual mask to highlight the spatial position where the color features are most consistent with the depth features, the high-frequency color guidance is used to supplement the depth features in the form of residual attention, thereby better restoring the details. The high-level semantic guidance module is used to generate a semantic mask that encodes global semantic guidance information using the semantic information of the top layer of the HR color map, and further perform semantic correction on the initial deep reconstruction features.

8. The color-guided depth map super-resolution reconstruction method according to claim 7, characterized in that: The multi-scale content enhancement block consists of a stack of four convolutional layers with different dilation rates to capture details at different receptive field scales.

Images