A Super-Resolution Reconstruction Method for Depth Images Combining Monocular Depth Estimation

Abstract

The present invention belongs to the technical field of image processing, and relates to a depth image super-resolution reconstruction method combined with monocular depth estimation, including: S1, inputting a low-resolution depth image into a depth image super-resolution reconstruction sub-network, and inputting a high-resolution color image corresponding to the low-resolution depth image into a monocular depth estimation sub-network; S2, training the depth image super-resolution reconstruction sub-network and the monocular depth estimation sub-network simultaneously; S3, the depth image super-resolution reconstruction sub-network outputs a high-resolution reconstructed depth image, and the monocular depth estimation sub-network outputs a high-resolution estimated depth image. This application selects high-frequency information from the monocular depth estimation sub-network to guide the depth image super-resolution reconstruction sub-network, and avoids texture replication by suppressing redundancy. Without introducing other supervision information, the method of the present invention achieves competitive performance on multiple publicly available benchmark datasets and can accurately restore the details of the depth image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a depth image super-resolution reconstruction method combined with monocular depth estimation. Background Technique

[0002] The emergence and popularization of portable consumer depth cameras have provided great convenience for accurately and quickly obtaining scene depth. However, due to the limitations of the current depth camera imaging capabilities, the resolution of depth images is usually low. In the face of the demand for high-quality depth images in many application fields, depth image super-resolution reconstruction technology has received extensive attention as a solution. Depth image super-resolution reconstruction is to restore the high-frequency information outside the camera cut-off frequency through algorithms without changing the depth camera, while improving the blurring phenomenon during imaging and effectively suppressing the random noise in the image, so as to reconstruct high-quality and high-resolution depth images. In recent years, with the rapid development of deep learning, the performance of this task has been greatly improved. In practical applications, high-resolution color images are easy to obtain and have strong structural similarity with depth images, so they can provide some prior information for depth image super-resolution reconstruction. Hui et al. [1] designed a multi-scale guided convolutional neural network, and used the rich hierarchical features extracted from color images to improve the blurring phenomenon of images during the depth image super-resolution reconstruction process. Huang et al. [2] proposed a deep dense residual network with a pyramid structure. Through dense connection and residual learning, multi-scale features were fully utilized to predict high-frequency residuals, realizing the super-resolution reconstruction of depth images. Guo et al. [3] proposed a hierarchical feature-driven residual learning depth image super-resolution reconstruction method. With the help of hierarchical color guidance, and through a residual U-Net structure to extract multi-level hierarchical feature representations of depth images, the super-resolution reconstruction of depth images was realized. Wen et al. [4] proposed a color-guided coarse-to-fine depth image super-resolution reconstruction network, which mainly includes: a color-guided information preprocessing module, a coarse-level depth image super-resolution reconstruction module, and a fine-level depth image super-resolution reconstruction module. Zuo et al. [5] proposed a depth image super-resolution reconstruction network based on global and local residual learning. Under the guidance of high-resolution intensity images, the low-resolution depth images were gradually upsampled and reconstructed at multiple scales. Ye et al. [6] proposed a progressive multi-branch aggregation network, and gradually optimized the high-resolution depth image obtained by deconvolution by fusing the reconstruction branch and the guidance branch. Wang et al. [7] proposed an edge-guided depth image super-resolution reconstruction network, and used the edge image learned from color images to upsample the depth image. Song et al. [8]A network for iterative residual depth image super-resolution reconstruction based on channel attention is proposed, which adopts a channel attention strategy to enhance the feature channels with rich high-frequency components.

[0003] The following disadvantages and deficiencies exist in the prior art: (1) Most existing methods learn the reconstruction features of depth images under the guidance of residual frameworks and color images to achieve depth image super-resolution reconstruction. However, there will be certain inconsistencies in cross-modal reconstruction features, and the learned features often have certain information redundancy; (2) In existing multi-task learning frameworks for depth images, additional training labels (such as semantic labels) are usually introduced, which will obviously reduce the generality of the algorithm.

[0004] The present invention aims to associate monocular depth estimation with depth image super-resolution reconstruction through a joint learning method, and replace the color branch in the color-guided depth image super-resolution reconstruction method with a monocular depth estimation network. Since monocular depth estimation can achieve cross-modal information conversion from color images to depth images during continuous training and learning, the color image features learned for monocular depth estimation are closer to the depth modality. Compared with direct feature guidance or edge feature guidance, the features learned by monocular depth estimation are more suitable for guiding depth image super-resolution reconstruction, thus avoiding obvious artifacts and achieving better reconstruction performance.

[0005] List of Definitions of Key Terms

[0006] 1. Depth image super-resolution reconstruction: It is to reconstruct a high-quality and high-resolution depth image through an algorithm on the premise that the depth camera remains unchanged, and effectively suppress the blurring phenomenon and random noise during the imaging process.

[0007] 2. Monocular depth estimation: It is to estimate the distance of each pixel in the image relative to the shooting source by using a single or unique-view color image. Summary of the Invention

[0008] The present invention aims to explore solutions to problems such as color guidance, detail restoration, and modality interaction in the task of depth image super-resolution reconstruction. Specifically, from the perspective of multi-task learning, a depth image super-resolution network that combines monocular depth estimation is studied, and monocular depth estimation and depth image super-resolution are incorporated into a unified framework, and the interactive guidance relationship between the two tasks is explored to achieve the effect of mutual promotion and win-win cooperation. The specific technical solutions are as follows:

[0009] A method for depth image super-resolution reconstruction that combines monocular depth estimation, and the method for depth image super-resolution reconstruction that combines monocular depth estimation is based on a network for depth image super-resolution reconstruction that combines monocular depth estimation;

[0010] The depth image super-resolution reconstruction network that combines monocular depth estimation includes: two sub-networks;

[0011] The two sub-networks are: a monocular depth estimation sub-network and a depth image super-resolution reconstruction sub-network;

[0012] The connection between the monocular depth estimation sub-network and the depth image super-resolution reconstruction sub-network is realized through a bridge;

[0013] The depth image super-resolution reconstruction method that combines monocular depth estimation includes the following steps:

[0014] S1. Input the low-resolution depth image into the depth image super-resolution reconstruction sub-network, and input the high-resolution color image corresponding to the low-resolution depth image into the monocular depth estimation sub-network;

[0015] S2. Train the depth image super-resolution reconstruction sub-network and the monocular depth estimation sub-network simultaneously;

[0016] Integrate the two sub-networks into a unified framework for joint learning;

[0017] S3. The depth image super-resolution reconstruction sub-network outputs a high-resolution reconstructed depth image, and uses the high-resolution reconstructed depth image as the final output.

[0018] Based on the above technical solution, both the monocular depth estimation sub-network and the depth image super-resolution reconstruction sub-network adopt an encoder-decoder structure form, and existing network models can be used as baselines;

[0019] The bridge includes: a high-frequency attention bridge and a content guidance bridge;

[0020] The high-frequency attention bridge is applied to the encoding stage of the encoders of the two sub-networks;

[0021] The content guidance bridge is applied to the decoding stage of the decoders of the two sub-networks to achieve different information interaction and guidance;

[0022] The monocular depth estimation sub-network passes the high-frequency information learned from the high-resolution color image through the high-frequency attention bridge to provide color guidance closer to the depth modality for the depth image super-resolution reconstruction sub-network;

[0023] The depth image super-resolution reconstruction sub-network provides content guidance for the monocular depth estimation sub-network through the content guidance bridge in the depth feature space.

[0024] Reviewing the existing methods for color-guided depth image super-resolution reconstruction, it can be found that the guidance of color images mainly includes two ways: direct guidance of corresponding features or guidance of edge details. Although color images and depth images have strong structural similarities, the rich textures and edges of color images are not always consistent with depth images. Therefore, these methods may lead to problems such as texture replication and depth loss. Monocular depth estimation takes a color image as input, maps the scene from a photometric representation to a geometric representation, and thus generates the corresponding depth image. Therefore, the features of the color image provided by the monocular depth estimation encoder are closer to the feature representation of the depth modality, and thus can be used to guide the depth image super-resolution reconstruction task to avoid obvious artifacts.

[0025] In other words, in the encoder interaction stage of the two sub-tasks, the present invention intends to use the monocular depth estimation features to guide the depth image super-resolution reconstruction features. After clarifying the transmission direction of the guidance information, the next question to consider is how to effectively achieve information interaction and guidance. The simplest and most intuitive method is to directly transfer the features of the corresponding layers of the monocular depth estimation sub-network to the depth image super-resolution reconstruction sub-network through cascading or addition, but this is obviously not a wise choice. In the encoder of the monocular depth estimation sub-network, as the network deepens, the resolution of the feature map gradually decreases. Among them, the high-level features have rich semantic information, while the low-level features have more structural information. Since the low-resolution depth image contains less high-frequency information, the high-resolution color image can provide more important high-frequency information (such as edge details) for the depth image super-resolution reconstruction, rather than the semantic information of the image.

[0026] Based on the above technical solution, the high-frequency attention bridge uses the attention mechanism to screen out valuable high-frequency information from the original features of the monocular depth estimation sub-network (i.e., the features extracted by the encoder) to obtain guiding features and guide the encoder features of the depth image super-resolution reconstruction sub-network.

[0027] Based on the above technical solution, the specific steps to obtain the guiding features are as follows:

[0028] First, perform blurring operations on the original features of the monocular depth estimation sub-network using average pooling and deconvolution operations, as shown in Equation (1).

[0029]

[0030] Among them, is the original feature of the i-th layer of the encoder of the monocular depth estimation sub-network, is the blurred feature of the i-th layer of the encoder of the monocular depth estimation sub-network, represents: for Perform average pooling operation, It means: perform deconvolution operation on ;

[0031] Then, by subtracting the original features from the blurred features, high-frequency information is obtained, and then the attention of the high-frequency information is generated, as shown in Equation (2),

[0032]

[0033] where, is the high-frequency attention of the i-th layer of the encoder of the monocular depth estimation sub-network, It means: perform a parametric rectified linear unit transformation operation on , that is, the activation function;

[0034] Next, the obtained high-frequency attention is used to correct and optimize the original features of the monocular depth estimation sub-network. Through residual connection, the optimized guiding features are finally obtained, as shown in Equation (3),

[0035]

[0036] where, is the optimized guiding feature of the i-th layer of the encoder of the monocular depth estimation sub-network.

[0037] The reason for defining the above operations is to highlight the high-frequency information in the original features of the monocular depth estimation sub-network, so that the low-resolution depth image can make the most of the high-frequency information during feature fusion.

[0038] On the basis of the above technical solution, in order to use the optimized guiding features from the encoder of the monocular depth estimation sub-network to guide the encoder features of the depth image super-resolution reconstruction sub-network, the specific steps are as follows:

[0039] First, the guiding features are concatenated with the features of the corresponding layer of the encoder of the depth image super-resolution reconstruction sub-network in the channel dimension to generate the composite features of this layer of the encoder

[0040] There will be a lot of redundancy in the above simple feature fusion in the spatial dimension and the channel dimension. Therefore, then, an attention block is introduced to enhance the feature fusion ability;

[0041] The attention block includes: a channel attention and a spatial attention;

[0042] The channel attention is used to: learn the importance of each feature channel;

[0043] The spatial attention is used to highlight important positions in the feature map, and the above process is expressed by Equations (4) and (5).

[0044]

[0045]

[0046] Wherein, is the feature of the i-th layer of the encoder of the deep image super-resolution reconstruction sub-network, is the feature after fusing high-frequency information of the i-th layer of the encoder of the deep image super-resolution reconstruction sub-network, CA is channel attention, SA is spatial attention, and conv 1×1 is a convolutional layer with a convolution kernel size of 1×1. represents concatenating and in the channel dimension;

[0047] The feature after fusing high-frequency information is used as the input of the next layer of the encoder of the deep image super-resolution reconstruction sub-network.

[0048] For the feature decoding stages of the deep image super-resolution reconstruction sub-network and the monocular depth estimation sub-network, their roles are to further extract task-oriented features to respectively complete super-resolution reconstruction and depth estimation. Finally, corresponding estimated or super-resolution reconstructed depth images can be obtained from the two sub-networks. Comparing the two tasks, monocular depth estimation is widely recognized as an ill-posed inverse problem due to its scale ambiguity. For example, many three-dimensional scenes observed in the world can correspond to exactly the same two-dimensional plane, that is, they are not one-to-one. Therefore, training a model that can map well from a color image to a depth image is a very difficult task. Although deep image super-resolution reconstruction is also an ill-posed problem, it still learns the mapping relationship in the same domain and focuses on restoring the details of the image, which is relatively simpler than monocular depth estimation. Considering the difference in the difficulty levels of the two tasks, the features generated by the decoder of the monocular depth estimation sub-network are no longer suitable for providing guidance information for the decoder of the deep image super-resolution reconstruction sub-network. Following the principle of the simple task guiding the difficult task, the present invention intends to exchange the guiding identities of the two sub-networks in the decoding stage, that is, to let the deep image super-resolution reconstruction sub-network provide content guidance for the monocular depth estimation sub-network in the depth feature space.

[0049] Based on the above technical solution, corresponding depth images are obtained through the decoder features of the two sub-networks.

[0050] Based on the above technical solution, the specific steps for the depth image super-resolution reconstruction sub-network to provide content guidance for the monocular depth estimation sub-network through the content guidance bridge in the depth feature space are as follows:

[0051] First, convolutional layers with a kernel size of 1×1 are applied to the decoders of the depth image super-resolution reconstruction sub-network and the monocular depth estimation sub-network respectively, so as to obtain the super-resolution reconstructed depth image and the estimated depth image, as shown in Equations (6) and (7).

[0052]

[0053]

[0054] Among them, is the reconstructed depth image generated by the i-th layer of the decoder of the depth image super-resolution reconstruction sub-network, is the estimated depth image generated by the i-th layer of the decoder of the monocular depth estimation sub-network, and are the features of the i-th layer of the decoders of the depth image super-resolution reconstruction sub-network and the monocular depth estimation sub-network respectively;

[0055] Then, the difference map between the reconstructed depth image and the estimated depth image is calculated. The difference map highlights the positions in the estimated depth image that need to be further optimized relative to the super-resolution reconstructed depth image, and it is hoped that this difference will become smaller and smaller as the network is trained;

[0056] Next, the difference weights are learned by applying convolutional operations and softmax activation to the difference map, so as to provide content guidance for the monocular depth estimation sub-network. The above operations are described as: Equations (8) and (9).

[0057]

[0058]

[0059] Among them, is the difference weight, is the content guidance feature of the i-th layer, and softmax is the softmax activation function;

[0060] Finally, an attention block is introduced to optimize the composite feature of this layer of the decoder

[0061] The attention block includes: a channel attention and a spatial attention;

[0062] The above process is shown in Equations (10) and (11).

[0063]

[0064]

[0065] Among them, CA is channel attention and SA is spatial attention. It means: and are concatenated in the channel dimension, is the optimized feature;

[0066] The optimized feature is used as the input of the next layer in the decoder of the monocular depth estimation sub-network.

[0067] Different from the loss functions of other multi-task learning which are the weighted sum of all branch loss functions, the present invention respectively assigns different optimizers to the loss functions of depth image super-resolution reconstruction and monocular depth estimation. This is because the learning difficulties of depth image super-resolution reconstruction and monocular depth estimation are quite different, resulting in different convergence speeds of the two tasks, so it is difficult to find a suitable weight setting to ensure that both tasks achieve the best performance. Therefore, in terms of the design of the loss function, the present invention proposes a strategy of respectively optimizing the relevant parts of depth image super-resolution reconstruction and monocular depth estimation.

[0068] Based on the above technical solutions, the loss functions for optimizing the depth image super-resolution reconstruction task and the monocular depth estimation task are respectively shown in Equations (12) and (13),

[0069]

[0070]

[0071] Among them, and are the per-pixel L1 losses of the depth image super-resolution reconstruction task and the monocular depth estimation task respectively; D SR and D DE are the outputs of the depth image super-resolution reconstruction sub-network and the monocular depth estimation sub-network respectively, that is, respectively: the reconstructed depth image with high resolution and the estimated depth image with high resolution, D HR is the high-resolution depth image corresponding to the input image of the two sub-networks;

[0072] The per-pixel L1 loss refers to: the loss of comparing differences per pixel; ||D SR -D HR ||1 means: taking the 1-norm of the difference between the output of the depth image super-resolution reconstruction sub-network and the high-resolution depth image corresponding to the input image; ||DDE -D HR ||1 represents: taking the 1-norm of the difference between the output of the monocular depth estimation sub-network and the high-resolution depth image corresponding to the input image.

[0073] The beneficial technical effects of the present invention are as follows:

[0074] Due to cross-modal reasons, the texture and edges of color images are not always consistent with depth images. Therefore, direct feature or edge guidance may lead to problems such as texture replication. For the monocular depth estimation task, the network maps the color image to the corresponding depth image, can learn features related to scene depth from the color image, and weaken the influence of irrelevant textures. In other words, the color features learned in the monocular depth estimation task are closer to the depth modality and can reduce obvious artifacts under its guidance. However, simple interaction methods (such as loss function constraints) are not sufficient to effectively achieve multi-task learning. Therefore, the high-frequency attention bridge aims to select more valuable high-frequency information from the monocular depth estimation sub-network to guide the depth image super-resolution reconstruction sub-network, thereby avoiding texture replication by suppressing unnecessary redundancy.

[0075] Without introducing other supervision information, the method of the present invention has achieved competitive performance on multiple publicly available benchmark datasets. The method of the present invention can accurately restore the details of the depth image. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The present invention has the following drawings:

[0077] Figure 1 is a schematic diagram of the overall architecture of the depth image super-resolution reconstruction network for joint monocular depth estimation;

[0078] Figure 2 is a schematic diagram of a visualization example of the application of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0079] In order to describe the present invention more specifically, the technical solutions of the present invention will be described in more detail below in conjunction with the drawings and specific embodiments. It should be emphasized that the following description is merely exemplary and not intended to limit the scope of the present invention and its applications.

[0080] Such as Figure 1As shown in the figure, the depth image super-resolution reconstruction network that combines monocular depth estimation consists of two sub-networks (i.e., the depth image super-resolution reconstruction sub-network and the monocular depth estimation sub-network) and two bridges (i.e., the high-frequency attention bridge and the content-guided bridge). The depth image super-resolution reconstruction sub-network (DSRNet) and the monocular depth estimation sub-network (MDENet) are integrated into a unified framework to achieve joint learning of depth image super-resolution reconstruction and monocular depth estimation. The high-frequency attention bridge (HABdg) is applied to the encoders of the two sub-networks, and the content-guided bridge (CGBdg) is applied to the decoders of the two sub-networks to bridge the two tasks (i.e., the monocular depth estimation task and the depth image super-resolution reconstruction task).

[0081] Due to cross-modal reasons, the texture and edges of color images are not always consistent with depth images. Therefore, direct feature or edge guidance may lead to problems such as texture replication. For the monocular depth estimation task, the monocular depth estimation sub-network maps the color image to the corresponding depth image, can learn features related to the scene depth from the color image, and weaken the influence of irrelevant textures. In other words, the color features learned in the monocular depth estimation task are closer to the depth modality, and obvious artifacts can be reduced under its guidance. However, simple interaction methods (such as loss function constraints) are not sufficient to effectively achieve multi-task learning. Therefore, the high-frequency attention bridge aims to select more valuable high-frequency information from the monocular depth estimation sub-network to guide the depth image super-resolution reconstruction sub-network, thereby avoiding texture replication by suppressing unnecessary redundancy.

[0082] Without introducing other supervision information, the method of the present invention has achieved competitive performance on multiple publicly available benchmark datasets. As Figure 2 shown in the figure, a visualization example of the technical method of the present invention is given. The first column is the color image, the second column is the low-resolution depth (the depth image has been enlarged for clearer visualization) image, the third column is the high-resolution depth image reconstructed by the method of the present invention, and the fourth column is the ground truth of the high-resolution depth image. It can be seen from the results that the method of the present invention can accurately restore the details of the depth image. For example, the contour of the toy head in the Doll image (the first row) is more accurate, and the edges of the small objects in the Laundry image (the second row) are more distinct.

[0083] The key points and points to be protected by the present invention are as follows:

[0084] (1) In the joint learning network of the present invention, the deep image super-resolution reconstruction task and the monocular depth estimation task are associated to improve the performance of deep image super-resolution reconstruction. The joint learning network proposed by the present invention includes: a deep image super-resolution reconstruction subnet (DSRNet) and a monocular depth estimation subnet (MDENet), as well as two bridges for joint learning, namely a high-frequency attention bridge (HABdg) and a content-guided bridge (CGBdg). The overall network structure of the present invention has high portability and can provide an example for associating the deep image super-resolution reconstruction and the monocular depth estimation tasks. In addition, different from other multi-task learning, the two tasks used for joint learning in the present invention do not require the introduction of other supervision information.

[0085] (2) The high-frequency attention bridge (HABdg) in the feature encoding stage transmits the color high-frequency information learned from the monocular depth estimation subnet to the deep image super-resolution reconstruction subnet, so as to provide color guidance information closer to the depth modality. Following the principle of simple tasks guiding difficult tasks, the present invention switches the guiding roles of the two tasks in the feature decoding stage and proposes a content-guided bridge (CGBdg), so that the deep image super-resolution reconstruction subnet can provide content guidance for the monocular depth estimation subnet in the depth feature space.

[0086] References (such as patents / papers / standards) are listed as follows:

[0087] [1] Hui T W, Loy C C, Tang X. Depth map super-resolution by deep multi-scale guidance[C] / / European conference on computer vision. Springer, Cham, 2016:353-369.

[0088] [2] L. Huang, J. Zhang, Y. Zuo and Q. Wu, “Pyramid-structured depth map super-resolution based on deep dense-residual network,” IEEE Signal Process. Lett., 2019, vol. 26, no. 12, pp. 1723–1727.

[0089] [3] C. Guo, C. Li, J. Guo, R. Cong, H. Fu and P. Han, “Hierarchical featuresdriven residual learning for depth map super-resolution,” IEEE Trans. Image Process., 2019, vol. 28, no. 5, pp. 2545–2557.

[0090] [4] Y. Wen, B. Sheng, P. Li, W. Lin and D. D. Feng, “Deep color guided coarse-to-fine convolutional network cascade for depth image super-resolution,” IEEE Trans. Image Process., 2019, vol. 28, no. 2, pp. 994–1006.

[0091] [5] Y. Zuo, Q. Wu, Y. Fang, P. An, L. Huang and Z. Chen, “Multi-scale frequencyreconstruction for guided depth map super-resolution via deep residualnetwork,” IEEE Trans. Circuits Syst. Video Techn., 2020, vol. 30, no. 2, pp. 297–306.

[0092] [6] X. Ye, B. Sun and Z. Wang, “Pmbanet: Progressive multi-branchaggregation network for scene depth super-resolution,” IEEE Trans. Image Process, 2020, vol. 29, pp. 7427-7442.

[0093] [7] Z. Wang, X. Ye and B. Sun, “Depth upsampling based on deep edge-awarelearning,” Pattern Recognit., 2020, vol. 103, pp. 107274.

[0094] [8]X. Song, Y. Dai and D. Zhou, “Channel attention based iterative residual learning for depth map super-resolution,” in CVPR, 2020, pp. 5631-5640.

[0095] As described above, it is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

[0096] The content not described in detail in this specification belongs to the prior art well-known to those skilled in the art.

Claims

1. A super-resolution reconstruction method for depth images combined with monocular depth estimation, characterized in that: The super-resolution reconstruction method of depth image based on joint monocular depth estimation is based on the super-resolution reconstruction network of depth image based on joint monocular depth estimation; The super-resolution reconstruction network of depth image based on joint monocular depth estimation includes: two sub-networks; The two sub-networks are: a monocular depth estimation sub-network and a super-resolution reconstruction sub-network of depth image; The association between the monocular depth estimation sub-network and the super-resolution reconstruction sub-network of depth image is realized through a bridge; The super-resolution reconstruction method of depth image based on joint monocular depth estimation includes the following steps: S1. Input the low-resolution depth image into the super-resolution reconstruction sub-network of depth image, and input the high-resolution color image corresponding to the low-resolution depth image into the monocular depth estimation sub-network; S2. Train the super-resolution reconstruction sub-network of depth image and the monocular depth estimation sub-network simultaneously; S3. The super-resolution reconstruction sub-network of depth image outputs a high-resolution reconstructed depth image, and takes the high-resolution reconstructed depth image as the final output; Both the monocular depth estimation sub-network and the super-resolution reconstruction sub-network of depth image adopt the encoder-decoder structure form; The bridge includes: a high-frequency attention bridge and a content guidance bridge; The high-frequency attention bridge is applied to the encoding stage of the encoders of the two sub-networks; The content guidance bridge is applied to the decoding stage of the decoders of the two sub-networks; The monocular depth estimation sub-network provides color guidance for the super-resolution reconstruction sub-network of depth image through the high-frequency attention bridge with the high-frequency information learned from the high-resolution color image; The super-resolution reconstruction sub-network of depth image provides content guidance for the monocular depth estimation sub-network through the content guidance bridge in the depth feature space; The high-frequency attention bridge uses the attention mechanism to filter out high-frequency information from the original features of the monocular depth estimation sub-network and uses it as the guiding feature to guide the encoder features of the super-resolution reconstruction sub-network of depth image.

2. The super - resolution reconstruction method of depth images for joint monocular depth estimation according to claim 1, wherein: The specific steps of filtering out high-frequency information from the original features of the monocular depth estimation sub-network and using it as the guiding feature to guide the encoder features of the super-resolution reconstruction sub-network of depth image are as follows: First, perform a blurring operation on the original features of the monocular depth estimation sub-network using average pooling and deconvolution operations, as shown in Equation (1), ; Among them, is the original feature of the i th layer of the encoder of the monocular depth estimation sub-network, is the blurred feature of the i th layer of the encoder of the monocular depth estimation sub-network, avgpool denotes: performing average pooling operation on , deconv ( avgpool ( )) denotes: performing deconvolution operation on avgpool . Then, subtract the original features from the blurred features to obtain high-frequency information, and further generate the attention of high-frequency information, as shown in Equation (2), ; Among them, is the high-frequency attention of the i layer of the encoder of the monocular depth estimation sub-network, indicating: performing a parametric rectified linear unit transformation operation on ; Next, use the obtained high-frequency attention to correct and optimize the original features of the monocular depth estimation sub-network, and finally obtain the optimized guiding feature through residual connection, as shown in Equation (3), ; Among them, is the optimized guiding feature of the i layer of the encoder of the monocular depth estimation sub-network.

3. The depth image super-resolution reconstruction method for joint monocular depth estimation according to claim 2, characterized in that: The specific steps of guiding the encoder features of the super-resolution reconstruction sub-network of depth image are as follows: First, the guiding feature is concatenated with the feature of the corresponding layer of the encoder of the depth image super-resolution reconstruction sub-network at the channel dimension to generate the composite feature of this layer of the encoder. ; Then, introduce an attention block; The attention block includes: a channel attention and a spatial attention; The channel attention is used for: learning the importance of each feature channel; The spatial attention is used to highlight important positions in the feature map, and the above process is expressed as equations (4) and (5). ; ; Among them, is the feature of the i th layer of the encoder of the depth image super-resolution reconstruction sub-network, is the feature after fusing high-frequency information of the i th layer of the encoder of the depth image super-resolution reconstruction sub-network, CA is the channel attention, SA is the spatial attention, is a convolutional layer with a convolution kernel size of 1×1, means: perform concatenation in the channel dimension on and ; The feature that incorporates high-frequency information is used as the input for the next layer of the encoder of the deep image super-resolution reconstruction sub-network.

4. The depth image super-resolution reconstruction method for combined monocular depth estimation according to claim 3, wherein: Obtain the corresponding depth image through the decoder features of the two sub-networks.

5. The depth image super-resolution reconstruction method combined with monocular depth estimation according to claim 4, characterized in that: The specific steps for the depth image super-resolution reconstruction sub-network to provide content guidance for the monocular depth estimation sub-network through the content guidance bridge in the depth feature space are as follows: First, convolutional layers with a kernel size of 1×1 are applied to the decoders of the depth image super-resolution reconstruction sub-network and the monocular depth estimation sub-network respectively, so as to obtain the super-resolution reconstructed depth image and the estimated depth image, as shown in equations (6) and (7). ; ; Among them, is the reconstructed depth image generated by the i th layer of the decoder of the depth image super-resolution reconstruction sub-network, is the estimated depth image generated by the i th layer of the decoder of the monocular depth estimation sub-network, and are the features of the i th layer of the decoders of the depth image super-resolution reconstruction sub-network and the monocular depth estimation sub-network respectively; Then, a reconstructed depth image is calculated. and the estimated depth image to obtain a difference map therebetween; Next, the difference weights are learned by applying convolutional operations and softmax activation to the difference map, so as to provide content guidance for the monocular depth estimation sub-network. The above operations are described as equations (8) and (9). ; ; Among them, is the difference weight, is the content guidance feature of the i th layer, softmax is the softmax activation function; Finally, an attention block is introduced to optimize the composite features of this layer of the decoder ; The attention block includes a channel attention and a spatial attention. The above process is shown in equations (10) and (11). ; ; Among them, CA is the channel attention, SA is the spatial attention, means: concatenate and along the channel dimension, is the optimized feature; Use the optimized features as the input to the next layer in the decoder of the monocular depth estimation sub-network.

6. The depth image super-resolution reconstruction method for joint monocular depth estimation according to claim 5, characterized in that: The loss functions for optimizing the depth image super-resolution reconstruction task and the monocular depth estimation task respectively are shown in equations (12) and (13). ； ； Among them, and are the per-pixel losses for the depth image super-resolution reconstruction task and the monocular depth estimation task, respectively; and are the outputs of the depth image super-resolution reconstruction sub-network and the monocular depth estimation sub-network, respectively, is the high-resolution depth image corresponding to the input images of the two sub-networks; The per-pixel loss refers to: the loss of comparing differences per pixel; It means: calculating the 1-norm of the difference between the output of the depth image super-resolution reconstruction sub-network and the high-resolution depth image corresponding to the input image; It means: calculating the 1-norm of the difference between the output of the monocular depth estimation sub-network and the high-resolution depth image corresponding to the input image.

Images