A real-time monocular depth estimation method integrating global features

Abstract

The present invention relates to the field of deep learning-based monocular depth estimation algorithms and discloses a real-time monocular depth estimation method that integrates global features. The method comprises the following steps: a three-channel color image is input into a multi-scale feature extractor based on a block embedding layer for feature extraction. The image is gradually divided into four scales in the multi-scale feature extractor. At each scale, a standard convolution layer is used to extract basic features of the image to obtain a feature map. A block embedding layer is used for downsampling, reducing the number of parameters in the extraction network. The block embedding layer is followed by a residual block consisting of two layers of standard convolution and an additive short circuit for information fusion. Due to the design of the present invention using a small number of convolution layers, the feature extractor has the characteristics of low latency and can quickly complete feature extraction at four scales.

Description

Technical Field

[0001] The present invention relates to the technical field of monocular depth estimation algorithms based on deep learning, and in particular to a real-time monocular depth estimation method integrating global features. Background Art

[0002] Depth estimation, a fundamental task in computer vision, has important applications in autonomous driving, 3D scene reconstruction, augmented reality, robotics, and the Internet of Things. Currently, effective methods for acquiring depth information include lidar (LiDAR), depth cameras, and depth estimation based on specialized algorithms and cameras. As active sensors, lidar and depth cameras rely on dedicated hardware components, resulting in high costs. In contrast, depth estimation algorithms using monocular cameras are more cost-effective. Monocular depth estimation methods can run on relatively low-cost devices, but their accuracy is limited. In real-world applications, both accuracy and speed are critical. Therefore, research on high-precision, lightweight, and real-time monocular depth estimation algorithms on memory- or power-constrained devices is of great practical significance. Traditional monocular depth estimation algorithms leverage multi-view geometry theory to perform depth estimation. However, due to the lack of geometric information in a single image, traditional methods cannot accurately predict the depth of unconnected objects. In recent years, the powerful feature extraction capabilities of deep convolutional networks, capable of extracting high-dimensional feature information, and their success in high-level vision tasks have driven progress in deep learning-based monocular depth estimation. However, since current high-precision monocular depth estimation algorithms often have problems such as high computational cost, high power consumption, and delay, current methods are difficult to be deployed in real time on devices with limited power or memory.

[0003] The existing real-time monocular depth estimation algorithm uses a low-latency, high-precision classification network backbone. For example, Wofk et al. implemented real-time monocular depth estimation on an NVIDIA Jetson TX2 embedded device. This method improved the decoder based on the MobileNet classification network and rebuilt a lightweight depth estimation algorithm (see D. Wofk, F. Ma, T. -J. Yang, S. Karaman and V. Sze, “FastDepth: Fast Monocular Depth Estimation on Embedded Systems,” in Proc. Int. Conf. Robots Autom., Montreal, QC, Canada, 2019, pp. 6101-6108). Yucel et al. used a subnetwork learned by the auxiliary network and used a pixel-level regression loss to reduce the loss of the network's intermediate prediction layer, guiding the network to output a more accurate depth map (see MK Yucel, V. Dimaridou, A. Drosou and A. Saa-Garriga, “Real-time Monocular Depth Estimation with Sparse Supervision on Mobile,” in Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit., Nashville, TN, USA, 2021, pp. 2428-2437). Currently, monocular depth estimation algorithms based on convolutional neural networks can achieve real-time inference with a certain degree of accuracy on embedded devices. However, during image processing, images undergo multi-scale convolutional layers, which results in a loss of spatial and channel information. Furthermore, the limited receptive field of convolution fragments the global image, allowing only local features to be extracted. Therefore, there is little room for improvement in the accuracy of real-time monocular depth estimation methods based on convolutional neural networks.Among the emerging computer vision methods, the Transformer is capable of processing images with a global receptive field and outperforms conventional convolutional neural networks in image classification tasks (see A. Dosovitskiy, L. Beyer, A. Kolesnikov, D.Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” 2020, arXiv:2010.11929). Mehta et al. leverage the high accuracy of the Transformer and the low latency of convolutional networks to propose a method that fuses local and global information, achieving a good balance between accuracy and speed. However, these methods still cannot achieve ideal performance on devices with limited power or memory.

[0004] In summary, the shortcomings of the existing technology are as follows:

[0005] (1) Most existing monocular depth estimation methods are only implemented through convolutional networks, and deep convolutional networks usually lose more image feature information;

[0006] (2) Most existing monocular depth estimation methods require long inference time and cannot run in real time on edge devices;

[0007] (3) Existing real-time algorithms cannot strike a good balance between inference speed and accuracy, and cannot generate more accurate depth maps on edge devices. To this end, we propose a real-time monocular depth estimation method that integrates global features. Summary of the Invention

[0008] (1) Technical problems solved

[0009] In view of the shortcomings of the existing technology, the present invention provides a real-time monocular depth estimation method integrating global features to solve the above problems.

[0010] (2) Technical solution

[0011] To achieve the above-mentioned purpose, the present invention provides the following technical solutions:

[0012] A real-time monocular depth estimation method integrating global features comprises the following steps:

[0013] Step 1: The three-channel color image is input into the multi-scale feature extractor based on the block embedding layer for feature extraction. The image is gradually divided into four scales in the multi-scale feature extractor. At each scale, the basic features of the image are extracted through the standard convolution layer to obtain the features. Figure 1 ;

[0014] Step 2: Semantic feature extractor receives features Figure 1 , feature extraction is performed under different receptive fields, the features extracted by multi-layer dilated convolution are spliced ​​together, and feature weights are assigned through channel attention. At the same time, the obtained weights are multiplied with the feature matrix to obtain feature 2;

[0015] Step 3: The global feature extractor receives feature 2 and fuses the local features with the global features. In the global feature extractor, the three-dimensional feature matrix is ​​transformed by flattening and folding to extract global features under the Transformer. In addition, the local features are spliced ​​with the global features through skip connections and fused through standard convolution to obtain features. Figure 3 ;

[0016] Step 4: The decoder restores the low-resolution features extracted by the semantic feature extractor to the resolution of the input image through upsampling. At the same time, the decoder combines the basic features extracted by the multi-scale feature extractor, performs image reconstruction and completes depth allocation, and outputs the final depth map.

[0017] Preferably, the backbone of the feature extractor in the first step consists of a block embedding layer and two 3×3 convolutional layers, respectively Figure 1 / 16, 1 / 8, 1 / 4 and 1 / 2 resolutions, and generate the original Figure 1 Coarse feature maps with 1 / 16, 1 / 8, 1 / 4, and 1 / 2 resolutions.

[0018] Preferably, the semantic feature extractor in the second step includes an adaptive pooling layer, a 1×1 convolution layer and three dilated convolution layers.

[0019] Preferably, the global feature extractor in the third step includes standard convolution, feature block flattening and folding and Transformer. The feature matrix first passes through a 1×1 convolution layer. After the feature matrix is ​​expanded, it is sent to the Transformer module for feature extraction under the global receptive field. The feature is then folded and restored to its original dimension, and then sent to a 5×5 standard convolution for fusion of local and global information.

[0020] Preferably, in the expansion and folding parts, the resolution of the input feature is H×W, the resolution of the picture block is h×w+N, the dimensions C and P are the number of channels of the feature block and the number of picture blocks respectively, and folding is the opposite operation of expansion.

[0021] Preferably, the decoder in the fourth step is composed of 4 decoder basic backbones, and the backbone is composed of 1 bilinear interpolation and 1 5×5 convolution layer. After the bilinear interpolation is completed, the features obtained by the multi-scale feature extractor are connected and then feature fusion is performed to reduce the information loss caused by convolution. Finally, the 5×5 convolution layer in the basic backbone is replaced by a 1×1 convolution layer to directly output the final depth map.

[0022] Preferably, the combined loss function is used to perform network learning and training, and the overall expression of this loss function combination is:

[0023]

[0024] in =0.1, is the depth prediction value of a pixel, is the true depth value of a pixel, Represents the mean absolute error, which is the absolute average of the difference between the predicted depth value and the true depth value. It is the SSIM loss function.

[0025] Preferably, the loss function is obtained by comparing the similarity between the predicted value and the true value in three dimensions: brightness, contrast and structure.

[0026] (3) Beneficial effects

[0027] Compared with the existing technology, the real-time monocular depth estimation method integrating global features provided by the present invention has the following beneficial effects:

[0028] This real-time monocular depth estimation method, which integrates global features, uses a block embedding layer for downsampling, reducing the number of parameters in the extraction network. This block embedding layer is followed by a residual block consisting of two standard convolution layers and an additive short-circuit for information fusion. Due to its small number of convolutional layers, the feature extractor has low latency and can quickly complete feature extraction at four scales.

[0029] 2. This real-time monocular depth estimation method, which integrates global features, proposes a method that combines dilated convolution groups with channel attention to better integrate semantic features into the network. The dilated convolution groups extract large-area image features at different receptive fields, while the channel attention assigns weights to the combined features, selecting semantic features of greater interest to the network, thereby improving depth estimation accuracy.

[0030] 3. This real-time monocular depth estimation method integrates global features. The network transforms the original Figure 1 The global feature extractor uses 16x16 resolution feature inputs to improve depth estimation accuracy while maintaining low latency. This significantly reduces the number of parameters required for global feature extraction, integrates local and global features, and improves the network's sensitivity to image details.

[0031] 4. This real-time monocular depth estimation method, which integrates global features, uses four basic backbones to rapidly upsample to the original image resolution and assign depth. Furthermore, the decoder incorporates essential features from the multi-scale feature extractor, reducing information loss introduced by the convolutional layers and improving network accuracy. This method achieves a certain level of accuracy on the NYU Depth v2 indoor dataset, with evaluation metrics such as RMSE and δ1 reaching 0.554 and 78%, respectively. This method enables real-time disparity estimation on power- or memory-constrained edge devices (such as the NVIDIA Jetson Nano), which has significant implications for autonomous driving, robotics, smart homes, smart cities, and the Internet of Things. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the overall composition and flow chart of the real-time monocular depth estimation method integrating global features according to an embodiment of the present invention;

[0033] Figure 2 It is the basic backbone of the multi-scale feature extractor based on the block embedding layer of the embodiment of the present invention, extracting feature diagrams with resolutions of 1 / 16, 1 / 8, 1 / 4 and 1 / 2;

[0034] Figure 3 4 is a detailed flowchart of a global feature extractor based on MobileViT according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] Example

[0037] See also Figure 1-3 The embodiment of the present invention provides a real-time monocular depth estimation method integrating global features. The overall composition framework and process are as follows: Figure 1This method is a U-shaped structure of encoder and decoder, which is divided into four stages to extract features, fuse features, restore scenes and assign depth, and finally obtain a depth map.

[0038] In the first stage, the network uses a multi-scale feature extractor based on the block embedding layer to Figure 1 Feature extraction is performed at resolutions of 1 / 16, 1 / 8, 1 / 4, and 1 / 2. Since the multi-scale feature extractor has 14 layers of convolution operations, the running time of the first stage is very small.

[0039] In the second stage, the semantic feature extractor receives the feature map obtained from the first stage, which is sent to the ASPP module for feature extraction under receptive fields of different scales, and then sent to the standard convolution and channel attention to further filter the semantic features of interest to the network.

[0040] In the third stage, the global feature extractor receives the feature map obtained from the second stage. After the feature is dimensionally transformed, it is sent to the Transformer module for feature extraction under the global receptive field. The feature is then dimensionally transformed again, restored to its original dimension, and then sent to the standard convolution for fusion of local and global information.

[0041] In the fourth stage, the decoder receives the feature map obtained from the third stage, upsamples the features to the resolution of the original image under the four basic backbones, completes the depth allocation, and finally outputs the depth map.

[0042] The specific steps include the following:

[0043] 1) Build a multi-scale feature extractor based on the block embedding layer

[0044] This paper proposes a lightweight and high-precision multi-scale feature extraction network, such as Figure 2 As shown in the figure, the backbone of the feature extractor consists of a block embedding layer and two 3×3 convolution layers. The block embedding layer divides the input image with a resolution of n*n into n / 2 image blocks, and then sends these n / 2 image blocks to the 2D convolution to obtain the feature matrix, while increasing the number of channels of the feature matrix; the two 3×3 convolutions further fuse the spatial features in the feature matrix and use the addition shortcut to reduce the information loss characteristics of the convolution filter. The multi-scale feature extractor uses 4 backbones, each of which is located in the original Figure 1 Feature extraction is performed at resolutions of 1 / 16, 1 / 8, 1 / 4, and 1 / 2. In addition, to add more spatial features, two more 3×3 convolutional layers are added to the last backbone.

[0045] 2) Build a semantic feature extractor based on the ASPP module

[0046] This paper proposes a feature extractor for capturing image features in a large receptive field and filtering features through channel attention, such as Figure 1 As shown in Figure 2, the ASPP module is a parallel structure consisting of an adaptive pooling layer, a 1×1 convolutional layer, and three dilated convolutional layers. Furthermore, the network uses three dilated convolutions with different receptive fields (RFFs) of 6, 12, and 18. This feature extractor adds a 1×1 convolutional layer and a channel attention layer after the merged features to filter image features extracted with a large receptive field. By fusing semantic features, the network can provide more information for scene reconstruction, thereby improving the accuracy of depth estimation.

[0047] 3) Build a global feature extractor based on MobileViT

[0048] The global feature extractor based on MobileViT provided by the present invention consists of three parts: standard convolution, feature block flattening, folding and Transformer. Figure 3 As shown in the figure, the feature matrix first passes through a 1×1 convolution layer to reduce the number of channels. After the feature matrix is ​​expanded, it is sent to the Transformer module for feature extraction within the global receptive field. The features are then folded to restore their original dimensions and then sent to a 5×5 standard convolution to fuse local and global information. In the expansion and folding parts, the resolution of the input features is H×W, and the resolution of the image block is h×w (N). The dimensions C and P are the number of channels and image blocks, respectively. Folding is the opposite of expansion. By fusing local and global features, the network increases its sensitivity to detailed information, thereby further improving the accuracy of depth estimation.

[0049] 4) Build a decoder

[0050] The decoder provided by the present invention can quickly upsample and restore the input image scene while maintaining a certain degree of accuracy. Figure 1 As shown in the figure, the decoder consists of four basic decoder backbones, each consisting of a bilinear interpolation layer and a 5×5 convolutional layer. After the bilinear interpolation is completed, features obtained by the multi-scale feature extractor are integrated and then fused to reduce information loss caused by convolution. The final 5×5 convolutional layer in the basic backbone is replaced with a 1×1 convolutional layer, which directly outputs the final depth map.

[0051] 5) Loss Function

[0052] Since the L1 loss function is used alone, it is difficult for the network to obtain the optimal solution, and its depth accuracy is also reduced to a certain extent. The present invention uses a combination of loss functions to train the network, which can improve the accuracy of the network and make the output depth map more consistent with the human eye. The overall expression of this loss function combination is:

[0053]

[0054] in =0.1, is the depth prediction value of a pixel, is the true depth value of a pixel, Represents the mean absolute error, which is the absolute average of the difference between the predicted depth value and the true depth value. The SSIM loss function is obtained by comparing the similarity between the predicted value and the true value in three dimensions: brightness, contrast and structure.

[0055] In summary, this patent proposes an efficient encoder-decoder monocular depth prediction network for real-time depth estimation of embedded devices. The proposed method utilizes a semantic feature extractor and a global feature extractor to obtain key features of the scene for effective depth prediction. The self-attention of the global feature extractor provides feature enhancement of the global receptive field to improve the details of the depth estimation. The lightweight structure of the multi-scale feature extractor and decoder based on the block embedding layer proposed in this patent reduces the running time of the entire network. In addition, this patent improves the quality and accuracy of visualization through joint loss strategy training.

[0056] Most existing techniques are generally unable to perform fast and high-precision disparity estimation on power- or memory-constrained devices. In contrast, our invention can be deployed in real time on resource-constrained edge devices (NVIDIA Jetson Nano) and perform depth estimation with high accuracy.

[0057] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A real-time monocular depth estimation method integrating global features, characterized in that: The following steps are involved: Step 1: The three-channel color image is input into a multi-scale feature extractor based on a block embedding layer for feature extraction. The image is gradually divided into four scales in the multi-scale feature extractor. At each scale, the basic features of the image are extracted through a standard convolutional layer to obtain feature map 1. Step 2: The semantic feature extractor receives feature map 1 and extracts features under different receptive fields. The features extracted by multiple layers of dilated convolution are concatenated and feature weights are assigned through channel attention. At the same time, the obtained weights are multiplied by the feature matrix to obtain feature map 2. Step 3: The global feature extractor receives feature 2 and fuses the local features with the global features. In the global feature extractor, the 3D feature matrix is ​​transformed by flattening and folding to extract global features under the Transformer. In addition, the local features are concatenated with the global features through skip connections and fused through standard convolution to obtain feature map 3. Step 4: The decoder restores the low-resolution features extracted by the semantic feature extractor to the resolution of the input image through upsampling. At the same time, the decoder combines the basic features extracted by the multi-scale feature extractor, performs image reconstruction and completes depth allocation, and outputs the final depth map; The backbone of the feature extractor in the first step consists of a block embedding layer and two 3×3 convolutional layers, which extract features at resolutions of 1 / 16, 1 / 8, 1 / 4, and 1 / 2 of the original image, and generate coarse feature maps at resolutions of 1 / 16, 1 / 8, 1 / 4, and 1 / 2 of the original image; The global feature extractor in the third step includes standard convolution, feature block flattening and folding, and Transformer. The feature matrix first passes through a 1×1 convolution layer. After the feature matrix is ​​expanded, it is sent to the Transformer module for feature extraction under the global receptive field. The feature is then folded to restore it to its original dimension and then sent to a 5×5 standard convolution to fuse local and global information. In the expansion and folding part, the resolution of the input feature is H×W, the resolution of the image block is h×w+N, the dimensions C and P are the number of channels of the feature block and the number of image blocks, respectively, and folding is the opposite operation of expansion.

2. The real-time monocular depth estimation method integrating global features according to claim 1 is characterized in that: The semantic feature extractor in the second step includes an adaptive pooling layer, a 1×1 convolutional layer and three dilated convolutional layers.

3. The real-time monocular depth estimation method integrating global features according to claim 1, wherein: The decoder in the fourth step consists of four decoder basic backbones, each consisting of one bilinear interpolation layer and one 5×5 convolutional layer. After the bilinear interpolation is completed, the features obtained by the multi-scale feature extractor are connected and then feature fusion is performed to reduce the information loss caused by convolution. Finally, the 5×5 convolutional layer in the last basic backbone is replaced by a 1×1 convolutional layer, and the final depth map is directly output.

4. The real-time monocular depth estimation method integrating global features according to claim 1, characterized in that: A combined loss function is used for network learning and training. The overall expression of this loss function combination is: ； in =0.1, is the depth prediction value of a pixel, is the true depth value of a pixel, L depth Represents the mean absolute error, which is the absolute average of the difference between the predicted depth value and the true depth value. It is the SSIM loss function.

5. The real-time monocular depth estimation method integrating global features according to claim 4 is characterized in that: The loss function is obtained by comparing the similarity between the predicted value and the true value in three dimensions: brightness, contrast and structure.

Images