Efficient monocular depth estimation method and system based on attention and CNN

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to an attention and CNN-based efficient monocular depth estimation method and system, and the network architecture of the system comprises an encoder which carries out the fusion of a Transform branch module and a lightweight CNN branch module in a cascade mode; the fusion module can aggregate local features of the CNN layer and global features from the Transform layer; the feature pyramid module is used for performing feature aggregation by using multi-scale feature fusion and top-down and transverse connection; the decoder adopts a lightweight additive attention mechanism to realize fusion of coding multi-scale features and a decoding module; the Bins prediction module divides a depth range into a plurality of intervals to realize depth classification regression; according to the method, the long-range correlation of the Transform and the local feature extraction capability of the lightweight CNN are aggregated by the network, and precise prediction is realized by combining local details and a long-distance relationship during depth prediction.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically provides an efficient monocular depth estimation method and system based on attention and CNN. Background Art

[0002] Monocular depth estimation (MDE) is a long-standing and fundamental problem in geometric computer vision, widely used in fields such as autonomous driving, 3D reconstruction, and scene understanding. The core of this task is to infer pixel depth from a single RGB image. This is a complex problem, as the same 2D image can be projected from different 3D scenes. In recent years, a growing number of deep learning-based methods have been proposed, some of which leverage codec architectures to predict depth using regression, classification, and classification-regression methods, achieving competitive results. Other methods achieve state-of-the-art performance by implementing depth prediction based on pre-trained models such as mask image modeling (MIM) and diffusion models, but these methods come at the expense of inference time and training cycles. For a fair comparison, these methods are not discussed in this article.

[0003] Early regression-based depth prediction used supervised learning with regression loss to directly predict the depth value of an image. This method achieved groundbreaking success as a general paradigm. To further improve performance, subsequent research proposed a classification-based prediction model. This method discretizes the depth interval into multiple different intervals (bins) and uses the center depth value of each interval as the final depth value, significantly improving model performance. However, the discretization operation can cause pixel depth to jump. To overcome the artifacts caused by pixel depth jumps, some methods restructure the MDE as a pixel-by-pixel classification-regression problem, learning the probability distribution of each pixel. Ultimately, the probability score of the pixel in each bin is weighted by its center value as the final depth value.

[0004] As for network architecture, there are many works devoted to the convolutional neural network (CNN) architecture based on the encoder-decoder style. When this type of encoder faces high-resolution images, the global feature extraction defects of the convolution kernel lead to the loss of global receptive field, which has great limitations for the prediction and division of depth intervals in classification and regression tasks. In recent years, transformers have been introduced into depth estimation tasks due to their advantages in long-range correlation, but the effect of models based solely on the Transformer framework on indoor datasets is not satisfactory. Some studies have also used the method of fusion encoders to enhance feature extraction in the encoding stage, but there is a problem of insufficient fusion of local and global features in the depth estimation algorithm. In addition, the existing models often have insufficient expression of feature information due to upsampling and splicing operations in the decoding stage.

[0005] Therefore, to address the above problems, an efficient monocular depth estimation method and system based on attention and CNN is proposed. Summary of the Invention

[0006] (1) Technical problems solved In view of the shortcomings of the existing technology, the present invention provides an efficient monocular depth estimation method and system based on attention and CNN, which is used to solve the problems raised in the background technology.

[0007] (2) Technical solution To achieve the above objectives, the present invention provides the following technical solutions: an efficient monocular depth estimation method and system based on attention and CNN, the system network architecture includes: The encoder uses a cascade method to fuse the Transformer branch module and the lightweight CNN branch module; The fusion module aggregates the local features of the CNN layer with the global features from the Transformer layer; Feature pyramid module, which uses multi-scale feature fusion and top-down and lateral connections for feature aggregation; The decoder uses a lightweight additive attention mechanism to achieve the fusion of encoding multi-scale features and decoding modules; Bins prediction module, which implements depth classification regression by dividing the depth range into multiple intervals; The loss function constrains the error between the predicted value and the true value to achieve accurate prediction of depth.

[0008] Preferably, the Transformer branch module adopts the swin transformer. Due to its window-based attention mechanism and sliding window operation, the swin transformer can maintain only linear growth in computational complexity when processing high-resolution images, and in a gradually deepening network, a hierarchical feature map is constructed by merging patches, which can better perform feature fusion with the CNN branch.

[0009] Preferably, the lightweight CNN branch module, a lightweight CNN branch module in which some point convolution and multi-scale attention convolution modules are aggregated, realizes downsampling and feature extraction in the PPConv stage; The multi-scale attention convolution module combines the wavelet convolution and channel-space attention convolution modules.

[0010] Preferably, the fusion module can aggregate the local features of the CNN layer and the global features from the Transformer layer, and the feature size of the CNN layer output is {Ci, , }, the feature sequence of the Transformer layer is large {Ci× × }, where i∈{1,2,3,4}, in order to integrate these features, we first reshape the feature sequence of the Transformer layer to the same size as the CNN layer: {Ci, , }, and then concatenate it with the feature map, and use the point-by-point convolution layer to fuse the concatenated feature map.

[0011] Preferably, the multi-scale feature fusion is to construct a feature pyramid to aggregate the high semantic features of the deep network with the high resolution features of the shallow network to generate a multi-scale feature representation; Top-down and lateral connections: A top-down path is used to transmit high-level semantic information, and it is integrated with the underlying features through lateral connections to enhance the expressive power of the features.

[0012] Preferably, in the decoder, given the decoder feature map Xi and the encoder feature map Yi, they are first projected to the same channel dimension X by 3×3 convolution. ia , Y ia ∈R Hi×Wi×Cj,其中Cj∈{128, 256, 512, 1024}，Xia and Y ia Then, the weight matrix W q和Wk Mapped into query matrix (Q) and key matrix (K), in order to balance the computational complexity and attention range, the window attention mechanism is adopted to limit the attention of Q and K matrices to a local window of W×W, with W=30. For low-level feature maps, the attention will be spread over the entire feature map, and for high-level feature maps, the attention will be carried out within the divided window; If Qw, Kw∈R hiwi×cj为窗口内的查询矩阵(Q)和键矩阵(K)，则对应的高效加性注意力操作如公式所示： ; Among them, the learnable parameter vector ∈R hiwi , represents the attention weight of Qw, α∈R n represents the global attention query vector, q∈R hiwi represents a single global query vector, and T represents a linear transformation.

[0013] Preferably, the upsampling part of the decoding module uses a dy dynamic upsampling module to replace the traditional bilinear difference. Different from the previous dynamic upsampling, the dy dynamic upsampling returns the upsampling to the essential mode of point sampling, realizing ultra-lightweight and efficient dynamic upsampling.

[0014] Preferably, the Bins prediction module first uses the bottleneck layer features as the generation information of the Bins seed, and uses mapping to convert the bottleneck features into Bins seeds. The size of the seed is { , , }, then upsample the seed to the next scale size, and then generate constraint coefficients by expanding the receptive field and channel mapping the decoded scale features. The generated constraint coefficients and Bins seeds are multiplied point by point to correct and constrain the Bins width. Finally, the corrected width is merged into the next quantitative scale. Q represents the bottleneck feature after feature aggregation, and E represents the encoded multi-scale feature. The specific formula is: ; c represents the unit depth center, and its prediction formula is as follows: .

[0015] Preferably, a scaled version of the scale-invariant loss in the loss function is used to supervise the network, in, , n represents the number of real pixels in the image, λ is set to 0.85, and α is set to 10.

[0016] A method for an efficient monocular depth estimation system based on attention and CNN, characterized in that the method steps include: S1: The input image is fed into the Transformer branch module and the CNN branch module for feature extraction. The outputs of the two modules flow into the fusion module for interactive feature fusion. The fused feature map serves as the input of the next level CNN. S2: The fused scale is { The feature map of} is input into the feature pyramid module for feature aggregation to obtain bottleneck features; S3: The decoder uses an attention module to hierarchically aggregate bottleneck features and encoded multi-scale features, and uses a dynamic upsampling module to restore the resolution to predict the probability distribution of the final unit depth center at each pixel; S4: Bottleneck features and decoding side { The scale map is used as the unit depth center seed and unit depth center correction branch input into the decreasing unit depth center Bins prediction module to adaptively predict the unit depth center of each image.

[0017] Experiment and analysis: Dataset and evaluation metrics: NYU-Depth-v2: The NYU-Depth-v2 indoor scene dataset boasts a large data set and rich scene variety. Compared to other datasets, its depth image quality is higher, so this dataset was chosen as the depth dataset for indoor scenes. The NYU-Depth-v2 dataset consists of 464 frames of indoor scene video sequences captured by a Microsoft Kinect RGBD camera, containing 120K pairs of 480×640 RGB images and depth maps. This paper follows the official training and testing methods to split the dataset: 249 scenes, totaling 50K images, are used for training; 654 images from 215 scenes are used for testing. During data processing, the center cropping method proposed by Eigen et al. is used to limit the maximum depth of the depth maps to 10 meters. The depth images output by this method are scaled to one-quarter the original image size (120×160). To match the resolution of the ground-truth depth maps, they are upsampled by a factor of 4 during training and testing.

[0018] KITTI: For the outdoor scene depth dataset, this paper chooses the KITTI dataset for experiments. The KITTI data is collected and constructed by a vehicle-mounted multi-sensor system, and its data source covers 61 sets of real road scenes. The dataset uses a roof-integrated sensor array to obtain binocular vision data and simultaneously records high-precision lidar point cloud information. The original RGB image resolution is 1241×375 pixels, and each sample is equipped with corresponding three-dimensional spatial coordinate data. In terms of experimental settings, this paper follows the established division of 26k left-eye camera images as training samples and 697 as test samples. For task evaluation, data preprocessing is performed in accordance with the specifications proposed by the Garg team. The effective depth adopts a depth upper limit of 80 meters. In order to eliminate the impact of resolution differences on the evaluation results, all predicted outputs are upsampled to ensure dimensional consistency with the true depth map.

[0019] Evaluation Metrics: As in previous studies, in order to verify the effectiveness of the proposed method in this paper, a standard evaluation protocol was used for evaluation, including relative absolute error (Abs Rel), relative squared error (Sq Rel), root mean square error (RMSE), root mean square logarithmic error (RMSE log), logarithmic error (log10), threshold accuracy (δ < 1.25, δ < 1.252 and δ < 1.253) and scale-invariant square root of logarithmic error (SILog).

[0020] Implementation details: The method in this paper is implemented using the Pytorch deep learning framework and trained on an L40 graphics card with 48G video memory. The training is set for a total of 20 cycles with an initial learning rate of 4.e-5 , and linearly decreases to 4 during training e-6 The entire training process takes about 15 hours. In addition, this article uses the Adam optimizer and the training batch size is 8. In order to improve the training speed, the Transformer branch of the encoder backbone network is initialized using the pre-trained weights of Swin-L.

[0021] Model parameters and inference time: To measure the performance of different models in terms of accuracy, number of parameters, and inference time, the table shows the performance of several competitive classification-regression models in terms of various parameters. Thanks to the design of the lightweight CNN module and the additive skip attention decoding module, although the method in this paper does not have an advantage in terms of number of parameters, it achieves better results in terms of accuracy and inference time. Figure 4 shown.

[0022] Comparison of qualitative and quantitative analysis: NYU-Depth-v2: To demonstrate the superiority of the proposed method in indoor scenes, this paper evaluates it on the NYU-Depth-v2 dataset. The results are shown in Tables 1 and Figure 2 As shown in the figure, our algorithm outperforms other baseline algorithms in all indicators. In terms of two key evaluation metrics, AbsRel and RMSE, our method improves by 11.6% and 7.9% respectively compared to DepthFormer, which also uses a fusion encoder, and by 4.4% and 3.1% respectively compared to PixelFormer, demonstrating the effectiveness of the module designed in this paper. from Figure 5 As can be seen from the figure, our model has more obvious contour features when processing features such as chairs, pillows, and desks than the Transformer architecture. It is also more accurate than the CNN architecture model in predicting depth over large areas such as carpeted floors and walls covered with wall coverings. It can grasp long-range correlations while also effectively learning local features. Method Backbone Abs Rel ↓ Sq Rel ↓ RMSE ↓ log10 ↓ δ<1.25 ↑ δ<1.252↑ δ<1.253 ↑ Eigen et al. AlexNet 0.158 - 0.641 - 0.769 0.950 0.988 DORN ResNet-101 0.115 - 0.509 0.051 0.828 0.965 0.992 BTS DenseNet-161 0.110 0.066 0.392 0.047 0.885 0.978 0.994 TransDepth R-50+ViT-B / 16† 0.106 - 0.365 0.045 0.900 0.983 0.996 AdaBins E-B5+mini-ViT 0.103 - 0.364 0.044 0.903 0.984 0.997 P3Depth ResNet-101 0.104 - 0.356 0.043 0.898 0.981 0.996 LocalBins E-B5 0.099 - 0.357 0.042 0.907 0.987 0.998 DepthFormer Swin-Large†+ResNet-50 0.096 - 0.339 0.041 0.921 0.989 0.998 NewCRFs Swin-Large† 0.095 0.045 0.334 0.041 0.922 0.992 0.998 BinsFormer Swin-Large† 0.094 - 0.330 0.040 0.925 0.989 0.997 PixelFormer Swin-Large† 0.090 - 0.322 0.039 0.929 0.991 0.998 IEBins Swin-Large† 0.087 0.040 0.314 0.038 0.936 0.992 0.998 This article Swin-Tiny†+CNN 0.108 0.060 0.375 0.047 0.892 0.982 0.996 This article Swin-Large†+CNN 0.086 0.040 0.312 0.038 0.935 0.992 0.998 Table 1: Evaluation results on the NYU-Depth-v2 dataset, with a maximum depth limit of 10 meters This article provides training results based on two backbone networks: Swin-Large and Swin-Tiny. † indicates the use of a pre-trained model trained on ImageNet-22K; KITTI: This paper also conducts qualitative and quantitative evaluation on the outdoor scene KITTI dataset. The results are as follows Figure 6 As shown in Table 2, the white box marks the part that shows that our model has demonstrated its superiority in both detail and global processing; Method Backbone Abs Rel ↓ Sq Rel ↓ RMSE ↓ log10 ↓ δ<1.25 ↑ δ<1.252↑ δ<1.253 ↑ DORN ResNet-101 0.072 0.307 2.727 0.120 0.932 0.984 0.994 BTS DenseNet-161 0.060 0.249 2.798 0.096 0.955 0.993 0.998 AdaBins E-B5+mini-ViT 0.058 0.190 2.360 0.088 0.964 0.995 0.999 P3Depth ResNet-101 0.071 0.270 2.843 0.103 0.953 0.993 0.998 DepthFormer Swin-Large†+ResNet-50 0.052 0.158 2.143 0.079 0.975 0.997 0.999 NewCRFs Swin-Large† 0.052 0.155 2.129 0.079 0.974 0.997 0.998 BinsFormer Swin-Large† 0.058 0.151 2.098 0.079 0.974 0.997 0.999 PixelFormer Swin-Large† 0.051 0.149 2.081 0.077 0.976 0.997 0.999 IEBins Swin-Large† 0.050 0.142 2.011 0.075 0.978 0.998 0.999 This article Swin-Tiny†+CNN 0.056 0.172 2.276 0.089 0.969 0.995 0.999 This article Swin-Large†+CNN 0.050 0.145 2.021 0.077 0.978 0.998 0.999 Table 2: Quantitative comparison results on the KITTI dataset with a maximum depth limit of 80 meters This paper provides training results based on two backbone networks: Swin-Large and Swin-Tiny. † indicates the use of a pre-trained model trained on ImageNet-22K.

[0023] Ablation experiment: To better verify the impact and effect of each part of the model, this paper conducted several ablation studies, focusing on the cascade encoder, Bins prediction mode, and additive skip attention decoding module. Cascaded Encoder: To verify the effectiveness and advantages of the cascaded encoder, we compared the use of only Transformer as the encoder module and CNN as the encoder module. The Transformer module uses Swin-Large and the CNN module uses EfficientNet-B5 with a similar network depth. The results show that compared with the latter two methods, the cascaded fusion module has better feature extraction capabilities for dense prediction tasks such as depth estimation. In addition, to verify the effectiveness of cascaded fusion, this paper also designed a bottleneck fusion method. Referring to the DepthFormer method, feature fusion is performed in the last layer of the encoder. Experiments show that the cascaded fusion method can better interact between features. The specific details are shown in Table 3. encoder Abs Rel ↓ RMSE ↓ log10 ↓ δ<1.25 ↑ δ<1.252 ↑ δ<1.253 ↑ EfficientNet-B5 0.098 0.346 0.042 0.912 0.098 0.346 Swin-Large 0.090 0.321 0.039 0.930 0.090 0.321 Bottleneck Fusion Encoder 0.088 0.318 0.039 0.931 0.088 0.318 Cascaded Fusion Encoder 0.086 0.040 0.312 0.038 0.935 0.992 Table 3: Comparison of different encoder types on the NYU-Depth-v2 dataset Bins prediction mode: This paper verifies the necessity of the Bins module by comparing it with the standard regression method. Standard regression directly expresses depth prediction as an end-to-end regression task. Experiments show that compared with the standard regression method, expressing depth estimation as a classification regression task can achieve better results. In addition, this paper also performs Bins prediction in different modes and with different numbers of Bins predictions, as shown in Tables 4 and 5.

[0024] Prediction method Abs Rel ↓ RMSE ↓ log10 ↓ δ<1.25 ↑ δ<1.252 ↑ δ<1.253 ↑ Standard regression 0.093 0.329 0.040 0.926 0.093 0.329 AdaBins 0.091 0.323 0.041 0.929 0.091 0.323 LocalBins 0.090 0.321 0.040 0.930 0.090 0.321 Incremental Bins 0.089 0.318 0.039 0.931 0.089 0.318 Decreasing Bins† 0.088 0.315 0.039 0.933 0.088 0.315 Decreasing Bins 0.086 0.040 0.312 0.038 0.935 0.992 Table 4: Comparison of different Bins prediction methods on the NYU-Depth-v2 dataset Standard regression does not use bins for prediction. AdaBins uses an adaptive approach to predict bins, and LocalBins uses a binary split approach to predict bins. Incremental Bins and LocalBins are similar in prediction, but differ in the number of bins, and the number of bins changes in the same way as decremental Bins. This is used for comparative experiments. † This shows that in the process of generating constraint coefficients, the receptive field of scale features is not expanded. Number of Bins Abs Rel ↓ RMSE ↓ log10 ↓ δ<1.25 ↑ δ<1.252 ↑ δ<1.253 ↑ 256 0.087 0.319 0.039 0.932 0.991 0.998 128 0.086 0.040 0.312 0.038 0.935 0.992 64 0.089 0.320 0.040 0.930 0.991 0.998 Table 5: Comparison of different numbers of Bins on the NYU-Depth-v2 dataset Additive Skip Attention Decoding Module: Table 6 shows the proposed additive skip attention module. Compared with convolution-based methods such as additive convolution and concatenated convolution, the additive skip attention module is more effective in aggregating encoder and decoder features at a specific scale. Additive convolution adds encoder and decoder features pixel by pixel before convolution, while concatenated convolution concatenates encoder and decoder features along the channel dimension before convolution. Decoding module Abs Rel ↓ RMSE ↓ log10 ↓ δ<1.25 ↑ δ<1.252 ↑ δ<1.253 ↑ Additive convolution 0.090 0.322 0.040 0.930 0.090 0.322 Concatenated Convolution 0.089 0.319 0.040 0.929 0.089 0.319 Additive Skip Attention 0.086 0.040 0.312 0.038 0.935 0.992 Table 6: Comparison of different decoding modules on the NYU-Depth-v2 dataset.

[0025] Beneficial effects Compared with the existing technology, the present invention provides an efficient monocular depth estimation method and system based on attention and CNN, which has the following beneficial effects: 1. The present invention integrates the long-range correlation of Transformer and the local feature extraction capability of lightweight CNN, combining local details and long-range relationships to achieve accurate prediction during depth prediction.

[0026] 2. The present invention aggregates efficient additive attention and dynamic upsampling decoding modules to overcome the computational complexity of attention while more fully interactively fusing the features after the jump connection.

[0027] 3. In this invention, the lightweight CNN branch module uses a small amount of parameters to exchange for the accuracy of the traditional CNN module, thereby improving the inference efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is a network architecture diagram of the present invention; Figure 2 This is a detailed diagram of the implementation of the lightweight CNN branch module of the present invention; Figure 3 Schematic diagram of the cross-coding fusion module of the present invention; Figure 4 This is a data graph of the test results on the NYU-Depth-v2 dataset of the present invention; Figure 5 This is a qualitative depth comparison diagram on the NYU-Depth-v2 dataset of the present invention; Figure 6 This is a qualitative depth comparison diagram on the KITTI dataset of the present invention. DETAILED DESCRIPTION

[0029] The following is a collection of drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0030] Specific examples are given below.

[0031] Example 1: The present invention provides an efficient monocular depth estimation method and system based on attention and CNN. The system network architecture includes: The encoder uses a cascade method to fuse the Transformer branch module and the lightweight CNN branch module; The fusion module aggregates the local features of the CNN layer with the global features from the Transformer layer; Feature pyramid module, which uses multi-scale feature fusion and top-down and lateral connections for feature aggregation; The decoder uses a lightweight additive attention mechanism to achieve the fusion of encoding multi-scale features and decoding modules; Bins prediction module, which implements depth classification regression by dividing the depth range into multiple intervals; The loss function constrains the error between the predicted value and the true value to achieve accurate prediction of depth; The Transformer branch module uses the Swin transformer. Due to its window-based attention mechanism and sliding window operation, the Swin transformer can maintain a linear increase in computational complexity when processing high-resolution images. In a gradually deepening network, it constructs a hierarchical feature map by merging patches, which can better integrate features with the CNN branch. Lightweight CNN branch module, which aggregates partial point convolution and multi-scale attention convolution modules, implements downsampling and feature extraction in the PPConv stage; The multi-scale attention convolution module combines the wavelet convolution and channel space attention convolution modules; The lightweight CNN branch module achieves the accuracy of the traditional CNN module at the cost of a small number of parameters, thus improving the inference efficiency.

[0032] Example 2: The fusion module can aggregate the local features of the CNN layer and the global features from the Transformer layer. The feature size of the CNN layer output is {Ci, , }, the feature sequence of the Transformer layer is large {Ci× × }, where i∈{1,2,3,4}, in order to integrate these features, we first reshape the feature sequence of the Transformer layer to the same size as the CNN layer: {Ci, , }, then concatenate it with the feature map, and fuse the concatenated feature map using a point-by-point convolutional layer; The network aggregates the long-range correlation of Transformer and the local feature extraction capabilities of lightweight CNN, combining local details and long-range relationships to achieve accurate prediction when predicting depth.

[0033] Example 3: Multi-scale feature fusion: By constructing a feature pyramid, the high-semantic features of the deep network are aggregated with the high-resolution features of the shallow network to generate multi-scale feature representations; Top-down and lateral connections: Use a top-down path to transmit high-level semantic information and fuse it with low-level features through lateral connections to enhance the expressive power of features; In the decoder, given the decoder feature map Xi and the encoder feature map Yi, they are first projected to the same channel dimension Xia by 3×3 convolution, Yia∈R Hi×Wi×Cj , where Cj∈{{128, 256, 512, 1024}, Xia and Yia are then mapped to the query matrix (Q) and key matrix (K) through the weight matrices Wq and Wk respectively. In order to balance the computational complexity and attention range, the window attention mechanism is adopted to limit the attention of the Q and K matrices to a local window of W×W, with W=30. For low-level feature maps, the attention will be spread over the entire feature map, and for high-level feature maps, the attention will be carried out within the divided window; If Qw, Kw∈R hiwi×cjFor the query matrix (Q) and key matrix (K) in the window, the corresponding efficient additive attention operation is shown in the formula: ; Among them, the learnable parameter vector ∈R hiwi , represents the attention weight of Qw, α∈R n represents the global attention query vector, q∈R hiwi represents a single global query vector, and T represents a linear transformation.

[0034] The upsampling part of the decoding module uses the dy dynamic upsampling module to replace the traditional bilinear interpolation. Unlike previous dynamic upsampling, dy dynamic upsampling returns upsampling to the essence of point sampling, achieving ultra-lightweight and efficient dynamic upsampling. The decoding module aggregates efficient additive attention and dynamic upsampling to overcome the computational complexity of attention while more fully interactively fusing the features after the jump connection.

[0035] The Bins prediction module first uses the bottleneck layer features as the generation information of Bins seeds, and uses mapping to convert the bottleneck features into Bins seeds. The seed size is { , , }, then upsample the seed to the next scale size, and then generate constraint coefficients by expanding the receptive field and channel mapping the decoded scale features. The generated constraint coefficients and Bins seeds are multiplied point by point to correct and constrain the Bins width. Finally, the corrected width is merged into the next quantitative scale. Q represents the bottleneck feature after feature aggregation, and E represents the encoded multi-scale feature. The specific formula is: ; c represents the unit depth center, and its prediction formula is as follows: .

[0036] A scaled version of the scale-invariant loss in the loss function is used to supervise the network. in, , n represents the number of real pixels in the image, λ is set to 0.85, and α is set to 10; Top-down and lateral connections: A top-down path is used to transmit high-level semantic information, and it is integrated with the underlying features through lateral connections to enhance the expressive power of the features.

[0037] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention as claimed.

Claims

1. An efficient monocular depth estimation system based on attention and CNN, characterized by: The system network architecture includes: The encoder uses a cascade method to fuse the Transformer branch module and the lightweight CNN branch module; The fusion module aggregates the local features of the CNN layer with the global features from the Transformer layer; Feature pyramid module, which uses multi-scale feature fusion and upsampling modules to achieve feature interaction; The decoder uses a lightweight additive attention mechanism to achieve the fusion of encoding multi-scale features and decoding modules; Bins prediction module, which implements depth classification regression by dividing the depth range into multiple intervals; The loss function constrains the error between the predicted value and the true value to achieve accurate prediction of depth.

2. The efficient monocular depth estimation system based on attention and CNN according to claim 1, characterized in that The Transformer branch module adopts the swin transformer. Due to its window-based attention mechanism and sliding window operation, the swin transformer can maintain a linear increase in computational complexity when processing high-resolution images. In addition, in a gradually deepening network, a hierarchical feature map is constructed by merging patches, which can better fuse features with the CNN branch.

3. The efficient monocular depth estimation system based on attention and CNN according to claim 1, characterized in that The lightweight CNN branch module, which is an aggregation of the partial point convolution and multi-scale attention convolution modules, realizes downsampling and feature extraction in the PPConv stage; The multi-scale attention convolution module combines the wavelet convolution and channel-space attention convolution modules.

4. The efficient monocular depth estimation system based on attention and CNN according to claim 1, characterized in that The fusion module can aggregate the local features of the CNN layer and the global features from the Transformer layer. The feature size of the CNN layer output is {Ci, , }, the feature sequence of the Transformer layer is large {Ci× × }, where i∈{1,2,3,4}, in order to integrate these features, we first reshape the feature sequence of the Transformer layer to the same size as the CNN layer: {Ci, , }, and then concatenate it with the feature map, and use the point-by-point convolution layer to fuse the concatenated feature map.

5. The efficient monocular depth estimation system based on attention and CNN according to claim 1, characterized in that The multi-scale feature fusion: by constructing a feature pyramid, the high semantic features of the deep network are aggregated with the high-resolution features of the shallow network to generate a multi-scale feature representation, which is then upsampled to the bottleneck size to generate bottleneck features; Top-down and lateral connections: A top-down path is used to transmit high-level semantic information, and it is integrated with the underlying features through lateral connections to enhance the expressive power of the features.

6. The efficient monocular depth estimation system based on attention and CNN according to claim 1, characterized in that In the decoder, given the decoder feature map X i With the encoder feature map Y i , first project it to the same channel dimension X through 3×3 convolution ia , Y ia ∈R Hi×Wi×Cj , where C j ∈{{128, 256, 512, 1024},X ia and Y ia Then, the weight matrix W q and W k Mapped into query matrix (Q) and key matrix (K), in order to balance the computational complexity and attention range, the window attention mechanism is adopted to limit the attention of Q and K matrices to a local window of W×W, with W=30. For low-level feature maps, the attention will be spread over the entire feature map, and for high-level feature maps, the attention will be carried out within the divided window; If Qw, Kw∈R hiwi×cj For the query matrix (Q) and key matrix (K) in the window, the corresponding efficient additive attention operation is shown in the formula: ； Among them, the learnable parameter vector ∈R hiwi , represents the attention weight of Qw, α∈R n represents the global attention query vector, q∈R hiwi represents a single global query vector, and T represents a linear transformation.

7. The efficient monocular depth estimation system based on attention and CNN according to claim 6, characterized in that The upsampling part of the decoding module uses the dy dynamic upsampling module to replace the traditional bilinear interpolation. Different from the previous dynamic upsampling, the dy dynamic upsampling returns the upsampling to the essential method of point sampling, achieving ultra-lightweight and efficient dynamic upsampling.

8. The efficient monocular depth estimation system based on attention and CNN according to claim 1, characterized in that The Bins prediction module first uses the bottleneck layer features as the generation information of Bins seeds, and uses mapping to convert the bottleneck features into Bins seeds. The size of the seed is { , , }, then upsample the seed to the next scale size, and then generate constraint coefficients by expanding the receptive field and channel mapping the decoded scale features. The generated constraint coefficients and Bins seeds are multiplied point by point to correct and constrain the Bins width. Finally, the corrected width is merged into the next quantitative scale. Q represents the bottleneck feature after feature aggregation, and E represents the encoded multi-scale feature. The specific formula is: ； c represents the unit depth center, and its prediction formula is as follows: 。 9. The efficient monocular depth estimation system based on attention and CNN according to claim 1, characterized in that A scaled version of the scale-invariant loss in the loss function is used to supervise the network. in, , n represents the number of real pixels in the image, λ is set to 0.85, and α is set to 10.

10. A method for an efficient monocular depth estimation system based on attention and CNN, characterized in that: The method steps include: S1: The input image is fed into the Transformer branch module and the CNN branch module for feature extraction. The outputs of the two modules flow into the fusion module for interactive feature fusion. The fused feature map serves as the input of the next level CNN. S2: The fused scale is { The feature map of} is input into the feature pyramid module for feature aggregation to obtain bottleneck features; S3: The decoder uses an attention module to hierarchically aggregate bottleneck features and encoded multi-scale features, and uses a dynamic upsampling module to restore the resolution to predict the probability distribution of the final unit depth center at each pixel; S4: Bottleneck features and decoding side { The scale map is used as the unit depth center seed and unit depth center correction branch input into the decreasing unit depth center Bins prediction module to adaptively predict the unit depth center of each image.

Images