Attention-based monocular image depth estimation methods

Abstract

This invention discloses a monocular image depth estimation method based on an attention mechanism. It primarily addresses the problem of low depth estimation accuracy in existing technologies for regions with minimal illumination differences and pixel depth value variations. The implementation scheme involves: reading and preprocessing monocular image depth estimation datasets; using a Swing Transformer network as the encoder network to extract features from the preprocessed data; constructing an aggregation structure to optimize the output global information features of the preprocessed data; building an attention-based decoder network to optimize and decode the output features of the encoder and aggregation structure, obtaining the decoder output features; and constructing a depth prediction network to predict image depth using the decoder's output features. This invention significantly improves the accuracy of monocular image depth estimation, and performs better in regions with minimal illumination differences and pixel depth value variations, making it applicable to autonomous driving, robotics, and 3D reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a monocular image depth estimation method that can be used in autonomous driving, robotics, and 3D reconstruction. Background Technology

[0002] Monocular image depth estimation is one of the fundamental tasks in computer vision, with widespread applications in autonomous driving, robotics, and 3D reconstruction, demonstrating its high practical value. The task of monocular image depth estimation is to predict pixel-by-pixel depth values ​​for RGB images.

[0003] In recent years, the problem of monocular image depth estimation has received widespread attention. Existing algorithms mainly utilize encoder-decoder structures. In this structure, the encoder extracts image features, while the decoder iteratively optimizes the features extracted by the encoder to output the final prediction result. For encoder-decoder design, most methods employ convolutional neural networks; recently, many methods have utilized the Transformer structure to improve the network's feature extraction and processing capabilities, while others have used conditional random fields to predict the image's energy function.

[0004] Compared to traditional methods, convolutional neural networks (CNNs) possess powerful image feature extraction capabilities. Eigen et al. first applied CNNs to monocular depth estimation. Building upon this, various works have emerged that utilize CNNs to achieve and improve upon monocular depth estimation. Among these, the multi-scale method BTS proposes a local plane guiding layer designed to fuse features from all levels during decoding. The block attention network PWA designs a patch-based attention mechanism that focuses on each local region.

[0005] While convolutional neural network (CNN)-based methods remain popular, they also have limitations. The receptive field of CNNs is local, failing to model global image information, which leads to performance bottlenecks in monocular image depth estimation. Transformers, with their larger receptive field and ability to model long-term dependencies, are gaining increasing attention in computer vision tasks, and many works apply them to monocular depth estimation. Among these, the adaptive interval method AdaBins utilizes the miniViT micro-visual deformer structure to predict adaptive intervals. The Neural Window Conditional Random Field (NeWCRF) method leverages the powerful feature extraction capabilities of the Swin-Transformer moving window deformer structure. The interval construction method BinsFormer uses a Transformer as a decoder. Although these transformer-based methods achieve better monocular depth estimation performance, they primarily treat monocular depth estimation as a regression task, resulting in slow convergence and suboptimal performance.

[0006] To alleviate this problem, another research branch treats monocular depth estimation as a classification task. The ordinal regression network DORN first treats monocular depth estimation as an ordinal classification regression task and designs an effective ordinal classification regression depth estimation loss function. However, due to the discretization of depth values, the ordinal classification regression method proposed by DORN exhibits significant depth discontinuities in the predicted depth map, affecting visual quality. To address this issue, Adamins further treats monocular depth estimation as a classification regression task. It effectively alleviates the problem of uneven depth value transitions by linearly combining interval center points. The pixel construction method PixelFormer proposes a lightweight interval generation module to reduce model complexity. While these methods have made significant progress in depth estimation accuracy, they still suffer from inaccuracies in predicting pixel depth values ​​in areas with poor lighting or small depth variations due to the lack of modeling long-term dependencies between image pixels and attention to specific regions of the image. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the existing technologies by proposing a monocular image depth estimation method based on an attention mechanism, so as to improve the accuracy of pixel depth value prediction in areas with poor image illumination and small depth variations, and to obtain better monocular image depth estimation performance.

[0008] To achieve the above objectives, the technical approach of this invention is as follows: global information aggregation structure is used to enhance the global information of features; a self-attention network branch is designed in the decoder to establish correlations between pixels; a region attention network branch is designed in the decoder to improve the network's attention to specific regions of the image; and a classification and regression prediction structure is designed to improve the network's convergence speed and optimization effect. Through these structural designs, the accuracy of pixel depth value prediction in regions with poor image illumination and small depth variations is ultimately improved, resulting in better monocular image depth estimation performance.

[0009] Based on the above ideas, the technical solution of the present invention includes the following steps:

[0010] 1. A monocular image depth estimation method based on an attention mechanism, characterized by comprising the following steps:

[0011] (1) Read training data and test data from monocular image depth estimation dataset. Perform preprocessing on the images of training data by rotating, scaling, flipping, adjusting and normalizing in sequence to obtain training tensor data; perform preprocessing on the images of test data by adjusting and normalizing in sequence to obtain test tensor data.

[0012] (2) Using a Swing Transformer network with four cascaded Swing Transformer stage modules as the encoder network, the training tensor data and test tensor data are respectively input into the Swing Transformer network to obtain the training image features E1, E2, E3 and E4 and the test image features E1', E'2, E3' and E'4 output by the four cascaded modules respectively.

[0013] (3) Use aggregation structures to enhance the global information of features:

[0014] 3a) A pyramid pooling module is formed by connecting four average pooling layers with pooling ratios of 1, 2, 3 and 6 in parallel. The training image features E4 output by the top encoder are input into the pyramid pooling module to extract multi-scale information. The multi-scale information is concatenated with the input features E4 and then passed through a convolutional layer to generate new features with global information.

[0015] 3b) The new features with global information are optimized by using the existing convolutional self-attention module to obtain the optimized global information features X4;

[0016] (4) Construct an attention-based decoder network:

[0017] 4a) Establish a window-based self-attention module consisting of a window self-attention submodule, a shift window self-attention submodule, and a cascaded Pixel Shuffle layer;

[0018] 4b) Establish a region attention module consisting of a parallel average pooling layer and a max pooling layer, which are then cascaded with a convolutional layer, a sigmoid layer, and a transposed convolutional layer;

[0019] 4c) Connect the window-based self-attention module and the region attention module in parallel to form a decoder module.

[0020] 4d) Concatenate the four decoder modules to form an attention-based decoder network;

[0021] (5) Use an attention-based decoder network to perform layer-by-layer optimization decoding on the features E1, E2, E3, E4 output by the encoder network and the global information feature X4 output by the aggregation structure, so as to obtain features X3, X2, X1 and X0 in sequence, and finally output feature X0.

[0022] (6) Construct a depth map prediction network:

[0023] 6a) Establish an adaptive interval center prediction module that sequentially includes a convolutional layer and two parallel average pooling layers and max pooling layers with learnable parameters, for adaptively predicting the interval center of the depth values ​​of the input image.

[0024] 6b) Establish a probability head module consisting of a convolutional layer and a softmax layer to predict the probability vector corresponding to the center of the depth value interval of the input image;

[0025] 6c) Connect the adaptive interval center prediction module and the probability head module in parallel to form a depth map prediction network;

[0026] (7) Predict the depth map corresponding to the input image:

[0027] 7a) Input the output feature X0 of the decoder network into the adaptive interval center prediction module and the probability head module in the depth map prediction network, respectively, and output the adaptive interval center c(b) and probability vector v of the image depth value;

[0028] 7b) The adaptive interval center c(b) and the probability vector v are linearly combined to obtain the preliminary depth map. Then, the preliminary depth map is restored to the size of the input image through an upsampling operation to obtain the final depth map.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] 1) This invention improves the image depth estimation accuracy of the network by enhancing the global information of features through a global information aggregation structure;

[0031] 2) Because the present invention constructs a decoder network based on an attention mechanism, it is convenient to establish the correlation between image pixels and improve the network's attention to characteristic image regions, thereby improving the accuracy of the present invention in predicting pixel depth values ​​in areas with poor image illumination and small depth changes.

[0032] 3) Since the present invention constructs a depth map prediction network based on adaptive classification regression, it can alleviate the slow convergence speed and suboptimal problems of regression prediction and improve the image depth estimation accuracy of the network. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0034] Figure 2 This is a schematic diagram of the structure of the present invention;

[0035] Figure 3 This is a schematic diagram of the decoder module in this invention;

[0036] Figure 4 This is a schematic diagram of the depth map prediction network in this invention;

[0037] Figure 5 These are images showing the results of depth estimation of KITTI dataset images using the present invention and two existing methods, respectively.

[0038] Figure 6 These are the results of depth estimation of NYUV2 dataset images using the present invention and two existing methods, respectively. Detailed Implementation

[0039] The embodiments and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0040] Reference Figure 1 The implementation steps of this embodiment are as follows:

[0041] Step 1: Obtain training and test data and preprocess them.

[0042] 1.1) Read training and test data from the monocular image depth estimation dataset. The monocular image depth estimation dataset used in this example contains two different datasets: the KITTI dataset and the NYUV2 dataset.

[0043] 1.2) First, randomly rotate the training data images [-1°, 1°], then scale the image size according to the different dataset types. That is, for the KITTI dataset, scale the training data images to 352×1120, and for the NYUV2 dataset, scale the training data images to 480×640. Then, randomly flip the images horizontally with a probability of 0.5, and randomly adjust the brightness, contrast, saturation, and hue of the images with a probability of 0.5. Finally, normalize them according to the following formula:

[0044]

[0045] Where, x scale is the normalized pixel value of the corresponding channel, x is the pixel value of a single channel of the RGB image, S is the standard deviation of the pixel value of the corresponding channel, and μ is the mean of the pixel value of the corresponding channel. In this example, the normalized mean values ​​of the three RGB channels are set to [0.485, 0.456, 0.406], and the standard deviations of the three channels are set to [0.229, 0.224, 0.225].

[0046] 1.3) Scale the test data images according to the different dataset types. For the KITTI dataset, scale the test data images to 352×1216; for the NYUV2 dataset, scale the test data images to 480×640. Then normalize them using the following formula:

[0047]

[0048] Where, x scale is the normalized pixel value of the corresponding channel, x is the pixel value of a single channel of the RGB image, S is the standard deviation of the pixel value of the corresponding channel, and μ is the mean of the pixel value of the corresponding channel. In this example, the normalized mean values ​​of the three RGB channels are set to [0.485, 0.456, 0.406], and the standard deviations of the three channels are set to [0.229, 0.224, 0.225].

[0049] Step 2: Use the Swing Transformer network as the encoder network to extract features from the image.

[0050] like Figure 2 As shown, the Swin Transformer network is an existing network consisting of four cascaded Swin Transformer stage modules. Each stage module contains a downsampling layer and several repeated Swin Transformer Blocks. In this example, the Swin-L version of the Swin Transformer is selected as the encoder network.

[0051] The training data and test data after step 1 preprocessing are input into the Swing Transformer network, and the four cascaded modules of the Swing Transformer network output the four features E1, E2, E3, E4 of the training data and the four features E1', E'2, E3' and E'4 of the test data, respectively.

[0052] The encoder network structure and the image features extracted by each feature are shown in Table 1.

[0053] Table 1: Swing Transformer Network Structure and Extracted Image Features

[0054]

[0055] In Table 1, h and w represent the height and width of the input image, respectively.

[0056] Step 3: Use the aggregation structure to enhance the global information of the features.

[0057] 3.1) A pyramid pooling module is formed by connecting four average pooling layers with pooling ratios of 1, 2, 3 and 6 in parallel. The training image features E4 output by the top encoder are input into the pyramid pooling module to extract multi-scale information. The multi-scale information is then concatenated with the input features E4 and then passed through a convolutional layer to generate new features with global information.

[0058] 3.2) The new features with global information are optimized using the existing convolutional self-attention module to obtain the optimized global information features X4:

[0059] 3.2.1) Project the new features with global information through three 1×1 convolutions. The resulting intermediate features are processed by two parallel operations. One operation is to shift the intermediate features to obtain the shifted new features. The other operation is to use the intermediate features as a query, key and value matrix and calculate the self-attention features.

[0060] 3.2.2) Add the new shifted feature and the self-attention feature to obtain the optimized global information feature X4.

[0061] Step 4: Construct an attention-based decoder network.

[0062] Reference Figure 3 The specific implementation of this step is as follows:

[0063] 4.1) Establish a window-based self-attention module:

[0064] 4.1.1) Establish a window self-attention submodule, which sequentially contains two parallel convolutional layers with a kernel size of 3×3 and a stride of 1, a regularization layer, a window partitioning layer, a multilayer perceptron, a window-based multi-head self-attention layer, and another multilayer perceptron:

[0065] This convolutional layer is used to change the number of channels in the input features;

[0066] This regularization layer is used to regularize the input features;

[0067] This window partitioning layer is used to divide the input features into 7×7 windows;

[0068] This first multilayer perceptron was used to generate the query matrix Q, the key matrix K, and the value matrix V;

[0069] This window-based multi-head self-attention layer computes multi-head self-attention using the query matrix Q, the key matrix K, and the value matrix V.

[0070] The second multilayer perceptron is used to increase the network's nonlinear fitting capability;

[0071] 4.1.2) Establish a shift window self-attention submodule, whose structure is the same as the window self-attention submodule, the difference being: in the window partitioning layer, the position of each 7×7 window is moved 3×3 to the lower right;

[0072] 4.1.3) Create a Pixel Shuffle layer, which includes existing Pixel Shuffle operations for adjusting the number of channels and resolution of features;

[0073] 4.1.4) Concatenate the window self-attention submodule, the shift window self-attention submodule, and the Pixel Shuffle layer to form a window-based self-attention module;

[0074] 4.2) Constructing a region attention module:

[0075] 4.2.1) Establish an average pooling layer consisting of average pooling operations to reduce the feature map size;

[0076] 4.2.2) Construct a max-pooling layer consisting of max-pooling operations to reduce the feature map size;

[0077] 4.2.3) Convolutional layers consisting of 7×7 kernel operations are constructed to compress the output features of parallel average pooling and max pooling layers;

[0078] 4.2.4) Establish a sigmoid layer composed of sigmoid operations to generate weights corresponding to different pixels for the features of the convolutional layer, and then multiply the weights with the input features of the region attention module to obtain the output features with region attention.

[0079] 4.2.5) Construct a transposed convolutional layer consisting of transposed convolution operations to adjust the number of channels and resolution of features;

[0080] 4.2.6) The average pooling layer and the max pooling layer are connected in parallel, and then cascaded with the convolutional layer, the sigmoid layer and the transposed convolutional layer to form a region attention module.

[0081] 4.3) Connect the window-based self-attention module and the region attention module in parallel to form a decoder module;

[0082] 4.4) Concatenate the four decoder modules to form an attention-based decoder network.

[0083] Step 5: Use an attention-based decoder network to perform layer-by-layer optimization decoding on the features E1, E2, E3, E4 output by the encoder network and the global information feature X4 output by the aggregation structure, to obtain the decoder network output feature X0.

[0084] 5.1) Input the output feature E4 of the fourth layer encoder module and the output feature X4 of the aggregation structure into the fourth layer decoder module of the decoder network to obtain the output feature X3 of the fourth layer encoder module;

[0085] 5.2) Input the output feature E3 of the third layer encoder module and the output feature X3 of the fourth layer decoder module into the third layer decoder module of the decoder network to obtain the output feature X2 of the third layer encoder module;

[0086] 5.3) Input the output feature E2 of the second layer encoder module and the output feature X2 of the third layer decoder module into the second layer decoder module of the decoder network to obtain the output feature X1 of the second layer encoder module;

[0087] 5.4) Input the output feature E1 of the first layer encoder module and the output feature X1 of the second layer decoder module into the first layer decoder module of the decoder network to obtain the output feature X0 of the first layer encoder module, which is the output feature of the entire decoder network.

[0088] Step 6: Construct a depth map prediction network.

[0089] Reference Figure 4 The specific implementation of this step is as follows:

[0090] 6.1) Establish an adaptive interval center prediction module that sequentially includes a convolutional layer and two parallel average pooling layers and max pooling layers with learnable parameters, for adaptively predicting the interval center of the depth values ​​of the input image;

[0091] 6.2) Establish a probability head module consisting of a convolutional layer and a softmax layer to predict the probability vector corresponding to the center of the depth value interval of the input image;

[0092] 6.3) Connect the adaptive interval center prediction module and the probability head module in parallel to form a depth map prediction network;

[0093] Step 7: Use the depth map prediction network to predict the depth map corresponding to the input image.

[0094] 7.1) Predict the interval center c(b) of the depth values ​​of the input image:

[0095] 7.1.1) Input the decoder network output feature X0 into a convolutional layer with a kernel size of 1×1 to increase the number of channels of X0 to 256;

[0096] 7.1.2) The feature X0 with increased channel number is passed through two parallel average pooling layers and max pooling layers to obtain two corresponding tensors. Two learnable parameters ρ1 and ρ2 are introduced and multiplied by the two generated tensors respectively. The two tensors are then added together to obtain the interval width of the depth value of the input image. To simultaneously capture global information and extract salient information;

[0097] 7.1.3) Using the interval width b, calculate the interval center of the depth values ​​of the input image.

[0098]

[0099] Where, d min and d max These represent the minimum and maximum depth values, respectively.

[0100] 7.2) Predict the probability vector v corresponding to the center c(b) of the depth value interval of the input image:

[0101] 7.2.1) Input the decoder network output feature X0 into a convolutional layer with a kernel size of 3×3, a stride and padding of 1, and 256 output channels to change the number of channels of X0;

[0102] 7.2.2) The feature X0 after changing the number of channels is passed through a softmax layer to generate the probability vector v corresponding to the center of the depth value interval of the input image;

[0103] 7.3) Predict the depth map corresponding to the input image:

[0104] The initial depth map is obtained by linearly combining the adaptive interval center c(b) and the probability vector v. Then, the initial depth map is restored to the size of the input image by upsampling to obtain the final depth map.

[0105] The effects of the present invention will be further described below with reference to simulation experiments.

[0106] 1. Experimental conditions:

[0107] The computer used has an Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz processor, 128GB of RAM, and an NVIDIA GeForce RTX 3090 GPU with 24GB of video memory.

[0108] The operating system was 64-bit Ubuntu 18.04 (LTS), and the deep learning framework used was PyTorch (version 1.10.1). All network training employed backpropagation to calculate the residuals of each layer, and the Adam optimizer was used to update the network parameters. The Adam optimizer parameters were β1 = 0.9, β2 = 0.999, and the weight decay term was 0.01. A total of 20 epochs were set, with an initial learning rate of 1e-5, which was linearly reduced to 1e-6 during training iterations. SILog loss was used to train the network.

[0109] The experiments were evaluated using the KITTI and NYUV2 datasets. The error metrics used included mean absolute relative error (AbsRel), mean square relative error (SqRel), root mean square error (RMSE), root mean square logarithmic error (RMSElog), and mean logarithmic error (log10). A threshold of δ < 1.25 was used. i The model is evaluated using accuracy metrics for i = 1, 2, 3. Lower error metrics are better, and higher accuracy metrics are better.

[0110] The KITTI dataset is the most popular depth estimation benchmark, consisting of outdoor scenes captured by sensor devices on moving vehicles. This dataset includes stereo images and corresponding LiDAR scans. Training was performed using 23,158 images, and testing was conducted using 697 images, based on Eigen's partitioning. During training and testing, the input images were cropped, and the output was upsampled to the label size. Furthermore, the upper bound for depth prediction was limited to within 10 meters.

[0111] The NYUV2 dataset is an indoor dataset comprising 120,000 RGB and depth pairs collected from 464 indoor scenes. Training and testing are performed using the official split, with 36,253 images from 249 scenes used for training and 654 images from 215 scenes used for testing. Input images were cropped during training and testing, and the upper bound for depth prediction was limited to within 10 meters.

[0112] 2. Experiment Content:

[0113] Experiment 1: Under the experimental conditions described above, monocular image depth estimation was performed on the KITTI dataset using the method of this invention and 11 existing image depth estimation methods. The results of the objective evaluation metrics are shown in Table 2, and the subjective results are attached. Figure 5 As shown, where Figure 5 (a) is the input image. Figure 5 (b) is the depth map predicted by the existing method NeWCRFs. Figure 5 (c) is the depth map predicted by the existing method PixelFormer. Figure 5 (d) is the depth map predicted by this invention. Figure 5 (e) is the true depth map corresponding to the input image.

[0114] Table 2 shows the results of image depth estimation using the present invention and 11 existing methods on the test set of the KITTI dataset.

[0115]

[0116]

[0117] The 11 existing methods listed in Table 2 are as follows:

[0118] Eigen et al.: Eigen et al. first proposed a monocular image depth estimation algorithm based on convolutional neural networks;

[0119] DORN: A monocular image depth estimation algorithm based on ordinal classification, first proposed by Lo et al.

[0120] Yin et al.: A monocular image depth estimation algorithm based on virtual normal geometric constraints proposed by Yin et al.

[0121] BTS: A monocular image depth estimation algorithm based on a local planar guided layer proposed by Lee et al.

[0122] PWA: A monocular image depth estimation algorithm based on a patch attention mechanism proposed by Lee et al.

[0123] Adabins: A monocular image depth estimation algorithm based on adaptive depth intervals proposed by Farooq et al.

[0124] P3Depth: A monocular depth estimation algorithm based on piecewise plane priors proposed by Patil et al.

[0125] Depthformer: A monocular image depth estimation algorithm proposed by Agarwal et al. based on local and global information fusion;

[0126] BinsFormer: A monocular image depth estimation algorithm based on transformer, proposed by Kolesnikov et al.

[0127] NeWCRFs: A monocular image depth estimation algorithm based on conditional random fields proposed by Yuan et al.

[0128] PixelFormer: A monocular image depth estimation algorithm based on pixel query optimization proposed by Agarwal et al.

[0129] As can be seen from Table 2, compared with 11 existing comparative methods, the present invention has achieved the best or second-best results in all objective evaluation indicators.

[0130] from Figure 5 As can be seen, compared with 11 existing comparative methods, the present invention achieves better depth estimation results and generates clearer contours in image areas with poor lighting or small depth changes, such as walls, windows, and tree canopies.

[0131] Experiment 2: Under the experimental conditions described above, monocular image depth estimation was performed on the NYUV2 dataset using the method of this invention and 12 existing image depth estimation methods. The results of the objective evaluation metrics are shown in Table 3, and the subjective results are attached. Figure 6 As shown, where Figure 6 (a) is the input image. Figure 6 (b) is the depth map predicted by the existing method NeWCRFs. Figure 6 (c) is the depth map predicted by the existing method PixelFormer. Figure 6 (d) is the depth map predicted by this invention. Figure 6 (e) is the true depth map corresponding to the input image.

[0132] Table 3 shows the results of image depth estimation using the present invention and 12 existing methods on the test set of the NYUV2 dataset.

[0133]

[0134] The existing method DAV in Table 3 is a monocular image depth estimation algorithm based on a local planar guided layer proposed by Lee et al. The other 11 existing methods in Table 3 are consistent with the methods in Table 2.

[0135] As can be seen from Table 3, compared with 12 existing comparative methods, the present invention has achieved the best or second-best results in all objective evaluation indicators.

[0136] from Figure 6 As can be seen, compared with 12 existing comparative methods, the present invention achieves better depth estimation results and generates clearer contours in image areas with poor lighting or small depth changes, such as windows, monitors, and keyboards.

[0137] Experiment 3: By making different deletions or replacements to the various modules proposed in this invention, a variety of new methods were obtained. These methods were then used to perform monocular image depth estimation on the KITTI dataset. The results of their objective evaluation metrics are shown in Table 4.

[0138] Table 4. Ablation experiments of various new methods for image depth estimation on the KITTI dataset test set.

[0139]

[0140] In Table 4, "ABD" refers to the attention-based decoder network proposed in this invention, "CR" refers to the classification and regression deep prediction network proposed in this invention, and "GIA" refers to the global information aggregation structure proposed in this invention. "Baseline" refers to the new method of this invention after removing "GIA" and replacing "ABD" and "CR". "Abs Rel", "Sq Rel", "RMSE", and "RMSE log" are error indicators, with "↓" indicating that the lower the indicator value, the better; "δ<1.25", "δ<1.25" indicates the error. 2 "and"δ<1.25 3 "This is the accuracy indicator, and "↑" indicates that the higher the indicator value, the better.

[0141] Baseline: refers to the first novel method that uses the Swing Transformer network as the encoder, the decoder as the decoder of the existing UNet network, and the existing regression prediction network to estimate image depth;

[0142] Baseline+ABD: This refers to a second new method based on the Baseline method, which replaces the decoder of the Baseline method with the decoder network based on the attention mechanism proposed in this invention.

[0143] Baseline+ABD+CR refers to a third new method based on the "Baseline+ABD" method, which replaces the existing regression prediction network with the classification regression deep prediction network proposed in this invention.

[0144] Baseline+ABD+CR+GIA (this invention): refers to the method formed by adding the global information aggregation structure proposed in this invention to the "Baseline+ABD+CR" method, i.e., this invention.

[0145] As shown in Table 4, compared with all other methods, the present invention has the lowest error index and the highest accuracy index for monocular image depth estimation on the KITTI dataset. That is, the image depth estimation performance of the present invention is the best. Furthermore, after adding or replacing the attention-based decoder network, classification regression depth prediction network and global information aggregation structure proposed in the present invention, the error index of monocular image depth estimation on the KITTI dataset of other methods continuously decreases and the accuracy index continuously increases, that is, the image depth estimation performance continuously improves, which proves the effectiveness of the various modules proposed in the present invention.

[0146] In summary, this invention achieves the best monocular image depth estimation results on both the KITTI and NYUV2 datasets, demonstrating that this invention achieves optimal results in both subjective and objective evaluation metrics for the monocular image depth estimation task.

Claims

1. A monocular image depth estimation method based on an attention mechanism, characterized in that, Comprising the following steps: (1) Read training data and test data from monocular image depth estimation database, and sequentially perform rotation, scaling, flipping, adjustment and normalization preprocessing on the images of the training data to obtain training data; Preprocess the images of the test data sequentially by scaling and normalization to obtain test data; (2) using a Swin Transformer network comprising four cascaded Swin Transformer stage modules as an encoder network, inputting training data and test data into the Swin Transformer network respectively, and obtaining training image features output by the four cascaded Swin Transformer stage modules respectively , , and and test image features , , and ; (3) Use aggregated structure to enhance the global information of the features: 3a) the pyramid pooling module is composed of 4 average pooling layers with pooling ratios of 1, 2, 3 and 6 in parallel, and the training image features output by the top layer of the encoder are input into the pyramid pooling module to extract multi-scale information and the multi-scale information is concatenated with the input features after a convolutional layer to generate new features with global information; 3b) optimizing the new features with global information through the existing convolution self-attention module to obtain optimized global information features ; (4) Build a decoder network based on attention mechanism: 4a) Establish a window-based self-attention module composed of a window self-attention submodule, a shift window self-attention submodule, and a Pixel Shuffle layer in cascade; 4b) Establish a regional attention module composed of an average pooling layer and a maximum pooling layer in parallel, and then in cascade with a convolution layer, a sigmoid layer, and a transposed convolution layer; 4c) Parallelly connect the window-based self-attention module and the regional attention module to form a decoder module, 4d) Cascade 4 decoder modules to form a decoder network based on attention mechanism; (5) Use an attention-based decoder network to process the features output by the encoder network. , , , and global information features output by the aggregation structure Perform layer-by-layer optimization decoding to obtain features sequentially. , , and The final output features ; (6) Build a depth map prediction network: 6a) Establish an adaptive interval center prediction module containing a convolution layer and two parallel average pooling layers and maximum pooling layers with learnable parameters, for adaptively predicting the interval center of the input image depth value; 6b) Establish a probability head module composed of a convolution layer and a softmax layer, for predicting the probability vector corresponding to the interval center of the input image depth value; 6c) Parallelly connect the adaptive interval center prediction module and the probability head module to form a depth map prediction network; (7) Predict the depth map depth corresponding to the input image: 7a) inputting the output features of the decoder network to an adaptive bin center prediction module and a probability head module in the depth map prediction network, respectively, to output an adaptive bin center c(b) and a probability vector v of the image depth value; 7b) Linearly combine the adaptive interval center c(b) and the probability vector v to obtain a preliminary depth map, and then restore the preliminary depth map to the size of the input image through upsampling operation to obtain the final depth map depth.

2. The method of claim 1, wherein, The pre-processing of rotating, scaling, flipping, adjusting and normalizing the images of the training data in step (1) is to firstly randomly rotate the input image [-1 ° , 1 ° ], then scale the image size according to the data set, then randomly horizontally flip the image with a probability of 0.5, and randomly adjust the brightness, contrast, saturation and hue of the image with a probability of 0.5, and then normalize the image by using the following normalization formula; ； where x is the pixel value of a single channel of the RGB image, μ is the mean of the corresponding channel pixel values, S is the standard deviation of the corresponding channel pixel values, is the normalized corresponding channel pixel value.

3. The method of claim 1, wherein, The four Swin Transformer stage modules in step (2) have the same structure, and each Swin Transformer stage module contains a down-sampling layer and a plurality of repeated Swin Transformer Blocks.

4. The method of claim 1, wherein, In step 3b), the new features with global information are optimized by the convolution self-attention module, which realizes as follows: The input feature map is projected by 3 1x1 convolutions to obtain intermediate features, which are processed according to two parallel operations, one of which is to shift the intermediate features, and the other is to take the intermediate features as query, key and value matrices, and calculate self-attention; Then add the output features of the two parallel operations to obtain the optimized features.

5. The method of claim 1, wherein, In step 4a), the window self-attention submodule, the shift window self-attention submodule and the Pixel Shuffle layer in the self-attention module have the following structure and function: The window self-attention submodule sequentially contains two parallel convolution layers with a kernel size of 3x3 and a stride of 1, a regularization layer, a window division layer, a first multilayer perceptron, a window-based multi-head self-attention layer, and a second multilayer perceptron; wherein: The convolution layer is used to change the channel number of the input feature; The regularization layer is used to regularize the input feature; The window division layer is used to divide the input feature by a 7x7 window; The first multi-layer perception is used to generate a query matrix Q, a key matrix K and a value matrix V; The window-based multi-head self-attention layer calculates multi-head self-attention by using the query matrix Q, the key matrix K and the value matrix V; The second multi-layer perception is used to increase the nonlinear fitting ability of the network; The shift window self-attention sub-module has the same structure as the window self-attention sub-module, except that in the window division layer, the position of each 7x7 window is shifted 3x3 to the right and down; The Pixel Shuffle layer is an existing Pixel Shuffle operation, which is used to adjust the channel number and resolution of the feature.

6. The method of claim 1, wherein, Step 4b) The structure and function of each layer in the region attention module constructed in step 4a) are as follows: The average pooling layer is an average pooling operation, which is used to reduce the size of the feature map; The max pooling layer is a max pooling operation, which is used to reduce the size of the feature map; The convolution layer is a convolution operation with a kernel size of 7x7, which is used to compress the output features of the parallel average pooling layer and max pooling layer; The sigmoid layer is a sigmoid operation, which is used to generate weights corresponding to different pixels for the features of the convolution layer, and then point the weights with the input features of the region attention module to obtain output features with region attention; The transpose convolution layer is a transpose convolution operation, which is used to adjust the channel number and resolution of the feature. In step 6a), the adaptive interval center prediction module adaptively predicts the interval center of the input image depth value, which is implemented as follows:

7. The method of claim 1, wherein, Step (5) uses an attention mechanism based decoder network to perform layer-wise optimization decoding on the features output by the encoder network , , , and the global information features output by the aggregation structure ​ 5a) inputting the output features of the 4th layer encoder module and the output features of the aggregated structure to a 4th layer decoder module of the decoder network to obtain the output features of the 4th layer encoder module ; 5b) inputting the output features of the 3rd layer encoder module and the output features of the 4th layer decoder module into the 3rd layer decoder module of the decoder network, resulting in output features of the 3rd layer encoder module ; 5c) inputting the output features of the 2nd layer encoder module and the output features of the 3rd layer decoder module to the 2nd layer decoder module of the decoder network, resulting in output features of the 2nd layer encoder module ; 5d) inputting the output features of the 1st layer encoder module and the output features of the 2nd layer decoder module into the 1st layer decoder module of the decoder network to obtain the output features of the 1st layer encoder module i.e. the output features of the entire decoder network.

8. The method of claim 1, wherein, In step 6b), the probability head module predicts the probability vector corresponding to the interval center of the input image depth value, which is implemented as follows: 6a1) Output features of the decoder network The input is fed into a convolutional layer with a kernel size of 1×1 to expand... The number of channels is increased to 256; 6a2) the features after expanding the number of channels Two corresponding tensors are obtained by two parallel average pooling layers and maximum pooling layers respectively, and two learnable parameters are introduced and The two tensors are multiplied respectively with the generated two tensors, and then the two tensors are added to obtain the interval width of the depth value of the input image to capture global information and extract significant information simultaneously; 6a3) Calculate the interval center of the input image depth value using the interval width b : ； where d min and d max respectively denote the minimum and maximum values of the depth values.

9. The method of claim 1, wherein, ​ 6b1) the decoder network output features A convolutional layer with kernel size 3x3, stride and padding 1, and 256 output channels is used to change the number of channels of the input ; 6b2) the features after the number of channels is changed A probability vector v corresponding to the center of the depth value interval of the input image is generated through a softmax layer.

Images