A monocular image depth estimation method based on multi-scale cross attention

Abstract

A monocular image depth estimation method based on multi-scale cross attention, which utilizes the complementary advantages of convolutional neural networks and visual Transformers, takes advantage of the local feature capture advantage of convolutional neural networks and the long-distance dependence advantage of global feature establishment of Transformers, improves the information retention capability, and thus improves the quality of the generated depth map; through the cross attention mechanism, the attention calculation intensity of different regional pixels is effectively controlled, the ability to capture local features and global features is balanced, the overall structure of the depth map is optimized, the fine granularity of the estimated depth map is improved, and the estimated depth is more natural; the capture ability of local features and global features of the image is improved, the problem of unnatural depth structure in monocular depth estimation methods is solved, the expression ability of detailed features is significantly improved, and the problem of insufficient details in monocular depth estimation is solved; the problem of huge consumption of computing resources in monocular depth estimation methods based on visual Transformers is alleviated, and the weak sensitivity of Transformers to local features is compensated.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a monocular image depth estimation method based on multi-scale cross-attention. Background Technology

[0002] With the improvement of computing power and the reduction of hardware costs, people have developed more and more intelligent robots and devices to replace or assist humans in performing complex and dangerous tasks. In the application of artificial intelligence, the importance of depth information is becoming increasingly prominent. For example, applications such as assisted driving technology, 3D reconstruction, and robot localization all rely heavily on high-precision depth information. However, currently widely used imaging devices, such as handheld cameras and mobile phones, often produce two-dimensional images, corresponding to RGB values, lacking depth values. These practical needs highlight the importance of research related to monocular depth estimation.

[0003] Traditional methods estimate image depth using visual cues such as texture, occlusion, and shadow variations. These methods largely rely on specific visual information within the scene and require significant computational resources, making them difficult to meet practical industrial needs. In contrast, deep learning-based methods have achieved significant breakthroughs. On one hand, the inductive bias and local feature capture of convolutional neural networks further improve prediction accuracy. Theoretically, deeper networks yield higher prediction accuracy; however, as networks become deeper, problems such as gradient vanishing occur. Furthermore, the inherent limited capacity of convolutional networks determines the upper limit of the amount of information stored and generated. On the other hand, the attention mechanism of visual transformers demonstrates superior performance in capturing global features compared to convolutional networks, but it is less effective at local perception. Moreover, monocular depth estimation is a pixel-level regression problem, and the computational complexity introduced by attention becomes a new pain point.

[0004] Furthermore, existing monocular depth estimation methods still suffer from problems such as unclear contours, uneven depth smoothness, and difficulty in preserving details in regions with small depth gradients. This is mainly reflected in the unequal representation capabilities of local and global features, leading to an unnatural overall structure in the generated depth maps. Moreover, existing methods partially rely on the inherent advantages of deep learning frameworks, neglecting the auxiliary and guiding roles of features such as shape, color, and texture in monocular depth estimation, and failing to fully utilize the role of network modules such as attention mechanisms. Summary of the Invention

[0005] To address the problems existing in the above-mentioned background technology, this invention proposes a monocular image depth estimation method based on multi-scale cross-attention, which improves the quality of the generated depth map.

[0006] A monocular image depth estimation method based on multi-scale cross-attention includes the following steps:

[0007] Step 1: Obtain deep datasets from various scenarios for training;

[0008] Step 2: Map the ground truth values ​​in the depth dataset from Step 1 to the same domain, and perform regularization on all ground truth values ​​to achieve scale scaling;

[0009] Step 3: Input the images from the training dataset processed in Step 2 into the encoder consisting of a convolutional module and a Transformer module to generate image feature vectors at different scales;

[0010] Step 4: Input the image features from Step 3 into the cross-attention mechanism module for advanced feature extraction;

[0011] Step 5: The features extracted in Step 4 are fed into the feature fusion module for feature fusion layer by layer, and then fed into the depth information output block to obtain the network prediction depth value.

[0012] Step 6: Use the least squares algorithm to perform scale matching between the depth value predicted in Step 5 and the true value preprocessed in Step 2 to obtain the scale factor and translation factor.

[0013] Step 7: Use the scaling factor and translation factor obtained in Step 6 to scale the network predictions, and use the scale-invariant loss function and the marginal gradient-invariant loss function to compare with the true values ​​to calculate the loss, and then optimize the loss to the global minimum to obtain the trained network model.

[0014] Step 8: Perform monocular depth estimation on any scene image based on the trained model. Input a single arbitrary image into the model trained in Step 7 to generate the corresponding depth map.

[0015] The beneficial effects achieved by this invention are as follows:

[0016] (1) This invention utilizes the complementary advantages of convolutional neural networks and visual Transformers, leveraging the advantages of convolutional neural networks in capturing local features and Transformers in establishing long-distance dependencies on global features, thereby improving the ability to retain information and thus enhancing the quality of the generated depth map.

[0017] (2) This invention effectively controls the attention calculation intensity of pixels in different regions through a cross-attention mechanism, balances the ability to capture local and global features, optimizes the overall structure of the depth map, improves the fineness of the estimated depth map, and makes the estimated depth more natural.

[0018] (3) Based on the advantages of convolutional neural networks and Transformer modules, this invention improves the ability to capture local and global features of images, optimizes the problem of unnatural depth structure in monocular depth estimation methods, and significantly improves the ability to express detailed features, thus solving the key problem of insufficient detail in monocular depth estimation.

[0019] (4) This invention alleviates the problem of huge computational resource consumption of monocular depth estimation method based on visual Transformer by using cross-attention mechanism, and at the same time makes up for the weak sensitivity of Transformer to local features. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall architecture of the monocular image depth estimation method in this embodiment of the invention.

[0021] Figure 2 This is a schematic diagram of feature extraction using a ResNet-based convolutional module in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of feature extraction of the Transformer module in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of the cross-attention mechanism in an embodiment of the present invention. Detailed Implementation

[0024] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0025] A monocular image depth estimation method based on multi-scale cross-attention includes the following steps:

[0026] Step 1: Obtain deep learning datasets for various scenarios used in training. Specifically, several mainstream deep learning datasets are used. To achieve generalization performance of the depth estimation network, the dataset scenarios include indoor, outdoor, street scenes, daily life scenes and objects, with a total size of approximately 200,000.

[0027] Step 2: Map the true values ​​of the depth dataset from Step 1 to the same range, thereby achieving domain unification, including the disparity domain and the depth domain. Then, perform regularization on all true values, that is, scale the values ​​in the dataset proportionally, thereby achieving scale scaling.

[0028] Step 3: Input the images from the training dataset processed in Step 2 into the encoder consisting of a convolutional module and a Transformer module to generate image feature vectors at different scales.

[0029] The encoder in step 3 includes a convolution module and a Transformer module.

[0030] The convolutional module consists of convolutional layers, pooling layers, fully connected layers, non-linear layers, and softmax layers. It performs convolutional calculations on images to extract information, undergoing cross-processing through multiple convolutional layers to achieve a mapping from low-level features to high-level features, such as... Figure 2 As shown, feature map A is obtained, and its expression is as follows:

[0031] x∈R H×W×C →x∈R h×w×c

[0032] Where x is the image; R is the set of real numbers; H is the length of the image; W is the width of the image; C is the number of channels of the image; and h, w, and c are the height, width, and number of channels of the encoded image, respectively.

[0033] The Transformer module consists of an image segmentation module, an image embedding module, and a location embedding module, all based on the Transformer layer architecture.

[0034] Because the input to the Transformer is a sequence of one-dimensional token embeddings, processing the two-dimensional image yields features B1, B2, B3, and B4 from the four Transformer layers, which together form feature B (the subsequent process is the calculation process for feature B). Figure 3 As shown, first, the image is reshaped into a patch, and its expression is as follows:

[0035] x∈R H×W×C →x∈R N×(P×P×C)

[0036] Where x is the image; R is the set of real numbers; H is the length of the image; W is the width of the image; C is the number of channels in the image; and (P,P) is the resolution of each patch. This is the total number of patches.

[0037] Subsequently, patch embedding is achieved by linear projection mapping of the image patch. Simultaneously, image location information and a learnable embedding block are added. Its expression is as follows:

[0038]

[0039] E∈R (P×P×C)×D E pos ∈R (P×P×C)×D

[0040] Where z0 is the output sequence of the encoder; x class It is a learnable embedding block; It is a sequence of image patches, with a number of N; E is the embedding operation; E pos It is the embedding of image location information; D is the mapping dimension; C is the number of image channels; P is the image patch resolution.

[0041] It is then fed into a Transformer layer consisting of a multi-head attention module (MHSA), which also includes layer normalization (LN) and a multilayer perceptron (MLP), as shown in the following expression:

[0042] Z′ l =MSA(LN(Z) l-1 ))+Z l-1 l = 1, 2, ..., L

[0043] Z l =MLP(LN(Z′) l ))+Z′ l l = 1, 2, ..., L

[0044] Among them, Z l It is the mapped token sequence, i.e., feature B; Z' l It is the output sequence after multi-head attention; MSA is multi-head self-attention; LN is layer normalization; MLP is multilayer perceptron; L is the length of the token sequence.

[0045] Step 4: Input the image features from Step 3 into the cross-attention mechanism module for advanced feature extraction.

[0046] Step 4 receives feature A obtained from the convolutional module in Step 3 and feature B obtained from the Transformer module. First, feature A output from the convolutional neural network is processed using the block embedding step of the Transformer module, followed by token C. Then, the Transformer's cls token is linked to token C. l (·) is the projection matrix, and then... and X' l Cross-attention (CA) is performed between the query blocks, where the cls token is the unique query block q. Therefore, the computational complexity and memory consumption of the attention graph generated by cross-attention are linear, rather than quadratic, making the entire process more efficient. Its expression is as follows:

[0047]

[0048]

[0049]

[0050] CA(X' l)=Av

[0051] Among them, f l (·) is the projection matrix; It is a learnable sequence of cls tokens for the Transformer; It is a sequence of convolutional features A; X' l It is a sequence and The concatenated sequence; q, k, v are sequences X' l ,X' l Passing through W respectively q W k W v The learnable parameters are: Softmax is the activation function; c is the embedding dimension; n is the number of attention heads; A is the attention map; CA is cross attention.

[0052] Step 5: The features extracted in Step 4 are fed into the feature fusion module for feature fusion layer by layer, and then fed into the depth information output block to obtain the network prediction depth value.

[0053] In step 5, the feature map processed by the attention module in step 4 is received and feature fusion is performed. The feature fusion module includes two residual convolutional layers, an upsampling module, and a linear projection. One residual convolutional layer receives shallow features, calculates the residual features, adds them to the deep features, and then sends them to the other residual convolutional layer for upsampling and projection. The residual convolutional layer includes a ReLU activation function layer, a convolutional module, a batch normalization module, a ReLU activation function layer, a convolutional module, and a batch normalization module, arranged in sequence. The convolutional kernels of the two convolutional modules are both 3×3.

[0054] In step 5, the depth information output block includes a convolutional module with a kernel size of 3×3, an upsampling module, a convolutional module with a kernel size of 3×3, a ReLU activation function, a convolutional module with a kernel size of 1, and a ReLU activation function.

[0055] Step 6: Use the least squares algorithm to perform scale matching between the depth value predicted in step 5 and the true value preprocessed in step 2 to obtain the scale factor s and the translation factor t.

[0056] In step 6, the expression for scale matching for the scale factor s and the translation factor t is:

[0057]

[0058] Where d i To predict depth values ​​for the model, Let i be the ground truth depth value, i be the i-th pixel, and V be the number of valid pixels. After alignment, the model's predicted value is... The actual value is

[0059] Step 7: Use the scaling factor and translation factor obtained in Step 6 to scale the network predictions, and use the scale-invariant loss function and the marginal gradient-invariant loss function to compare with the true values ​​to calculate the loss, and then optimize the loss to the global minimum to obtain the trained network model.

[0060] The expression for the scale-invariant loss function in step 7 is:

[0061]

[0062] Where W is the number of pixels in the image with valid truth values; j is the j-th pixel in the image with a valid truth value. and Let represent the depth estimate and the valid true value corresponding to the j-th pixel, respectively.

[0063] The expression for the marginal gradient-invariant loss function is:

[0064]

[0065] Where K represents different scales. R k The disparity mapping difference at scale k; and represents the offset of a pixel on the x-axis and y-axis, respectively.

[0066] The overall training loss function is as follows:

[0067]

[0068] Where N l This indicates the number of images processed in each batch, and 'a' represents the weight of the gradient loss function.

[0069] Step 8: Perform monocular depth estimation on any scene image based on the trained model. Input a single arbitrary image into the model trained in Step 7 to generate the corresponding depth map.

[0070] Regarding the above method and process, the following should be noted:

[0071] 1. Figure 1 The cross-attention mechanism can be replaced in different ways, such as replacing multi-head attention with channel attention or spatial self-attention, but the operational purpose remains the same.

[0072] 2. Figure 2The residual neural network in the model can be replaced with another type, such as LeNet or AlexNet, to accomplish the same feature capture task.

[0073] 3. Figure 3 The Transformer can be improved by selecting modules such as Swing Transformer and TwinsTransformer, which are strategy improvement modules for Transformer. In essence, they still belong to the category of establishing long-distance dependencies of global features.

[0074] 4. Figure 4 The cross-attention mechanism in [the text] involves three input branches. Features captured by the convolutional module and features captured by the Transformer module can be arbitrarily combined and input into different branches, all adhering to the same cross-attention concept. (The input to the cross-attention mechanism is features. Feature maps are relatively abstract; low-level feature transformations reveal observable content, but high-level feature transformations into graphical representations are invisible to the naked eye. The processing flow corresponds to step 4.)

[0075] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.

Claims

1. A monocular image depth estimation method based on multi-scale cross-attention, characterized in that: The method includes the following steps: Step 1: Obtain deep datasets from various scenarios for training; Step 2: Map the ground truth values ​​in the depth dataset from Step 1 to the same domain, and perform regularization on all ground truth values ​​to achieve scale scaling; Step 3: Input the images in the dataset processed in Step 2 into the encoder consisting of a convolutional module and a Transformer module to generate image feature vectors at different scales; Step 4: The image features from Step 3 are fed into the cross-attention mechanism module for advanced feature extraction. The cross-attention mechanism adopts the concept of cross-interaction and includes three input branches. The features captured by the convolution module and the features captured by the Transformer module can be arbitrarily combined and input into different branches. Step 5: The features extracted in Step 4 are fed into the feature fusion module for feature fusion layer by layer, and then fed into the depth information output block to obtain the network prediction depth value. Step 6: Use the least squares algorithm to perform scale matching between the depth value predicted in Step 5 and the true value preprocessed in Step 2 to obtain the scale factor and translation factor. Step 7: Use the scaling factor and translation factor obtained in Step 6 to scale the network predictions, and use the scale-invariant loss function and the marginal gradient-invariant loss function to compare with the true values ​​to calculate the loss, and then optimize the loss to the global minimum to obtain the trained network model. Step 8: Perform monocular depth estimation on any scene image based on the trained model. Input a single arbitrary image into the model trained in Step 7 to generate the corresponding depth map.

2. The monocular image depth estimation method based on multi-scale cross-attention according to claim 1, characterized in that: In the encoder of step 3, the convolution module consists of convolutional layers, pooling layers, fully connected layers, nonlinear layers and softmax layers. It performs convolution calculations on the image to extract information. After the cross processing of multiple convolutional layers, it realizes the mapping from low-level features to high-level features and obtains feature A. The Transformer module consists of an image segmentation module, an image embedding module, and a location embedding module, all designed based on the Transformer layer architecture.

3. The monocular image depth estimation method based on multi-scale cross-attention according to claim 2, characterized in that: In step 3, the input to the Transformer is a sequence of one-dimensional token embeddings. Therefore, the two-dimensional image is processed, and the four Transformer layers yield features B1, B2, B3, and B4, which are features B. First, the image is reshaped into a patch, the expression of which is as follows: Where R is the set of real numbers; H is the length of the image; W is the width of the image; C is the number of channels in the image; and (P, P) is the resolution of each patch. This is the total number of patches; Subsequently, the image patch is linearly projected and mapped to achieve patch embedding; simultaneously, the image's positional information and a learnable embedding block are added. Its expression is as follows: , in, It is the output sequence of the encoder; It is a learnable embedding block; It is a sequence of image patches, with a number of N; E is the embedding operation; It is the embedding of image location information; D is the mapping dimension; C is the number of image channels; P is the image patch resolution; It is fed into a Transformer layer consisting of multi-head attention modules, which includes layer normalization and multi-layer perceptrons, and its expression is as follows: in, It is the mapped token sequence, i.e., feature B; It is the output sequence after multi-head attention; MSA is multi-head self-attention; LN is layer normalization; MLP is multilayer perceptron; L is the length of the token sequence.

4. The monocular image depth estimation method based on multi-scale cross-attention according to claim 1, characterized in that: In step 4, features A and B from step 3 are received. First, feature A, output by the convolutional neural network, is processed using token C, similar to the block embedding step of Transformer. Then, the Transformer's cls token is linked to token C. It is the projection matrix, and then... and Cross-attention (CA) is performed between them, where the cls token is the unique query block q, and its expression is as follows: in, It is the projection matrix; It is a learnable sequence of cls tokens for the Transformer; It is a sequence of convolutional features A; It is a sequence and The concatenated sequence; q, k, v are sequences , , Each through learnable parameters , , The results are as follows: Softmax is the activation function; c is the dimension of the sequence embedding; n is the number of attention heads; A is the attention map; CA is cross-attention.

5. The monocular image depth estimation method based on multi-scale cross-attention according to claim 1, characterized in that: In step 5, the features processed by the attention module in step 4 are fused. The feature fusion module includes two residual convolutional layers, an upsampling module, and a linear projection module. One residual convolutional layer receives shallow features, calculates the residual features, adds them to the deep features, and then feeds them into the other residual convolutional layer for upsampling and projection. The residual convolutional layer includes a ReLU activation function layer, a convolutional module, a batch normalization module, a ReLU activation function layer, a convolutional module, and a batch normalization module, arranged sequentially. The convolutional kernels of both convolutional modules are... .

6. The monocular image depth estimation method based on multi-scale cross-attention according to claim 1, characterized in that: In step 5, the depth information output block includes the convolution kernel as follows: The size of the convolutional module, upsampling module, and convolutional kernel are as follows: Convolutional modules of size 1, ReLU activation function, convolutional modules with kernel size 1, and ReLU activation function.

7. The monocular image depth estimation method based on multi-scale cross-attention according to claim 1, characterized in that: In step 6, the expression for scale matching for the scale factor s and the translation factor t is: in To predict depth values ​​for the model, Let i be the ground truth depth value, i be the i-th pixel, and V be the number of valid pixels. After alignment, the model's predicted value is... The actual value is .

8. The monocular image depth estimation method based on multi-scale cross-attention according to claim 1, characterized in that: The expression for the scale-invariant loss function in step 7 is: Where V is the number of valid pixels, W is the number of pixels with valid truth values ​​in the image, and j is the j-th pixel with a valid truth value in the image. Let represent the depth estimate and the valid true value corresponding to the j-th pixel, respectively; The expression for the marginal gradient-invariant loss function is: Where K represents different scales. , The disparity mapping difference at scale k. and represents the offset of the pixel on the x-axis and y-axis, respectively; The overall training loss function is as follows: in This indicates the number of images processed in each batch, and 'a' represents the weight of the gradient loss function.

Images