A monocular image depth estimation algorithm based on self-attention neural network

Abstract

This invention discloses a monocular image depth estimation algorithm based on a self-attention neural network. In this method, low-level features of the monocular image are preprocessed, long-range connections are established using the self-attention neural network, and high-level features are encoded from the low-level features of the image to quickly obtain global image information. Token attention is also used to enhance the feature expression capability of the self-attention network. Finally, low-level features are fused with high-level features and depth information is decoded. To address the scale differences in depth ranges of different scenes and the blurred edges of depth maps, the invention uses a scale-invariant loss and an edge gradient-invariant loss function to supervise the network to learn affine-invariant depth information. The invention can quickly generate depth maps with accurate depth information and clear edges.

Description

Technical Field

[0001] The present invention relates to the field of 3D vision technology, and in particular to a monocular image depth estimation algorithm based on a self-attention neural network. Background Art

[0002] Monocular depth estimation involves estimating the three-dimensional spatial information of an image from a single image. This information is essential for numerous vision tasks, such as scene reconstruction and viewpoint generation. Depth information acquisition can be categorized into two main types: early capture and late acquisition. Early capture typically relies on specialized equipment, such as LiDAR or depth cameras. While this method can quickly acquire depth information, it requires specialized equipment, making it difficult for ordinary teams to obtain, and the captured depth maps are sparse, low-resolution, and difficult to obtain. In contrast, the later acquisition method, which is widely used, uses existing images to obtain three-dimensional information from existing two-dimensional images. Early monocular depth estimation primarily relied on image cues, including occlusion (generally, objects appear farther away when occluded); objects that are closer appear larger than objects that are farther away; surface texture (which appears denser at greater distances); and light and shadow relationships, all of which serve as cues for estimating depth. However, these manual inferences are labor-intensive and complex, and do not address the real challenges in this field.

[0003] With the development of deep learning and the emergence of numerous depth datasets, monocular depth estimation based on deep learning has addressed most challenges in this field. Using deep datasets to supervise networks, researchers have learned monocular depth models, significantly reducing the amount of effort and resources required. These methods, requiring only a single image as input, can generate dense depth maps, significantly reducing the amount of effort and resources required. Current monocular depth estimation methods based on supervised learning with deep networks primarily rely on convolutional neural networks. However, convolutional networks suffer from a limited receptive field (RF) constraint. In the shallow layers of the network, they can only capture local information. As the network deepens, their receptive field gradually expands, allowing them to capture global information. However, as the convolutional network deepens, it continuously downsamples, leading to the loss of fine-grained image features that cannot be recovered during the decoding phase. Consequently, current monocular depth estimation algorithms based on convolutional networks still suffer from issues such as loss of depth information details and low resolution. Furthermore, due to the varying annotations between depth datasets, most depth estimation algorithms are tailored to a single dataset. While they can perform well in certain scenes of the same type, they fail in others. This limits their application to a limited number of scenarios.

[0004] In response to the problem of limited prediction scenes, the prior art has also disclosed relevant technical solutions. For example, the Chinese patent application with application publication number: CN112288788A discloses a monocular image depth estimation method, which includes performing a joint loss calculation on the obtained predicted depth map and the corresponding GT depth map using a joint loss function of sorting loss, multi-scale structural similarity loss, and multi-scale scale-invariant gradient matching loss to obtain the corresponding monocular depth estimation map, solving the problems of poor generalization of monocular image depth estimation methods and insufficient depth map accuracy. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides an algorithm for depth estimation of monocular images based on a self-attention neural network, which can effectively predict different scenes.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] The present invention is a monocular image depth estimation algorithm based on a self-attention neural network, comprising the following steps:

[0008] Step 1: Obtain a depth dataset of various scenarios involved in training;

[0009] Step 2: Unify the domain of the real values in the depth dataset in step 1 and scale all the real values through regularization;

[0010] Step 3: Send the images in the training dataset to the image preprocessing network to generate low-level image features;

[0011] Step 4: Feed the low-level image features from step 3 into the self-attention neural network for high-level feature extraction;

[0012] Step 5: The image features extracted in steps 3 and 4 are sent to the feature fusion module for feature fusion, and then sent to the depth information decoder to obtain the network predicted depth value;

[0013] Step 6: Use the least squares algorithm to perform scale matching on the depth value predicted in step 5 and the real value preprocessed in step 2 to obtain the scale factor and translation factor;

[0014] Step 7: Use the scale factor and translation factor obtained in step 6 to scale the network prediction value, calculate the loss through the scale-invariant loss function and the edge gradient-invariant loss function, perform network backpropagation, and obtain a trained network with the optimized loss as the global minimum;

[0015] Step 8: Input a single arbitrary image into the network trained in step 7 to generate the corresponding depth information map.

[0016] A further improvement of the present invention is that: in step 3, the image preprocessing network includes a convolutional network and a block coding network;

[0017] Convolutional network transforms image I∈R (H×W×3) Perform low-level feature encoding to obtain the feature vector I∈R (h×w×c) , where H, W are the original height and width of the input image, h, w, c are the height, width and number of channels after encoding respectively; the block coding network transforms the feature vector I∈R (h×w×c) Perform block position encoding and reduce the three-dimensional vector to a token vector I∈R N×M , and embed positional encodings of the same size, where N is the number of final token vectors and M is the number of token vector feature dimensions.

[0018] A further improvement of the present invention is that the self-attention neural network for receiving the image features in step 3 for feature encoding in step 4 includes 12 layers, each layer includes a multi-head self-attention network, a multi-layer perceptron, and layer normalization. The layers are connected in series, and token attention TA(I) is added in several layers. The expression is as follows:

[0019] TA(I)=sigmoid(MLP(Maxpool(I))+MLP(AvgPool(I)))

[0020] Among them, MLP is a multi-layer perceptron, MaxPool is the maximum pooling layer, AvgPool is the average pooling layer, and sigmoid is the activation function.

[0021] A further improvement of the present invention is that: in step 5, the feature fusion module includes two residual convolution layers, an upsampling module and a linear projection in sequence, one residual convolution layer receives shallow features, calculates residual features and adds deep features, and then sends them to another residual convolution layer for upsampling and projection. The residual convolution layer includes a ReLu activation function layer, a convolution module, a batch normalization module, a ReLu activation function layer, a convolution module and a batch normalization module arranged in sequence, and the convolution kernels of the two convolution modules are both 3*3.

[0022] A further improvement of the present invention is that: in step 5, the depth information decoder includes a convolution module with a convolution kernel of 3*3 size, an upsampling module, a convolution module with a convolution kernel of 3*3 size, a ReLu activation function, a convolution module with a convolution kernel size of 1 and a ReLu activation function.

[0023] A further improvement of the present invention is that the expression for scale matching in step 6 is:

[0024]

[0025] where d iPredict depth values for the network, is the true depth value, i is the i-th pixel, and V is the number of valid pixels. The network prediction value after alignment is The true value is

[0026] A further improvement of the present invention is that the expression of the scale-invariant loss function in step 7 is:

[0027]

[0028] Where V is the number of effective pixels, W = 0.8V, excluding the maximum difference of 20%, is the depth prediction value after alignment, is the true depth value after alignment, and the subscript j represents the depth value of the j-th point.

[0029] The expression of the edge gradient invariant loss function is:

[0030]

[0031] Where V is the number of effective pixels, Represents the depth difference of point i, R k Indicates different scales k, the scale size decreases by half, k=4 means there are 4 scale sizes, Calculate the horizontal and vertical gradients at each scale separately.

[0032] The total training loss function is as follows:

[0033]

[0034] where N l Represents the number of images processed in each batch, and α represents the weight of the gradient loss function.

[0035] The beneficial effects of the present invention are: 1. The present invention utilizes the self-attention neural network to obtain high-level features in the shallow stage of the network, reduces unnecessary downsampling, and retains more fine features; and further enhances the feature expression of the self-attention neural network through token attention, which can generate a depth map with accurate depth information and complete image details.

[0036] 2. The present invention provides a training strategy and loss function that combines multiple depth data sets. Through preprocessing and scale matching, the model can be effectively trained to obtain a robust depth prediction model that can adapt to various prediction environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1Schematic diagram of the algorithm for depth estimation of monocular images based on self-attention neural network of the present invention.

[0038] Figure 2 This is a diagram of the self-attention network structure implemented in the present invention.

[0039] Figure 3 This is a diagram of the multi-head self-attention network structure implemented in the present invention.

[0040] Figure 4 This is a diagram of the token attention network structure implemented in the present invention.

[0041] Figure 5 This is a network structure diagram of the feature fusion module implemented in the present invention.

[0042] Figure 6 This is a network structure diagram of the depth information decoder implemented in the present invention.

[0043] Figure 7 It is an experimental comparison diagram between the method of the present invention and DPT. DETAILED DESCRIPTION

[0044] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them.

[0045] like Figure 1 As shown, the present invention is a monocular image depth estimation algorithm based on self-attention neural network, and the specific steps are as follows:

[0046] Step 1: Obtain a depth dataset of various scenarios involved in training;

[0047] The dataset used in this step is several existing mainstream depth datasets. To achieve generalization performance of the depth estimation network, the dataset scenes include indoor, outdoor, street scenes, daily life scenes and objects, and the dataset size reaches approximately 200,000. To improve the training efficiency of the self-attention network, this example uses network parameters pre-trained on the large image dataset ImageNet, and then performs parameter training on the depth training dataset, which significantly reduces the network's feature learning time.

[0048] Step 2: Unify the domains of the real values in the depth dataset in step 1, including the disparity domain and the depth domain, and scale all the real values through regularization;

[0049] Because the true values of each depth dataset contain both depth and disparity values, it is necessary to convert both the disparity domain and the inverse depth domain to a unified depth domain for supervision. Furthermore, because the datasets used in this embodiment include both indoor and outdoor scenes, with significantly different depth scales, the true depth values need to be scaled to grayscale levels of 0-255.

[0050] Step 3: Send the images in the training dataset to the image preprocessing network to generate low-level image features;

[0051] Step 4: Feed the low-level image features from step 3 into the self-attention neural network for high-level feature extraction;

[0052] Step 5: The image features extracted in steps 3 and 4 are sent to the feature fusion module for feature fusion, and then sent to the depth information decoder to obtain the network predicted depth value;

[0053] Step 6: Use the least squares algorithm to scale-match the depth value predicted in step 5 with the true value preprocessed in step 2 to obtain the corresponding scale factor s and translation factor t;

[0054] Step 7: Use the scale factor and translation factor obtained in step 6 to scale the network prediction value, calculate the loss through the scale-invariant loss function and the edge gradient-invariant loss function, perform network backpropagation, and obtain a trained network with the optimized loss as the global minimum;

[0055] Step 8: Input a single arbitrary image into the network trained in step 7 to generate the corresponding depth information map. The preprocessing network used in step 3 includes a convolutional network ResNet50 and a block coding network. The monocular color image is fed into the preprocessing network to extract low-level image features. The specific operations are as follows:

[0056] Convolutional network transforms image I∈R (H×W×3) Perform low-level feature encoding to obtain the feature vector I∈R (h×w×c) In this embodiment, H and W are the original height and width of the input image, both of which are 384 pixels. h, w, and c are the height, width, and number of channels after encoding, respectively, which are 32, 32, and 144.

[0057] The eigenvector I∈R (h×w×c) Block position encoding is performed to reduce the dimension of feature vectors and identify feature positions. The three-dimensional vector is reduced to a two-dimensional token vector I∈R N×M , while embedding the same size P∈R N×M The position encoding in this embodiment is a trainable random vector, which is randomly generated when initialized.

[0058] Step 4: Feed the low-level image features into the self-attention neural network for high-level feature extraction;

[0059] The self-attention neural network used in this embodiment is as follows Figure 2 As shown, it is composed of a 12-layer self-attention neural network cascade, where each layer of the self-attention neural network includes a multi-head self-attention layer MSA, layer normalization LN, a multi-layer perceptron MLP and cross-connections between features. The structure of the multi-head self-attention layer MSA is as follows Figure 3 As shown in the figure, it consists of m self-attention heads in parallel. The learned m pairs of Q, K, V matrices are sent to the corresponding self-attention heads for calculation, and the calculation results are spliced and projected.

[0060] The mathematical expression formula of a single-layer self-attention neural network is as follows:

[0061] I′=MLP(LN(I+MSA(LN(I))))+(I+MSA(LN(I)))

[0062] MSA(I)=concat(AH1;AH2;...;AH m )×W o

[0063]

[0064] Where AH is the self-attention head, Q = I × W Q , K=I×W K , V=I×W V They are the input token vectors I∈R N×M By the corresponding learnable matrix vector W∈R M×d MSA is a multi-head self-attention, which combines multiple self-attention heads in the feature dimension for feature learning. o ∈md×M is the normalized projection matrix, m is the number of self-attention heads, d is the feature dimension of the self-attention head, T is the matrix transpose, I′∈R N×M is the output of a single-layer self-attention neural network.

[0065] In the self-attention neural network formulation stage, this embodiment embeds token attention in the 3rd, 6th, 9th and 12th layers of the network respectively, such as Figure 4As shown in the figure, before the token vector is subjected to multi-head attention calculation, the token vector is subjected to maximum pooling and average pooling operations respectively, and then the corresponding pooled token vector is obtained by multi-layer perceptron. The processed token vector is then added and weighted using the sigmoid function to obtain TA(I), which is then multiplied with the input token vector I to improve the feature expression ability of the token vector, emphasize important features, and suppress unimportant features.

[0066] The token attention TA(I) formula is as follows:

[0067] TA(I)=sigmoid(MLP(Maxpool(I))+MLP(AvgPool(I)))

[0068] Among them, MLP is a multi-layer perceptron, MaxPool is the maximum pooling layer, AvgPool is the average pooling layer, sigmoid is the activation function, and finally the token vector weight value is obtained.

[0069] The feature fusion module in step 5 is as follows Figure 5 As shown in the figure, there are two residual convolution layers, an upsampling module, and a projection module. The residual convolution layer includes a ReLu activation function layer, a convolution module, a batch normalization module, a ReLu activation function layer, a convolution module, and a batch normalization module, in that order. The convolution kernels of both convolution modules are 3*3. One residual convolution layer receives shallow-layer features, calculates the residual features, adds them to the deep-layer features, and then feeds them into another residual convolution layer for upsampling and projection.

[0070] For the feature vectors of the self-attention network, projection, unflattening, and upsampling are used to restore the token vectors from two-dimensional features to three-dimensional image features, and then feature fusion is performed. The first feature fusion module receives the self-attention network features from the 9th and 12th layers for the first feature fusion output. The second feature fusion module receives the output features of the first feature fusion module and the low-level features of the convolutional preprocessing layer for the second feature fusion.

[0071] Depth information decoder such as Figure 6 As shown in the figure, it includes two ReLu activation functions, two convolution modules with a convolution kernel size of 3*3, an upsampling module, and a convolution module with a convolution kernel size of 1. It receives the feature vector from the last feature fusion module, integrates the features and projects them as depth information output.

[0072] The specific scale matching formula in step 6 is as follows:

[0073]

[0074] where d i Predict depth values for the network, is the true depth value, i is the i-th pixel, and V is the number of valid pixels. The network prediction value after alignment is The true value is

[0075] The scale-invariant loss function formula in step 7 is as follows:

[0076]

[0077] Where V is the number of effective pixels, W = 0.8V, excluding the maximum difference of 20%, is the depth prediction value after alignment, is the true depth value after alignment, and the subscript j represents the depth value of the j-th point.

[0078] The edge gradient invariant loss function formula is as follows:

[0079]

[0080] Where V is the number of effective pixels, Represents the depth difference of point i, R k Indicates different scales k, the scale size decreases by half, k=4 means there are 4 scale sizes, Calculate the horizontal and vertical gradients at each scale separately.

[0081] The final total training loss function is as follows:

[0082]

[0083] where N l Indicates the number of images processed in each batch, and α = 0.5 indicates the weight of the gradient loss function.

[0084] In step 8, an arbitrary scene picture is selected and input into the self-attention neural depth prediction network. After network coding, low-level feature extraction, high-level feature extraction, feature fusion, and depth decoding, a grayscale image (depth map) is output. Its size is 0-255. The grayscale value of 0 (darkest) indicates that the pixel of the scene is closest to the camera, and the grayscale value of 255 (brightest) indicates that the pixel of the scene is closest to the camera.

[0085] The experimental results are shown in Figure 7The present invention was subjectively compared with a currently mainstream depth prediction algorithm (Ranftl, René, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction, abbreviated as DPT) across six different test datasets. The results show that the present invention recovers more detailed depth information, such as the bookcase in the NYU dataset and the roof of the house in the DIODE dataset. Furthermore, the depth results are closer to the true depth values, such as the figurine and cart in the Sintel dataset. This demonstrates that the present invention can generate depth maps with fine details and accurate depth information, while being adaptable to different scenarios.

[0086] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other variation to the technical solution and technical contents disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.

Claims

1. A monocular image depth estimation algorithm based on a self-attention neural network, characterized by: The steps include: Step 1: Obtain a depth dataset of various scenarios involved in training; Step 2: Unify the domain of the real values in the depth dataset in step 1 and scale all the real values through regularization; Step 3: Send the images in the training dataset to the image preprocessing network to generate low-level image features; Step 4: Feed the low-level image features from step 3 into the self-attention neural network for high-level feature extraction; The self-attention neural network that receives the image features in step 3 for feature encoding in step 4 consists of 12 layers. Each layer includes a multi-head self-attention network, a multi-layer perceptron, and layer normalization. The layers are connected in series, and token attention is added in several layers. ; Token attention is embedded in the 3rd, 6th, 9th and 12th layers of the self-attention neural network. Before the token vector is subjected to multi-head attention calculation, the token vector is subjected to maximum pooling and average pooling operations respectively. Then, the corresponding pooled token vector is obtained through a multi-layer perceptron. The processed token vectors are then added and weighted using the sigmoid function to obtain TA(I), which is then multiplied with the input token vector I. Token attention The expression is as follows: ； Where MLP is a multi-layer perceptron, MaxPool is the maximum pooling layer, AvgPool is the average pooling layer, and sigmoid is the activation function; Step 5: The image features extracted in steps 3 and 4 are sent to the feature fusion module for feature fusion, and then sent to the depth information decoder to obtain the network predicted depth value; Step 6: Use the least squares algorithm to perform scale matching on the depth value predicted in step 5 and the real value preprocessed in step 2 to obtain the scale factor and translation factor; Step 7: Use the scale factor and translation factor obtained in step 6 to scale the network prediction value, calculate the loss through the scale-invariant loss function and the edge gradient invariant loss function, perform network backpropagation, and obtain a trained network with the optimized loss as the global minimum; Step 8: Input a single arbitrary image into the network trained in step 7 to generate the corresponding depth information map.

2. The monocular image depth estimation algorithm based on self-attention neural network according to claim 1, characterized in that: The image preprocessing network in step 3 includes a convolutional network and a block coding network; Convolutional networks transform images Perform low-level feature encoding to obtain feature vectors , where H, W are the original height and width of the input image, h, w, c are the height, width and number of channels after encoding respectively; The block coding network transforms the feature vector Perform block position encoding and reduce the three-dimensional vector to a token vector , while embedding positional codes of the same size, where is the final number of token vectors, is the number of token vector feature dimensions.

3. The monocular image depth estimation algorithm based on self-attention neural network according to claim 1, characterized in that: In step 5, the feature fusion module includes two residual convolution layers, an upsampling module and a linear projection in sequence. One residual convolution layer receives shallow features, calculates residual features and adds them to deep features, and then sends them to another residual convolution layer for upsampling and projection. The residual convolution layer includes a ReLu activation function layer, a convolution module, a batch normalization module, a ReLu activation function layer, a convolution module and a batch normalization module in sequence. The convolution kernels of the two convolution modules are both 3*3.

4. The monocular image depth estimation algorithm based on self-attention neural network according to claim 3, characterized in that: In step 5, the depth information decoder includes a convolution module with a convolution kernel of 3*3 size, an upsampling module, a convolution module with a convolution kernel of 3*3 size, a ReLu activation function, a convolution module with a convolution kernel size of 1, and a ReLu activation function.

5. The monocular image depth estimation algorithm based on self-attention neural network according to claim 1, characterized in that: The expression for scale matching in step 6 is: ； in, Predict depth values for the network, is the true depth value, For the pixels, V is the number of valid pixels, and the network prediction value after alignment is , the true value is .

6. The monocular image depth estimation algorithm based on self-attention neural network according to claim 5, characterized in that: The expression of the scale-invariant loss function in step 7 is: ； in, , that is, excluding the 20% largest differences; The expression of the edge gradient invariant loss function is: ； Among them, K represents different scales; Representing different scales , the scale size decreases by half; Calculate the horizontal and vertical gradients of each scale respectively; The total training loss function is as follows: ； in, Represents the number of images processed in each batch, and α represents the weight of the gradient loss function.

Images