A self-supervised monocular depth estimation method based on convolutional neural network

Abstract

The present invention relates to a self-supervised monocular depth estimation method based on a convolutional neural network, comprising: using a monocular camera to shoot continuous frame images as a data set; constructing a depth estimation network, outputting a depth estimation image; building a camera pose network, the camera pose network outputting a pose change matrix; calculating the depth estimation image and the pose change matrix, obtaining a reconstructed view input diffusion model, and outputting a new reconstructed view; comparing the new reconstructed view with the target view, designing a loss function, and using back propagation to update the parameters of the depth estimation network and the diffusion model. The present invention improves the depth estimation accuracy, and by introducing a dilated convolution in the depth estimation network, enhances the ability to extract local features, increases the receptive field, and more accurately captures the detail information in the image, uses a diffusion model to process the reconstructed view, gradually removes noise, optimizes the quality of the reconstructed view, and effectively improves the detail retention ability of the depth map in complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of power system optimization dispatching and integrated operation, and in particular to a self-supervised monocular depth estimation method based on a convolutional neural network. Background Art

[0002] Depth prediction is one of the important research directions in the field of computer vision. It enables computers to estimate the depth information of a scene through a two-dimensional image. At present, image-based depth estimation methods can be divided into monocular depth estimation and multi-camera depth estimation according to the number of required input images.

[0003] Multi-camera depth estimation methods estimate depth based on multiple images observed. Ideally, the depth information of the scene is directly calculated based on the principle of binocular ranging. However, estimation methods based on binocular images are often limited by the baseline length, resulting in the inability to match the device volume with the vehicle platform, which restricts its application in some situations.

[0004] In contrast, the monocular depth estimation method uses a single image for depth estimation, which is more flexible. The easiest way to obtain it is to use a monocular image taken by a monocular camera. However, due to the lack of reliable depth clues such as motion and stereo vision, and the fact that the depth value of any pixel in the monocular view can theoretically have countless solutions, it is an ill-posed problem, so this task is very challenging.

[0005] With the rise and development of deep learning, it has achieved outstanding results in many visual fields. Eigen et al. first applied convolutional neural networks to the problem of monocular depth estimation, and obtained a relatively accurate depth map based on a single image during testing. In subsequent studies, combined methods such as multi-scale feature fusion, conditional random field (CRF), ordinal relationship-based, and multivariate image information-based methods have all improved monocular depth estimation.

[0006] For self-supervised monocular depth estimation, there is no need for the hard-to-obtain real depth value data. Although the form is simple, the disadvantage is that it cannot guarantee the integrity of the fine structure of the depth map, especially in occluded and low-texture areas, resulting in poor quality depth estimation maps. In addition, the current monocular depth estimation methods based on deep learning usually cannot obtain the correlation between long-distance features, and thus cannot obtain better feature expression, resulting in problems such as loss of details in the estimated depth map. Summary of the invention

[0007] In order to solve the problems that accurate depth estimation maps cannot be generated in occluded and low-texture areas and the network cannot obtain long-distance features, the purpose of the present invention is to provide a self-supervised monocular depth estimation method based on convolutional neural networks that captures global information, optimizes the problem of depth map detail loss, and improves prediction accuracy.

[0008] To achieve the above object, the present invention adopts the following technical solution: a self-supervised monocular depth estimation method based on convolutional neural network, the method comprising the following steps in order:

[0009] (1) Obtaining a data set: Using a monocular camera to capture continuous frame images as a data set;

[0010] (2) Build a depth estimation network, select an image from the dataset as the target view and input it into the depth estimation network, which then outputs a depth estimation image.

[0011] (3) Build a camera pose network, select the previous frame image and the next frame image of the target view from the data set and input them into the camera pose network, which outputs the pose change matrix;

[0012] (4) Use the depth estimation network and camera pose network to calculate the depth estimation image and pose change matrix to obtain the reconstructed view

[0013] (5) Reconstruct the view The diffusion model is input and the diffusion model outputs a new reconstructed view;

[0014] (6) Compare the new reconstructed view and the target view, design a loss function, and use back propagation to update the parameters of the depth estimation network and the diffusion model to obtain the depth estimation network and the diffusion model after parameter update;

[0015] (7) The image to be estimated is input into the depth estimation network after parameter update, and the depth estimation network after parameter update outputs a depth estimation image.

[0016] The step (2) specifically refers to: the depth estimation network adopts a jump-connected encoder-decoder structure, the encoder is composed of a ResNet101 network, a dilated convolution is introduced between the third residual block and the fourth residual block of the ResNet101 network to enhance the extraction of local features and increase the receptive field, and an expansion rate parameter is introduced into the convolution layer of the dilated convolution, and the expansion rate parameter defines the spacing between the values ​​when the convolution kernel processes data; the decoder includes a convolution layer, an upsampling layer and four identical upsampling modules;

[0017] A self-attention module is added between the encoder and the decoder to explore non-continuous image areas. The image passes through the encoder to obtain a low-resolution feature map X, which is input into the self-attention module. First, the query vector Q, key vector K and value vector V are formed. The formula is as follows;

[0018] Q=W Q X

[0019] K=WK X

[0020] V=W V X

[0021] Where W Q , W K , W V are all weight matrices;

[0022] Calculate Q and K, and then normalize them through the softmax function to get the attention weight S:

[0023] S = softmax(Q T K)

[0024] Finally, we add the weighted sum with V to get the attention output Attention(Q,K,V):

[0025] Attention(Q,K,V)=softmax(Q T K)·V.

[0026] The step (3) specifically refers to: the camera posture network includes an encoder and a decoder, the encoder adopts a ResNet network for capturing high-level image features; the decoder passes the image features output from the encoder through a compression layer, compresses each feature into 256 channels, and concatenates the compressed features in the channel dimension, and then passes through three convolutional layers. Except for the last convolutional layer, the other two convolutional layers apply a ReLU activation function to introduce nonlinearity, and then obtain a feature vector through a global average pooling operation, and finally adjust the output shape to obtain a posture change matrix.

[0027] The step (4) specifically refers to: using the depth estimation network and the camera pose network to calculate the depth estimation image and the pose change matrix, and obtain the depth value and the pose change matrix of the target view converted to the reconstructed view:

[0028]

[0029] Among them, ps is the pixel point on the reconstructed view, pt is the pixel point on the target view, K is the camera intrinsic parameter matrix, is the depth value, The pose change matrix from the target view to the reconstructed view;

[0030] According to the depth value, the target view is converted to the pose change matrix on the reconstructed view, and bilinear interpolation is used to obtain the reconstructed view.

[0031]

[0032] Among them, ps ijare the four adjacent pixels of ps, t, b, l, r represent the top, bottom, left and right of ps respectively, w ij is the weight.

[0033] The step (5) specifically refers to: inputting the reconstructed view into the diffusion model, the diffusion model adopts the DDPM model, the input of the diffusion model is the reconstructed view, the output of the diffusion model is the new reconstructed view, and the DDPM model is divided into two processing stages: a forward process and a reverse process;

[0034] The forward process is to transform the input reconstructed view Gradually add Gaussian noise and finally transform it into pure noise X T , T is the number of iterations, the reverse process is from pure noise X T Stepwise denoising, the output is the new reconstructed view I d , through the inverse process, it is possible to predict and remove the noise in each iteration, thus realizing the process of generating an image from noise;

[0035] The forward process is used to generate training data by gradually adding Gaussian noise to the reconstructed view, converting it into a series of noisy images, and calculating and saving the intermediate results. The formula is as follows:

[0036]

[0037] In the formula, q(X t |X t-1 ) is the conditional probability distribution of the noisy image from the iteration t-1 to the iteration t; X t is the noised image of the tth iteration; X t-1 is the image of the t-1th iteration; N is the Gaussian distribution; β t The variance parameter of the noise added at the tth iteration; is the mean; β t I is the covariance matrix; I is the identity matrix;

[0038] By recursively obtaining the above formula, we can get the reconstructed view To the t-th iteration of the noisy image X t :

[0039]

[0040] α t =1-β t

[0041]

[0042] In the formula, It is the cumulative product of all noise factors from the first iteration to the tth iteration, ∈ represents a noise vector sampled from a standard normal distribution;

[0043] The inverse process is used to predict the noise added in each iteration and remove it, the formula is:

[0044] p θ (X t-1 |X t )=N(X t-1 ;μ θ (X t ,t),∑ θ (X t ,t))

[0045] In the formula, p θ (X t-1 |X t ) is the probability distribution from the noisy image of iteration t to the noisy image of iteration t-1; μ θ (X t ,t) is the mean;∑ θ (X t ,t) is the covariance matrix.

[0046] In step (6), the loss function of the design is:

[0047] L=γLp+βLs+δL d ;

[0048] Where γ, β, δ are parameters, Lp is the pixel-by-pixel luminosity loss, Ls is the smoothness loss, and L d is the denoising loss;

[0049] The formula for pixel-by-pixel luminosity loss Lp is:

[0050]

[0051] In the formula, I t is the target view, is the reconstructed view; SSIM is the structural similarity, which is an indicator used to measure the similarity between two images; is the L1 norm, which is used to measure the absolute difference between two images at each pixel position; the parameter α is used to balance the contribution of SSIM and L1 norm in the loss and is set to 0.85;

[0052] Optimize Lp loss and use the minimum reprojection method:

[0053]

[0054] In the formula, I s is the adjacent frame of the target view;

[0055] The formula for the smoothing loss Ls is:

[0056]

[0057] In the formula, Depth map Gradient in the x and y axis directions; δ x I t , δ y I t Represents image I t Gradient in the x and y axis directions; Represents the weight term to ensure that the depth map is smoother;

[0058] Denoising loss L d The formula is as follows:

[0059] L d =||I t -I d || 2

[0060] In the formula, I d Rebuild the view for the new one.

[0061] It can be seen from the above technical scheme that the beneficial effects of the present invention are as follows: first, the present invention improves the accuracy of depth estimation. By introducing a dilated convolution in the depth estimation network, the ability to extract local features is enhanced, the receptive field is increased, and the detail information in the image is captured more accurately. The introduction of the self-attention module can explore non-continuous areas in the image and capture the relationship between distant pixels, which not only enhances the global information capture ability of the image, but also ensures that more detail information in the depth estimation is retained; second, the present invention optimizes the detail retention of the depth map and improves the quality of the reconstructed image. The diffusion model is used to process the reconstructed view, and the noise is gradually removed through the forward and reverse processes, which further optimizes the quality of the reconstructed view. The application of the diffusion model effectively improves the detail retention ability of the depth map in complex scenes and reduces the problem of detail loss caused by noise; third, the present invention combines the optimization effect of the loss function and designs a comprehensive loss function including pixel-by-pixel photometric loss, smoothing term loss and denoising loss. By reasonably adjusting the contribution ratio of these losses to the total loss, the network training is made more stable and efficient, and the accuracy and robustness of the depth estimation are finally improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0063] like Figure 1As shown, a self-supervised monocular depth estimation method based on convolutional neural network includes the following steps in order:

[0064] (1) Obtaining a data set: Using a monocular camera to capture continuous frame images as a data set;

[0065] (2) Build a depth estimation network, select an image from the dataset as the target view and input it into the depth estimation network, which then outputs a depth estimation image.

[0066] (3) Build a camera pose network, select the previous frame image and the next frame image of the target view from the data set and input them into the camera pose network, which outputs the pose change matrix;

[0067] (4) Use the depth estimation network and camera pose network to calculate the depth estimation image and pose change matrix to obtain the reconstructed view

[0068] (5) Reconstruct the view The diffusion model is input and the diffusion model outputs a new reconstructed view;

[0069] (6) Compare the new reconstructed view and the target view, design a loss function, and use back propagation to update the parameters of the depth estimation network and the diffusion model to obtain the depth estimation network and the diffusion model after parameter update;

[0070] (7) The image to be estimated is input into the depth estimation network after parameter update, and the depth estimation network after parameter update outputs a depth estimation image.

[0071] The step (2) specifically refers to: the depth estimation network adopts a jump-connected encoder-decoder structure, the encoder is composed of a ResNet101 network, a dilated convolution is introduced between the third residual block and the fourth residual block of the ResNet101 network to enhance the extraction of local features and increase the receptive field, and an expansion rate parameter is introduced into the convolution layer of the dilated convolution, and the expansion rate parameter defines the spacing between the values ​​when the convolution kernel processes data; the decoder includes a convolution layer, an upsampling layer and four identical upsampling modules;

[0072] A self-attention module is added between the encoder and the decoder to explore non-continuous image areas. The image passes through the encoder to obtain a low-resolution feature map X, which is input into the self-attention module. First, the query vector Q, key vector K and value vector V are formed. The formula is as follows;

[0073] Q=W Q X

[0074] K=W K X

[0075] V=WV X

[0076] Where W Q , W K , W V are all weight matrices;

[0077] Calculate Q and K, and then normalize them through the softmax function to get the attention weight S:

[0078] S = softmax(Q T K)

[0079] Finally, we add the weighted sum with V to get the attention output Attention(Q,K,V):

[0080] Attention(Q,K,V)=softmax(Q T K)·V.

[0081] The step (3) specifically refers to: the camera posture network includes an encoder and a decoder, the encoder adopts a ResNet network for capturing high-level image features; the decoder passes the image features output from the encoder through a compression layer, compresses each feature into 256 channels, and concatenates the compressed features in the channel dimension, and then passes through three convolutional layers. Except for the last convolutional layer, the other two convolutional layers apply a ReLU activation function to introduce nonlinearity, and then obtain a feature vector through a global average pooling operation, and finally adjust the output shape to obtain a posture change matrix.

[0082] The step (4) specifically refers to: using the depth estimation network and the camera pose network to calculate the depth estimation image and the pose change matrix, and obtain the depth value and the pose change matrix of the target view converted to the reconstructed view:

[0083]

[0084] Among them, ps is the pixel point on the reconstructed view, pt is the pixel point on the target view, K is the camera intrinsic parameter matrix, is the depth value, The pose change matrix from the target view to the reconstructed view;

[0085] According to the depth value, the target view is converted to the pose change matrix on the reconstructed view, and bilinear interpolation is used to obtain the reconstructed view.

[0086]

[0087] Among them, ps ij are the four adjacent pixels of ps, t, b, l, r represent the top, bottom, left and right of ps respectively, wij is the weight.

[0088] The step (5) specifically refers to: inputting the reconstructed view into the diffusion model, the diffusion model adopts the DDPM model, the diffusion model can remove noise from the input reconstructed view, and output a clearer view; the input of the diffusion model is the reconstructed view, the output of the diffusion model is a new reconstructed view, and the DDPM model is divided into two processing stages: a forward process and a reverse process;

[0089] The forward process is to transform the input reconstructed view Gradually add Gaussian noise and finally transform it into pure noise X T , T is the number of iterations, the reverse process is from pure noise X T Stepwise denoising, the output is the new reconstructed view I d , through the inverse process, it is possible to predict and remove the noise in each iteration, thus realizing the process of generating an image from noise;

[0090] The forward process is used to generate training data by gradually adding Gaussian noise to the reconstructed view, converting it into a series of noisy images, and calculating and saving the intermediate results. The formula is as follows:

[0091]

[0092] In the formula, q(X t |X t-1 ) is the conditional probability distribution of the noisy image from the iteration t-1 to the iteration t; X t is the noised image of the tth iteration; X t-1 is the image of the t-1th iteration; N is the Gaussian distribution; β t The variance parameter of the noise added at the tth iteration; is the mean; β t I is the covariance matrix; I is the identity matrix;

[0093] By recursively obtaining the above formula, we can get the reconstructed view To the t-th iteration of the noisy image X t :

[0094]

[0095] α t =1-β t

[0096]

[0097] In the formula, It is the cumulative product of all noise factors from the first iteration to the tth iteration, ∈ represents a noise vector sampled from a standard normal distribution;

[0098] The inverse process is used to predict the noise added in each iteration and remove it, the formula is:

[0099] p θ (X t-1 |X t )=N(X t-1 ;μ θ (Xt,t),∑ θ (X t ,t))

[0100] In the formula, p θ (X t-1 |X t ) is the probability distribution from the noisy image of iteration t to the noisy image of iteration t-1; μ θ (X t ,t) is the mean;∑ θ (X t ,t) is the covariance matrix.

[0101] In step (6), the loss function of the design is:

[0102] L=γLp+βLs+δL d ;

[0103] Where γ, β, δ are parameters, Lp is the pixel-by-pixel luminosity loss, Ls is the smoothness loss, and L d is the denoising loss;

[0104] The formula for pixel-by-pixel luminosity loss Lp is:

[0105]

[0106] In the formula, I t is the target view, is the reconstructed view; SSIM is the structural similarity, which is an indicator used to measure the similarity between two images; is the L1 norm, which is used to measure the absolute difference between two images at each pixel position; the parameter α is used to balance the contribution of SSIM and L1 norm in the loss and is set to 0.85;

[0107] Optimize Lp loss and use the minimum reprojection method:

[0108]

[0109] In the formula, I s is the adjacent frame of the target view;

[0110] The formula for the smoothing loss Ls is:

[0111]

[0112] In the formula, Depth map Gradient in the x and y axis directions; δ x I t , δ y I t Represents image I t Gradient in the x and y axis directions; Represents the weight term to ensure that the depth map is smoother;

[0113] Denoising loss L d The formula is as follows:

[0114] L d =||I t -I d || 2

[0115] In the formula, I d Rebuild the view for the new one.

[0116] In summary, the present invention improves the accuracy of depth estimation. By introducing hollow convolution in the depth estimation network, the ability to extract local features is enhanced, the receptive field is increased, and the detail information in the image is captured more accurately. The introduction of the self-attention module can explore non-continuous areas in the image and capture the relationship between distant pixels, which not only enhances the global information capture ability of the image, but also ensures that more detail information in the depth estimation is retained; the present invention optimizes the detail retention of the depth map and improves the quality of the reconstructed image. The diffusion model is used to process the reconstructed view, and the noise is gradually removed through the forward and reverse processes, which further optimizes the quality of the reconstructed view. The application of the diffusion model effectively improves the detail retention ability of the depth map in complex scenes and reduces the problem of detail loss caused by noise; the present invention integrates the optimization effect of the loss function and designs a comprehensive loss function including pixel-by-pixel luminosity loss, smoothing term loss and denoising loss. By reasonably adjusting the contribution ratio of these losses to the total loss, the network training is made more stable and efficient, and ultimately the accuracy and robustness of the depth estimation are improved.

Claims

1. A self-supervised monocular depth estimation method based on convolutional neural network, characterized by: The method comprises the following steps in the following order: (1) Obtaining a data set: Using a monocular camera to capture continuous frame images as a data set; (2) Build a depth estimation network, select an image from the dataset as the target view and input it into the depth estimation network, which then outputs a depth estimation image. (3) Build a camera pose network, select the previous frame image and the next frame image of the target view from the data set and input them into the camera pose network, which outputs the pose change matrix; (4) Use the depth estimation network and camera pose network to calculate the depth estimation image and pose change matrix to obtain the reconstructed view (5) Reconstruct the view Input a diffusion model, and the diffusion model outputs a new reconstructed view; input the reconstructed view into the diffusion model, the diffusion model adopts the DDPM model, the input of the diffusion model is the reconstructed view, and the output of the diffusion model is the new reconstructed view; (6) Compare the new reconstructed view and the target view, design a loss function, and use back propagation to update the parameters of the depth estimation network and the diffusion model to obtain the depth estimation network and the diffusion model after parameter update; (7) inputting the image to be estimated into the depth estimation network after parameter update, and the depth estimation network after parameter update outputs the depth estimation image; In step (6), the loss function of the design is: L=γLp+βLs+δL d ; In the formula, γ, β, δ are parameters, Lp is the pixel-by-pixel luminosity loss, Ls is the smoothness loss, L d is the denoising loss; the formula for pixel-by-pixel luminosity loss Lp is: In the formula, I t is the target view, To reconstruct the view; SSIM is structural similarity, an indicator used to measure the similarity between two images; is the L1 norm, which is used to measure the absolute difference between two images at each pixel position; the parameter α is used to balance the contribution of SSIM and L1 norm in the loss and is set to 0.85; Optimize Lp loss and use the minimum reprojection method: In the formula, I s is the adjacent frame of the target view; The formula for the smoothing loss Ls is: In the formula, Depth map Gradient in the x and y axis directions; δ x I t , δ y I t Represents image I t Gradient in the x and y axis directions; Represents the weight term to ensure that the depth map is smoother; Denoising loss L d The formula is as follows: L d =||I t -I d || 2 In the formula, I d Rebuild the view for the new one.

2. The self-supervised monocular depth estimation method based on convolutional neural network according to claim 1, characterized in that: The step (2) specifically refers to: the depth estimation network adopts a jump-connected encoder-decoder structure, the encoder is composed of a ResNet101 network, a dilated convolution is introduced between the third residual block and the fourth residual block of the ResNet101 network to enhance the extraction of local features and increase the receptive field, and an expansion rate parameter is introduced into the convolution layer of the dilated convolution, and the expansion rate parameter defines the spacing between the values ​​when the convolution kernel processes data; the decoder includes a convolution layer, an upsampling layer and four identical upsampling modules; A self-attention module is added between the encoder and the decoder to explore non-continuous image areas. The image passes through the encoder to obtain a low-resolution feature map X, which is input into the self-attention module. First, the query vector Q, key vector K and value vector V are formed. The formula is as follows; Q=W Q X K=W K X V=W V X Where W Q , W K , W V are all weight matrices; Calculate Q and K, and then normalize them through the softmax function to get the attention weight S: S=softmax(Q T K) Finally, we add the weighted sum with V to get the attention output Attention(Q,K,V): Attention(Q,K,V)=softmax(Q T K)·V。 3. The self-supervised monocular depth estimation method based on convolutional neural network according to claim 1, characterized in that: The step (3) specifically refers to: the camera posture network includes an encoder and a decoder, the encoder adopts a ResNet network for capturing high-level image features; the decoder passes the image features output from the encoder through a compression layer, compresses each feature into 256 channels, and concatenates the compressed features in the channel dimension, and then passes through three convolutional layers. Except for the last convolutional layer, the other two convolutional layers apply a ReLU activation function to introduce nonlinearity, and then obtain a feature vector through a global average pooling operation, and finally adjust the output shape to obtain a posture change matrix.

4. The self-supervised monocular depth estimation method based on convolutional neural network according to claim 1, characterized in that: The step (4) specifically refers to: using the depth estimation network and the camera pose network to calculate the depth estimation image and the pose change matrix, and obtain the depth value and the pose change matrix of the target view converted to the reconstructed view: Among them, ps is the pixel point on the reconstructed view, pt is the pixel point on the target view, K is the camera intrinsic parameter matrix, is the depth value, The pose change matrix from the target view to the reconstructed view; According to the depth value, the target view is converted to the pose change matrix on the reconstructed view, and bilinear interpolation is used to obtain the reconstructed view. Among them, ps ij are the four adjacent pixels of ps, t, b, l, r represent the top, bottom, left and right of ps respectively, w ij is the weight.

5. The self-supervised monocular depth estimation method based on convolutional neural network according to claim 1, characterized in that: The step (5) specifically refers to: the DDPM model is divided into two processing stages: a forward process and a reverse process; The forward process is to transform the input reconstructed view Gradually add Gaussian noise and finally transform it into pure noise X T , T is the number of iterations, the reverse process is from pure noise X T Stepwise denoising, the output is the new reconstructed view I d , through the inverse process, it is possible to predict and remove the noise in each iteration, thus realizing the process of generating an image from noise; The forward process is used to generate training data by gradually adding Gaussian noise to the reconstructed view, converting it into a series of noisy images, and calculating and saving the intermediate results. The formula is as follows: In the formula, q(X t |X t-1 ) is the conditional probability distribution of the noisy image from the iteration t-1 to the iteration t; X t is the noised image of the tth iteration; X t-1 is the image of the t-1th iteration; N is the Gaussian distribution; β t The variance parameter of the noise added at the tth iteration; is the mean; β t I is the covariance matrix; I is the identity matrix; By recursively obtaining the above formula, we can get the reconstructed view To the t-th iteration of the noisy image X t : α t ＝1-β t In the formula, It is the cumulative product of all noise factors from the first iteration to the tth iteration, ∈ represents a noise vector sampled from a standard normal distribution; The inverse process is used to predict the noise added in each iteration and remove it, the formula is: p θ (X t-1 |X t )=N(X t-1 ;μ θ (X t ,t),∑ θ (X t ,t)) In the formula, p θ (X t-1 |X t ) is the probability distribution from the noisy image of iteration t to the noisy image of iteration t-1; μ θ (X t ,t) is the mean;∑ θ (X t ,t) is the covariance matrix.

Images