Image depth estimation method based on label reconstruction-coding and decoding neural network

Abstract

An image depth estimation method based on a label reconstruction-coding and decoding neural network comprises the steps that a label reconstruction-coding and decoding neural network model is firstly built, the model comprises a label reconstruction module and a coding and decoding module, and each module comprises a convolution layer, a corresponding sampling layer and an activation layer; then selecting a training set and a test set, and setting training parameters of the label reconstruction-coding and decoding neural network model; training the label reconstruction-coding and decoding neural network model by taking minimization of a loss function as a target according to the label reconstruction-coding and decoding neural network model and training parameters thereof; and finally, inputting an image to be processed into the image depth estimation model based on the label reconstruction-coding and decoding neural network, and outputting a corresponding depth map. The depth values of the obtained depth image in global and local ranges are relatively accurate, and the method has the advantages of high precision, fast calculation, low resource consumption and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and specifically relates to an image depth estimation method based on a label reconstruction-encoding-decoding neural network. Background Art

[0002] The depth image estimation problem refers to calculating the straight-line distance between the distance of all objects in the scene and the shooting point by computational means. Traditional methods for solving the depth image estimation problem have multiple input methods, including using continuous color images from multiple perspectives, binocular color images and monocular color images, as well as color images and other input methods such as radar and heat maps as joint input methods.

[0003] The methods for estimating the depth information of monocular color images based on supervised deep neural networks are mainly divided into two-stage neural network estimation methods and end-to-end neural network estimation methods based on encoding and decoding; the end-to-end method uses a deep neural network to regress the depth information of each pixel from the color image. Although this method has a simple process, it has a low accuracy rate due to the large resolution of the color image and the irregular distribution of the depth information corresponding to the image pixels. The two-stage method uses a global estimation followed by a fine estimation method to achieve depth information estimation.

[0004] Existing neural network-based methods (J.Hu, M.Ozay, Y.Zhang, and T.Okatani, "Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries," in 2019 IEEE winter conference on applications of computer vision (WACV), 2019: IEEE, pp.1043-1051.) directly regress the depth information corresponding to the pixel points. Due to the different image depth ranges of different data sets, and the continuity of depth information and the irregularity of depth image distribution, the accuracy of directly estimating depth information through color images is low. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an image depth estimation method based on label reconstruction-encoding and decoding neural network, which has the advantages of high precision, fast calculation and low resource consumption, and the depth values ​​of the obtained depth image in the global and local ranges are more accurate.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for image depth estimation based on label reconstruction-encoding and decoding neural network, comprising the following steps:

[0008] Step 1: Build a label reconstruction-encoding and decoding neural network model. The label reconstruction-encoding and decoding neural network model includes multiple different convolutional layers, upsampling layers, downsampling layers, and activation layers.

[0009] Step 2: Select the training set and test set, and set the training parameters of the label reconstruction-encoding-decoding neural network model;

[0010] Step 3: According to the label reconstruction-encoding-decoding neural network model and its training parameters, the prescribed loss function is minimized, thereby completing the training of the label reconstruction-encoding-decoding neural network model;

[0011] Step 4: Input the image to be processed into the trained label reconstruction-encoding-decoding neural network model to obtain the corresponding depth image.

[0012] The label reconstruction-encoding and decoding neural network model built in step 1 includes a label reconstruction module and an encoding and decoding module. The label reconstruction module has four convolutional layers. The number of repetitions of each convolutional layer is 3, 4, 6, and 3 respectively. The convolution kernel sizes are 1×1, 3×3, and 1×1 respectively. The number of channels of each convolutional layer is 64, 64, 256, 128, 128, 512, 256, 256, 1024, 512, 512, and 2048 respectively.

[0013] In the encoding and decoding module of the label reconstruction-encoding and decoding neural network model built in step 1, the encoding structure part includes 4 convolutional layers with a convolution kernel of 3×3. Each convolutional layer is followed by a corresponding downsampling layer. The number of channels of the 4 convolutional layers are 256, 512, 1024 and 2048 respectively.

[0014] In the encoding and decoding module of the label reconstruction-encoding and decoding neural network model built in step 1, the decoding structure includes 4 convolutional layers with a convolution kernel of 3×3. Each convolutional layer is followed by a corresponding upsampling layer. The number of channels of the 4 convolutional layers are 2048, 1024, 512 and 256 respectively.

[0015] In the encoding and decoding module of the label reconstruction-encoding and decoding neural network model built in step 1, the input of the decoding structure part comes from two parts, including the output of the previous layer encoding structure module and the output of the corresponding encoding structure part.

[0016] In the label reconstruction-encoding-decoding neural network model built in step 1, a corresponding ReLu activation layer is added after each convolutional layer.

[0017] Step 1: During the training of the label reconstruction-encoding-decoding neural network model, the weight initialization of the label reconstruction-encoding-decoding neural network model is performed by He Kaiming's initialization method.

[0018] In step 2, the training set and test set adopt the training / test split in the NYU DepthV2 dataset and the KITTI dataset. The network is trained in the training sets of the two different datasets, and the network is tested in the test sets of the two different datasets.

[0019] The loss function in step 3 is a combined loss function: loss total =γL pixel +η L gradx +η L grady ,in, Represents the error between the depth logarithm of the real depth map and the predicted depth map at pixel i, Represents the gradient error in the x direction between the depth value of the real depth map and the predicted depth map at pixel i, Represents the gradient error in the y direction between the depth value of the real depth map and the predicted depth map at pixel i, It represents the logarithmic difference between the predicted depth map and the real depth map at pixel i and the real depth map at pixel i. It represents the gradient value in the x direction of the logarithmic difference between the depth value of the predicted depth map at pixel i and the depth value of the true depth map at pixel i. represents the gradient value in the y direction of the logarithmic difference between the depth value of the predicted depth map at pixel i and the true depth map at pixel i, p i Represents the predicted depth value of the predicted depth map at pixel i, represents the actual depth value of the real depth map at pixel i, n represents the number of all valid pixels in the depth image, and γ, η and λ are weight coefficients.

[0020] In step 3, the depth estimation of the graph is established based on the network weights obtained by minimizing the loss function, and the Adam optimization method is used to minimize the loss function.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] (1) The depth estimation method of the present invention is based on a deep neural network. By introducing a label reconstruction-encoding and decoding network structure, the learning and generalization capabilities of the network are greatly enhanced, and a strong mapping relationship from color image to depth image can be established in different usage environments;

[0023] (2) The label reconstruction-encoding and decoding neural network established by the present invention can perform accurate depth estimation on different data sets, and can effectively improve the estimation accuracy and efficiency in actual use scenarios;

[0024] (3) The label reconstruction-encoding and decoding neural network constructed by the present invention adopts the idea of ​​classification regression to transform the depth estimation problem into an image multi-classification problem, thereby avoiding the problem of inaccurate estimation results and failure of model convergence caused by direct depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of an image depth estimation method based on a label reconstruction-encoding and decoding neural network according to an embodiment of the present invention.

[0026] Figure 2 Schematic diagram of the internal structure of the encoding and decoding network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention is described in detail below in conjunction with embodiments and drawings.

[0028] This embodiment provides an image depth estimation method based on a label reconstruction-encoding and decoding neural network. The neural network establishes a label reconstruction module and an encoding and decoding module, and obtains image features with the help of the learning ability of the convolutional layer and the expression ability of the activation layer, which greatly enhances the learning ability and generalization ability of the network, and accurately learns the mapping relationship from color images to depth images, so that the depth of the color image can be estimated through the learned mapping relationship; the flowchart is as follows Figure 1 As shown, the following steps are included:

[0029] Step 1: Building a label reconstruction-encoding and decoding neural network model, wherein the label reconstruction-encoding and decoding neural network model includes a plurality of different convolutional layers, upsampling layers, downsampling layers and activation layers;

[0030] This embodiment adopts a self-supervised learning method. In order to accurately estimate the image depth value, this method includes two main parts: a label reconstruction module and a coding and decoding module; the label reconstruction module of this embodiment has four convolutional layers, the number of repetitions of each convolutional layer is 3, 4, 6, and 3 respectively, the convolution kernel sizes are 1×1, 3×3, and 1×1 respectively, and the number of channels of each convolutional layer is 64, 64, 256, 128, 128, 512, 256, 256, 1024, 512, 512, and 2048 respectively; the activation layer after each convolutional layer uses the rectified linear unit ReLu function: Where α = 10, β = 0.1; the above label reconstruction means: re-establishing the depth label corresponding to the original color image using the semantic segmentation method, reconstructing the depth label into a depth classification label, thereby converting the depth estimation into image multi-classification, and using the image multi-classification result as prior knowledge for depth estimation;

[0031] Reference Figure 2 In the encoding and decoding module, both the encoding structure and the decoding structure include 4 convolutional layers with 3×3 convolution kernels; the convolutional layer of the encoding structure is followed by a corresponding downsampling layer, and the convolutional layer of the decoding structure is followed by a corresponding upsampling layer; the input of the decoding structure comes from two parts, including the output of the previous encoding structure module and the output of the corresponding encoding structure part; the above correspondence means that the size and number of channels of the feature map from the output of the encoding structure to the input of the decoding structure are corresponding, that is, the same height, width and depth; for example, the output feature map size of the 4th convolutional layer of the encoding structure is H e4 *W e4 *C e4 , the output feature map size of the first convolutional layer of the decoding structure is H d1 *W d1 *C d1 , then, H e4 =H d1 ,W e4 =W d1 And C e4 =C d1 At the same time, the output feature map size of the second convolutional layer of the decoding structure is H d1 *W d1 *(C e4 +C d1 ), this operation effectively increases the utilization rate of feature maps and improves the accuracy of the neural network model in estimating image depth; among them, the decoding structure is particularly important, which combines the output of the corresponding encoding structure with the output of the previous convolutional layer as input, thus retaining the global features and calculating the local features;

[0032] Step 2: Select the training set and test set, and set the training parameters of the label reconstruction-encoding-decoding neural network model;

[0033] The training set images and test set images used in this embodiment include color images and corresponding depth images. The color images have three RGB channels, and the depth images have only one channel, that is, the color images containing depth labels.

[0034] In this embodiment, the training / test splits in the NYU DepthV2 dataset and the KITTI dataset widely used in the field of image recognition are selected as the training set and the test set; before adopting the method of estimating image depth based on label reconstruction-encoding and decoding neural network, the neural network needs to be pre-trained, and then the training parameters of the neural network model are set; the learning rate is set to 0.001, the decay rate of the learning rate is set to 0.1, and the strategy of decaying once every 10 training times is adopted; the initial value of the weight parameter of the neural network model is initialized by He Kaiming's initial method, the batch size is 64, and the total number of training times is 200;

[0035] Step 3: According to the label reconstruction-encoding-decoding neural network model and its training parameters, the prescribed loss function is minimized, thereby completing the training of the label reconstruction-encoding-decoding neural network model;

[0036] In order to make an effective prediction of the depth image, this embodiment uses a combined loss function: the size of the original depth image is H*W, and the size of the generated predicted depth image is the same as the original depth image size; the depth value at the pixel point i in the predicted depth image is p i , the depth value at pixel i in the actual depth image is The depth error corresponding to each pixel is expressed as: in represents the logarithmic difference between the predicted depth map and the true depth map at pixel i, p i Represents the depth value of the predicted depth map at pixel i, represents the depth value of the true depth map at pixel i, n represents the number of all valid pixels in the depth image, λ is the weight coefficient, and the gradient error of the depth value of the true depth map and the predicted depth map at pixel i in the x direction is expressed as: It represents the gradient value in the x direction of the logarithmic difference between the depth value of the predicted depth map at pixel i and the true depth map at pixel i. The gradient error in the y direction between the depth value of the true depth map and the predicted depth map at pixel i is: It represents the gradient value in the y direction of the logarithmic difference between the depth value of the predicted depth map at pixel i and the true depth map at pixel i. The final loss function is loss total =γL pixel +ηL gradx +ηL grady , where L pixel Represents the error between the real depth map and the predicted depth map at the pixel i, loss total is the total loss function, γ and η are weight coefficients.

[0037] Step 4: Input the image to be processed into the trained label reconstruction-encoding-decoding neural network model and output the corresponding depth image.

[0038] The image depth estimation method of this embodiment greatly enhances the learning ability of the network by introducing a label reconstruction-encoding and decoding structure, and establishes a strong mapping relationship from color images to depth images; by introducing a label reconstruction module, the depth label of the original depth image can be converted into an image multi-classification label, and the image multi-classification label is used as prior knowledge for subsequent depth estimation; by introducing an encoding and decoding module, the global features can be retained while retaining local features, thereby improving the local accuracy and global accuracy of depth estimation; the network model based on label reconstruction-encoding and decoding is trained on the NYU DepthV2 dataset and the KITTI dataset to construct a stable mapping from color images to depth images, and finally a depth map close to the real one is obtained, which proves that the present invention has good accuracy and versatility.

Claims

1. A method for image depth estimation based on label reconstruction-encoding and decoding neural network, characterized in that: The following steps are involved: Step 1: Build a label reconstruction-encoding and decoding neural network model. The label reconstruction-encoding and decoding neural network model includes multiple different convolutional layers, upsampling layers, downsampling layers, and activation layers. Step 2: Select the training set and test set, and set the training parameters of the label reconstruction-encoding-decoding neural network model; Step 3: According to the label reconstruction-encoding-decoding neural network model and its training parameters, the prescribed loss function is minimized, thereby completing the training of the label reconstruction-encoding-decoding neural network model; Step 4: Input the image to be processed into the trained label reconstruction-encoding-decoding neural network model to obtain the corresponding depth image.

2. The method according to claim 1, characterized in that: The label reconstruction-encoding and decoding neural network model built in step 1 includes a label reconstruction module and an encoding and decoding module. The label reconstruction module has four convolutional layers. The number of repetitions of each convolutional layer is 3, 4, 6, and 3 respectively. The convolution kernel sizes are 1×1, 3×3, and 1×1 respectively. The number of channels of each convolutional layer is 64, 64, 256, 128, 128, 512, 256, 256, 1024, 512, 512, and 2048 respectively.

3. The method according to claim 2, characterized in that: In the encoding and decoding module of the label reconstruction-encoding and decoding neural network model built in step 1, the encoding structure part includes 4 convolutional layers with a convolution kernel of 3×3. Each convolutional layer is followed by a corresponding downsampling layer. The number of channels of the 4 convolutional layers are 256, 512, 1024 and 2048 respectively.

4. The method according to claim 3, characterized in that: In the encoding and decoding module of the label reconstruction-encoding and decoding neural network model built in step 1, the decoding structure includes 4 convolutional layers with a convolution kernel of 3×3. Each convolutional layer is followed by a corresponding upsampling layer. The number of channels of the 4 convolutional layers are 2048, 1024, 512 and 256 respectively.

5. The method according to claim 4, characterized in that: In the encoding and decoding module of the label reconstruction-encoding and decoding neural network model built in step 1, the input of the decoding structure part comes from two parts, including the output of the previous layer encoding structure module and the output of the corresponding encoding structure part.

6. The method according to claim 1, characterized in that: In the label reconstruction-encoding-decoding neural network model built in step 1, a corresponding ReLu activation layer is added after each convolutional layer.

7. The method according to claim 1, characterized in that: Step 1: During the training of the label reconstruction-encoding-decoding neural network model, the weight initialization of the label reconstruction-encoding-decoding neural network model is performed by He Kaiming's initialization method.

8. The method according to claim 1, characterized in that: In the step 2, the training set and the test set adopt the training / test split in the NYUDepth V2 dataset and the KITTI dataset, and the network training is performed in the training sets of the two different datasets, and the network test is performed in the test sets of the two different datasets.

9. The method according to claim 1, characterized in that: The loss function in step 3 is a combined loss function: loss total =γL pixel +ηL gradx +ηL grady ,in, Represents the error between the depth logarithm of the real depth map and the predicted depth map at pixel i, Represents the gradient error in the x direction between the depth value of the real depth map and the predicted depth map at pixel i, Represents the gradient error in the y direction between the depth value of the real depth map and the predicted depth map at pixel i, It represents the logarithmic difference between the predicted depth map and the real depth map at pixel i and the real depth map at pixel i. It represents the gradient value in the x direction of the logarithmic difference between the depth value of the predicted depth map at pixel i and the depth value of the true depth map at pixel i. represents the gradient value in the y direction of the logarithmic difference between the depth value of the predicted depth map at pixel i and the true depth map at pixel i, p i Represents the predicted depth value of the predicted depth map at pixel i, represents the actual depth value of the real depth map at pixel i, n represents the number of all valid pixels in the depth image, and γ, η and λ are weight coefficients.

10. The method according to claim 9, characterized in that: In step 3, the graph depth estimation neural network model is established based on the network weights obtained by minimizing the loss function, and the Adam optimization method is used to minimize the loss function.

Images