An indoor monocular depth estimation method based on self-supervised deep learning

Abstract

The application discloses an indoor monocular depth estimation method based on self-supervised deep learning, which is used for predicting the depth of an indoor scene from a single image and relates to the field of computer vision. The application introduces a self-supervised optical flow estimation network to provide effective supervision information for depth estimation. Firstly, the original photometric loss of the optical flow estimation network is optimized into a photometric loss based on a local pixel block, the optical flow estimation network is fine-tuned by using a public indoor scene dataset, and the fine-tuned optical flow network parameters are fixed; then, the optical flow predicted by the optical flow network is used to design an optical flow consistency loss, and a multi-scale feature map synthesis loss is designed based on a feature pyramid generated by the optical flow network to supervise the training of the depth estimation network; finally, the trained depth estimation model is used to predict the depth of a single image. The application improves the depth estimation precision of the indoor low-texture area by designing effective loss function items.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of indoor monocular depth estimation method based on self-supervised deep learning, the present application can be used to carry out depth estimation to single indoor scene image.The present application belongs to computer vision field. BACKGROUND

[0002] Monocular depth estimation aims to recover scene depth from single image, and has been widely applied in three-dimensional reconstruction, environment perception and other fields.Supervised method based on deep learning can obtain very high prediction accuracy, but needs a large number of label data for training.Self-supervised method can get rid of the dependence on label data, so it has got more and more attention.However, existing self-supervised method is mostly good in outdoor scene dataset, and it is very difficult to achieve satisfactory performance in indoor scene.

[0003] Since existing self-supervised method mainly relies on photometric consistency assumption, therefore for indoor monocular depth estimation, the existence of a large number of low-texture regions is a great challenge.These regions often lack significant features, are prone to cause pixel mismatch, and lead to depth prediction error.In the literature“Z.Yu, L.Jin, and S.Gao. P 2 Net:Patch-Match andPlane-Regularization for Unsupervised Indoor Depth Estimation.In ECCV,2020.”, Zehao Yu et al.extract significant feature pixel block to calculate photometric loss, and use plane consistency of non-texture region to design supervision information.

[0004] Accurate optical flow information can provide accurate pixel matching for depth estimation, thereby improving depth estimation accuracy.Existing self-supervised optical flow estimation method is also based on photometric consistency assumption, and performance will also decline in low-texture region. SUMMARY

[0005] In order to solve the problem of poor depth estimation accuracy in indoor low-texture region, the present application designs an indoor monocular depth estimation method based on optical flow learning by using the supervision information provided by self-supervised optical flow estimation network.

[0006] The present application firstly improves the photometric loss of the optical flow estimation network, calculates the photometric loss based on the extracted feature points and the 3*3 pixel blocks around the feature points, fine-tunes the optical flow estimation model based on the loss, and improves the optical flow estimation accuracy in the low texture area. The optical flow calculated by the fine-tuned optical flow estimation network is used to supervise the rigid flow generated by the depth estimation network, and the optical flow consistency loss is calculated. The target image is convolved by the optical flow estimation network to obtain a multi-scale feature map, and after being remapped by the rigid flow, a synthesized source image feature map is generated, and the multi-scale feature map synthesis loss is calculated together with the actual source image feature map.

[0007] The technical problem solved by the present application is that the existing indoor depth estimation method based on self-supervised deep learning has poor accuracy in low texture areas.

[0008] The technical solution of the present application is: an indoor monocular depth estimation method based on self-supervised deep learning, characterized in that it comprises the following steps:

[0009] (1) dividing a monocular sequence image dataset taken in an indoor environment into a training set and a test set, and preprocessing the data;

[0010] (2) constructing a depth estimation network overall framework based on self-supervised optical flow learning, using the optical flow and multi-scale feature map predicted by the optical flow network, and designing corresponding loss function terms;

[0011] (3) optimizing the original photometric loss of the optical flow network into a photometric loss based on local pixel blocks, and using the divided training set to fine-tune the pre-trained optical flow estimation network combined with the optimized photometric loss;

[0012] (4) taking the training set as input, using the fine-tuned optical flow network to predict, and taking the obtained optical flow as one of the supervision information for depth estimation, and designing a corresponding optical flow consistency loss;

[0013] (5) taking the training set as input, using the fine-tuned optical flow network to calculate the multi-scale feature map as another supervision information for depth estimation, and designing a corresponding multi-scale feature map synthesis loss;

[0014] (6) based on the constructed depth estimation network framework and the designed loss function, training the network with the training set;

[0015] (7) taking a single image as input, and obtaining the depth result through model prediction.

[0016] In step (2), the optical flow learning is introduced for supervision for depth estimation, and the optical flow and multi-scale feature map generated by the optical flow estimation network are used as supervision information for depth estimation. The loss function of the depth estimation network is represented as:

[0017] Ldepth = L ph + λ1L sm + λ2L spp + L rigid + λ3L feature

[0018] where L ph represents the photometric consistency loss based on the pixel block, L sm represents the smoothness loss, L spp represents the planar consistency loss, L rigid is the optical flow consistency loss, and L feature is the multi-scale feature map synthesis loss. λ1, λ2 and λ3 are the weights of the smoothness loss, the planar consistency loss and the feature map synthesis loss, respectively.

[0019] In the step (3), the photometric loss of the optical flow estimation network is optimized. The present application firstly extracts feature points, and selects 3x3 local pixel blocks around the feature points to participate in the calculation of the photometric loss, and the pixel block is represented as:

[0020] P = {(x+x k ,y+y k ), x k ∈{-N, 0, N}, y k ∈{-N, 0, N}}

[0021] where (x, y) is the pixel coordinate of the feature point, and the size of the pixel block can be set by adjusting N. For the predicted optical flow f t→s from the target image to the source image, the photometric loss based on the local pixel block is represented as:

[0022]

[0023] where F(·) represents a function for calculating the similarity between pixels, and in order to reduce the influence of the change of ambient light on the photometric consistency as much as possible, a ternary Census transform loss is used to describe the similarity between pixels. I s (P i ) represents the pixel brightness of the pixel block in the source image, represents the remapped pixel brightness, and the calculation formula is I t (P i ) is the pixel brightness of the pixel block in the target image.

[0024] In the step (4), the optical flow predicted by the fine-tuned optical flow network is used as the supervision information for depth estimation, and an optical flow consistency loss is designed, and the optical flow consistency loss is represented as:

[0025] L rigid = |frigid (p t )-f flow (p t )|

[0026] wherein, f rigid (p t ) represents the rigid flow calculated by the depth estimation network in the training process, f flow (p t ) represents the optical flow predicted by the optical flow estimation network.

[0027] In the step (5), the training set is input into the network, and is subjected to convolution through the Siamese feature pyramid network to obtain multi-scale feature maps of the target image and the source image. The feature map of the target image is subjected to remapping to obtain a synthesized source image feature map, and the difference between the synthesized source image feature map and the actual feature map of the source image is calculated as a multi-scale feature map synthesis loss, which can be expressed as:

[0028]

[0029] wherein, F src (p t ) represents the actual feature map of the source image, represents the synthesized feature map of the source image, and the calculation formula is as follows:

[0030]

[0031] wherein, F tgt (p t ) represents the feature map of the target image.

[0032] Compared with the prior art, the present application has the following advantages:

[0033] (1) The present application introduces self-supervised optical flow learning to provide supervision information for depth estimation, optimizes the photometric loss of the optical flow estimation network, and calculates the photometric loss by using feature significant feature points and neighborhood pixels, thereby improving the accuracy of optical flow estimation in low-texture regions.

[0034] (2) The fine-tuned optical flow estimation network provides high-precision optical flow as supervision information for depth estimation, and a photometric consistency loss function is designed, thereby improving the monocular depth estimation accuracy in indoor low-texture regions.

[0035] (3) The fine-tuned optical flow estimation network generates multi-scale feature maps through convolution, and simultaneously supervises the depth estimation with the high-precision optical flow, and a multi-scale feature map synthesis loss is designed, thereby further improving the depth estimation accuracy in indoor scenes. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of the method of the present application.

[0037] Figure 2 Schematic diagram for calculating the photometric loss based on local pixel block for the optical flow estimation network.

[0038] Figure 3 Schematic diagram for calculating the multi-scale feature map synthesis loss for the depth estimation network.

[0039] Figure 4 (a) is the result of the algorithm of the present application, Figure 4 (b) is the result of the algorithm of Zehao Yu, Figure 4 (c) is the result of the algorithm of the present application, Figure 4 (d) is the visualization result of the depth reference value. DETAILED DESCRIPTION

[0040] In order to better understand the present application, the technical solutions of the present application are described in detail below in combination with the drawings and examples.

[0041] Referring to the drawings Figure 1 An indoor monocular depth estimation method based on self-supervised deep learning comprises the following steps:

[0042] 1. 283 scenes are selected from a monocular indoor image dataset, and a total of 21483 target images are selected. Every 10 frames of target images are selected as a sequence image after distortion correction, and 21483 groups of sequence images are formed as a training set of an optical flow estimation network and a depth estimation network. 654 images are selected as a test set of the depth estimation network. All training set images are converted to 288x384 size, and random color enhancement and left-right flipping are performed.

[0043] 2. Based on the PyTorch code framework, a ResNet18s pre-trained on ImageNet is used as an encoder-decoder structure, and Adam is used as an optimizer of the network to construct a basic framework of the depth estimation network. A self-supervised optical flow learning network is introduced, and the predicted optical flow and multi-scale feature maps are used to design corresponding loss function items. The loss function of the depth estimation network is represented as:

[0044] L depth = L ph + λ1L sm + λ2L spp + L rigid + λ3L feature

[0045] Wherein, L ph represents the photometric consistency loss based on the pixel block, L sm represents the smoothness loss, L spp represents the plane consistency loss, L rigid is the optical flow consistency loss, and L featureThe multi-scale feature map synthesis loss. λ1, λ2 and λ3 are the weights of the smoothness loss, the plane consistency loss and the feature map synthesis loss, respectively, which are set to 0.001, 0.05 and 3, respectively.

[0046] 3. Refer to the attached drawings Figure 2 The original photometric loss of the optical flow estimation network is optimized. The original photometric loss uses all pixels in the image to participate in the calculation, and the optimized photometric loss only uses the pixels of the feature points with significant features and their neighborhoods to participate in the calculation. First, the Direct Sparse Odometry (DSO) is used to extract the feature points, and the 3x3 local pixel block around the feature points is selected to participate in the calculation of the photometric loss, which is represented as:

[0047] P={(x+x k ,y+y k ),x k ∈{-N,0,N},y k ∈{-N,0,N}}

[0048] Where (x, y) is the pixel coordinate of the feature point, and the size of the pixel block can be set by adjusting N. For the predicted optical flow f t→s from the target image to the source image, the photometric loss based on the local pixel block is represented as:

[0049]

[0050] Where F(·) represents a function for calculating the similarity between pixels, and in order to reduce the influence of ambient light changes on photometric consistency as much as possible, a ternary Census transform loss is used to describe the similarity between pixels. I s (P i ) represents the pixel brightness of the pixel block in the source image, represents the remapped pixel brightness, and its calculation formula is I t (P i ) is the pixel brightness of the pixel block in the target image.

[0051] The loss function of the optical flow estimation network is composed of the photometric loss based on the local pixel block and the smoothness loss, which can be represented as:

[0052] L flow =L patch (f t→s )+λL sm (f t→s )

[0053] Where f t→sdenotes the predicted optical flow from the target image to the source image, and the smoothness loss is defined as:

[0054]

[0055] where I t denotes the target image, and the smoothness loss ensures that the prediction results of adjacent pixels are similar.

[0056] Based on the model parameters pre-trained on the MPI Sintel optical flow dataset of animated movies, combined with the optimized loss function, the training set divided in step 1 is used to fine-tune the optical flow estimation network. The 0th frame and the +1th frame in each group of sequence images are used as the target image and the source image.

[0057] 4. The optical flow network parameters fine-tuned in step 3 are frozen, and the training set images are inputted. The predicted optical flow is used to supervise the rigid flow calculated by the depth estimation network. The rigid flow from the target image to the source image is defined as:

[0058] f rigid t )=p s -p t

[0059] where p t and p s represent the pixel coordinates of the corresponding points in the target image and the source image, respectively. The projection relationship of the predicted depth of the target image and the homogeneous coordinates of the corresponding points can be written as:

[0060] p s =KT t→s D t (p t )K -1 p t

[0061] where K is the camera intrinsic matrix, T t→s denotes the relative pose from the target image to the source image, which can be obtained by the relative pose estimation convolutional neural network PoseCNN, and D t (p t ) represents the predicted depth of the target image, which is obtained by the depth estimation convolutional neural network DepthCNN in the depth estimation network. To realize the supervision of the rigid flow, an optical flow consistency loss is designed, which is represented as:

[0062] L rigid =|f rigid (p t )-f flow (p t )|

[0063] ​wherein f rigid (p t ) represents the rigid flow calculated by the depth estimation network during the training process, f flow (p t ) represents the optical flow predicted by the optical flow network.

[0064] 5. Refer to the attached Figure 3 , input the target image and the source image in the training set into the fine-tuned optical flow network, and perform convolution through the Siamese feature pyramid network to obtain the multi-scale feature maps of the target image and the source image as another supervision information of the depth estimation network. The rigid flow is down-sampled to the same scale as the feature map, and the feature map of the target image is remapped through the rigid flow of the corresponding scale to obtain the synthesized source image feature map, and the difference between the synthesized source image feature map and the actual feature map of the source image is calculated as the multi-scale feature map synthesis loss, which can be expressed as:

[0065]

[0066] wherein F src (p t ) represents the actual feature map of the source image, the synthesized feature map of the source image, and the calculation formula is:

[0067]

[0068] wherein F tgt (p t ) represents the feature map of the target image.

[0069] 6. Based on the depth estimation network framework established in steps 1-5 and the designed loss function, the network is trained, and each group of sequence images in the training set uses the 0th frame as the target image and the-1th frame and the 1st frame as the source image.

[0070] 7. Input a single indoor scene image, and the depth result can be obtained through the model prediction.

[0071] The effects of the present application are further illustrated by the following test results.

[0072] The test set divided in step 1 is used as input to calculate the depth prediction result of the network, and the prediction result is compared with that of Zehao Yu, and the accuracy evaluation indicators are as follows:

[0073] · Absolute relative error:

[0074] · Root mean square error:

[0075] · Average Log10 error:

[0076] Accuracy:

[0077] wherein represents the ground truth of depth, D represents the predicted result of depth, N represents the number of pixels, and thr is a threshold (1.25, 1.25 2 , 1.25 3 ).

[0078] The test results are shown in Table 1.

[0079] Table 1

[0080] Method REL RMS Log10 δ < 1.25 <1.25 2 ]]> <1.25 3 ]]> Zehao Yu 0.159 0.599 0.068 0.772 0.942 0.984 The invention 0.158 0.583 0.067 0.779 0.947 0.987

[0081] The three indexes REL, RMS and Log10 in Table 1 measure the error of depth estimation, and the lower the results of the three indexes are, the better the results are. The last three indexes δ<1.25, δ<1.25 2 and δ<1.25 3 represent the accuracy of depth estimation, and the higher the results are, the better the results are. Compared with the unsupervised algorithm proposed by Zehao Yu, the results of all indexes of the present application are improved. The visual comparison results of depth prediction are shown in the attached Figure 4 In low-texture areas such as walls and curtains, the present application has achieved better depth estimation results than Zehao Yu.

Claims

1. A self-supervised deep learning based indoor monocular depth estimation method, characterized in that, The method comprises the following steps: (1) dividing monocular sequence image data sets taken indoors into a training set and a test set, and preprocessing the data; (2) constructing a deep estimation network overall framework based on self-supervised optical flow learning, using the optical flow predicted by the optical flow network and multi-scale feature maps to design corresponding loss function terms; (3) optimizing the original photometric loss of the optical flow network into a photometric loss based on local pixel blocks, using the divided training set to fine-tune the pre-trained optical flow estimation network in combination with the optimized photometric loss; (4) taking the training set as input to predict the optical flow using the fine-tuned optical flow network, taking the obtained optical flow as one of the supervision information for depth estimation, and designing a corresponding optical flow consistency loss; (5) taking the training set as input to calculate multi-scale feature maps using the fine-tuned optical flow network, taking the multi-scale feature maps as another supervision information for depth estimation, and designing a corresponding multi-scale feature map synthesis loss; (6) training the network using the training set based on the constructed depth estimation network framework and the designed loss function; (7) taking a single image as input, and obtaining the depth result through model prediction; In the step (3), the photometric loss of the optical flow estimation network is optimized, and the optimized photometric loss only uses the pixels of the feature points and their neighborhoods with significant features to participate in the calculation; first, the feature points are extracted using a direct sparse odometry calculation method, and a 3*3 local pixel block around the feature points is selected to participate in the calculation of the photometric loss, and the pixel block is represented as: P = {(x+x k ,y+y k ), x k ∈ {-N,0,N}, y k ∈ {-N,0,N}}; where (x, y) is the pixel coordinate of the feature point, and the size of the pixel block can be set by adjusting N; for the prediction optical flow f from the target image to the source image t→s The photometric loss based on the local pixel block is represented as: wherein F(·) represents a function for calculating the similarity between pixels, in order to reduce the influence of ambient light changes on photometric consistency as much as possible, a ternary Census transform loss is used to describe the similarity between pixels; I s (P i ) represents the pixel brightness of the pixel block in the source image, represents the pixel brightness of the pixel block in the target image, I t (P i ) is the pixel brightness of the pixel block in the target image; The loss function of the optical flow estimation network is composed of the photometric loss based on the local pixel block and the smoothness loss, and is represented as: L flow = L patch (f t→s ) + λL sm (f t→s ); where f t→s denotes the predicted optical flow from the target image to the source image, λ is set to 50 in training, and the smoothness loss is defined as: wherein I t represents the target image; based on the pre-trained model parameters on the optical flow dataset, combined with the optimized loss function, the training set divided in step 1 is used to fine-tune the optical flow estimation network, and the 0th frame and the +1th frame in each group of sequence images are used as the target image and the source image; In the step (4), the optical flow predicted by the fine-tuned optical flow network is taken as the supervision information for depth estimation, and the optical flow consistency loss is designed, and the optical flow consistency loss is represented as: L rigid = |f rigid (p t )-f flow (p t )|; wherein f rigid (p t ) represents the rigid flow calculated by the depth estimation network during the training process, f flow (p t ) represents the optical flow predicted by the optical flow estimation network. In the step (5), the training set is input into the network, and the Siamese feature pyramid network is used for convolution to obtain the multi-scale feature maps of the target image and the source image, which are taken as another supervision information for the depth estimation network; the rigid flow is down-sampled to the same scale as the feature maps, the feature maps of the target image are remapped through the rigid flow of the corresponding scale to obtain the synthesized source image feature maps, and the difference between the synthesized source image feature maps and the actual feature maps of the source image is calculated as the multi-scale feature map synthesis loss, and the loss can be represented as: wherein F src (p t ) represents the actual feature map of the source image, represents the synthesized feature map of the source image, and the calculation formula is: where F tgt (p t ) denotes a feature map of the target image.

2. The indoor monocular depth estimation method based on self-supervised deep learning according to claim 1, characterized in that: In the step (2), the optical flow learning is introduced for supervision for depth estimation, and the optical flow generated by the optical flow estimation network and the multi-scale feature maps are both taken as the supervision information for depth estimation; and the loss function of the depth estimation network is represented as: L depth = L ph + λ1L sm + λ2L spp + L rigid + λ3L feature ; wherein, L ph represents the photometric consistency loss based on the pixel block, L sm represents the smoothness loss, L spp represents the planar consistency loss, L rigid is the optical flow consistency loss, L feature is the multi-scale feature map synthesis loss; λ1, λ2 and λ3 are the weights of the smoothness loss, the planar consistency loss and the feature map synthesis loss, respectively.

Images