A depth estimation method based on improved MVSNet

Abstract

The application discloses a depth estimation method based on an improved MVSNet.The application has universality in target depth estimation in a small scene, a depth map set is obtained by estimating a scene multi-view, and the application can be used for three-dimensional reconstruction.The depth size recovered by the MVSNet is smaller than the input picture size, an inverse convolution upsampling is performed in a depth map optimization step, and a binary edge detection map of a reference image is fused to complete the optimization process of the initial depth map;in view of the fact that feature extraction is not performed on a multi-scale semantic information by the MVSNet through a convolution network, a FPN image pyramid is used to construct a multi-scale space domain to iteratively construct a recovered depth map.The improved method can effectively improve the accuracy of the recovered depth map, and a larger scale depth map can more clearly describe small objects in a scene.

Description

Technical Field

[0001] This invention relates to the field of depth estimation in deep learning, specifically depth estimation techniques for small-scale scenarios. Background Technology

[0002] 3D printing technology often requires us to manually build the structure of the object model. When the surface of the object to be printed is relatively complex, it is difficult to build the model manually. We can use a camera to take a set of pictures of the target to estimate the depth, and use the depth map to reconstruct the three-dimensional structure of the target to obtain point cloud information for 3D printing.

[0003] Traditional algorithms for 3D reconstruction in small scenes are inferior to deep learning-based methods in terms of both accuracy and completeness of point clouds. Furthermore, traditional algorithms rely on manually extracted features, such as corners and edges, making them somewhat rigid. Deep learning, on the other hand, can extract high-level semantic information for depth estimation and exhibits better robustness in complex scenes with noise interference and varying lighting conditions. However, deep learning methods place higher demands on machine performance, such as significant memory consumption. In particular, deep learning methods based on MVSNet, which recover depth maps with a resolution of only 1 / 16 of the input, result in the loss or absence of some scene details in the depth map, leading to high memory requirements. Additionally, there is still considerable room for improvement in the accuracy and completeness of the recovered point clouds.

[0004] This invention aims to address the aforementioned problems of the deep learning-based depth estimation model MVSNet, improving the accuracy of depth estimation, reducing memory consumption and runtime, and accurately reconstructing scene details without omissions or losses. This invention demonstrates good practicality and versatility in the field of depth estimation for small scenes. Summary of the Invention

[0005] To overcome the problems of MVSNet, this invention proposes an improved depth estimation technique. This technique improves upon the deep learning framework MVSNet to address issues such as small estimated depth map size, increased depth estimation accuracy, and reduced memory consumption and runtime. The specific network improvement framework structure is shown in the appendix. Figure 1 As shown.

[0006] The specific technical solution of this invention is as follows:

[0007] Step 1: Feature extraction uses a CNN convolutional network with an image pyramid structure, divided into three levels. The first stage uses the input image set after two downsampling steps. The network has eight layers: the first two layers use Conv+BN+ReLU with a stride of 1, the third layer uses Conv+BN+ReLU with a stride of 2, the fourth and fifth layers use Conv+BN+ReLU with a stride of 1, the sixth layer uses Conv+BN+ReLU with a stride of 2, the seventh layer uses Conv+BN+ReLU with a stride of 1, and the eighth layer uses Conv with a stride of 1. The feature extraction network parameters are shared between different pyramid levels.

[0008] Step 2: Perform homography transformation on the features extracted by convolution in step 1 to obtain cost volume V1. Set the transformation depth plane to 192, and set the depth interval between planes according to the actual situation. The default value is 1.06 mm.

[0009] Step 3: Perform 3D U-net convolution on the cost body V1 to obtain the post-cost body VV1. Then, perform softargmax on the convolutional cost body to obtain the probability distribution result, and the probability body P. 1；

[0010] Step 4: Calculate the expectation along the depth direction of the probability volume to obtain the initial depth map D. 1；

[0011] Step 5: Perform Canny edge detection on the input reference image of this level of pyramid to obtain a binary mask image M. 1； The initial depth map D1 is combined with two deconvolution layers, each with a stride of 2, to obtain an upsampled depth map. A second channel is added to this upsampled depth map as a binary mask map M1, followed by a 4-layer convolutional network. The first three layers of the network are Conv+BN+ReLU with a stride of 1, and then a Conv convolution with a stride of 1 is performed to obtain the optimized depth map DD of the first pyramid layer. 1；

[0012] Step 6: In the second-level pyramid network, the input is the result of the first pyramid downsampling of the image set. The feature extraction network has the same structure as the previous layer, and all parameters are shared.

[0013] Step 7: After feature extraction, the feature body F2 is used to estimate the depth along the depth direction based on the cost body VV1 of the previous layer network. First, the variance of the cost body VV1 is calculated by combining the probability body P1 to obtain the depth estimation interval A1 through uncertainty estimation. Then, the upsampling depth estimation interval B1 is obtained by nearest neighbor interpolation. Homography transformation is performed on F2 using B1. At this time, the number of hypothetical planes for homography transformation is changed to 40. The homography transformation depth interval for each pixel position of F2 is performed according to B1.

[0014] Step 8: Perform 3D U-net convolution on the adaptive cost volume V' obtained in the previous step to obtain VV'2, and perform softargmax operation to obtain the probability volume P2. Then perform two-stage depth estimation, which is the same as step 5.

[0015] Step Nine: Perform the third stage of depth estimation. Input the original size map into the network for feature extraction. The network structure and specific parameters are the same as in Step One, resulting in the feature volume F. 3；

[0016] Step 10: Based on the feature body F3 obtained in the previous step, calculate the depth estimation interval A2 by combining the cost body VV'2 of the second stage with the probability body P2, and use the nearest neighbor interpolation algorithm to upsample to obtain B2; use B2 to perform homography transformation on F3, assuming the depth interval is 40, then perform 3D U-net convolution, use softargmax to calculate the probability body P3, and then calculate the expectation along the depth interval as the initial depth map; then use the same method as in step 5 to obtain the optimized depth map.

[0017] Compared with the prior art, the beneficial effects of the present invention are:

[0018] (1) Our improvement makes the final output depth map size consistent with the input image size, and the depth map information is more accurate and detailed.

[0019] (2) The overall network running time is shorter and the memory consumption is lower. Attached Figure Description

[0020] Figure 1 For: An improved MVSNet network framework.

[0021] Figure 2 For: Feature extraction network structure.

[0022] Figure 3 For: Assume a planar reference.

[0023] Figure 4 It is a 3D U-net structure.

[0024] Figure 5 For: The process of constructing an adaptive cost body.

[0025] Figure 6 For: Input to output result display.

[0026] Figure 7 For: Depth map optimization process.

[0027] Figure 8 For: Depth map effect comparison. Detailed Implementation

[0028] The invention will be further described below with reference to the accompanying drawings.

[0029] Our improved method produces depth maps with significantly higher detail compared to the original network. Figure 8 The image shows the improved results. It's clear that our depth map depicts the depth information of the bricks more clearly and delicately, with obvious color differences representing strong depth value variations. In contrast, the original network's depth depiction of the bricks is rather blurry, and the distinction between near and far depths is not very clear. This is thanks to our network's use of a multi-scale network for feature extraction, which effectively captures subtle differences. Furthermore, the adaptive cost volume constructed using uncertainty estimation can more accurately locate the depth value of each pixel.

[0030] Specific methods

[0031] 1. The network input is a set of four images: one reference image (the target for depth estimation) and three images with similar viewpoints to the reference image. A series of images of the target scene are captured. After selecting one image as the reference image, similarity estimation is performed on the other images in the dataset and the reference image. Based on the similarity scores, the top three views are selected as the nearest neighbor views.

[0032] 2. Our network utilizes a CNN (Convolutional Neural Network) to extract features from images. Each level of the pyramid has the same network structure and shares network parameters. After downsampling the four input images as needed, features are extracted separately. The specific feature extraction network structure is as follows: Figure 2 As shown, the blue layer of the network performs 3x3 convolutions, Batch Normalization (BN), and ReLU activation with a stride of 1, maintaining the same image size for both input and output. The orange-red layer performs 3x3 convolutions, BN, and ReLU activation with a stride of 2, reducing the input and output dimensions to half their original size. The green layer performs 3x3 convolutions with a stride of 1. The first layer expands the input image from 3 channels to 8 channels. Each subsequent orange-red layer doubles the number of channels, ultimately resulting in a 32-channel feature body.

[0033] 3. For the extracted feature volumes, homography transformation is performed. We first set the depth estimation interval for the reference image. The number of depth assumption planes d for the reference image is... num We default to 192 depth planes, with a depth interval d between adjacent depth planes. interval We default to 1.06 mm. Figure 3 This demonstrates the hypothetical planar structure. We have thus obtained the initial hypothetical depth range [0, 203.52]. In practical applications, we need to adjust d according to specific circumstances. num and d intervalParameters. For the feature volumes of the reference image, only the same feature volume content is copied in each depth plane, without homography transformation. Based on the depth values ​​of the reference image in each hypothetical plane, the homography transformation matrix from adjacent images to different depth planes of the reference image is calculated. Using the homography matrix, the feature volumes of each adjacent image are transformed onto the hypothetical depth plane of each reference image feature volume. This constitutes d. num There are three depth planes, each containing a feature volume of a reference image and feature volumes of three adjacent images. We call this data structure the cost volume.

[0034] 4. Perform a 3D U-Net convolution operation on the cost volume to obtain the post-cost volume. The biggest advantage of U-Net is its suitability for distinguishing and segmenting differences in image content. The 3D U-Net volume structure is similar, capable of calculating the score of each pixel position in the cost volume for each hypothetical plane; a higher score indicates that the pixel position belongs to that depth plane. Because this part of the network consumes a lot of GPU memory, the 32-channel cost volume is downsampled to 8 channels in the first stage, and then downsampled in each subsequent stage, until the top-level cost volume is only a single channel. The 3D U-Net network structure is as follows: Figure 4 As shown, there are four levels in total. H, W, and D represent the length, width, and number of hypothesis planes of the input feature volume. Our cost volume input size is HxWxDxF, and the output is HxWxD. F represents the number of channels, which is 32. After passing through the network, it is reduced to a single channel.

[0035] 5. After obtaining the post-cost body, we perform a softargmax operation to calculate the probability of each pixel position in each depth plane, obtaining the overall probability distribution, i.e., the probability body. Using the post-cost body and the probability body, we can calculate the depth estimation interval for each pixel. In the second and third stages of homography transformation, we use the depth interval calculated for each pixel in the previous stage, subdivide it into 40 layers, and then perform homography transformation to obtain the adaptive cost body. This part of the process is as follows: Figure 5 As shown, the main focus is on the construction of the adaptive cost body in the second and third stages. We mainly replace the cost body with the adaptive cost body for 3D U-net convolution.

[0036] 6. Using the probability volume, we calculate the expected value along the depth direction to obtain the initial depth map. Figure 6 The input reference map, probability map, and initial depth map are shown from left to right. The reference map has dimensions H x W, so the initial depth map has dimensions H / 4 x W / 4. We perform two deconvolutions on the initial depth map, doubling its size after each deconvolution. The final dimensions are the same as the reference map. Figure 1Next, we perform Canny edge detection on the reference image to obtain a binary edge map, which serves as the second channel of the deconvolutioned depth map. This second channel is then processed by the final convolutional network for depth map edge detail sharpening. Sharpening the depth map is crucial because the boundaries of the depth map gradually become blurred and smoothed during the large receptive field and upsampling process of the convolutional network. Appropriate boundary information needs to be incorporated for adjustment to improve the accuracy of the depth map. The depth map optimization process is as follows: Figure 7 As shown in the figure, the traditional optimization process has been selected by box, and the comparison highlights our improvements.

Claims

1. A depth estimation method based on an improved MVSNet, comprising the following steps: Step 1: Feature extraction uses a CNN convolutional network with an image pyramid structure, divided into three levels. The first stage uses the input image set after two downsampling steps. The network has eight layers: the first two layers use Conv+BN+ReLU with a stride of 1, the third layer uses Conv+BN+ReLU with a stride of 2, the fourth and fifth layers use Conv+BN+ReLU with a stride of 1, the sixth layer uses Conv+BN+ReLU with a stride of 2, the seventh layer uses Conv+BN+ReLU with a stride of 1, and the eighth layer uses Conv with a stride of 1. The feature extraction network parameters are shared between different pyramid levels. Step 2: Perform homography transformation on the features extracted by convolution in step 1 to obtain cost volume V1. Set the transformation depth plane to 192, and set the depth interval between planes according to the actual situation. The default value is 1.06 mm. Step 3: Perform 3D U-net convolution on the cost body V1 to obtain the post-cost body VV1, and perform softargmax on the post-cost body VV1 to obtain the probability distribution result, probability body P1; Step 4: Calculate the expectation along the depth direction of the probability volume to obtain the initial depth map D. 1； Step 5: Perform Canny edge detection on the input reference image of this pyramid level to obtain a binary mask image M1; combine it with the initial depth image D1 and perform two deconvolution layers with a stride of 2 for each layer to obtain an upsampled depth image. Add a second channel to the obtained upsampled depth image to form a binary mask image M1, and then perform a 4-layer convolutional network; the first three layers of the network are Conv+BN+ReLU with a stride of 1, and then perform Conv convolution with a stride of 1 to obtain the optimized depth image DD1 of the first pyramid level; Step 6: In the second-level pyramid network, the input is the result of the first pyramid downsampling of the image set. The feature extraction network has the same structure as the previous layer, and all parameters are shared. Step 7: After feature extraction, the feature body F2 is used to estimate the depth along the depth direction based on the post-cost body VV1 of the previous layer network. First, the variance of the post-cost body VV1 is calculated by combining the probability body P1 to obtain the depth estimation interval A1 through uncertainty estimation. Then, the upsampled depth estimation interval B1 is obtained by nearest neighbor interpolation. Homography transformation is performed on F2 using B1. At this time, the number of hypothetical planes for homography transformation is changed to 40. The homography transformation depth interval for each pixel position of F2 is performed according to B1. Step 8: Perform 3D U-net convolution on the adaptive cost volume V' obtained in the previous step to obtain the post-cost volume VV'2, and perform softargmax operation to obtain the probability volume P2. Then perform two-stage depth estimation, which is the same as step 5. Step 9: Perform the third stage of depth estimation. Input the original size map into the network for feature extraction. The network structure and specific parameters are the same as in Step 1, and the feature volume F3 is obtained. Step 10: Based on the feature body F3 obtained in the previous step, calculate the depth estimation interval A2 by combining the post-cost body VV'2 of the second stage with the probability body P2, and use the nearest neighbor interpolation algorithm to upsample to obtain B2; use B2 to perform homography transformation on F3, assuming the depth interval is 40, then perform 3D U-net convolution, use softargmax to calculate the probability body P3, and then calculate the expectation along the depth interval as the initial depth map; then use the same method as in step 5 to obtain the optimized depth map.

2. The method as described in claim 1, characterized in that, Step seven describes the process of combining probability body P1 with subsequent cost body VV1 to calculate the variance, perform uncertainty estimation to obtain depth estimation interval A1, and use nearest neighbor interpolation to obtain upsampled depth estimation interval B1. This is then used for homography transformation of feature body F2 in the second stage to obtain cost body V2.

3. The method as described in claim 1, characterized in that, Step 10 describes the process of combining the post-cost body VV'2 of the second stage with the probability body P2 to obtain the depth estimation interval A2, and using the nearest neighbor interpolation algorithm to upsample to obtain B2. Then, using B2, a homography transformation is performed on F3 to obtain the cost body V3.

Images