Depth estimation network based on dynamic multi-scale features and content aware aggregation

Abstract

The invention designs a depth estimation network based on dynamic multi-scale features and content awareness aggregation aiming at the problems of low model detail reduction degree, high requirements on computing power and storage resources, long modeling time and the like when a multi-view unmanned aerial vehicle sequence image is used for performing a three-dimensional reconstruction task of a large-range scene. High-precision depth estimation is an important guarantee for realizing high-precision three-dimensional reconstruction, the network is based on a multi-view three-dimensional network MVSNet, a dynamic multi-scale feature extraction network, a lightweight cost body, a content-aware cost body aggregation module and a cyclic convolution hybrid network are introduced, richer and deeper depth information in an image can be extracted, and the image quality is improved. And the method has excellent performance in the aspect of operation efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of computer image processing. It designs a lightweight depth estimation network for dynamic multi-scale convolutional feature extraction and content aggregation based on deep learning, so as to achieve the purpose of effectively utilizing high-level semantic features of images, reducing the amount of calculation, and improving the speed and accuracy of depth estimation. Background Art

[0002] With the development of smart cities and digital twins, 3D reconstruction of large scenes plays an increasingly important role in many fields. Many applications rely on high-precision and high-fidelity 3D models. However, the current 3D reconstruction methods based on deep learning generally have problems such as low model detail restoration, high computing power and storage resource requirements, and long modeling time, which brings many challenges to the 3D reconstruction of large scenes. How to save computing power resources while ensuring high model accuracy has become an urgent problem to be solved in the current reconstruction of large scenes. Among them, depth estimation is a key link in 3D reconstruction, and its accuracy greatly affects the accuracy of 3D reconstruction. Depth refers to the distance from the camera focus to the 3D point corresponding to a certain pixel on the image. The depth map is a data format containing the depth information of all pixels on the image, which can reflect the distance relationship of objects in the image. According to the depth map, the pixels can be restored to the three-dimensional space to form a sheet point cloud. By using multiple depth maps, more depth information can be obtained, thereby achieving higher-precision 3D reconstruction. Therefore, the depth map plays an important role in 3D reconstruction and is one of the key steps in converting images into 3D models. Traditional deep learning-based depth estimation networks often use single-scale feature extraction, which performs poorly for depth estimation of large scenes. At the same time, it requires a lot of computing resources and computing time, and has high computational complexity. Therefore, the current research trend is to develop new depth estimation methods based on deep learning technology, and use neural networks combined with semantic information to improve the accuracy and robustness of depth estimation. This method can effectively utilize the high-level semantic features of the image, while reducing the amount of calculation and improving the speed and accuracy of depth estimation. In this context, the present invention is based on MVSNet, and has made a series of improvements to design a lightweight depth estimation network based on dynamic multi-scale convolutional feature extraction and content aggregation. The network is characterized by utilizing the multi-angle and multi-view information of multi-view drone sequence images, and using a dynamic multi-scale feature extraction network to extract high-resolution image features of different scales. Then, the cost volume is constructed using feature maps of different resolutions. At the same time, the network adopts a mean measurement method based on group similarity to construct a lightweight cost volume, which can greatly speed up the network operation while retaining as much information as possible. In addition, the content-aware cost volume aggregation module realizes the upsampling and aggregation of cost volumes of different resolutions. Finally, the RNN-CNN hybrid network is used to aggregate and regularize information in spatial context and depth direction to generate a probability volume, and finally a depth map is generated from the probability volume. Summary of the invention

[0003] The present invention discloses a depth estimation network based on dynamic multi-scale features and content-aware aggregation. The invention adopts a dynamic multi-scale feature extraction network to extract high-resolution image features of different scales. Then, the cost body is constructed using feature maps of different resolutions. At the same time, the network adopts a method based on the mean measurement of group similarity to construct a lightweight cost body, which can greatly speed up the operation of the network while retaining as much information as possible. In addition, the cost body aggregation module based on content awareness realizes the upsampling and aggregation of cost bodies of different resolutions. Finally, the information aggregation and regularization in the spatial context and depth direction are realized through the RNN-CNN hybrid network to generate a probability body, and finally a depth map is generated from the probability body.

[0004] The technical solution adopted by the present invention is to design a lightweight depth estimation network based on deep learning for dynamic multi-scale convolution feature extraction and content aggregation, so as to effectively utilize high-level semantic features of images, reduce the amount of calculation, and improve the speed and accuracy of depth estimation. The method includes:

[0005] Step 1: A dynamic multi-scale feature extraction network guided by normal curvature is used to extract multi-scale deep features of UAV images.

[0006] Step 1.1: Input an image of size H×W and its estimated epipolar lines e to the network, and use the encoder to downsample step by step to extract image features. In the first layer, the scale is set to 3, 5, 7, 11, the step size is set to 1, and the convolutional neural network is used to extract image features and output an image of size H×W;

[0007] Step 1.2: In the second layer of the encoder, receive the image input of the first layer with a size of H×W, set the scale to 3, 5, 7, 9, and the step size to 1, use the convolutional neural network to extract image features again, and output an image of size H×W;

[0008] Step 1.3: In the third layer of the encoder, receive the image input of the previous layer with a size of H×W, downsample it, set the step size to 2, and finally output the size of H / 2×W / 2;

[0009] Step 1.4: In the fourth layer of the encoder, receive the image input of the previous layer with a size of H / 2×W / 2, set the scale to 3, 5, 7, set the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 2×W / 2;

[0010] Step 1.5: In the fifth layer of the encoder, receive the image input of the previous layer with a size of H / 2×W / 2, set the scale to 3, 5, 7, set the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 2×W / 2;

[0011] Step 1.6: In the sixth layer of the encoder, receive the image input of the previous layer with a size of H / 2×W / 2, downsample it, set the step size to 2, and finally output the size of H / 4×W / 4;

[0012] Step 1.7: In the seventh layer of the encoder, receive the image input of the previous layer with a size of H / 4×W / 4, set the scale to 3, 5, and the step size to 1, use the convolutional neural network to extract image features, and output an image with a size of H / 4×W / 4;

[0013] Step 1.8: In the eighth layer of the encoder, receive the image input of the previous layer with a size of H / 4×W / 4, set the scale to 3, 5, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 4×W / 4;

[0014] Step 1.9: At the ninth layer of the encoder, receive the image input of the previous layer with a size of H / 4×W / 4, downsample it, set the step size to 2, and finally output the size of H / 8×W / 8;

[0015] Step 1.10: The decoder receives the output of the encoder as input, and restores the resolution of the feature map through step-by-step upsampling and feature reuse strategies and outputs the final result. In the first layer, the scale is set to 1, 3, the step size is set to 1, and the convolutional neural network is used to extract image features, and the output size is H / 8×W / 8.

[0016] Step 1.11: In the second layer of the decoder, receive the image input of the previous layer with a size of H / 8×W / 8, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 8×W / 8;

[0017] Step 1.11: In the third layer of the decoder, receive the image input of the previous layer with a size of H / 8×W / 8, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 8×W / 8;

[0018] Step 1.12: In the fourth layer of the decoder, receive the image input of the previous layer with a size of H / 8×W / 8, upsample it, set the step size to 2, and perform a cross-layer jump connection with the image features output in step 1.8, and finally output an image with a size of H / 4×W / 4;

[0019] Step 1.13: In the fifth layer of the decoder, receive the image input of the previous layer with a size of H / 4×W / 4, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 8×W / 8;

[0020] Step 1.14: In the sixth layer of the decoder, receive the image input of the previous layer with a size of H / 4×W / 4, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 4×W / 4;

[0021] Step 1.15: In the seventh layer of the decoder, receive the image input of the previous layer with a size of H / 4×W / 4, upsample it, set the step size to 2, and perform a cross-layer jump connection with the image features output in step 1.5, and finally output an image with a size of H / 2×W / 2;

[0022] Step 1.16: In the eighth layer of the decoder, receive the image input of the previous layer with a size of H / 2×W / 2, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 2×W / 2;

[0023] Step 1.17: In the ninth layer of the decoder, receive the image input of the previous layer with a size of H / 2×W / 2, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 2×W / 2.

[0024] Step 2: Construct the cost volume based on the group similarity mean metric.

[0025] Step 2.1: Transform the adjacent feature map to the reference view through homography to obtain the transformed feature map;

[0026] Step 2.2: Group the reference feature map and the transformed feature map by channel, calculate their similarity, and then find the similarity information within each group and perform the average processing;

[0027] Step 2.3: Concatenate the similarity means of each group into a vector as the output of the cost body.

[0028] Step 3: After constructing cost volumes of different resolutions based on feature maps of different resolutions, the next step is to upsample the cost volumes of different resolutions to improve the resolution and aggregate them.

[0029] Step 4: Regularize the cost volume using the RNN-CNN hybrid network.

[0030] Step 4.1: Slice the cost volume according to the depth dimension, and multiple horizontally parallel convolutional neural networks regularize the slices of their respective dimensions;

[0031] Step 4.2: The RNN network passes the output of the previous dimension to the next one in the vertical direction.

[0032] Step 5: Perform Softmax operation along the depth direction, normalize the convolution output, and obtain the probability volume P.

[0033] Step 6: Take the mathematical expectation of the pixel depth value as the derived depth value to generate the final depth map. Compared with the prior art, the beneficial effects of the present invention are:

[0034] (1) The normal curvature information is used as a learnable parameter to guide the dynamic convolutional network to adjust the convolution kernel size.

[0035] (2) The encoding-decoding structure and skip connection of the dynamic multi-scale feature extraction network can realize the construction of semantic feature maps at different levels and extract deeper and richer features from the image.

[0036] (3) Based on the mean measurement method of group similarity, the similarity information within the group is averaged as the output of the cost body, which greatly reduces the computational load of the regularization link.

[0037] (4) Cost volume aggregation based on content-aware strategy retains more detailed information and has a larger receptive field.

[0038] (5) The RNN-CNN hybrid network can further reduce network memory consumption by performing regularization in both horizontal and vertical directions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a depth estimation network structure based on dynamic multi-scale features and content-aware aggregation. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings.

[0041] As shown in the attached network structure Figure 1 As shown in the figure, a dynamic multi-scale feature extraction network is first used to extract image features. The size of the convolution kernel is adaptively adjusted based on the learnable normal curvature to increase the receptive field and improve the feature extraction effect of details, thin structures, weak texture areas and non-rigid deformations. Then, the image features of different scales are used to construct a lightweight cost body. The mean measurement method based on group similarity divides elements into different groups using similar groups, and then calculates the mean of each group as the representative value of the group. The cost body established based on this method can be used to evaluate problems of different scales, and has a smaller volume and higher efficiency.

[0042] Then, based on the content-aware cost volume aggregation strategy, the cost volumes of different scales are aggregated into one cost volume to retain details and depth information to the greatest extent. Then, the RNN-CNN hybrid network is used to realize information aggregation and regularization in the spatial context and depth direction to obtain the probability volume. The network can further reduce the network memory consumption by performing regularization in the horizontal and vertical directions at the same time. Then, the Softmax operation is performed on the cost volume along the depth direction to obtain the probability volume P. The value of each voxel in the probability volume represents the probability that the corresponding depth hypothesis plane is the true depth. That is, it represents the probability size that the depth hypothesis plane is most likely to be the true depth. Finally, the mathematical expectation of the pixel depth value is used as the derived depth value to generate the final depth map.

Claims

1. A depth estimation network based on dynamic multi-scale features and content-aware aggregation, characterized in that: The following steps are involved: Step 1: Extract multi-scale deep features of UAV images based on the dynamic multi-scale feature extraction network guided by normal curvature; Step 1.1: Input an image of size H×W and its estimated epipolar lines e to the network, and use the encoder to downsample step by step to extract image features; In the first layer, the scale is set to 3, 5, 7, and 11, the step size is set to 1, and the convolutional neural network is used to extract image features and output an image of size H×W; Step 1.2: In the second layer of the encoder, receive the image input of the first layer with a size of H×W, set the scale to 3, 5, 7, 9, and the step size to 1, use the convolutional neural network to extract image features again, and output an image of size H×W; Step 1.3: In the third layer of the encoder, receive the image input of the previous layer with a size of H×W, downsample it, set the step size to 2, and finally output the size of H / 2×W / 2; Step 1.4: In the fourth layer of the encoder, receive the image input of the previous layer with a size of H / 2×W / 2, set the scale to 3, 5, 7, set the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 2×W / 2; Step 1.5: In the fifth layer of the encoder, receive the image input of the previous layer with a size of H / 2×W / 2, set the scale to 3, 5, 7, set the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 2×W / 2; Step 1.6: In the sixth layer of the encoder, receive the image input of the previous layer with a size of H / 2×W / 2, downsample it, set the step size to 2, and finally output the size of H / 4×W / 4; Step 1.7: In the seventh layer of the encoder, receive the image input of the previous layer with a size of H / 4×W / 4, set the scale to 3, 5, and the step size to 1, use the convolutional neural network to extract image features, and output an image with a size of H / 4×W / 4; Step 1.8: In the eighth layer of the encoder, receive the image input of the previous layer with a size of H / 4×W / 4, set the scale to 3, 5, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 4×W / 4; Step 1.9: At the ninth layer of the encoder, receive the image input of the previous layer with a size of H / 4×W / 4, downsample it, set the step size to 2, and finally output the size of H / 8×W / 8; Step 1.10: The decoder receives the output of the encoder as input, restores the resolution of the feature map through step-by-step upsampling and feature reuse strategies, and outputs the final result; In the first layer, the scale is set to 1, 3, the step size is set to 1, and the convolutional neural network is used to extract image features and output an image of size H / 8 × W / 8; Step 1.11: In the second layer of the decoder, receive the image input of the previous layer with a size of H / 8×W / 8, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 8×W / 8; Step 1.11: In the third layer of the decoder, receive the image input of the previous layer with a size of H / 8×W / 8, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 8×W / 8; Step 1.12: In the fourth layer of the decoder, receive the image input of the previous layer with a size of H / 8×W / 8, upsample it, set the step size to 2, and perform a cross-layer jump connection with the image features output in step 1.8, and finally output an image with a size of H / 4×W / 4; Step 1.13: In the fifth layer of the decoder, receive the image input of the previous layer with a size of H / 4×W / 4, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 8×W / 8; Step 1.14: In the sixth layer of the decoder, receive the image input of the previous layer with a size of H / 4×W / 4, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 4×W / 4; Step 1.15: In the seventh layer of the decoder, receive the image input of the previous layer with a size of H / 4×W / 4, upsample it, set the step size to 2, and perform a cross-layer jump connection with the image features output in step 1.5, and finally output an image with a size of H / 2×W / 2; Step 1.16: In the eighth layer of the decoder, receive the image input of the previous layer with a size of H / 2×W / 2, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 2×W / 2; Step 1.17: In the ninth layer of the decoder, receive the image input of the previous layer with a size of H / 2×W / 2, set the scale to 1, 3, and the step size to 1, use the convolutional neural network to extract image features, and output an image of size H / 2×W / 2; Step 2: Construct the cost volume based on the group similarity mean metric; Step 2.1: Transform the adjacent feature map to the reference view through homography to obtain the transformed feature map; Step 2.2: Group the reference feature map and the transformed feature map by channel, calculate their similarity, and then find the similarity information within each group and perform the average processing; Step 2.3: Concatenate the similarity means of each group into a vector as the output of the cost body; Step 3: After constructing cost volumes of different resolutions based on feature maps of different resolutions, the next step is to upsample the cost volumes of different resolutions to improve the resolution and aggregate them; Step 4: Regularize the cost volume using the RNN-CNN hybrid network; Step 4.1: Slice the cost volume according to the depth dimension, and multiple horizontally parallel convolutional neural networks regularize the slices of their respective dimensions; Step 4.2: The RNN network passes the output of the previous dimension to the next one in the vertical direction; Step 5: Perform Softmax operation along the depth direction, normalize the convolution output, and obtain the probability volume P; Step 6: Use the mathematical expectation of the pixel depth value as the derived depth value to generate the final depth map.

2. The method according to claim 1, characterized in that: Step 1 uses a dynamic multi-scale feature extraction network to extract image features. Based on the learnable normal curvature, the size of the convolution kernel is adaptively adjusted to increase the receptive field and improve the feature extraction effect for details, thin structures, weak texture areas, and non-rigid deformations.

3. The method according to claim 1, characterized in that: Step 2 uses the image features of different scales obtained in step 1 to construct a lightweight cost body. The mean measurement method based on group similarity uses similar groups to divide elements into different groups, and then calculates the mean of each group as the representative value of the group. The cost body established based on this method can be used to evaluate problems of different scales, and has a smaller volume and higher efficiency.

4. The method according to claim 1, characterized in that: In step 3, cost volumes of different scales are aggregated into one cost volume based on the content-aware cost volume aggregation strategy to preserve details and depth information to the greatest extent.

5. The method according to claim 1, characterized in that: In step 4, a hybrid RNN-CNN network is used to aggregate and regularize information in spatial context and depth direction to obtain a probability body. The network can further reduce network memory consumption by performing regularization in both horizontal and vertical directions.

6. The method according to claim 1, characterized in that: Step 5 performs Softmax operation on the cost volume along the depth direction to obtain the probability volume P; the value of each voxel in the probability volume represents the probability that the corresponding depth hypothesis plane is the true depth; that is, it represents the probability that the depth hypothesis plane is most likely to be the true depth.

7. The method according to claim 1, characterized in that: Step 6 uses the mathematical expectation of the pixel depth value as the derived depth value. This method can be effectively applied to deep neural networks.

Images