Multi-view stereo reconstruction method based on adaptive learning and aggregation

Abstract

The present invention relates to a multi-view stereo reconstruction method based on adaptive learning and aggregation, comprising the following steps: S1. Feature extraction, extracting image features from an input image and removing unimportant information from a large amount of information; S2. Matching cost construction, calculating the matching cost between each pixel in a reference camera and the corresponding matching pixel in its adjacent camera under each sampling depth assumption; S3. Matching cost regularization, denoising the cost volume in the matching cost; S4. Depth map estimation, weighting the result of the regularized matching cost using function regression to obtain an initial depth map; S5. Depth map optimization, reducing the effects of oversmoothing on the edges of the initial depth map. The present invention aims to provide a multi-view stereo reconstruction method based on adaptive learning and aggregation, which has higher reconstruction accuracy and completeness than MVSNet and significantly reduces time consumption and graphics card memory consumption.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-view stereo reconstruction method based on adaptive learning and aggregation. Background Art

[0002] Multi-view stereo (MVS) is an important branch of computer vision. Its main content is to reconstruct a high-precision 3D geometric model of the observed scene or object based on a series of known images and the corresponding camera poses and intrinsic parameters (internal and external parameters of the camera).

[0003] The reconstruction methods of MVS mainly include voxel-based methods, point cloud-based methods and depth map-based methods. The voxel-based method divides the 3D space into a regular grid and estimates whether each voxel is attached to the surface. The main disadvantages are spatial discreteness errors and huge memory consumption; the point cloud-based method operates directly on the point cloud, usually relying on a propagation strategy to gradually make the reconstruction denser. Since the propagation of the point cloud is carried out step by step and cannot be parallelized, the reconstruction takes a long time; the depth map-based method uses the estimated depth map as an intermediate layer to decompose the complex MVS problem into smaller depth estimation problems for each view, and only focuses on one reference image and several source images at a time, and then regresses and fuses each depth map to form the final 3D point cloud model.

[0004] Nowadays, there are many MVS reconstruction methods based on depth maps. The technologies with good performance are mainly based on deep neural networks. The only difference between them is the network structure and other details. The more popular MVS networks based on deep learning are DeepMVS, MVSNet, R-MVSNet, CVP-MVSNet, M3VSNet, etc. They all have their own advantages and disadvantages. For example, DeepMVS network is the first open source MVS reconstruction system based on deep learning. Its advantage is that it divides image serialization into patches, which enables it to better handle areas with weak textures and thin structures. The disadvantage is that the patch-based method causes the network to lose the global information of the image during training and cannot handle scenes with rich details.

[0005] MVSNet is an MVS reconstruction network proposed by Professor YaoYao of the Hong Kong University of Science and Technology in 2018. Although this algorithm has fast reconstruction speed, strong generalization ability, and can reconstruct good results without fine-tuning, it does not take into account the pixel visibility between input images and the high consumption of video memory, resulting in incomplete reconstructed scenes and a high demand for video memory. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-view stereo reconstruction method based on adaptive learning and aggregation, which has higher reconstruction accuracy and completeness than MVSNet and greatly reduces time consumption and graphics card memory consumption.

[0007] The object of the present invention is achieved through the following technical solution: a multi-view stereo reconstruction method based on adaptive learning and aggregation, which comprises the following steps:

[0008] S1. Feature extraction: extract image features from the input image, remove unimportant information from the large amount of information, and retain only the key information of the scene;

[0009] S2. Matching cost construction, under each sampling depth hypothesis, calculate the matching cost between each pixel in the reference camera and the corresponding matching pixel in its neighboring camera;

[0010] S3. Matching cost regularization, denoising the cost volume in the matching cost;

[0011] S4. Depth map estimation: the result of the regularized matching cost is weighted using a regression function to obtain the initial depth map;

[0012] S5. Depth map optimization, reducing the effect of over-smoothing on the edges of the initial depth map;

[0013] In step S1, a multi-scale feature aggregation extractor is constructed. The multi-scale feature aggregation extractor contains three convolutional units. Each convolutional unit consists of three convolutional layers. Except for the last convolutional layer, the first two convolutional layers are followed by a batch normalization layer and a ReLU activation function layer.

[0014] Each input image generates three feature maps of sizes H / 4xW / 4, H / 2xW / 2, and HxW, which are then fed into the deformable convolutional layer. When the feature map of size H / 4xW / 4 is output, the feature maps of the other two scales are downsampled to H / 4xW / 4. The feature maps of the three scales are then concatenated along the feature dimension to obtain a multi-scale aggregated feature map with a resolution of H / 4xW / 4x64.

[0015] The same applies to other resolutions, and the final output feature maps are H / 4xW / 4x64, H / 2xW / 2x64, and HxWx64 respectively.

[0016] Compared with the prior art, the advantages of the present invention are:

[0017] The present invention uses a novel cascade structure to gradually recover from a rough global estimate to a more refined estimate. It changes the feature extraction module, matching cost construction module and matching cost regularization module in the original algorithm, and solves the problems of pixel visibility and high video memory consumption in MVSNet.

[0018] 1. The network uses a cascade structure to first estimate a small-resolution depth map, and then use the obtained small-resolution depth map as prior information to predict a larger-resolution depth map.

[0019] 2. The feature extraction module adds deformable convolution to enable the module to adaptively aggregate feature information of different scales so that the feature results contain more feature information.

[0020] 3. Pixel visibility prediction has been added to the matching cost module. By calculating the occlusion map of pixels in adjacent viewports, visible pixels are retained while occluded pixels are filtered out. Furthermore, the cost volume is constructed by calculating the inner product of the source and reference images instead of using a variance-based approach.

[0021] 4. The matching cost regularization module transforms the 3D convolutional neural network into a Transformer-based VT-UNet network (voxel-based U-shaped self-attention transformation network), greatly reducing the number of model parameters and lowering the consumption of video memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a visual structure diagram of the feature extraction module in the MVSNet network.

[0023] Figure 2 This is a visualization result of the regularization module in the MVSNet network.

[0024] Figure 3 Schematic diagram of a feature extraction module of a multi-view stereo reconstruction method based on adaptive learning and aggregation according to the present invention.

[0025] Figure 4 It is a schematic diagram of the matching cost construction in the present invention.

[0026] Figure 5 Schematic diagram of the matching cost regularization module in the present invention.

[0027] Figure 6 It is a structural diagram of the entire network of the present invention.

[0028] Figure 7 This is a comparison chart of the actual effects of image processing (the left side is the MVSNet reconstruction result, and the right side is the network reconstruction result of the present invention). DETAILED DESCRIPTION

[0029] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0030] like Figure 3-7 FIG2 is a schematic diagram of an embodiment of a multi-view stereo reconstruction method based on adaptive learning and aggregation provided by the present invention.

[0031] A multi-view stereo reconstruction method based on adaptive learning and aggregation, comprising the following steps:

[0032] Step S1. Feature extraction: extract image features from the input image, remove unimportant information from the numerous pieces of information, and retain only the key scene information;

[0033] In step S1, a multi-scale feature aggregation extractor is constructed. The multi-scale feature aggregation extractor contains three convolutional units. Each convolutional unit consists of three convolutional layers. Except for the last convolutional layer, the first two convolutional layers are followed by a batch normalization layer and a ReLU activation function layer.

[0034] Each input image generates three feature maps of size H / 4xW / 4, H / 2xW / 2, and HxW, which are then fed into the deformable convolution layer. When the feature map of size H / 4xW / 4 is output, the feature maps of the other two scales are downsampled to H / 4xW / 4. The feature maps of the three scales are then concatenated along the feature dimension to obtain a multi-scale aggregated feature map with a resolution of H / 4xW / 4x64.

[0035] The same applies to other resolutions. The final output feature maps are H / 4xW / 4x64, H / 2xW / 2x64, and HxWx64 respectively. Figure 3 .

[0036] The feature extraction module in the MVSNet network consists of eight convolutional layers. The specific composition is shown in Table 1. The visual structure diagram of the feature extraction module can be seen in Figure 1 .

[0037] Table 1

[0038]

[0039]

[0040] Step S2. Matching cost construction: Under each sampling depth hypothesis, the matching cost between each pixel in the reference camera and the corresponding matching pixel in its neighboring camera is calculated;

[0041] In step S2, the feature map and the internal and external parameters of the corresponding camera are combined to construct the matching cost in the three-dimensional space using homography transformation;

[0042] Given a reference image I1 and a set of source images S, S is defined as:

[0043]

[0044] Where N represents the number of input images, H and W are the height and width of the images respectively.

[0045] Taking the reference image I1 as the benchmark, the feature map F corresponding to the i-th perspective is transformed using homography i Transformed into a plane parallel to the feature map F1 corresponding to the reference image I1, the homography transformation formula is as follows:

[0046]

[0047] Among them, I1 is the reference image, is the matching view for multi-view stereo matching with I1, are the camera intrinsic parameters, rotation matrix and translation vector corresponding to each view, d is the uniform sampling value within the depth range, n1 is the main axis of the camera corresponding to the reference image, the superscript T is the transpose inside the matrix, and I is the identity matrix.

[0048] For the feature map F1 corresponding to the reference image I1, its homography transformation matrix is ​​a 3x3 identity matrix, which ensures that the homography transformation is universal for all views.

[0049] Then, the homography matrix is ​​used to establish the correspondence between the feature maps of the source image and the reference image. The formula is as follows:

[0050]

[0051] in, Represents the feature map of the source image mapped to the reference image perspective;

[0052] Recalculate The F1 similarity, which measures the degree of match between two views, is defined as the following equation:

[0053]

[0054] Among them, <·,·> represents the inner product, S i The dimensions of (d) are WxHxDxC;

[0055] After obtaining the per-view cost volume, it passes through a three-layer 3D convolutional network, with the last layer outputting a channel dimension of 1. Then, the sigmoid function is used to convert the cost volume into a visibility probability map for each pixel. This operation is used to calculate the visibility of pixels under different viewpoints. The closer the viewpoint of the source image is to that of the reference image, the more identical information the source image and the reference image contain. Therefore, it can be determined that the matching cost volume constructed by this source image and the reference image is more important than other matching cost volumes.

[0056] Finally, each matching cost volume is multiplied by a weight value and then summed to obtain the final aggregated matching cost volume. The aggregated matching cost is defined as:

[0057]

[0058] where w i Indicates the pixel visibility between the source view and the reference view, see Figure 4 .

[0059] The MVSNet network uses homography to transform the feature map F corresponding to the i-th perspective i Mapped to a plane parallel to the feature map F1 corresponding to the reference view I1, in order to be able to adaptively input any number of matching views, when calculating the matching cost based on the feature volume mapping, the matching cost construction of any number of input views is realized by using the variance-based mapping relationship. The calculation formula is as follows:

[0060]

[0061] Where N is the number of input views, is the mean of all features.

[0062] S3. Matching cost regularization, denoising the cost volume in the matching cost;

[0063] Since in real scenes, the surface of objects is not an ideal Lambertian body and there are occlusions between different objects, the matching cost will be inaccurate. Therefore, after obtaining the matching cost, regularization processing is required.

[0064] In the MVSNet network, the matching cost regularization module is composed of a three-dimensional convolutional neural network, and its structure is a U-Net network composed of three-dimensional convolutional layers. The specific composition is shown in Table 2 below. Figure 2 This is the visual result diagram of the MVSNet network regularization module.

[0065] Table 2

[0066]

[0067]

[0068] In step S3 of the present invention, a voxel-based U-shaped self-attention transform network is used to regularize the matching cost. This module abandons the original 3D convolutional neural network and instead uses a Transformer (self-attention transform) to build a U-shaped network to regularize the matching cost. While improving the accuracy of the matching cost, it also significantly reduces the demand for video memory.

[0069] We introduced VT-UNet (Volumetric TransformerUNetwork, voxel-based U-type self-attention transformation network), which is built on the basis of Transformer and can directly process 3D matching cost volume. The model architecture of VT-UNet is as follows Figure 5 . The overall segmentation process refers to UNet (U-shaped convolutional neural network), but the difference is that VT-UNet has designed a brand new Encoder (encoder) and Decoder (decoder). VT-Encoder (voxel self-attention transform encoder) can effectively learn local and global information. VT-Encoder uses a two-layer (flipped) parallel structure similar to VT-Decoder (voxel self-attention transform decoder) to fuse high-resolution information from the encoder and low-resolution information from the decoder, restore the features lost during the downsampling process, and improve segmentation accuracy. In addition, a unique jump connection mechanism is used to make the fusion of low-resolution and high-resolution features more efficient, while maintaining a low number of parameters and computational complexity.

[0070] S4. Depth map estimation: the result of the regularized matching cost is weighted using a regression function to obtain the initial depth map;

[0071] By processing the matching cost regularization result with softmax along the depth dimension, we can obtain the probability distribution of depth estimation at different depth sampling values, which is called the probability body P. For any point (x, y, d) on the probability body P, the corresponding value represents the confidence of this depth estimation value. When the probability body is known, the simplest method is to directly estimate the depth map according to the "winner takes all" method. However, this method cannot estimate the depth value at the sub-pixel level, and it is impossible to obtain the optimal result through error back propagation in the network. Therefore, the softargmin function is used to process the probability body. The principle is to calculate the weighted average at each depth. The specific formula is as follows:

[0072]

[0073] Among them, d min and d maxDenote the minimum depth and maximum depth respectively, d is the number of uniform samples within the depth range, and P(d) is the probability estimate corresponding to the depth sampling value d.

[0074] S5. Depth map optimization, reducing the effect of over-smoothing on the edges of the initial depth map;

[0075] After performing pixel-by-pixel depth estimation, an initial predicted depth map is obtained. However, due to the large receptive field of deep learning networks, the boundaries of the initial depth map D may be too smooth. This module uses the input image as a guide and utilizes a deep residual network to fine-tune the initial estimated depth map, thereby improving the depth estimation results at the boundaries. Specifically, the module first concatenates the initial estimated depth map D and the reference view into a 4-channel input, which is then fed into a deep residual network consisting of three 32-channel convolutional layers and one 1-channel convolutional layer for learning.

[0076] After the depth map is optimized, it can also be trained through the loss function module. This module uses the L1 function as the loss function. During network training, the loss function is used to measure the difference between the current output result and the expected result, and backpropagation is performed to learn the parameters of the network model. The loss function formula is as follows:

[0077]

[0078] Among them, p is a set of useful pixels, d(p) is the true depth value corresponding to pixel p, is the depth value of the corresponding initial depth map, is the depth value of the corresponding optimized depth map, μ1 and μ2 are weight coefficients, which measure the influence of the initial depth map and the optimized depth map on network learning.

[0079] The algorithm flow of the present invention is intuitively described as follows:

[0080] The network of the present invention aims to infer the depth map of the reference image with the help of camera parameters. Figure 6 The detailed structural diagram of the network is given. The network of the present invention utilizes a cascade structure (from rough estimation to fine optimization) to generate depth maps in stages, and the depth map generated by the previous layer is used as a priori hypothesis for the estimation of the next layer. At each level, multiple per-view cost volumes are constructed based on all reference-source image pairs. The pixel visibility between each image pair is then calculated, and this result is weighted and summed with the corresponding cost volume to obtain the aggregated matching cost.

[0081] After the matching cost is input into the regularization module, the regularized result is processed by the soft-argmin function to obtain the probability distribution of each pixel at different depth sampling values, which is called the probability body P. For any point (x, y, d) on the probability body P, its corresponding value represents the confidence of the pixel at a certain depth. Finally, the final depth estimation map is obtained by weighted regression of the probability body P and all sampling values ​​within the depth range. After obtaining the depth map of each angle, after filtering outliers through photometric consistency and geometric consistency, the depth maps of all angles are fused and converted into a point cloud to obtain the three-dimensional reconstruction result of the scene. Note that except for the depth range of the first stage, which is predefined, the depth range of the remaining stages is obtained from the depth map generated in the previous stage.

Claims

1. A multi-view stereo reconstruction method based on adaptive learning and aggregation, comprising the following steps: S1. Feature extraction: extract image features from the input image, remove unimportant information from the large amount of information, and retain only the key information of the scene; S2. Matching cost construction, under each sampling depth hypothesis, calculate the matching cost between each pixel in the reference camera and the corresponding matching pixel in its neighboring camera; S3. Matching cost regularization, denoising the cost volume in the matching cost; S4. Depth map estimation: the result of the regularized matching cost is weighted using a regression function to obtain the initial depth map; S5. Depth map optimization, reducing the effect of over-smoothing on the edges of the initial depth map; It is characterized in that, in step S1, a multi-scale feature aggregation extractor is constructed, and the multi-scale feature aggregation extractor includes three convolution units, each of which is composed of three convolution layers. Except for the last convolution layer, the first two convolution layers are followed by a batch normalization layer and a ReLU activation function layer. Each input image generates three feature maps of sizes H / 4xW / 4, H / 2xW / 2, and HxW, which are then fed into the deformable convolutional layer. When the feature map of size H / 4xW / 4 is output, the feature maps of the other two scales are downsampled to H / 4xW / 4. The feature maps of the three scales are then concatenated along the feature dimension to obtain a multi-scale aggregated feature map with a resolution of H / 4xW / 4x64. The same applies to other resolutions, and the final output feature maps are H / 4xW / 4x64, H / 2xW / 2x64, and HxWx64 respectively; In step S2, the feature map and the internal and external parameters of the corresponding camera are combined to construct the matching cost in the three-dimensional space using homography transformation; Given a reference image And a set of source images S, S is defined as: ； Where N represents the number of input images, H and W are the height and width of the images respectively; Reference image As a benchmark, the feature map corresponding to the i-th perspective is transformed using homography Transformed to the reference image The corresponding feature map For parallel planes, the homography transformation formula is as follows: ； in, is the reference image, For Matching views for multi-view stereo matching, are the camera parameters, rotation matrix and translation vector corresponding to each view, d is the uniform sampling value within the depth range, is the principal axis of the camera corresponding to the reference image, the superscript T is the transpose inside the matrix, and I is the identity matrix; Then, the homography matrix is ​​used to establish the correspondence between the feature maps of the source image and the reference image. The formula is as follows: ； in, Represents the feature map of the source image mapped to the reference image perspective; Recalculate and Similarity, which measures how well two perspectives match, is defined as follows: ； in, represents the inner product, The dimensions are WxHxDxC; After obtaining the per-view cost volume, it passes through a three-layer 3D convolutional network, with the output channel dimension of the last layer being 1. Then, the cost volume is converted into a visibility probability map for each pixel through a sigmoid function. Finally, each matching cost volume is multiplied by a weight value and then summed to obtain the final aggregated matching cost volume. The aggregated matching cost is defined as: ； in Indicates the pixel visibility between the source view and the reference view.

2. The multi-view stereo reconstruction method based on adaptive learning and aggregation according to claim 1, characterized in that: In step S3, a voxel-based U-shaped self-attention transformation network is used to regularize the matching cost.

Images