A multi-view stereo vision reconstruction method based on deep learning

Abstract

The present invention discloses a multi-view stereo vision reconstruction method based on deep learning, which belongs to the field of machine vision technology. The present invention constructs a network model that is more suitable for multi-view image three-dimensional reconstruction by changing the feature extraction network structure, designing a contribution algorithm, and obtaining hidden spatial information, thereby solving the problems of insufficient reconstruction accuracy and completeness. The improved MVSNet method of the present invention achieved Acc and Comp of 0.473 and 1.304 on the DTU dataset, with better accuracy and completeness; the method is suitable for complex large-scale image three-dimensional reconstruction environments, and can be well used in reverse engineering, ancient cultural relics restoration and other fields.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision, and in particular relates to a multi-view stereo vision reconstruction method based on deep learning. Background Art

[0002] With the continuous development of computer technology, three-dimensional models are valued and applied in more and more information fields, such as digital city construction, virtual reality (VR) technology, reverse engineering, restoration of ancient cultural relics and autonomous driving systems, so the research and development of three-dimensional reconstruction technology is a general trend. Among the many three-dimensional reconstruction methods, the three-dimensional reconstruction technology using multi-view image depth map has become one of the most mainstream research directions due to its low equipment cost, high operational flexibility and good reconstruction accuracy.

[0003] For the 3D reconstruction technology of multi-view image depth map, it is generally divided into traditional computer vision-based methods and deep learning-based methods. The basic idea of ​​the traditional method is to find the matching pixel of each pixel point and obtain the depth information of this point through disparity. The specific method is to use manually designed features to calculate the cost of stereo matching, aggregate the cost, calculate and optimize the disparity, and finally obtain the depth value. It can achieve good 3D reconstruction effect in ideal Lambertian scenes, but for areas with sparse texture and non-diffuse reflection, dense matching will be difficult to perform due to the difficulty of local feature extraction, so the reconstruction result is often poor. However, the deep learning method is based on a large amount of training data and automatically learns the high-level and global features of the input image through a deep convolutional neural network. It can comprehensively learn the information in the image and abstract it into high-level semantic features. Compared with traditional methods, the extracted features are more semantic and avoid human errors. Although some recent studies have shown that image depth estimation technology based on deep learning can effectively improve the quality of reconstruction, there is still room for further improvement in accuracy and completeness. Summary of the invention

[0004] The current mainstream three-dimensional reconstruction technology of multi-view image depth maps based on deep learning often has problems with insufficient reconstruction accuracy and completeness. The present invention provides a multi-view stereo vision reconstruction method based on deep learning.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A multi-view stereoscopic vision reconstruction method based on deep learning, the reconstruction method comprising the following steps:

[0007] Step 1, process the DTU data set;

[0008] Step 2, design the network;

[0009] Step 3, read the data set in step 1 into the designed network in step 2 for training, and obtain the trained network weights;

[0010] Step 4: read the network weights trained in step 3 into the network designed in step 2, read the scene information to be predicted into the network for calculation, obtain the depth map, probability map and mask map of each picture, and generate point cloud data for each predicted scene;

[0011] Step 5: Analyze the accuracy and completeness of the point cloud data of each prediction scene to obtain the average accuracy and completeness.

[0012] Furthermore, the specific method of processing the DTU data set in step 1 is: normalizing the image data using Python language, and converting the data types of other data to obtain a processed data set.

[0013] Furthermore, the specific method of designing the network in step 2 is: it also includes the following steps:

[0014] Step 2.1 Design a special three-layer feature extraction pyramid network;

[0015] Step 2.2: Design a contribution algorithm to calculate the contribution for each feature body;

[0016] Step 2.3: Design a Gaussian process regression algorithm and perform 3D regularization on the Cost volume.

[0017] Further, the step 2.1 designs a special three-layer feature extraction pyramid network. The specific method is as follows: the feature extraction network is a special three-layer pyramid model. The difference from the ordinary three-layer pyramid model is that a gradient image of the source image with the same image resolution obtained by a downsampling function is added after each layer, wherein the input image is downsampled using the F.interpolate() function, and the third and fourth dimensions of the input image are respectively calculated using the np.gradient() and np.abs() functions, and the final gradient image is obtained by the summation function;

[0018] The first layer superimposes the gradient image of the source image with the same image resolution in the channel dimension to obtain an input with 6 channels, followed by two 3×3 convolution kernels with a stride of 1, 8 channels, and an output size of 512×640;

[0019] The second layer first uses a convolution kernel with a stride of 2, a convolution kernel of 5×5, and 16 channels, and then superimposes the gradient image of the source image with the same resolution as the image in the channel dimension to obtain an input with 19 channels. Then, two convolution kernels with a stride of 1, a convolution kernel of 3×3, and 16 channels are used, and the output size is 256×320.

[0020] The third layer first uses a convolution kernel with a stride of 2, a convolution kernel of 5×5, and 32 channels, and then superimposes the gradient image of the source image with the same resolution as the image in the channel dimension to obtain an input with 35 channels, followed by two convolution kernels with a stride of 1, a 3×3 kernel, 32 channels, and an output size of 128×160; each convolution operation is followed by a batch operation and a nonlinear activation function, and finally a feature map is obtained.

[0021] Furthermore, the step 2.2 designs a contribution algorithm, and the specific method for calculating the contribution for each feature body is as follows: when the cost volume is formed, the feature map is transformed by a differentiable homography to obtain the feature body, which results in the same source image pixel on the same depth plane. The corresponding pixel points are different due to the different camera postures of the feature map, and based on the different views of the visibility of the three-dimensional point at the same coordinate from different viewpoints, that is, different views should make different contributions to the matching cost, a contribution algorithm is designed to calculate the contribution body for each feature body. First, use the torch.abs() and torch.sub() functions to obtain the absolute difference between the feature volume of the source image and the feature volume of the reference image, and use the slicing function and reshape(1,b*c*h*w) to process all the feature volume difference data to obtain the two-dimensional tensor data (1,b*c*h*w) of the face set of all feature volume differences in each depth plane, and then use the torch.cat() function to connect on the first dimension to obtain data of dimension (l,b*c*h*w), and use the softmax() function and torch.neg() function to calculate the opposite of the tensor, and then use the unsqueeze(-1) and reshape(b,c,h,w) functions twice to obtain the four-dimensional tensor (b,c,h,w), and form the contribution body corresponding to the feature body respectively, and then multiply it with the face patch of the corresponding feature body to obtain feature bodies with different contributions, and finally fuse them into a new Cost volume in the form of variance.

[0022] Furthermore, the step 2.3 designs a Gaussian process regression algorithm, and the specific method for 3D regularization of the Cost volume is as follows: first, design a distance function pose_distance(pose1, pose2), use the slicing function to obtain the rotation matrix R and the translation matrix t from the camera pose pose, and use np.linalg.norm() and np.matrix.trace() to perform different calculations on t and R respectively, and finally get the distance between the two camera poses, so as to calculate the distance D between the camera poses of all read pictures, with a dimension of (n, n); secondly, design a radial basis kernel function K, define three hyperparameters, and use np.expand_dims() to expand the camera pose distance D into three dimensions (1, n, n), so as to obtain K, and use torch.sum() to add the Cost volume of all pictures in the third dimension to obtain a tensor with a dimension of (b, l, h, w), and then use torch.stack() to add all Cost The volume is connected in the second dimension to obtain Y with dimension (b, n, l, h, w), and the view(b, l, -1) function is used on Y so that Y can be adaptively used as the result. Finally, the relationship function between the camera posture distance D and the Cost volume is obtained to predict the hidden spatial information brought by different camera postures, enriching the Cost volume. Finally, the Cost volume is 3D regularized to obtain a depth map.

[0023] Furthermore, the specific method of generating the point cloud data of each predicted scene in step 4 is as follows: after obtaining the depth map and probability map of each image, a simple version of the depth map filter / fusion is performed to obtain the fused point cloud data.

[0024] The method of the present invention is applicable to three-dimensional reconstruction projects of a large number of images, such as cultural relic restoration, reverse engineering, etc.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] When the MVSNet network reconstructs an image, the reconstruction accuracy and completeness are related to the quality of the image. The reconstruction effect is often not good for image scenes with strong illumination and large posture changes, which is reflected in the lack of reconstruction accuracy and completeness. The present invention studies the depth estimation of multi-view images and the improvement of the accuracy of depth estimation. It proposes an improved MVSNet algorithm based on deep learning. By improving a single feature extraction network module, the first-order gradient image of the image is added to the feature extraction network module to extract features that are less affected by illumination. At the same time, a contribution algorithm is designed to obtain a new Cost volume to improve the accuracy of depth estimation. In addition, a Gaussian process regression algorithm is used to design a spatial information acquisition module to predict the hidden spatial information brought by different camera postures, thereby improving the accuracy of the depth map and having a good reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is the network design diagram of this method;

[0028] Figure 2 It is the structural diagram of feature extraction module;

[0029] Figure 3 It is the flow chart of contribution calculation module;

[0030] Figure 4 It is the flow chart of spatial information acquisition module;

[0031] Figure 5 2 is an example diagram of the reconstruction result of the present invention. DETAILED DESCRIPTION

[0032] Example 1

[0033] Step 1, data set processing: Use Python to process the public data set DTU, read in images, camera parameters, mask images, etc., and convert them into single-precision tensor data types to adapt to network reading.

[0034] Step 2: read the training set after data processing in step 1 into the designed network for training: 79 scenes, i.e. 27,097 images, are used as the training set, 18 scenes, i.e. 6,174 images, are used as the test set, batch size is 4, numdepth = 192, learning rate is 0.001, and weight decay rate is 0. Finally, stop after 16 iterations.

[0035] The network structure is designed as follows: First, a three-layer feature extraction pyramid network is designed. In the first step, the input image is downsampled using the F.interpolate() function, and the third and fourth dimensions of the input image are calculated using the np.gradient() and np.abs() functions, respectively, and the final gradient image is obtained by the summation function; In the second step, a three-layer feature extraction pyramid network is designed; The first layer superimposes the gradient image of the source image with the same image resolution in the channel dimension to obtain an input with 6 channels, followed by two 3×3 convolution kernels with a step size of 1, 8 channels, and an output size of 512×640. The second layer first uses a convolution kernel with a step size of 2, a convolution kernel of 5×5, and 16 channels, and then superimposes the gradient image of the source image with the same image resolution in the channel dimension to obtain an input with 19 channels, followed by two convolution kernels with a step size of 1, a convolution kernel of 3×3, and 16 channels, and an output size of 256×320. The third layer first uses a convolution kernel with a stride of 2, a convolution kernel of 5×5, and 32 channels, and then superimposes the gradient image of the source image with the same resolution as the image in the channel dimension to obtain an input with 35 channels, followed by two convolution kernels with a stride of 1, a 3×3 convolution kernel, 32 channels, and an output size of 128×160. Each convolution operation is followed by a batch operation and a nonlinear activation function, and finally a feature map is obtained.

[0036] Secondly, a contribution algorithm is designed. When the cost volume is formed, the feature map is transformed into a feature body through a differentiable homography. This results in the same source image pixel on the same depth plane corresponding to different pixels due to different camera poses of the feature map. In addition, based on the different views of the three-dimensional point at the same coordinate, different views should make different contributions to the matching cost. A contribution algorithm is designed to calculate the contribution body for each feature body. First, use the torch.abs() and torch.sub() functions to obtain the absolute difference between the feature volume of the source image and the feature volume of the reference image, and use the slicing function and reshape(1,b*c*h*w) to process all the feature volume difference data to obtain the two-dimensional tensor data (1,b*c*h*w) of the face set of all feature volume differences in each depth plane, and then use the torch.cat() function to connect on the first dimension to obtain data of dimension (l,b*c*h*w), and use the softmax() function and torch.neg() function to calculate the opposite of the tensor, and then use the unsqueeze(-1) and reshape(b,c,h,w) functions twice to obtain the four-dimensional tensor (b,c,h,w), and form the contribution body corresponding to the feature body respectively, and then multiply it with the face patch of the corresponding feature body to obtain feature bodies with different contributions, and finally merge them into a new Costvolume in the form of variance.

[0037] Before performing 3D regularization on the Cost volume, a spatial information acquisition module is designed. The Gaussian process regression algorithm is used to obtain hidden spatial information using camera pose information and Cost volume information. First, a distance function pose_distance(pose1, pose2) is designed, and the slicing function is used to obtain the rotation matrix R and translation matrix t from the camera pose pose, and np.linalg.norm() and np.matrix.trace() are used to perform different calculations on t and R, and finally the distance between the two camera poses is obtained. The distance D between the camera poses of all read images is calculated in this way, and the dimension is (n, n); secondly, a radial basis kernel function K is designed, three hyperparameters are defined, and np.expand_dims() is used to expand the camera pose distance D into three dimensions (1, n, n) to obtain K, and torch.sum() is used to add the Cost volumes of all images in the third dimension to obtain a tensor with a dimension of (b, l, h, w), and then torch.stack() is used to stack all Cost. The volume is connected in the second dimension to obtain Y with dimensions (b, n, l, h, w), and the view(b, l, -1) function is applied to Y so that Y can be adaptively used as the result. Finally, the relationship function between the camera posture distance D and Costvolume is obtained to predict the hidden spatial information brought by different camera postures.

[0038] Step 3: Input the test image into the network for prediction: Select 22 scenes, i.e. 7564 images, from the remaining scenes of the DTU dataset for prediction, with a batch size of 2 and numdepth = 96. Finally, a point cloud data file is obtained.

[0039] Step 4: Analyze the obtained point cloud data file to obtain the final average acc and comp data: By using the standard MatLab evaluation code, calculate Acc and Comp for the point cloud model data of all scenes, and finally obtain the accuracy, completeness and overall score of the reconstructed point cloud of all predicted scenes to evaluate the accuracy of the depth map.

[0040] Table 1 Efficiency analysis

[0041]

[0042] Table 1 shows the quantitative results of the reconstruction quality of the four methods on the DTU dataset (the lower the better). It can be seen that when numdepth=96, in terms of Acc (accuracy), the accuracy of the original MVSNet algorithm is 0.496, the accuracy of the R-MVSNet algorithm is 0.478, the accuracy of the Point-MVSNet algorithm is 0.462, and the accuracy of the improved method of the present invention is 0.483, which is better than the accuracy of the original MVSNet algorithm and the R-MVSNet algorithm; in terms of Comp (completeness), the integrity of the original MVSNet algorithm is 1.378, the integrity of the R-MVSNet algorithm is 1.341, the integrity of the Point-MVSNet algorithm is 1.326, and the integrity of the improved method of the present invention is 1.304, which is the best among the four methods; in terms of Overall (average accuracy), the average accuracy of the original MVSNet algorithm is 0.937, the average accuracy of the R-MVSNet algorithm is 0.910, the average accuracy of the Point-MVSNet algorithm is 0.894, and the average accuracy of the improved method of the present invention is 0.889, which is the best among the four methods.

[0043] The present invention studies the reconstruction of multi-view images, the problems of insufficient reconstruction accuracy and completeness, and proposes an improved MVSNet algorithm based on deep learning. By modifying the feature extraction module, designing the contribution calculation module and the spatial information acquisition module, a network model is constructed to improve the accuracy of the depth map, and the depth estimation ability of the image is enhanced. The experimental results show that the network model proposed by the present invention effectively solves the problem of poor accuracy and completeness when reconstructing images, and has a good reconstruction effect.

[0044] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For ordinary technicians in the field of this technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A multi-view stereo vision reconstruction method based on deep learning, characterized by: The reconstruction method comprises the following steps: Step 1, process the DTU data set; Step 2, design the network; Step 3, read the data set in step 1 into the network designed in step 2 for training, and obtain the trained network weights; Step 4: read the network weights trained in step 3 into the network designed in step 2, read the scene information to be predicted into the network for calculation, obtain the depth map, probability map and mask map of each picture, and generate point cloud data for each predicted scene; Step 5, analyzing the accuracy and completeness of the point cloud data of each prediction scene to obtain the average accuracy and completeness; The specific method of designing the network in step 2 is: it also includes the following steps: Step 2.1 Design a special three-layer feature extraction pyramid network; Step 2.2: Design a contribution algorithm to calculate the contribution for each feature body; Step 2.3: Design a Gaussian process regression algorithm and perform 3D regularization on the Cost volume. The specific method of designing a special three-layer feature extraction pyramid network in step 2.1 is as follows: the first layer superimposes the gradient image of the source image with the same image resolution in the channel dimension to obtain an input with 6 channels, and then uses two 3×3 convolution kernels with a step size of 1, the number of channels is 8, and the output size is 512×640; the second layer first uses a convolution kernel with a step size of 2, a convolution kernel of 5×5, and the number of channels is 16, and then superimposes the gradient image of the source image with the same image resolution in the channel dimension to obtain an input with 19 channels, and then uses Two convolution kernels with a stride of 1, a convolution kernel of 3×3, and 16 channels, with an output size of 256×320; the third layer first uses a convolution kernel with a stride of 2, a convolution kernel of 5×5, and 32 channels, and then superimposes the gradient image of the source image with the same resolution as the image in the channel dimension to obtain an input with 35 channels, followed by two convolution kernels with a stride of 1, a convolution kernel of 3×3, and 32 channels, with an output size of 128×160; each convolution operation is followed by a batch operation and a nonlinear activation function, and finally a feature map is obtained; The step 2.2 designs a contribution algorithm, and the specific method for calculating the contribution for each feature body is as follows: first, use the torch.abs() and torch.sub() functions to obtain the absolute difference between the feature body of the source image and the feature body of the reference image, and use the slicing function and reshape(1,b*c*h*w) to process all feature body difference data to obtain a two-dimensional tensor data (1,b*c*h*w) of the face set of all feature body differences in each depth plane, and then connect them on the first dimension through the torch.cat() function to obtain data with a dimension of (l,b*c*h*w), and calculate the opposite number of the tensor through the softmax() function and the torch.neg() function, and then obtain a four-dimensional tensor (b,c,h,w) through two unsqueeze(-1) and reshape(b,c,h,w) functions, and respectively form the contribution body corresponding to the feature body, and then multiply it with the face patch of the corresponding feature body to obtain feature bodies with different contributions, and finally merge them into a new Costvolume in the form of variance; The specific method of designing a Gaussian process regression algorithm in step 2.3 to perform 3D regularization on the Cost volume is as follows: first, design a distance function pose_distance(pose1, pose2), use the slicing function to obtain the rotation matrix R and the translation matrix t from the camera pose pose, and use np.linalg.norm() and np.matrix.trace() to perform different calculations on t and R respectively, and finally obtain the distance between the two camera poses, thereby calculating the distance D between the camera poses of all read images, with a dimension of (n, n); secondly, design a radial basis kernel function K, define three hyperparameters, and use np.expand_dims() to expand the camera pose distance D into three dimensions (1, n, n), thereby obtaining K, and use torch.sum() to add the Cost volumes of all images in the third dimension to obtain a tensor with a dimension of (b, l, h, w), and then use torch.stack() to stack all Cost The volume is connected in the second dimension to obtain Y with dimensions (b, n, l, h, w), and the view(b, l, -1) function is used on Y so that Y can be adaptively used as the result. Finally, the relationship function between the camera posture distance D and the Cost volume is obtained to predict the hidden spatial information brought by different camera postures, enriching the Cost volume. Finally, the Cost volume is 3D regularized to obtain a depth map.

2. The multi-view stereo vision reconstruction method based on deep learning according to claim 1, characterized in that: The specific method of processing the DTU data set in step 1 is: normalizing the image data using Python language and converting the data types of other data to obtain the processed data set.

3. The multi-view stereo vision reconstruction method based on deep learning according to claim 1, characterized in that: The specific method of generating the point cloud data of each predicted scene in step 4 is as follows: after obtaining the depth map and probability map of each image, a simple version of the depth map filter / fusion is performed to obtain the fused point cloud data.

Images