A multi-view three-dimensional reconstruction method based on deep learning

Abstract

The application discloses a kind of multi-view three-dimensional reconstruction methods based on deep learning, comprising: using camera to reconstruct target multi-angle image acquisition and select an image as reference image;The image collected is input into image feature extraction module to generate corresponding feature map;By homography transformation algorithm, the feature map is converted into the feature map under the perspective of reference image, and the cost volume is constructed by pixel-by-pixel calculation image consistency;The cost volume constructed is input into cost volume regularization module to generate the sparse depth map of reference image;Sparse depth map is input into depth map refinement module to generate dense depth map;According to the spatial point cloud structure of dense depth map generation reconstruction target, three-dimensional reconstruction is realized.The above-mentioned method is a kind of deep learning reconstruction method based on image, with high stability, not affected by environmental factors such as illumination intensity, good reconstruction effect, and using end-to-end reconstruction mode, improve reconstruction efficiency.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and image processing technology, specifically to a method for three-dimensional reconstruction of multi-view RGB images based on deep learning. Background Technology

[0002] 3D reconstruction is a classic computer vision problem. Image-based 3D reconstruction algorithms have made great strides in recent years, but many problems still exist and there is room for improvement.

[0003] First, traditional 3D reconstruction methods rely excessively on sophisticated equipment, which is impractical in real-world applications. For example, low lighting conditions, complex target shapes with occlusion, or weak surface texture features can all negatively impact the 3D reconstruction results. Furthermore, traditional 3D reconstruction techniques are not suitable for large-scale scene reconstruction.

[0004] Second, deep learning-based 3D reconstruction algorithms have higher robustness and can effectively avoid the problem of low reconstruction efficiency caused by changes in illumination, such as GCNet proposed by Kendall et al.: Kendall A, Martirosyan H, Dasgupta S, et al. End-to-end learning of geometry and context for deep stereoregression[C] / / Proceedings of the IEEE international conference on computervision.2017:66-75. This algorithm is relatively complex to operate, has a long reconstruction time, poor reconstruction effect, and high requirements for the computing power of the equipment.

[0005] Therefore, there is an urgent need for a robust end-to-end deep learning-based multi-view 3D reconstruction algorithm. Summary of the Invention

[0006] Purpose of the invention: To address the shortcomings of existing technologies, this invention proposes a multi-view 3D reconstruction method based on deep learning, which has high robustness, simple operation, fast reconstruction speed, high efficiency, and good results.

[0007] Technical solution: This invention discloses a multi-view 3D reconstruction method based on deep learning, the 3D reconstruction method comprising the following steps:

[0008] Step 1: Calibrate the camera to obtain relevant parameters, and use the camera to acquire images of the object to be reconstructed from different angles. The images include a reference image and several source images.

[0009] Step 2: Build an image feature extraction module to extract features from the image in Step 1 and obtain a feature map;

[0010] The image feature extraction module mainly includes four downsampling layers and four block blocks. Each of the four downsampling layers uses a 2×2 convolutional layer with a stride of 2 followed by a batch regularization layer, named Conv0, Conv1, Conv2, and Conv3 respectively. The block blocks use a 7×7 dilated convolutional layer followed by two 1×1 convolutional layers, an activation function, and a batch regularization layer, named Block0, Block1, Block2, and Block3 respectively. The overall connection of the image feature extraction module is: Input-Conv0-Block0-Conv1-Block1-Conv2-Block2-Conv3-Block3-Output.

[0011] Step 3: Design a homography transformation algorithm based on feature maps. Perform homography transformation and similarity calculation on the feature maps obtained in Step 2 to construct the cost volume. Perform global variance operation on the cost volume to obtain the probability volume.

[0012] Step 4: Build a cost volume regularization module to perform regularization and regression operations on the probability volume in Step 3 to obtain a sparse depth map of the reference image.

[0013] Step 5: Build a depth map refinement module, stitch together the reference image obtained in Step 1 and the initial depth map in Step 4, and then input the image into the refinement module to generate a dense depth map;

[0014] Step 6: Generate a spatial point cloud structure based on the dense depth map obtained in Step 5 to achieve three-dimensional reconstruction of the target.

[0015] Furthermore, before performing homography transformation in step 3, it is necessary to calculate the homography matrix H between adjacent images based on the acquired camera-related parameters. The formula is as follows:

[0016]

[0017] Among them, K i Let Ri be the intrinsic parameter matrix of the camera, and t be the rotation relationship matrix between the two views. i Let K0 be the translation matrix between views, K0 be the camera intrinsic parameter matrix of the reference image, R0 be the rotation relation matrix of the reference image, t0 be the translation relation matrix of the reference image, and d be the translation relation matrix between views. j Let be the depth value from the j-th depth plane to the reference image, n be the normal vector along the reference image, and I be the identity matrix.

[0018] Furthermore, the specific operation steps in step 3 are as follows:

[0019] Step 3.1: Set the number of depth planes between the reference image and the reconstructed object to 192;

[0020] Step 3.2: Calculate the homography matrix H between adjacent images based on the camera-related parameters obtained in Step 1;

[0021] Step 3.3: Convert the feature map F obtained in Step 2 into a single image. i (i = 1, ..., n) are projected onto each depth plane through the homography matrix H to obtain the viewpoint transformation feature map F′. i The formula is:

[0022] F′ i =H i (d j )×F i (2)

[0023] Step 3.4: Based on the transformed feature map F′ obtained in Step 3.3 i Calculate the similarity between the current viewpoint image and the reference feature map F0, obtain the matching cost, and construct the cost body C, as shown in the formula:

[0024] C i =∑|F′ i -F0|(i=1,...,n) (3)

[0025] Step 3.5: Calculate the global variance of the cost body C to obtain the probability body C. p The formula is:

[0026]

[0027] Where n represents the number of images, It represents the average value in the probability volume.

[0028] Furthermore, the specific structure of the cost body regularization module in step 4 is as follows:

[0029] The cost volume regularization module includes a feature extraction structure, an encoding structure, and a decoding structure;

[0030] The feature extraction structure includes a 3D convolutional layer and an activation function layer;

[0031] The encoder consists of a total of 4 layers. The first two layers are composed of two 3D convolutional layers, and the last two layers are composed of two 3D capsule convolutional layers. Each layer is named en-Conv1, en-Conv2, Caps1, and Caps2 in sequence. The specific connection method of the encoder is: en-Conv1 connects to en-Conv2 connects to Caps1 connects to Caps2.

[0032] The decoder consists of 5 layers. The first 4 layers correspond to the 4-layer structure of the encoder. Layers 1, 3, and 5 are 3D convolutional layers, and layers 2 and 4 are deconvolutional layers. Each layer is named as follows: de-Conv1, Unconv2, de-Conv3, Unconv4, and de-Conv5. The specific connection method is: de-Conv1 connects to Unconv2, de-Conv3 connects to Unconv4, and de-Conv5.

[0033] Furthermore, the overall connection method of the cost body regularization module is to connect the feature extraction structure, the encoding structure and the decoding structure in sequence, that is: the feature extraction structure is connected to en-Conv1, en-Conv2, Caps1, Caps2, de-Conv1, Unconv2, de-Conv3, Unconv4 and de-Conv5. At the same time, the four-layer structure of the encoder is connected to the first four layers of the decoder, that is: en-Conv1 is connected to Unconv4, en-Conv2 is connected to de-Conv3, Caps1 is connected to Unconv2 and Caps2 is connected to de-Conv1.

[0034] Furthermore, the specific operation steps of the depth map refinement module in step 5 are as follows:

[0035] Step 5.1: Perform a stitching operation between the sparse depth map from Step 4 and the reference image from Step 1;

[0036] Step 5.2: Input the stitched image from Step 5.1 into the depth map thinning module to obtain a dense depth map. This module includes four 2D convolutional layers and connects them using a residual structure.

[0037] Beneficial effects

[0038] First: This invention utilizes deep learning to achieve 3D reconstruction of objects. The reconstruction method is an end-to-end generation method, which has high reconstruction efficiency and speed, can generate denser 3D models, and has high robustness and is not sensitive to environmental factors such as lighting. It can be used for indoor scenes and reconstruction targets or smaller scales.

[0039] Second: The feature extraction module of this invention adopts the most advanced deep learning technology, and the feature map it generates contains more global image features.

[0040] Third: In this invention, the cost volume regularization module utilizes capsule networks for decoding, and the dynamic routing mechanism in the capsule network can generate depth maps more accurately. Attached Figure Description

[0041] Figure 1This is an overall flowchart of the three-dimensional reconstruction method of the present invention;

[0042] Figure 2 This is a structural diagram of the image feature extraction module of the present invention;

[0043] Figure 3 Here is a structural diagram of the downsampling layer and block module in the image feature extraction module of this invention;

[0044] Figure 4 This is a structural diagram of the cost body regularization module of the present invention;

[0045] Figure 5 These are comparison images of the three-dimensional reconstruction effects of embodiments of the present invention, where (a) is the Colmap reconstruction effect, (b) is the MVSNet reconstruction effect, and (c) is the reconstruction effect of the present invention. Detailed Implementation

[0046] The present invention will be described in detail below with reference to the accompanying drawings. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0047] See Figure 1 The flowchart of the deep learning-based multi-view 3D reconstruction method provided by this invention is shown below, with specific steps as follows:

[0048] Step 1: Calibrate the camera and calculate its internal and external parameters; after obtaining the relevant parameters of the camera, use the camera to acquire multi-angle images of the object to be reconstructed, and at the same time establish an image set. Select one image from the image set as the reference image, and the rest of the images are the source images. The number of images n in the image set should be greater than or equal to 3 (n≥3). In this invention, the number of images n in the image set is 48 (n=48).

[0049] Step 2: Build an image feature extraction module, such as... Figure 2 As shown, the images from the image set in step 1 are input into the feature extraction module to extract image features, resulting in the depth map F corresponding to the images in the image set. i (i = 0, ..., 47), where F0 is the reference feature map, F i (i = 1, ..., 47) represents the feature map of the source image. The image feature extraction module utilizes a convolutional neural network to extract features from the image, comprising four downsampling layers and four sets of block modules, with the downsampling layers and block modules alternately connected. The image feature extraction module is described in further detail below:

[0050] The image feature extraction module's four downsampling layers each employ a convolutional layer with a stride of 2, followed by a batch regularization layer. For example... Figure 3As shown in (a), the four downsampling layers are named Conv0, Conv1, Conv2 and Conv3, respectively. Conv0 is a 4×4 convolution with a stride of 2, and Conv1, Conv2 and Conv3 are all 2×2 convolutions with a stride of 2. The number of input channels of each downsampling layer is [3, 8, 16, 64], and the corresponding number of output channels is [8, 16, 64, 32].

[0051] like Figure 3 As shown in (b), each of the four Block modules uses a 7×7 dilated convolutional layer, followed by two 1×1 convolutional layers, an activation function, and a batch regularization layer. The four Block modules are named Block0, Block1, Block2, and Block3, respectively. The number of input channels for each Block module is [8, 16, 64, 32], and the corresponding number of output channels is [8, 16, 64, 32].

[0052] The downsampling layers and Block modules are connected alternately, i.e.: Conv0-Block0-Conv1-Block1-Conv2-Block2-Conv3-Block3. The image is input from Conv0 and output from Block3, with an input resolution of 1024×1024 and an output feature map F. i The dimensions are 64×64.

[0053] Step 3: Design a homography transformation algorithm based on feature maps to transform the feature map F obtained in Step 2. i Homography transformation is performed, and pixel-by-pixel similarity is calculated based on the epipolar geometry principle to obtain the matching cost value. This cost value is then used to construct the cost volume, and a global variance operation is performed on the cost volume to obtain the probability volume. Further detailed description follows:

[0054] Step 3.1: Set the number of depth planes between the reference image and the reconstructed object to 192.

[0055] Step 3.2: Calculate the homography transformation H between adjacent images based on the camera-related parameters obtained in Step 1. The homography transformation formula is:

[0056]

[0057] Among them, K i R represents the intrinsic parameter matrix of the camera. i Let t be the rotation relationship matrix between the two views. i Let K0 be the translation matrix between views, K0 be the camera intrinsic parameter matrix of the reference image, R0 be the rotation relation matrix of the reference image, t0 be the translation relation matrix of the reference image, and d be the translation relation matrix between views. jLet be the depth value from the j-th depth plane to the reference image, n be the normal vector along the reference image, and I be the identity matrix.

[0058] Step 3.3: Convert the feature maps F corresponding to each source image obtained in Step 2. i (i = 1, ..., n) are projected one by one onto each depth plane through homography transformation H, resulting in the feature map F′ after view transformation. i The formula is:

[0059] F′ i =H i (d j )×F i (2)

[0060] Step 3.4: Based on the transformed feature map F′ obtained in Step 3.3 i By utilizing the epipolar geometry principle, a successive similarity calculation is performed between the current viewpoint image and the reference feature map F0 to obtain the matching cost value and simultaneously construct the cost body C, as shown in the formula:

[0061] C i =∑|F′ i -F0|(i=1,...,n) (3)

[0062] Step 3.5: Calculate the global variance of the cost body C to obtain the probability body C. p The formula is:

[0063]

[0064] Step 4: Build the cost volume regularization module, such as... Figure 4 As shown, this module is a symmetric encoder-decoder structure. The probability volume C obtained in step 3... p Inputting this regularization module, after regularization and regression operations, yields a sparse depth map of the reference image. The cost volume regularization module is described in further detail below:

[0065] The cost volume regularization module consists of three parts: a feature extraction structure, an encoding structure, and a decoding structure. The feature extraction structure includes a 3D convolutional layer and an activation function layer; the encoder has 4 layers, with inputs of [32, 8, 16, 32] and outputs of [8, 16, 32, 32]. The first two layers use a 3D convolutional network to regularize the cost volume, and the last two layers use a 3D capsule network to extract more depth spatial semantic information. The layers are named en-Conv1, en-Conv2, Caps1, and Caps2, respectively. The decoder has a 5-layer structure. The 1st, 3rd, and 5th layers are 3D convolutional networks, and the 2nd and 4th layers are deconvolutional networks. The number of input channels for the 1st, 3rd, and 5th layers is [32, 32, 16], and the number of output channels is [32, 8, 1]. The number of input channels for the 2nd and 4th layers is [64, 8], and the number of output channels is [16, 8]. The layers are named de-Conv1, Unconv2, de-Conv3, Unconv4, and de-Conv5, respectively.

[0066] The overall connection method of cost volume regularization connects the feature extraction structure, encoding structure, and decoding structure sequentially. The four layers inside the encoder are connected sequentially: en-Conv1 connects to en-Conv2 connects to Caps1 connects to Caps2. The five layers inside the decoder are connected sequentially: de-Conv1 connects to Unconv2 connects to de-Conv3 connects to Unconv4 connects to de-Conv5. At the same time, the four layers inside the encoder are connected to the first four layers inside the decoder in a corresponding manner: en-Conv1 connects to Unconv4, en-Conv2 connects to de-Conv3, Caps1 connects to Unconv2, and Caps2 connects to de-Conv1.

[0067] Step 5: Construct a depth map refinement module. The reference image from Step 1 and the sparse depth map generated in Step 4 are concatenated and input into this module to obtain a dense depth map. This module includes four 2D convolutional layers and preserves image features through residual connections.

[0068] Step 6: Generate a spatial point cloud structure based on the dense depth map obtained in Step 5, and realize the three-dimensional reconstruction of the target through image rendering.

[0069] The following is based on Figure 5 The experiment is illustrated using the image shown in the figure. Figure 5 In the figures, (a) shows the Colmap reconstruction result, (b) shows the MVSNet reconstruction result, and (c) shows the reconstruction result of the present invention. As can be seen from the figures, the present invention outperforms classic reconstruction algorithms such as Colmap and MVSNet in terms of both reconstruction completeness and effectiveness.

[0070] Regarding Table 1, FE is the image feature extraction module used in the MVSNet reconstruction algorithm, and FENet-T is the image feature extraction module in this invention. The feature extraction module of this invention uses a larger convolutional kernel than FE to expand the model's receptive field. Simultaneously, this invention employs grouped convolution and dilated convolution to improve the efficiency of image feature extraction.

[0071] Table 1 Comparison of FE and FENet-T networks

[0072]

[0073] Ablation experiments, also known as comparative experiments, demonstrate the effectiveness of the proposed model by controlling variables. In Table 2, Acc represents accuracy, Comp represents completeness, and Overall represents overall accuracy. Acc, Comp, and Overall are two metrics that measure the ground truth of the point cloud generated by the model. Acc represents the Euclidean distance between the generated point cloud and the ground truth of the point cloud; therefore, the smaller the Acc, the higher the reconstruction accuracy. The same applies to Comp and Overall.

[0074] As shown in Table 2, through the comparative experiments of Group 1 and Group 2 or Group 3 and Group 4, it can be seen that the regularization effect of the cost volume regularization module (3D-CapsCNN) of the present invention is better than that of the cost volume regularization module (UNet) of the MVSNet reconstruction algorithm; through the comparative experiments of Group 1 and Group 3 or Group 2 and Group 4, it can be seen that the image feature extraction effect of the feature extraction module (FENet-T) of the present invention is better than that of the feature extraction module (FE) of the MVSNet reconstruction algorithm.

[0075] Table 2 Comparison of results from ablation experiment 2

[0076]

[0077] Acc represents accuracy, Comp represents completeness, and Overall represents wholeness. Colmap, Tola, Camp, Gipuma, and Furu are traditional classic 3D reconstruction algorithms that achieve 3D reconstruction by manually designing algorithms to detect and match feature points in multiple images. SurfaceNet, MVSNet, and the 3D reconstruction algorithm proposed in this invention are end-to-end multi-view algorithms based on deep learning. Figure 3 3D Reconstruction Algorithm. As can be seen from the table, the 3D reconstruction algorithm proposed in this invention achieves optimal values ​​for Comp and Overal, and performs well in Acc, thus proving the effectiveness of the 3D reconstruction algorithm described in this invention.

[0078] Table 3 Comparison of experimental results

[0079] / mm

[0080]

[0081] It will be understood by those skilled in the art that the above description is merely a preferred embodiment of the invention and is not intended to limit the invention. Although the invention has been described in detail, those skilled in the art can still modify the technical solutions described in the examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention are included within the scope of protection of the invention.

Claims

1. A multi-view 3D reconstruction method based on deep learning, characterized in that, The three-dimensional reconstruction method includes the following steps: Step 1: Calibrate the camera to obtain relevant parameters, and use the camera to acquire images of the object to be reconstructed from different angles. The images include a reference image and several source images. Step 2: Build an image feature extraction module to extract features from the image in Step 1 and obtain a feature map; The image feature extraction module includes four downsampling layers and four block groups. Each of the four downsampling layers uses a step size of 2. Each convolutional layer is followed by a batch regularization layer, named Conv0, Conv1, Conv2, and Conv3 respectively; a single block is used in each block. Hollow convolutional layer, followed by two The convolutional layers, activation functions, and batch regularization layers are named Block0, Block1, Block2, and Block3, respectively. The overall connection of the image feature extraction module is: Input-Conv0-Block0-Conv1-Block1-Conv2-Block2-Conv3-Block3-Output; Step 3: Design a homography transformation algorithm based on feature maps. Perform homography transformation and similarity calculation on the feature maps obtained in Step 2 to construct the cost volume. Perform global variance operation on the cost volume to obtain the probability volume. Step 4: Build a cost volume regularization module to perform regularization and regression operations on the probability volume in Step 3 to obtain a sparse depth map of the reference image. The cost volume regularization module includes a feature extraction structure, an encoding structure, and a decoding structure; The feature extraction structure includes a 3D convolutional layer and an activation function layer; The encoder consists of a total of 4 layers. The first two layers are composed of two 3D convolutional layers, and the last two layers are composed of two 3D capsule convolutional layers. Each layer is named en-Conv1, en-Conv2, Caps1, and Caps2 in sequence. The specific connection method of the encoder is: en-Conv1 connects to en-Conv2 connects to Caps1 connects to Caps2. The decoder consists of 5 layers. The first 4 layers correspond to the 4-layer structure of the encoder. Layers 1, 3, and 5 are 3D convolutional layers, and layers 2 and 4 are deconvolutional layers. Each layer is named as follows: de-Conv1, Unconv2, de-Conv3, Unconv4, and de-Conv5. The specific connection method is: de-Conv1 connects to Unconv2, de-Conv3 connects to Unconv4, and de-Conv5. The overall connection method of the cost body regularization module is to connect the feature extraction structure, the encoding structure and the decoding structure in sequence, that is: the feature extraction structure is connected to en-Conv1, then to en-Conv2, then to Caps1, then to Caps2, then to de-Conv1, then to Unconv2, then to de-Conv3, then to Unconv4, then to de-Conv5. At the same time, the four-layer structure of the encoder is connected to the first four layers of the decoder, that is: en-Conv1 is connected to Unconv4, en-Conv2 is connected to de-Conv3, Caps1 is connected to Unconv2, and Caps2 is connected to de-Conv1. Step 5: Build a depth map refinement module, stitch together the reference image obtained in Step 1 and the initial depth map in Step 4, and then input the result into the depth map refinement module to generate a dense depth map; Step 6: Generate a spatial point cloud structure based on the dense depth map obtained in Step 5 to achieve three-dimensional reconstruction of the target.

2. The deep learning-based multi-view 3D reconstruction method according to claim 1, characterized in that, Before performing homography transformation in step 3, the homography matrix H between adjacent images needs to be calculated based on the acquired camera-related parameters. The formula is as follows: ； in, This represents the intrinsic parameter matrix of the camera. The rotation relationship matrix between the two views is... The translation matrix between views. For the reference image camera intrinsic parameter matrix, For the reference image rotation relation matrix, As the translation relationship matrix of the reference image, Let be the depth value from the j-th depth plane to the reference image, m be the normal vector along the reference image, and I be the identity matrix.

3. The deep learning-based multi-view 3D reconstruction method according to claim 2, characterized in that, The specific steps in step 3 are as follows: Step 3.1: Set the number of depth planes between the reference image and the reconstructed object to 192; Step 3.2: Calculate the homography matrix H between adjacent images based on the camera-related parameters obtained in Step 1; Step 3.3: Convert the feature map obtained in Step 2 into a single image. The viewpoint transformation feature map is obtained by projecting the homography matrix H onto each depth plane. The formula is: ； Step 3.4: Based on the transformation feature map obtained in Step 3.3 Calculate the current viewpoint image and reference feature map The similarity between them is used to obtain the matching cost and construct the cost body C, as shown in the formula: ； Step 3.5: Calculate the global variance of the cost body C to obtain the probability body. The formula is: ； Where n represents the number of images, It represents the average value in the probability volume.

4. The deep learning-based multi-view 3D reconstruction method according to claim 1, characterized in that, The specific operation steps of the depth map refinement module in step 5 are as follows: Step 5.1: Perform a stitching operation between the sparse depth map from Step 4 and the reference image from Step 1; Step 5.2: Input the stitched image from Step 5.1 into the depth map thinning module to obtain a dense depth map. This module includes four 2D convolutional layers and connects them using a residual structure.

Images