A 3D reconstruction method considering multi-view fusion strategy

Abstract

The present invention discloses a 3D reconstruction method considering a multi-view fusion strategy, comprising: in a first stage, N-1 source feature maps and a reference feature map are subjected to homography transformation and distortion to construct a cost volume, each cost volume is transmitted to View-Net, a weight map of the view relative to the reference map is obtained, the weight map is weightedly fused with the cost volume, and finally a cost volume with useful information elements is obtained; the cost volume is regularized by conventional 3D convolution, and a low-resolution depth map is output, the depth information of the depth map will provide initialization for the preset depth in the next stage, and after three stages, a final predicted depth map is obtained to complete the 3D reconstruction; the present invention provides a new expression method for the cost volume fusion of multiple views, improves the availability of matched pixels and reduces the interference of unmatched pixels, thereby obtaining a more accurate depth estimation map and improving the completeness of the 3D point cloud reconstruction.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a three-dimensional reconstruction method considering a multi-view fusion strategy. Background Art

[0002] As people's demand for images and scene recognition continues to increase, three-dimensional space vision is becoming more and more important. Through the observation of three-dimensional space, various scene structures and detail differences that are missing in the two-dimensional plane can be obtained, making the user experience more real and reliable. Conventional three-dimensional reconstruction equipment can reconstruct three-dimensional space from pictures taken by cameras, but its price is very expensive, not enough to support daily use, and there is no way to popularize it to the public.

[0003] Multi-view stereo (MVS) aims to recover 3D scene geometry from a set of RGB images using known camera poses and obtain a 3D dense model of the real-world scene from multiple images. It has many important applications, such as artifact reconstruction, virtual reality, autonomous driving, defect detection, etc. Compared with traditional MVS methods that use hand-crafted matching metrics for image consistency checks, deep learning-based MVS methods tend to use frontal plane scanning, evaluate the depth of the same candidate set based on the same image for each pixel, and achieve higher accuracy and completeness on many MVS benchmarks compared to the state of the art. Although learning-based MVS has achieved unexpected results, there are still many areas that need to be addressed and optimized to further improve the quality of point cloud reconstruction. Convolutional neural networks (CNNs) have been widely used in 3D reconstruction and broader computer vision tasks. Recent multi-view stereo matching algorithms usually compute 3D cost volumes based on a set of hypothesized depths, and apply 3D convolutions to the cost volumes to regularize and regress the final scene depth.

[0004] Many studies have demonstrated the importance of cost volume construction for depth prediction accuracy. From the previous MVS pipeline process, we know that the input is N-1 source images and 1 reference image. The two-view cost volume is constructed through the homography relationship between each source image and the reference image, and then the N-1 two-view cost volumes are compressed into a final cost volume in a specific fusion method. Finally, 3DCNN or RNN regularization is used for depth prediction. Among them, a very important issue is to judge the visibility of pixels in the view and the selection of views. The interference of irrelevant views and the introduction of erroneous pixels will make the verification of the final score invalid or even have a negative impact, resulting in errors in depth estimation.

[0005] The original MVSNet used the traditional view fusion strategy, calculating the variance distribution of the cost bodies of two different views, and using the variance as the weight to combine the cost bodies of different views. Many subsequent variant algorithms also adopted this scheme. Other methods apply operations such as averaging or maximum pooling to sum up the matching costs, or directly use some specific implicit network layers to obtain the so-called adaptive weight map. They have not received a reasonable theoretical explanation. Although the network may implicitly learn how to discard unmatched pixels in the view, the interference of unmatched pixels in the view still inevitably deteriorates the final reconstruction. Summary of the invention

[0006] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art and to provide a three-dimensional reconstruction method that takes into account a multi-view fusion strategy, so as to solve the problem that interference from irrelevant views and the introduction of erroneous pixels in the existing three-dimensional reconstruction will make the verification of the final score invalid or even have a negative impact, thereby causing depth estimation errors.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A 3D reconstruction method considering a multi-view fusion strategy comprises the following steps:

[0009] S1, obtain multiple pictures collected by the camera;

[0010] S2, extract features from the image and obtain feature maps of three different stages;

[0011] S3, using homography transformation to map the N-1 source feature maps in one stage to the hypothesized depth plane of the reference view;

[0012] S4, construct N-1 initial cost volumes based on N-1 source feature maps and one reference view respectively;

[0013] S5, perform adaptive weight training on the N-1 initial cost bodies through View-Net to obtain the weight map corresponding to the initial cost body;

[0014] S6, performing weighted fusion on the weight map and the initial cost volume to obtain the cost volume to be regularized;

[0015] S7, regularizing the cost volume in step S6 to obtain a probability map, and generating a depth map of this stage based on the probability map;

[0016] S8, using the depth map at the stage in step S7 as the initialization of the depth preset of the second stage, and looping through steps S3 to S7 to generate a depth map of the second stage;

[0017] S9, using the depth map obtained in the second stage as the initialization of the depth preset in the third stage, and looping through steps S3 to S7 to generate a final predicted depth map in the third stage;

[0018] S10: Based on the final predicted depth map, complete the three-dimensional reconstruction of the cost volume.

[0019] Furthermore, in step S3, the homography transformation is used to map the N-1 source feature maps in one stage to the assumed depth plane of the reference view to obtain the implicit depth relationship between images of different viewing angles:

[0020]

[0021] in, For the The homography relationship between the feature map of the view and the reference feature map with depth d; K i , R i , t i are the camera intrinsic properties, rotation and translation of the i-th view, respectively. is the principal axis of the reference camera; I, K0, R0 are the remaining common factors extracted from the determinant, the source view camera intrinsic parameters, and the source view rotation and translation extrinsic parameters, respectively.

[0022] Furthermore, step S4 specifically includes:

[0023] According to 1 reference feature map and N-1 source feature maps and the corresponding camera and pose parameters, The Zhang Yuan feature map is transformed homographically to the preset plane of the reference camera to construct N-1 initial cost volumes.

[0024] Furthermore, in step S5, the N-1 initial cost bodies are adaptively weighted through View-Net to obtain a weight map corresponding to the initial cost body:

[0025]

[0026] Among them, V(x) is the view weight map, and exp(-x) is e to the power of -x.

[0027] Furthermore, in step S6, the weight map is weightedly fused with the initial cost volume to obtain the cost volume to be regularized, including:

[0028]

[0029] Among them, Vt otal (k) is the cost volume after fusion in the kth stage, k∈{1, 2, 3}; Warp n(k) is the initial cost volume after homographic distortion of the nth source view and the reference view in the kth stage; View n (k) is the one-dimensional weight information of the nth initial cost body output by View-Net in the kth stage.

[0030] Furthermore, the smooth L1 loss is used to calculate the mean absolute difference between the real depth map and the estimated depth map, and the three-stage losses are accumulated as the final loss Loss:

[0031]

[0032] Where k is the cascade stage, is a set of valid real pixels, d(p, k) is the real depth value of pixel p at the kth stage, is the predicted depth value of pixel p at the kth stage.

[0033] The 3D reconstruction method considering the multi-view fusion strategy provided by the present invention has the following beneficial effects:

[0034] The present invention proposes a universal aggregation network, which will be used as a universal subnetwork of the MVS pipeline network to train the pixel-level confidence of the cost volume of two views, and provide a new expression for the cost volume fusion of multiple views to improve the availability of matching pixels and reduce the interference of unmatched pixels, thereby obtaining a more accurate depth map and improving the completeness of 3D point cloud reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the present invention.

[0036] Figure 2 This is the network architecture of the present invention.

[0037] Figure 3 The present invention maps N-1 source feature maps to the assumed depth plane of the reference view through homography transformation.

[0038] Figure 4 This is a two-view fusion strategy diagram of the present invention.

[0039] Figure 5 This is the two-view weight module of the present invention.

[0040] Figure 6 The corresponding fusion weights are obtained after passing multiple two-view cost bodies through View-Net, and the input is the view of the DTU test set Scan1.

[0041] Figure 7 Line graph for increasing number of views vs. different levels of error accuracy. DETAILED DESCRIPTION

[0042] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0043] Example 1

[0044] This embodiment provides a 3D reconstruction method considering a multi-view fusion strategy, which is used to solve the problem of uncertainty in view selection and pixel homography distortion in the existing method, resulting in the addition of deterioration factors in the cost volume fusion process. Figure 1 , this embodiment specifically includes the following steps:

[0045] Step S1, obtaining multiple pictures captured by a camera;

[0046] refer to Figure 2 ,The input of this embodiment includes multiple captured pictures and the corresponding camera parameters.

[0047] Step S2: extract features from the image to obtain feature maps of three different stages;

[0048] This embodiment uses the FPN architecture as the feature extraction layer to obtain feature maps of different sizes (N×C×H×W) in three stages;

[0049] In all stages of this embodiment, the depth hypothesis is uniformly sampled from a depth range (e.g., 1-192 mm). The first stage acquires image features at low resolution and constructs a cost volume through homography mapping with a predetermined depth range and a larger depth interval, while the subsequent stages use high spatial resolution, a narrower depth range, and a smaller depth interval to obtain a more detailed depth prediction map.

[0050] This embodiment takes the first stage as an example to describe the subnetwork, where the 0th one is the reference feature map, N-1 source feature maps, and the N-1 source feature maps and the reference feature map are respectively subjected to homography transformation and distortion to construct a cost body, and each cost body is passed into View-Net to obtain a weight map of the view relative to the reference map; it is similar to the function of the attention mechanism, and the weight map can suppress non-matching information in the view and enhance the transformable information, as shown in steps S3 to S6:

[0051] Step S3, using homography transformation to map N-1 source feature maps in one stage to the assumed depth plane of the reference view;

[0052] refer to Figure 3According to the epipolar principle, the N-1 source feature maps are mapped to d hypothetical planes of the reference perspective through homography transformation distortion, so as to obtain the implicit depth relationship between images of different perspectives.

[0053] The coordinate mapping is determined by the homography:

[0054]

[0055] in, For the The homography relationship between the feature map of the view and the reference feature map with depth d; in addition, K i , R i , t i are the camera intrinsic properties, rotation and translation of the i-th view, respectively. are the principal axes of the reference camera; then, a differentiable homography is used to warp the 2D feature map into the hypothesized plane of the reference camera to form multiple two-view cost volumes.

[0056] Step S4, constructing N-1 initial cost volumes based on N-1 source feature maps and one reference view respectively;

[0057] refer to Figure 3 The input is 1 reference feature map and N-1 source feature maps and the corresponding camera and pose parameters. The Zhang Yuan feature map is transformed homographically to the preset plane of the reference camera to construct N-1 initial cost volumes.

[0058] Step S5, the weight module of the two views of this embodiment performs adaptive weight training on the N-1 initial cost volumes through View-Net to obtain a weight map (1×H×W) corresponding to the initial cost volume;

[0059] refer to Figure 5 The input of the View-Net module is the two-view cost body. After a Conv3d layer, the number of channels is reduced to 1 / 2 of the original. In order to obtain fast convergence in the initial stage, a BatchNorm3d-Relu is used as the activation layer of this layer. Then, a Conv3d layer is used to reduce the number of channels to one dimension (1×D×H×W). Here, it can be considered that the above operation obtains a temporary cost body x, and it is converted into the final usefulness map (view weight map) V(x)(1×H×W) through the following formula.

[0060]

[0061] Of course, in order to comply with the coarse-to-fine training strategy, three independent View-Nets are set for three different cascade stages. The specific setting parameters are shown in Table 1:

[0062] Table 1. View-Net setting parameters for the three stages. The original resolution of the input here is 512×640. The input obtained in each stage is 1 / 4, 1 / 2, and 1 times of the original resolution.

[0063]

[0064] Step S6: Reference Figure 4 , the two-view fusion strategy of this embodiment performs weighted fusion of the weight map and the initial cost volume to obtain the cost volume to be regularized;

[0065] This step uses unsqueeze dimension multiplication and addition with the cost body, and finally divides by the accumulated amount to obtain the final fused cost body. The specific implementation formula is:

[0066]

[0067] Among them, Warp n (k) is the initial cost volume after homographic distortion of the nth source view and the reference view in the kth stage, View n (k) is the one-dimensional weight information of the nth initial cost body output by View-Net in the kth stage, V total (k) is the cost body k∈{1,2,3} after fusion in the kth stage.

[0068] The fusion strategy of this step can improve the effective matching pixel sampling under different source views, suppress the interference of non-matching pixels, and finally obtain the fusion weight map of the source view and the reference view; and all the cost bodies of the two views are fused with pixel-level weights to reduce the interference information of the final cost body.

[0069] Step S7, regularizing the cost volume in step S6 to obtain a probability map, and generating a depth map of this stage based on the probability map;

[0070] The regularization processing in this step to obtain the probability map and to generate the predicted depth map of this stage based on the probability map are both directly carried out by conventional means, so the specific process will not be described in detail in this embodiment.

[0071] Step S8, using the depth map at the stage in step S7 as the initialization of the depth preset of the second stage, and looping through steps S3 to S7 to generate a predicted depth map of the second stage;

[0072] Step S9, using the depth map obtained in the second stage as the initialization of the depth preset in the third stage, and looping through steps S3 to S7 to generate a final predicted depth map in the third stage;

[0073] Step S10: completing the three-dimensional reconstruction of the cost volume based on the final predicted depth map;

[0074] This step is based on the final predicted depth map to complete the three-dimensional reconstruction of the cost volume, and also adopts conventional means in the art, so the detailed process will not be repeated.

[0075] This embodiment is aimed at three stages, that is, the loss calculation is performed on the three-layer cascade output results. Each layer uses smooth L1 loss to calculate the mean absolute difference between the real depth map and the estimated (predicted) depth map, and the losses of the three stages are accumulated as the final loss Loss:

[0076]

[0077] Where k is the cascade stage, is a set of valid real pixels, d(p, k) is the real depth value of pixel p at the kth stage, is the predicted depth value of pixel p at the kth stage.

[0078] Example 2

[0079] This example is used to evaluate and verify the method steps in Example 1;

[0080] Specifically, this embodiment trains it on the DTU training set; in the training stage, consistent with the conventional MVS operation, the number of input images N=3 is set, and the image resolution is 512×640. For coarse-to-fine regularization, the depth hypothesis is sampled from 425 mm to 935 mm; the number of plane scanning depth hypotheses in each stage is 48, 32, and 8, respectively; the corresponding depth intervals decay by 0.25 and 0.5 from the coarsest stage to the finest stage. The model was trained with Adam for 16 epochs, with an initial learning rate of 0.001, which decayed by 0.5 times after 6, 8, and 12 epochs, respectively, using a batch size of 2 on an NVIDIA RTX 3090 GPU, and one batch occupies 6GB of memory.

[0081] The proposed method is evaluated on the evaluation set of the DTU dataset using the official evaluation criteria. In the evaluation stage, N=5 is set and the input resolution is 864×1152. The quantitative evaluation is shown in Table 2.

[0082] Table 2 Quantitative evaluation of the DTU dataset (the lower the better). The method in this paper surpasses our baseline network and most other advanced methods in terms of completeness.

[0083]

[0084] From the partial multi-view stereo qualitative results on the DTU dataset in the above table, it can be seen that the method of the present invention is significantly denser and more complete in point cloud reconstruction.

[0085] In order to verify the advantages of the fusion strategy proposed in this invention and to intuitively show the contribution of this strategy to MVS, the visibility graph of each two views after passing through View-Net is visualized as follows: Figure 6 As shown in the figure, it is obvious that different perspectives contribute different values ​​to the reference perspective. The brighter area means that there are more matching pixels between this part of the area and the reference view, while the lighter area means that this part of the area has little relationship with the reference view. In other words, the amount of information observed from different perspectives on the same object has different emphases. This is used to allocate the importance of different pixels in the homography distortion of the reference perspective;

[0086] From Vis-MVSNet, we know that a good multi-view fusion strategy should not be affected by the number of views, and the results will not degrade due to the increase of source views. Therefore, the performance of the proposed fusion strategy is tested on large scene and small scene datasets with different numbers of source views.

[0087] Verification tests were conducted on the DTU dataset and BlenderMVS dataset, and the experimental results are shown in Table 3 and Figure 7 As shown in the experiment, it can be seen that with the increase in the number of views, the depth estimation accuracy of this method gradually improves, especially for high-precision ranges such as 2mm and 4mm.

[0088] Table 3 tests the errors of different precisions of the depth maps introduced from 2 to 8 views from left to right.

[0089]

[0090] It can be seen that based on the above verification, the method proposed in the present invention highlights the filtering of the quality of image information, avoids the calculation of invalid image pixels, improves the estimation of correct pixels, and thus improves the completeness of the reconstructed point cloud. The multi-view matching method of the present invention only makes a depth map, and the three-dimensional reconstruction scheme basically uniformly follows a certain open source method, that is, directly adopts conventional means in this field.

[0091] Although the specific implementation of the invention is described in detail in conjunction with the drawings, it should not be understood as limiting the scope of protection of this patent. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this patent.

Claims

1. A 3D reconstruction method considering a multi-view fusion strategy, characterized in that: The following steps are involved: S1, obtain multiple pictures collected by the camera; S2, extract features from the image and obtain feature maps of three different stages; S3, using homography transformation to map the N-1 source feature maps in one stage to the hypothesized depth plane of the reference view; S4, construct N-1 initial cost volumes based on N-1 source feature maps and one reference view respectively; S5, perform adaptive weight training on the N-1 initial cost bodies through View-Net to obtain the weight map corresponding to the initial cost body; S6, performing weighted fusion on the weight map and the initial cost volume to obtain the cost volume to be regularized; S7, regularizing the cost volume in step S6 to obtain a probability map, and generating a depth map of this stage based on the probability map; S8, using the depth map at the stage in step S7 as the initialization of the depth preset of the second stage, and looping through steps S3 to S7 to generate a depth map of the second stage; S9, using the depth map obtained in the second stage as the initialization of the depth preset in the third stage, and looping through steps S3 to S7 to generate a final predicted depth map in the third stage; S10, completing the three-dimensional reconstruction of the cost volume based on the final predicted depth map; In step S3, a homography transformation is used to map the N-1 source feature maps in one stage to the assumed depth plane of the reference view, so as to obtain the implicit depth relationship between images of different viewing angles: in, For the The homography relationship between the feature map of the view and the reference feature map with depth d; , , Respectively i camera intrinsics, rotations and translations of the views, is the principal axis of the reference camera; I , , They are respectively the determinant extraction common factor remainder, source view camera intrinsic parameter, source view rotation and translation extrinsic parameter; In step S5, the N-1 initial cost bodies are trained with adaptive weights through View-Net to obtain a weight map corresponding to the initial cost body: in, is the view weight map, for of Power; In step S6, the weight map and the initial cost volume are weightedly fused to obtain the cost volume to be regularized, including: in, is the cost body after fusion in the kth stage, ; It is Stage The initial cost volume after homographic distortion of the source view and the reference view; For the Phase The one-dimensional weight information of the initial cost body output by View-Net.

2. The three-dimensional reconstruction method considering the multi-view fusion strategy according to claim 1, characterized in that: The step S4 specifically includes: According to 1 reference feature map and N-1 source feature maps and the corresponding camera and pose parameters, The Zhang Yuan feature map is transformed homographically to the preset plane of the reference camera to construct N-1 initial cost volumes.

3. The three-dimensional reconstruction method considering the multi-view fusion strategy according to claim 1, characterized in that: Smooth L1 loss is used to calculate the mean absolute difference between the true depth map and the estimated depth map, and the three-stage losses are accumulated as the final loss Loss: in, For the cascade stage, is the set of effective real pixels, For the Phase Pixels The real depth value, For the Phase Pixels The predicted depth value.

Images