Multi-view stereo method based on dual uncertainty estimation
By introducing dual uncertainty estimation technology in the multi-view stereo reconstruction method, multi-stage depth map and confidence map are calculated, the problem of low computing cost and efficiency in large-scale image data processing is solved, and high-precision and efficient three-dimensional reconstruction is achieved , suitable for multi-view stereo reconstruction of outdoor scenes.
Patent Information
- Application Number
- CN202410242011.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-04
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-03-04
AI Technical Summary
The existing multi-view stereo reconstruction method has high computing cost, low efficiency and low precision when processing large-scale outdoor scene image data, which is difficult to meet the needs of real-time and high-level applications, especially the high performance requirements for computing equipment and general Insufficient sex.
Using a multi-view stereoscopic method based on dual uncertainty estimation, multi-scale features are calculated by referring to the image and source images, adjacent stages and pairwise stage uncertainty estimators are constructed, multi-stage depth maps and confidence maps are calculated, and no Deterministic loss and error distribution perceived loss, generating high-quality point cloud models.
Improve the accuracy and time efficiency of depth maps, reduce the demand for computing resources, is suitable for high-precision 3D reconstruction of large-scale image data, enhances the model's learning ability for high-uncertainty areas, and meets the outdoor large-scale 3D Reconstruction application requirements.
Smart Images

Figure CN118229872B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to computer vision and image processing technology, and in particular to a multi-view stereo method based on dual uncertainty estimation. Background Art
[0002] Multi-view stereo (MVS) is a technology that calculates a dense point cloud model of a scene or target object from multi-view image data. Existing multi-view stereo reconstruction methods are mainly divided into the following categories: (1) Multi-view stereo reconstruction methods based on seed point diffusion; (2) Multi-view stereo reconstruction methods based on Markov random fields; (3) Multi-view stereo reconstruction methods based on energy optimization, such as the classic graph cut method; (4) Multi-view stereo reconstruction methods based on depth map fusion. Since multi-view stereo reconstruction methods based on depth map fusion have high flexibility and scalability, such methods have attracted widespread attention from researchers in academia and industry, and a large number of novel methods have been proposed. The main focus of these methods is: how to calculate high-quality depth maps, thereby improving the quality of dense point cloud models.
[0003] At present, there are two main methods for calculating depth maps: (1) using stereo matching methods to calculate depth maps; (2) using deep learning methods to calculate depth maps. The former has high time efficiency, and the latter can calculate high-precision depth maps. Existing multi-view stereo reconstruction methods have made great progress in the problem of reconstructing dense point clouds of small-scale scene image data collected by handheld cameras. However, for large-scale outdoor scenes, existing multi-view stereo reconstruction methods need to be further improved.
[0004] In addition, with the popularity and widespread use of mobile devices such as drones, it has become very convenient for people to obtain large-scale outdoor scene image data. These large amounts of outdoor scene image data have brought a series of new challenges to multi-view stereo reconstruction methods: (a) Existing multi-view stereo reconstruction methods based on deep learning technology are very time-consuming. Especially when processing large-scale image data, these methods are difficult to calculate high-quality dense point cloud models within a limited time. Therefore, they cannot meet the real-time requirements of some application software systems in the field of high-level computer vision and computer graphics. (b) Existing multi-view stereo reconstruction methods based on deep learning technology require high-performance computing devices as the basis for training models and running inference, otherwise research and testing work cannot be carried out; (c) The versatility of existing multi-view stereo reconstruction methods based on deep learning technology needs to be further improved and enhanced; for example, some algorithms are only applicable to image data collected by handheld cameras, but it is difficult to obtain expected results on aerial image data, and vice versa.
[0005] In summary, the above problems have seriously hindered the theoretical development and application promotion of multi-view stereo reconstruction methods based on deep learning technology, and revealed the shortcomings of existing multi-view stereo reconstruction methods based on deep learning technology. Therefore, people are in urgent need of a fast and effective multi-view stereo reconstruction method to quickly reconstruct high-quality point cloud models of scenes or target objects from large-scale image data.
[0006] In recent years, deep learning technology has achieved good application results in image recognition, image matching, image rendering, autonomous driving, virtual reality and augmented reality. Inspired by this, some researchers at home and abroad have tried to apply deep learning technology to the problem of multi-view stereo reconstruction, and have made good research progress. The following related research papers have been published: "GeoMVSNet: Learning Multi-View Stereo with Geometry Perception" discloses a multi-view stereo reconstruction method based on geometry perception; "RayMVSNet: Learning Ray-based 1D ImplicitFields for Accurate Multi-View Stereo" discloses a multi-view stereo reconstruction method based on ray implicit fields; "Cascade Cost Volume for High-Resolution Multi-View Stereo and StereoMatching" discloses a multi-view stereo reconstruction and stereo matching method based on cascade structure; the above multi-view stereo reconstruction methods focus on improving the accuracy of the predicted depth map, thereby improving the quality of the point cloud model.
[0007] However, the above methods still face the following challenges when applied to large-scale image data: (1) When processing large-scale image data, the point cloud models output by these methods have low accuracy and are difficult to meet the needs of some high-level application systems, such as unmanned driving, visual navigation, metaverse, virtual reality, digital protection of cultural heritage, digital twins, and impact assessment of natural disasters; (2) Existing multi-view stereo reconstruction methods based on deep learning technology require high-performance computing devices as a basis, and as the amount of image data increases, the performance requirements of these methods for computing devices increase exponentially; (3) The operating efficiency of existing multi-view stereo reconstruction methods when processing large-scale image data is too low, making it difficult to meet the time efficiency requirements of large-scale 3D reconstruction. Summary of the invention
[0008] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and to provide a multi-view stereo method based on dual uncertainty estimation, which utilizes a reference image and a source image to restore a depth map corresponding to the reference image, thereby further making it possible to reconstruct a dense point cloud model of a large-scale outdoor scene.
[0009] Technical solution: A multi-view stereo method based on dual uncertainty estimation of the present invention comprises the following steps:
[0010] Step S1: input the image data of a multi-view scene and the internal and external parameters of the camera, and each scene contains at least 3 images, each image has the internal and external parameters of the camera; each set of image data There are N images with a width of W and a height of H in each image. i represents the sequence number of the image. When i=0, I0 refers to the reference image. refers to the source image, and the camera internal parameters of the i-th image are K i , the external parameters of the camera are T i The image data in the present invention may have a variety of sources, such as image data taken by a mobile device (including a mobile phone, a mobile camera), image data taken by a drone, or satellite image data;
[0011] Step S2, calculating the multi-scale features of the multi-view image, by using the feature pyramid network FPN to extract features of different scales in the image, and each convolution layer uses convolution kernels of different sizes;
[0012] For each image, three different scale image features are extracted The sizes of the three image features are After H×W, these three features are used in different stages of the cascade network structure, s=0, 1, 2 represent the first stage, the second stage and the third stage respectively, and the resolution of the depth map of each stage is H represents the height of the image, and W represents the width of the image;
[0013] Step S3: construct an adjacent stage uncertainty estimator, and use the features obtained in step S2 to construct a probability body to further calculate the multi-scale depth map and confidence map. The specific calculation steps are as follows:
[0014] First, based on the features in step S2 Construct the probability body P0 of the first stage and predict the generation The depth map D0 and confidence map Conf0 of the predicted depth map are obtained; then, the probability body P0 and the predicted depth map D0 are input into the uncertainty estimator of the adjacent stage to generate the depth hypothesis value for the second stage; the above steps are repeated in the third stage, and finally a multi-stage depth map set is obtained. and confidence map collection j represents the sequence number of the stage, s=0, 1, 2 represent the first stage, the second stage and the third stage respectively;
[0015] The adjacent stage uncertainty estimator includes multiple layers of two-dimensional convolution, activation functions and adjustment parameters;
[0016] Step S4: construct a two-stage uncertainty estimator, which includes a multi-layer perceptron (MLP) and an activation function. The specific steps are as follows:
[0017] First, for the multi-stage depth map obtained in step S3 Then, the data after difference pairing are input into the two-stage uncertainty estimator to obtain the uncertainty map U t ;
[0018] Step S5, calculate the mixed loss Loss, Loss = Loss reg +Loss unc +Loss eda ;
[0019] Loss reg is the true depth value and the predicted depth value The depth map regression loss between unc Because of uncertainty graph U t The introduced uncertainty loss, Loss eda Because of uncertainty graph U t The introduced error distribution aware loss;
[0020] Step S6, repeating steps S2 to S5 to calculate all depth maps. After the depth maps of all images are calculated, the depth maps are filtered using photometric consistency and geometric consistency to improve the quality of the depth maps. The depth maps are fused to generate a point cloud model.
[0021] Furthermore, the specific steps of step S3 are:
[0022] Step S3.1, set the depth intervals of the first stage, the second stage and the third stage to L1, L2, L3 respectively, and the depth intervals to M1, M2, M3 respectively; for the first stage, adopt a uniform sampling strategy, starting from the minimum depth value with a depth interval of L1, uniformly sample M1 depth hypothesis values to the maximum depth value, denoted as For the second and third stages, an adaptive sampling strategy (i.e., the sampling strategy in S3.4) is adopted, and the corresponding depth hypothesis values are obtained;
[0023] Use homography to transform all image features Move to the parallel slice of the reference camera frustum, thus obtaining the projected feature volume of each image Therefore, the homography transformation formula of the i-th source view feature when the depth assumption value is d is as follows:
[0024]
[0025] Where N is the number of input images, i is the sequence number of the image, h is the sequence number of the depth hypothesis value, and K is i represents the camera internal parameters of the i-th image, T i is the camera extrinsic parameter of the i-th image, represents the inverse of the camera intrinsic parameters of the i-th image, represents the inverse of the camera extrinsic parameters of the i-th image, s = 0, 1, 2 represent the first stage, the second stage and the third stage respectively;
[0026] Step S3.2: For the projection feature set Each projected feature in {E i}, group them in the channel dimension, the number of groups is g, recorded as calculate and The specific calculation formula is as follows:
[0027]
[0028] Among them, <·,·> represents the dot product, which is used to measure the similarity of two vectors; N represents the number of input images, i represents the sequence number of the image, and g represents the number of channel dimension groups;
[0029] Step S3.3: Use a lightweight cost body regularization network to The first-stage cost body V0 obtained by combining is regularized to obtain the first-stage probability body P0; the probability body P0 and By performing regression operations, the predicted depth map D0 and confidence map Conf0 can be obtained; where i represents the sequence number of the image, h represents the sequence number of the depth hypothesis value, N represents the number of input images, and M1 represents the number of depth intervals in the first stage;
[0030] Step S3.4: Construct a neighbor uncertainty estimator using an 8-layer two-dimensional convolutional network and ReLU activation function.
[0031] Furthermore, the specific process of constructing the adjacent uncertainty estimator in step 3.4 is as follows:
[0032] Step 3.4.1, input the probability body P0 of the first stage into the uncertainty estimator of the adjacent stage, obtain the sampling weight map h by the uncertainty estimator, and use the sampling weight map h to perform point multiplication with the depth interval L2 to obtain a new depth interval NEW_L2; at the same time, because too small a depth interval is not conducive to the recognition of deep convolution features, an adjustment parameter is also used Prevent too small depth intervals; when When setting
[0033] Step 3.4.2: In the first stage, the depth map D0 is predicted to be sampled at the depth interval of NEW_L2 on both sides. The depth hypothesis value is obtained
[0034] Step 3.4.3, repeating steps S3.1 to S3.3 can obtain the predicted depth map D1 and confidence map Conf1 of the second stage;
[0035] Step 3.4.4: The calculation process of the third stage is the same as that of the second stage; wherein h represents the serial number of the depth hypothesis value, and M2 represents the depth interval number of the second stage.
[0036] Furthermore, in step S4, the steps of constructing a two-stage uncertainty estimator and calculating an uncertainty graph are as follows:
[0037] Step S4.1: First, the depth map generated by the cascade structure Upsampling is performed to obtain a depth map of the same resolution The resolution of the depth map is H×W; then, calculate n depth maps with the same resolution The square of the pairwise difference of , therefore, for n depth maps, The difference map is denoted as The calculation formula is as follows:
[0038]
[0039] Among them, t and j represent the lth stage and the jth stage respectively, and when calculating the diff t When l is not equal to j; n represents the number of depth maps generated by the cascade structure; m represents the number of difference maps, t represents the sequence number of difference maps and uncertainty maps; H represents the height of the input image, W represents the width of the input image, and s=0, 1, 2 represent the first stage, the second stage, and the third stage respectively;
[0040] Step S4.2: Use three linear layers, ELU activation function and Dropout to construct a two-stage uncertainty estimator; transform the difference map in S4.1 into Input into the two-stage uncertainty estimator, and finally generate the uncertainty graph U t .
[0041] Furthermore, the mixed loss Loss in step S5 includes three parts, namely, the regression loss of the depth map, the uncertainty loss and the error distribution perception loss. unc The purpose of Loss is to strengthen the model's learning of high uncertainty areas in the predicted depth map; eda Will not determine the figure U t The distribution of predicted errors is compared with the actual error distribution, thereby improving the model's overall perception of the scene);
[0042] The specific calculation process of the three parts is as follows:
[0043] Step S5.1, use L1 loss to measure the error between the real depth map and the predicted depth map. Since the cascade structure used produces depth maps of three scales, different loss weights are used for adjustment at different stages, and the calculation method of regression loss is as follows:
[0044]
[0045] Among them, λ j Indicates the loss weights at different stages, the three stages are 0.5, 1, and 2 respectively; j indicates the sequence number of the stage, L j represents the L1 loss of the jth stage, s = 0, 1, 2 represents the first stage, the second stage and the third stage;
[0046] Step S5.2: Use Laplace distribution to model uncertainty, and then calculate the uncertainty loss for the generated uncertainty graph:
[0047]
[0048] Where M is the number of valid pixels on the depth map, i represents the sequence number of the valid pixel, and D i is the predicted depth value, D_gt i is the true depth value, s i is the value of the corresponding pixel in the uncertain image;
[0049] Step S5.3, calculate the error loss between the true error distribution τ and the error distribution σ predicted by the two-stage uncertainty estimator. The specific calculation method is as follows:
[0050]
[0051] Where u represents the number of groups in the histogram, k represents the kth group in the histogram, and H τis the histogram of the true error, H σ is the histogram of prediction error, H τ (k) represents the kth group of data in the true error distribution histogram, H σ (k) represents the kth group of data of the prediction error distribution histogram.
[0052] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0053] (1) The present invention embeds the uncertainty estimator of the adjacent stage in the cascade structure, which not only retains the memory saving advantage of the cascade structure, but also reduces the dependence of the fine stage on the coarse stage, making the final predicted depth map more accurate.
[0054] (2) The present invention uses a two-stage uncertainty estimator to mine the valuable information contained in the multi-stage depth maps. The uncertainty loss and error distribution perception loss are introduced to strengthen the model's learning of high uncertainty areas in the predicted depth map and improve the model's overall perception of the scene.
[0055] (3) The present invention can not only solve the storage space overflow problem caused by the existing multi-view stereo reconstruction method based on deep learning technology when processing outdoor large-scale image data, but also improve the time efficiency of 3D reconstruction, which provides a new direction for the application of outdoor large-scale image data in the field of high-precision 3D reconstruction and multi-view stereo reconstruction. Figure 3 It has laid an important foundation for the development of maintenance and reconstruction technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 It is a schematic diagram of the overall process of the present invention;
[0057] Figure 2 is a sample schematic diagram of large-scale image data in an embodiment;
[0058] Figure 3 is a sample schematic diagram of a depth map in an embodiment;
[0059] Figure 4 Schematic diagram of the point cloud model output in the embodiment;
[0060] Figure 5 is a flow chart of uncertainty estimator in adjacent stages in the embodiment;
[0061] Figure 6 Flow chart of the two-stage uncertainty estimator in the embodiment. DETAILED DESCRIPTION
[0062] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.
[0063] Existing multi-view methods based on deep learning usually use a cascade structure. However, these methods often only use the depth map generated by the fine stage to fuse and generate point clouds, without mining the detailed information contained in the multi-stage depth maps. At the same time, when using this structure, the fine stage heavily relies on the accurate depth map predicted by the coarse stage for depth sampling. To solve the above problems, the present invention provides a multi-view stereo method based on dual uncertainty estimation. First, the multi-scale features of the multi-view image are calculated based on the input multi-view image and camera parameters; then, an adjacent stage uncertainty estimator is constructed to calculate the multi-stage depth map and confidence map; secondly, a pairwise stage uncertainty estimator is constructed to calculate the uncertainty map; thirdly, the mixed loss is calculated based on the uncertainty map and the multi-stage depth map; finally, the depth map is fused to calculate a high-quality point cloud.
[0064] The present invention fully exploits the potential of the deep neural network model with a cascade structure in multi-view stereo reconstruction, and reduces errors and improves the accuracy of the depth map by embedding adjacent stage uncertainty estimators and two-stage uncertainty estimators in the cascade structure, thereby generating high-quality point clouds. Since the network structure of the present invention is a coarse-to-fine cascade structure, it includes three stages. The front stage first generates a low-resolution depth map, and the rear stage gradually optimizes and generates a depth map corresponding to the final reference image. The adjacent stage uncertainty estimator is used between stages to produce a more reasonable depth interval, so that the depth map generated in the subsequent stage is more accurate. The two-stage uncertainty estimator compares the depth maps generated by all stages, extracts effective information from them, generates an uncertainty map, and introduces two new loss functions from the uncertainty map to improve the overall performance of the network.
[0065] In the above three-stage cascade structure, the first stage adopts a uniform sampling strategy, and the second and third stages both use the adaptive sampling strategies generated by the uncertainty estimators of the adjacent stages.
[0066] like Figure 1 As shown, the multi-view stereo method based on dual uncertainty estimation of this embodiment includes the following steps:
[0067] Step S1: input the image data of a multi-view scene and the internal and external parameters of the camera, and each scene contains at least 3 images, each image has the internal and external parameters of the camera; each set of image data There are N images with a width of W and a height of H in each image. i represents the sequence number of the image. When i=0, I0 refers to the reference image. refers to the source image, and the camera internal parameters of the i-th image are K i , the external parameters of the camera are T i ;
[0068] Step S2, calculating the multi-scale features of the multi-view image, by using a feature pyramid network to extract features of different scales in the image, and each convolution layer uses a convolution kernel of different sizes;
[0069] For each image, three different scale image features are extracted The sizes of the three image features are After H×W, these three features are used in different stages of the cascade network structure, s=0, 1, 2 represent the first stage, the second stage and the third stage respectively, and the resolution of the depth map of each stage is
[0070] Step S3: construct an adjacent stage uncertainty estimator and calculate the multi-scale depth map and confidence map. The specific calculation steps are as follows:
[0071] First, based on the image features in step S2 Construct the probability body P0 of the first stage and predict the generation The depth map D0 and confidence map Conf0 of the predicted depth map are obtained; then, the probability body P0 and the predicted depth map D0 are input into the uncertainty estimator of the adjacent stage to generate the depth hypothesis value for the second stage; the above steps are repeated in the third stage, and finally a multi-stage depth map set is obtained. and confidence map collection j represents the sequence number of the stage, s=0, 1, 2 represent the first stage, the second stage, and the third stage respectively;
[0072] Step S4: construct a two-stage uncertainty estimator, which includes a multi-layer perceptron MLP and an activation function. The specific steps are:
[0073] First, for the multi-stage depth map obtained in step S3 Then, the data after difference pairing are input into the two-stage uncertainty estimator to obtain the uncertainty map U t ;
[0074] Step S5, calculate the mixed loss Loss, Loss = Loss reg +Loss unc +Loss eda ;
[0075] Loss reg is the true depth value and the predicted depth value The depth map regression loss between unc Because of uncertainty graph U t The introduced uncertainty loss, Loss eda Because of uncertainty graph U tThe introduced error distribution aware loss;
[0076] Step S6: Repeat steps S2 to S5 to calculate all depth maps, and fuse the depth maps to obtain a point cloud model.
[0077] like Figure 5 As shown, the adjacent stage uncertainty estimator of this embodiment includes multiple layers of two-dimensional convolution, activation function and adjustment parameters The specific steps of multi-scale depth map and confidence map are:
[0078] Step S3.1, set the depth intervals of the first stage, the second stage and the third stage to L1, L2, L3 respectively, and the depth intervals to M1, M2, M3 respectively; for the first stage, adopt a uniform sampling strategy starting from the minimum depth value with a depth interval of L1, uniformly sample M1 depth hypothesis values to the maximum depth value, denoted as For the second and third stages, the sampling strategy in S3.4 is adopted, and the corresponding depth hypothesis values are obtained;
[0079] Use homography to transform all image features Move to the parallel slice of the reference camera frustum, thus obtaining the projected feature volume of each image Therefore, the homography transformation formula of the i-th source view feature when the depth assumption value is d is as follows:
[0080]
[0081] Where h represents the sequence number of the depth hypothesis value, represents the inverse of the camera intrinsic parameters of the i-th image, represents the inverse of the camera extrinsic parameters of the i-th image;
[0082] Step S3.2: For the projection feature set Each projected feature in {E i}, group them in the channel dimension, the number of groups is g, recorded as calculate and The specific calculation formula is as follows:
[0083] Where <·,·> represents the dot product, and g represents the number of channel dimension groups;
[0084] Step S3.3: Use a lightweight cost body regularization network to The first-stage cost body V0 obtained by combining is regularized to obtain the first-stage probability body P0; the probability body P0 and Perform regression operations to obtain the predicted depth map D0 and confidence map Conf0;
[0085] Step S3.4: Construct a neighbor uncertainty estimator using an 8-layer two-dimensional convolutional network and ReLU activation function.
[0086] like Figure 6 As shown, the steps of the two-stage uncertainty estimator and the calculation of the uncertainty graph of this embodiment are as follows:
[0087] Step S4.1: First, the depth map generated by the cascade structure Upsampling is performed to obtain a depth map of the same resolution The resolution of the depth map is H×W; then, calculate n depth maps with the same resolution The square of the pairwise difference of , therefore, for n depth maps, The difference map is denoted as The calculation formula is as follows:
[0088]
[0089] Among them, l and j represent two different stages; n represents the number of depth maps generated by the cascade structure; m represents the number of difference maps;
[0090] Step S4.2: Use three linear layers, ELU activation function and Dropout to construct a two-stage uncertainty estimator; transform the difference map in S4.1 into Input into the two-stage uncertainty estimator, and finally generate the uncertainty graph U t .
[0091] Example
[0092] In this embodiment, the sample of the input image data is as follows: Figure 2 As shown, Figure 2 The four images in the figure are all reference images. The two reference images in the first row belong to the same scene, but have different viewing angles. The two reference images in the second row belong to another scene, but have different viewing angles. The corresponding depth maps after processing are shown in Figure 1. Figure 3 As shown, for Figure 2 The corresponding point cloud model of the two scenes in the final output is as follows Figure 4 As shown, it can be seen that the dense point cloud model reconstructed from the image data has high geometric consistency and texture consistency with the real outdoor scene.
[0093] It can be seen from the above embodiments that the present invention first inputs a multi-view image and corresponding camera parameters; then, calculates a multi-scale depth map of the multi-view image; secondly, constructs an adjacent stage uncertainty estimator, calculates a multi-scale depth map and a confidence map; thirdly, constructs a two-stage uncertainty estimator, and calculates an uncertainty map; thirdly, calculates a mixed loss based on the uncertainty map and the multi-stage depth map; finally, the depth map is fused to obtain a high-quality point cloud model (such as Figure 4 shown).
[0094] The present invention embeds adjacent stage uncertainty estimators and two-stage uncertainty estimators in a cascaded multi-view stereo method, reduces the adverse effects of the cascade structure, and obtains valuable information between multi-stage depth maps. Thus, a more accurate depth map is calculated and a final high-quality point cloud model is generated.
[0095] According to the final experimental results ( Figure 4 ) It can be seen that the present invention can not only improve the time efficiency of the outdoor large-scale multi-view stereo reconstruction process, but also avoid consuming a large amount of storage space, making it possible to quickly reconstruct a high-quality point cloud model of a large-scale outdoor scene.
[0096] The evaluation results of the present invention and COLMAP on the Tanks and Temples dataset are compared. The experimental results are shown in Table 1. Table 1 shows the quantitative results of different methods on the Tanks and Temples dataset. The evaluation index is the mean F value (the higher the better).
[0097] Table 1 Comparison of evaluation results between the present invention and the prior art
[0098]
[0099] The existing multi-view stereo method (COLMAP) calculates depth maps from images of different perspectives, and then uses the depth map to map the pixels in the image to three-dimensional space through back-projection to generate a dense three-dimensional point cloud. Although this existing solution can generate relatively stable reconstruction results, it is computationally intensive and difficult to handle large-scale scenes. At the same time, the multi-view stereo method based on deep learning can better handle low-light and texture-free areas, forming a higher quality point cloud model.
Claims
1. A multi-view stereo method based on dual uncertainty estimation, characterized in that: The following steps are involved: Step S1: input the image data of a multi-view scene and the internal and external parameters of the camera, and each scene contains at least 3 images, each image has the internal and external parameters of the camera; each set of image data There are N images with a width of W and a height of H in each image. i represents the sequence number of the image. When i=0, I0 refers to the reference image. refers to the source image, and the camera internal parameters of the i-th image are K i , the external parameters of the camera are T i ; Step S2, calculating the multi-scale features of the multi-view image, by using the feature pyramid network FPN to extract features of different scales in the image, and each convolution layer uses convolution kernels of different sizes; For each image, three different scale image features are extracted The sizes of the three image features are After H×W, these three features are used in different stages of the cascade network structure, s=0, 1, 2 represent the first stage, the second stage and the third stage respectively, and the resolution of the depth map of each stage is Step S3: construct an adjacent stage uncertainty estimator, which includes an 8-layer two-dimensional convolutional network, an activation function and adjustment parameters. The specific calculation steps of the multi-scale depth map and confidence map are as follows: First, based on the image features in step S2 Construct the probability body P0 of the first stage and predict the generation The depth map D0 and confidence map Conf0 of the proposed method are obtained. Then, the probability body P0 and the predicted depth map D0 are input into the adjacent stage uncertainty estimator to generate a depth hypothesis value for the second stage. In the third stage, the above steps are repeated to finally obtain a multi-stage depth map set. and confidence map collection j represents the sequence number of the stage, s=0, 1, 2 represent the first stage, the second stage, and the third stage respectively; Step S4: construct a two-stage uncertainty estimator, which includes a multi-layer perceptron MLP and an activation function. The specific steps are: First, for the multi-stage depth map obtained in step S3 Then, the data after difference pairing are input into the two-stage uncertainty estimator to obtain the uncertainty map U t , where t represents the serial number of the uncertain graph; Step S5, calculate the mixed loss Loss, Loss = Loss reg +Loss unc +Loss eda ; Loss reg is the true depth value and the predicted depth value The depth map regression loss between unc Because of uncertainty graph U t The introduced uncertainty loss, Loss eda Because of uncertainty graph U t The introduced error distribution aware loss; Step S6: Repeat steps S2 to S5 to calculate all depth maps, and fuse the depth maps to obtain a point cloud model.
2. The multi-view stereo method based on dual uncertainty estimation according to claim 1, characterized in that: The specific steps of step S3 for calculating the multi-scale depth map and the confidence map are as follows: Step S3.1, set the depth intervals of the first stage, the second stage and the third stage to L1, L2, L3 respectively, and the depth intervals to M1, M2, M3 respectively; for the first stage, adopt a uniform sampling strategy starting from the minimum depth value with a depth interval of L1, uniformly sample M1 depth hypothesis values to the maximum depth value, denoted as For the second and third stages, an adaptive sampling strategy is used to obtain the corresponding depth hypothesis values; Use homography to transform all image features Move to the parallel slice of the reference camera frustum, thus obtaining the projected feature volume of each image Therefore, the homography transformation formula of the i-th source view feature when the depth assumption value is d is as follows: Where h represents the sequence number of the depth hypothesis value, represents the inverse of the camera intrinsic parameters of the i-th image, represents the inverse of the camera extrinsic parameters of the i-th image; Step S3.2: For the projection feature set Each projected feature in {E i }, group them in the channel dimension, the number of channel dimension groups is g, denoted as calculate and The specific calculation formula is as follows: Among them, <·,·> represents the dot product; Step S3.3: Use a lightweight cost body regularization network to The first-stage cost body V0 obtained by combining is regularized to obtain the first-stage probability body P0; the probability body P0 and Perform regression operation to obtain the predicted depth map D0 and confidence map Conf0; Step S3.4: Construct a neighbor uncertainty estimator using an 8-layer two-dimensional convolutional network and ReLU activation function.
3. The multi-view stereo method based on dual uncertainty estimation according to claim 2, characterized in that: The specific process of constructing the adjacent stage uncertainty estimator in step S3.4 is as follows: Step 3.4.
1. Input the probability body P0 of the first stage into the uncertainty estimator of the adjacent stage, obtain the sampling weight map h by the uncertainty estimator, and use the sampling weight map h to multiply the depth interval L2 to obtain the new depth interval NEW_L2; at the same time, use the adjustment parameter Prevent too small depth intervals; when When setting Step 3.4.2: In the first stage, the depth map D0 is predicted to be sampled at the depth interval of NEW_L2 on both sides. The depth hypothesis value is obtained Step 3.4.3, repeat steps S3.1 to S3.3 to obtain the predicted depth map D1 and confidence map Conf1 of the second stage; Step 3.4.4: The calculation process of the third stage is the same as that of the second stage.
4. The multi-view stereo method based on dual uncertainty estimation according to claim 1, characterized in that: In step S4, the steps of constructing a two-stage uncertainty estimator and calculating an uncertainty graph are as follows: Step S4.1: First, the depth map generated by the cascade structure Upsampling is performed to obtain a depth map of the same resolution The resolution of the depth map is H×W; then, calculate n depth maps with the same resolution The square of the pairwise difference of , therefore, for n depth maps, The difference map is denoted as The calculation formula is as follows: Among them, l and j represent the lth stage and the jth stage respectively, and when calculating the diff t When l is not equal to j; n represents the number of depth maps generated by the cascade structure; m represents the number of difference maps; t represents the sequence number of difference maps and uncertainty maps; Step S4.2: Use three linear layers, ELU activation function and Dropout to construct a two-stage uncertainty estimator; transform the difference map in S4.1 into Input into the two-stage uncertainty estimator, and finally generate the uncertainty graph U t .
5. The multi-view stereo method based on dual uncertainty estimation according to claim 1, characterized in that: The mixed loss Loss in step S5 includes three parts: regression loss of the depth map, uncertainty loss and error distribution perception loss. The specific calculation process of the three parts is as follows: Step S5.1, using L1 loss to measure the error between the true depth map and the predicted depth map, the regression loss is calculated as follows: Among them, λ j represents the loss weights at different stages, which are 0.5, 1, and 2 respectively; j represents the sequence number of the stage, L j represents the L1 loss of the jth stage, s=0, 1, 2 represents the first stage, the second stage and the third stage; Step S5.2: Use Laplace distribution to model uncertainty, and then calculate the uncertainty loss for the generated uncertainty graph: Where M is the number of valid pixels in the depth map, i represents the image number, and D i is the predicted depth value, D_gt i is the true depth value, s i is the value of the corresponding pixel in the uncertain image; Step S5.3, calculate the error loss between the true error distribution τ and the error distribution σ predicted by the two-stage uncertainty estimator. The specific calculation method is as follows: Where u represents the number of groups in the histogram, k represents the kth group in the histogram, and H τ is the histogram of the true error, H σ is the histogram of prediction error, H τ (k) represents the kth group of data in the true error distribution histogram, H σ (k) represents the kth group of data of the prediction error distribution histogram.
