A multi-view stereo reconstruction method that cooperates with depth edge and visibility priors

Abstract

The application provides a multi-view stereo reconstruction method cooperating with depth edge and visibility prior, comprising: acquiring multiple images collected from different views of the same scene, taking each image as a reference image and the rest as source images and performing multiple iterations, estimating the depth map of each reference image, each iteration comprising: updating the depth estimation value of reliable pixels in this iteration; reconstructing the depth estimation value of unreliable pixels in this iteration, comprising: dividing the reference image into multiple regions; evaluating whether all pixels in the reference image are visible in the source image; for unreliable pixels, only selecting multiple anchor points from a specified pixel set in the same region as the unreliable pixels for patch deformation to obtain an anchor point set corresponding to the source image, the specified pixel set refers to a pixel set that is visible and reliable relative to the source image; according to the anchor point set corresponding to each source image, reconstructing the depth estimation value of the unreliable pixels; and obtaining the depth map of this iteration based on the depth estimation value of all pixels.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to the fields of three-dimensional reconstruction and multi-view stereo matching, and more particularly to a multi-view stereo reconstruction method based on collaborative depth edges and visibility priors. BACKGROUND

[0002] Multi-View Stereo (MVS) is a core task in computer vision, aiming to densify the geometric representation of scenes or objects from some overlapping photos taken from different perspectives. It has been widely applied in various fields such as virtual reality, autonomous driving, object detection, etc. In recent years, the emergence of a large number of imaginative ideas has greatly improved its performance in various benchmark tests such as the ETH3D dataset. These innovations can be roughly divided into learning-based MVS and traditional MVS.

[0003] Learning-based MVS uses convolutional neural networks to extract high-dimensional features for reconstruction, but they require a large amount of training dataset and have poor generalization ability. In contrast, traditional MVS is extended from the PatchMatch algorithm, which propagates appropriate hypotheses from neighboring pixels and generates refined hypotheses to construct the solution space, and then selects the best hypothesis through the standard of multi-view matching cost definition. However, when the patch is located in a textureless area, its matching cost becomes unreliable due to the lack of distinguishable features within the receptive field.

[0004] In order to reconstruct textureless areas, two main types of traditional MVS are proposed: planarization-based and patch deformation-based methods. Planarization-based methods connect reliable pixels in well-textured areas to form regions that divide textureless areas, and then planarize these regions for reconstruction. Various strategies have been adopted, such as the superpixel TSAR-MVS algorithm, the triangulation ACMMP algorithm, and the KD-tree HPM-MVS algorithm, but these methods are usually affected by the size limit of connected regions and are prone to bias in the planarization process.

[0005] Different from the patch deformation-based methods, which deform the fixed patches to introduce enough features to reduce the matching cost. For example, API-MVS introduces entropy to increase the patch size and sampling interval, while APD-MVS algorithm adaptively searches for relevant reliable pixels around each unreliable pixel, and then constructs multiple sub-patches centered on them to reduce the matching cost. However, such methods mainly focus on exploring advanced reliable pixel search strategies to alleviate the matching ambiguity. When reconstructing unreliable red pixels (i.e., pixels with matching ambiguity), the traditional patch deformation-based methods usually divide the surrounding area of the unreliable red pixel into multiple fixed-angle sectors, and then search for reliable gray pixels within each sector to form a patch. However, in the process of searching for reliable pixels, unexpected edge jumps caused by shadows or occlusions destroy the depth continuity principle, which may lead to potential matching distortion.

[0006] Therefore, in the existing patch deformation methods, on the one hand, especially in the textureless area, the reliable pixels that are not continuous with the depth of the unreliable pixels are selected as anchor points for patch deformation, which makes the deformed patch inaccurate and leads to potential estimation bias. On the other hand, due to the pixel difference caused by complex view changes and occlusions, the deformed patch often contains pixels that do not exist in the image of other views, which increases the matching cost and leads to potential reconstruction distortion.

[0007] It should be noted that the background art is only used to introduce the related information of the present application, so as to help understand the technical solutions of the present application, but does not mean that the related information must be prior art. In the absence of evidence that the related information has been disclosed before the filing date of the present application, the related information should not be regarded as prior art. SUMMARY

[0008] Therefore, the purpose of the present application is to overcome the defects of the prior art, and to provide a multi-view stereo reconstruction method based on depth edge and visibility prior.

[0009] The purpose of the present application is achieved by the following technical solutions:

[0010] According to a first aspect of the present invention, a multi-view stereo reconstruction method is provided, the method comprising: S1, acquiring multiple images of the same scene captured from different viewpoints, taking each of the multiple images as a reference image and the remaining images as source images corresponding to the reference image; S2, performing multiple iterations based on the reference image and each source image to estimate a depth map of each reference image, wherein each iteration process comprises: S21, evaluating whether the depth estimation value of each pixel in the depth map obtained this time is reliable, and obtaining reliable pixels and unreliable pixels, wherein the initialized depth map is used for the first time, and then each time the depth map obtained by upsampling the previous depth map is used; S22, updating the depth estimation value of each reliable pixel by using the PatchMatch algorithm; S23, reconstructing the depth estimation value of each unreliable pixel by using a preset deformation algorithm, comprising: S231, dividing the reference image into different multi-view stereo images; S24 regions, each region is estimated to be a depth-continuous region; S232, evaluate whether all pixels in the reference image are visible in each source image, and obtain the visible pixels and invisible pixels of the reference image relative to each source image; S233, for each unreliable pixel and each source image, select multiple anchor points only from the specified pixel set in the same area as the unreliable pixel for patch deformation, and obtain the anchor point set corresponding to each unreliable pixel relative to each source image, wherein the specified pixel set refers to the set of pixels that are visible and reliable relative to the source image; S234, reconstruct the depth estimation value of each unreliable pixel according to the anchor point set corresponding to each unreliable pixel relative to each source image; S24, obtain the depth map of this time based on the depth estimation value of all pixels, and use the depth map obtained in the last iteration as the depth map of the reference image; S3, fuse the depth maps of all reference images to obtain the three-dimensional point cloud data of the scene.

[0011] In some embodiments of the present invention, in S231, the method of dividing the reference image into multiple different areas includes: using a depth estimation model to perform depth estimation on the reference image to obtain an initial depth map; using an edge detection algorithm to perform edge detection on the reference image to obtain an initial edge map; aligning the initial depth map and the initial edge map to generate a boundary image, which includes multiple different areas, and all pixels in each area have continuous depths in the corresponding initial depth map.

[0012] In some embodiments of the present invention, the initial edge map includes information about each initial scattered area, and the alignment processing method includes: dividing the initial scattered area according to the initial depth map and the information about the initial scattered area according to a preset division rule to obtain multiple areas, wherein the preset division rule is as follows:

[0013] ,

[0014] in, Indicates the Initial dispersion area, Indicates the first regions, Indicates the first regions, , represents the plane similarity function, And it means The estimated plane after the region is flattened, Indicates the The normal of the center pixel of the region, Indicates the The depth of the center pixel of the region, And it means The estimated plane after the region is flattened, Indicates the The normal of the center pixel of the region, Indicates the The depth of the center pixel of the region, It means taking the minimum of two values. Indicates taking the absolute value, Indicates the preset similarity threshold, Indicates the The ratio of the inlier pixels of a region to the total pixels of the region, Indicates the The ratio of the inlier pixels of a region to the total pixels of the region, Indicates the The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, Indicates the preset first ratio threshold.

[0015] In some embodiments of the present invention, the alignment processing method further includes: for any two initially scattered areas, based on the initial depth map and the information of the two initially scattered areas, merging the two initially scattered areas that meet a preset merging rule into one area, wherein the merging method is as follows:

[0016] ,

[0017] in, Indicates the Initial dispersion area, Indicates the Initial dispersion area, Indicates the merged regions, Indicates the The estimated plane after the initial scattered area is flattened, Indicates the The estimated plane after the initial scattered area is flattened, Indicates the The ratio of the inlier pixels of the initial scattered area to the total pixels of the area, Indicates the The ratio of the inlier pixels of the initial scattered area to the total pixels of the area, Indicates the preset second ratio threshold.

[0018] In some embodiments of the present invention, in S232, the method of evaluating whether all pixels in the reference image are visible in each source image includes:

[0019] The view selection strategy is used to initialize the visibility weights of all pixels in the reference image in the source image, where a visibility weight of 0 indicates that the corresponding pixel is invisible, and a visibility weight greater than 0 indicates that the corresponding pixel is visible. For pixels initially determined to be invisible, cross-view depth reprojection is used to re-determine whether the pixel is visible in the source image. If it is re-determined to be visible, the visibility weight of the pixel is updated to be greater than 0.

[0020] In some embodiments of the present invention, the method of re-determining whether the pixel is visible in the source image by adopting the cross-view depth reprojection method includes:

[0021] The pixel is projected into the source image to obtain another pixel corresponding to the pixel, and the other pixel is projected into the reference image to obtain a re-projected pixel; a distance between the invisible pixel and the re-projected pixel is calculated, and when the distance is less than or equal to a preset distance threshold, it is determined that the pixel is visible in the source image.

[0022] In some embodiments of the present invention, in S234, reconstructing the depth estimation value of each unreliable pixel includes minimizing a multi-view aggregation cost to reconstruct the depth estimation value of each unreliable pixel, wherein the multi-view aggregation cost is calculated as follows:

[0023] ,

[0024] in, represents the multi-view aggregation cost, Indicates unreliable pixels, represents the set of anchor points corresponding to all source images, represents the number of source images, Indicates the number of the source image, Indicates the Unreliable pixels in the reference image Relative Visibility weight of Zhang Yuan image, express and The matching cost of the deformation between Relative The anchor point set of source images, , Represents the anchor point set No. Anchor points, Indicates that only the pixels with unreliable An initial set of anchor points selected from reliable pixels in the same region, Indicates unreliable pixels The area where you are located, Indicates anchor point Relative The visibility weight of each source image.

[0025] In some embodiments of the present invention, the step S24 further includes setting, for each unreliable pixel, its normal range value and depth range value in the following manner to propagate and refine the depth map of this time:

[0026] Obtaining a visibility map of the reference image relative to each source image based on visible pixels and invisible pixels of the reference image relative to each source image, and obtaining a plurality of visible source images relative to the unreliable pixel based on the visibility map, wherein when the unreliable pixel is a visible pixel relative to the source image, the source image is regarded as a visible source image;

[0027] Mapping the unreliable pixel to a plurality of visible source images to obtain mapped pixels of the plurality of visible source images, connecting camera centers of the plurality of visible source images to corresponding mapped pixels, obtaining viewing cone directions of the respective source images for the pixel, and setting a normal range value of the pixel according to all viewing cone directions;

[0028] According to the maximum disturbance value and the minimum disturbance value preset for the mapping pixel of each visible source image, pixels are selected along the epipolar line of the source image where the mapping pixel is located and symmetrically about the mapping pixel, and the selected pixels are mapped to the reference image to obtain the depth range value of the pixel assumed in each visible source image. The corresponding depth range values ​​in all visible source images are aggregated to obtain the set depth range value.

[0029] According to a second aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of any one of the methods described in the first aspect of the present invention by executing the executable instructions.

[0030] Compared with the prior art, the advantages of the present invention are:

[0031] The present application uses the results of the judgment of whether the divided regions and the reference image pixels are visible in the source image as prior information to limit the subsequent patch deformation when using the preset deformation algorithm to estimate the depth of unreliable pixels; when performing patch deformation, for each unreliable pixel and each source image, an anchor set corresponding to each unreliable pixel relative to each source image is constructed, wherein the conditions that the anchors in the anchor set satisfy include: 1) the anchor is a reliable pixel in the same region as the unreliable pixel, and 2) the anchor is visible in the source image. The anchors selected under this condition make the deformed patch of the present application more in the same region with continuous depth, while more greatly reducing the pixels that do not exist in other source images, reducing the matching cost, greatly improving the accuracy of patch deformation, and thus more accurately estimating the depth value of the unreliable pixel and improving the success probability of three-dimensional reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0032] The embodiments of the present application are further described below with reference to the accompanying drawings, in which:

[0033] Figure 1 The flowchart of the multi-view stereo reconstruction method according to the embodiment of the present application;

[0034] Figure 2 The flowchart of the preset deformation algorithm according to the embodiment of the present application;

[0035] Figure 3 The deformation comparison result between the existing patch deformation-based method a and the method b of the present application according to the embodiment of the present application;

[0036] Figure 4 The principle diagram of dividing the image into different regions according to the embodiment of the present application;

[0037] Figure 5 The comparison diagram between the existing deformed patch principle and the patch deformation principle after judging whether the pixels are visible according to the embodiment of the present application;

[0038] Figure 6 The process principle diagram of propagation and refinement according to the embodiment of the present application;

[0039] Figure 7 The complete processing process principle diagram of the multi-view stereo reconstruction method according to the embodiment of the present application;

[0040] Figure 8 The result diagram of testing the method of the present application and other existing technologies in combination with the ETH3D data set according to the embodiment of the present application. DETAILED DESCRIPTION

[0041] For the purpose of the present application, the technical solutions and advantages are more clear and explicit, the following will be further detailed by specific examples in combination with the drawings. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0042] As mentioned in the background section, in the existing block deformation method, the first aspect, especially in the textureless region, the reliable pixels with depth discontinuity of the unreliable pixels are selected as anchor points for patch deformation, which makes the deformation patch inaccurate, leading to potential estimation bias problem, the second aspect, due to the pixel difference caused by complex view changes and occlusion, the deformed patch often contains pixels that do not exist in the image of other views, increasing the matching cost, thus leading to potential reconstruction distortion problem.

[0043] Based on the above problems, the inventors made the following analysis: for the first aspect problem, the inventors found that in the textureless region, the insufficient alignment of depth information and edge information leads to the fact that the patch cannot accurately follow the depth continuity principle during deformation, and is prone to produce inaccurate deformation results in the depth discontinuous region. The inventors found that by dividing the region during patch deformation, the depth continuity in the region can be ensured as much as possible, and the anchor point selection in the depth continuous region can improve the reconstruction accuracy. For the second aspect problem, the inventors found that the existing method usually relies on fixed view to select anchor points, which makes it difficult to accurately handle occlusion and view difference when facing view changes, thus leading to unstable patch deformation. The inventors found that by evaluating whether all pixels in the reference image are visible in each source image, for each source image, the visible pixels are selected as anchor points, so that the anchor point set after patch deformation under different views includes more visible pixels, reduces the matching cost, ensures the stability and accuracy of patch deformation, and thus improves the three-dimensional construction adaptability in diversified scenes.

[0044] Based on the above analysis process, according to one embodiment of the present application, a multi-view stereo reconstruction method based on depth edge and visibility prior is proposed. The method of the present application improves the deformation PM algorithm used in the existing APD-MVS algorithm on the basis of the existing APD-MVS algorithm, obtains the preset deformation algorithm of the present application, and then estimates the depth value of each unreliable pixel by using the preset deformation algorithm. Wherein, when the preset deformation algorithm is used to estimate the depth of the unreliable pixel, not only the reference image needs to be divided into different regions, each region is estimated as a depth continuous region, but also all the pixels in the reference image need to be evaluated whether they are visible in each source image, and the judgment results of the divided regions and whether the pixels in the reference image are visible in the source image are used as prior information to limit the subsequent patch deformation, which effectively helps the stability and accuracy of the patch deformation; when the patch deformation is performed, for each unreliable pixel and each source image, an anchor point set corresponding to each unreliable pixel relative to each source image is constructed, wherein when the anchor points are selected from the reference image to construct the anchor point set corresponding to each source image, the anchor points meet the following conditions: 1) the anchor points are reliable pixels in the same region as the unreliable pixel, 2) the anchor points are visible in the source image, and under this condition, the selected anchor points make the deformed patch of the present application more in the same depth continuous region, while more greatly reducing the pixels that do not exist in other source images, reducing the matching cost, greatly improving the accuracy of patch deformation, and thus more accurately estimating the depth value of the unreliable pixel.

[0045] According to one embodiment of the present application, referring to Figure 1 , which is a flowchart of the multi-view stereo reconstruction method of the present application, the method comprises steps S1, S2 and S3, and step S2 further comprises steps S21, S22 and S23. In order to better understand the present application, each step will be described in detail below in combination with specific embodiments.

[0046] In step S1, a plurality of images collected from different angles of view of the same scene are obtained, each image in the plurality of images is taken as a reference image, and the remaining images are taken as source images corresponding to the reference image.

[0047] According to one embodiment of the present application, given an image set formed by a plurality of images collected from different angles of view of the same scene, the image set is denoted as , , wherein any image is denoted as , the number of the image is denoted as , and the total number of images is denoted as The corresponding camera parameters include an intrinsic matrix , a rotation matrix , and a camera center . Each image in is taken as a reference image the remaining images as a reference image of the source images. Wherein, the homography principle and camera parameters of each image are used to reconstruct the depth map of the reference image by matching each reference image with the remaining images one by one.

[0048] In step S2, multiple iterations are performed based on the reference image and each source image to estimate the depth map of each reference image.

[0049] According to an embodiment of the present application, before the multiple iterations are performed, each image is down-sampled multiple times to achieve scaling at different levels and obtain multiple feature maps at different levels. Each iteration is based on the feature maps at the same level of the reference image and each source image to estimate the depth map of this iteration, and the iterations are performed in order from low to high resolution of the feature maps. Illustratively, for example, each image is down-sampled K times to obtain K+1 feature maps. In the K+1 feature maps, the first feature map is the original image, the second feature map is obtained after the first down-sampling on the original image, and so on. The K+1 feature map is obtained after the Kth down-sampling, and the resolution of the corresponding feature maps from the first to the K+1 decreases in order. The feature maps are divided into the first level, the second level, …, and the K+1 level in order from low to high resolution, so that the K+1 feature map is at the first level, the Kth feature map is at the second level, …, and the first feature map is at the K+1 level. Taking the K+1 feature maps at different levels formed as an example, the first iteration is based on the feature maps at the first level of the reference image and each source image to estimate the depth map of this iteration, the second iteration is based on the feature maps at the second level of the reference image and each source image to estimate the depth map of this iteration, and so on. The K+1 iteration is performed, and the depth map obtained by the K+1 iteration is taken as the depth map of the reference image.

[0050] According to an embodiment of the present application, after the reference image and each source image are respectively down-sampled multiple times, multiple feature maps at different levels corresponding to each image are obtained, and multiple iterations are performed based on the multiple feature maps of all images. Each iteration process includes the following steps S21, S22 and S23:

[0051] Step S21: evaluate whether the depth estimation value of each pixel in the depth map obtained this time is reliable to obtain reliable pixels and unreliable pixels. The initialized depth map is used for the first time, and the depth map obtained by up-sampling the depth map obtained in the previous time is used thereafter.

[0052] According to one embodiment of the present invention, the same method as in the APD-MVS algorithm is used to evaluate whether the depth estimation value of each pixel in the depth map is reliable, thereby obtaining reliable pixels and unreliable pixels.

[0053] Step S22: using the PatchMatch algorithm to update the depth estimation value of each reliable pixel.

[0054] According to one embodiment of the present invention, the PatchMatch algorithm, referred to as the PM algorithm, uses fixed patches to reconstruct and update the depth estimation value of reliable pixels. With the rest of the source images The depth map is reconstructed by pairwise matching of feature maps at the corresponding level. Specifically, given a reference image and the source image , for the reference image Each pixel in , first randomly generate a plane hypothesis ,in, and Represents normal and depth respectively. Through the principle of homography mapping, the plane hypothesis can be calculated for and Pixels between The projection matrix Then, use The reference image China-Israel Fixed-size patch centered Projection onto source image , mapped to the source image Mapping patch Finally, the matching cost can be calculated by and The NCC (Normalized Cross-Correlation) score between them is used to obtain the matching cost, which is calculated as follows:

[0055] , (1)

[0056] in, Represents the reference image Reliable pixels With the source image The correlation between the pixels in , to determine whether they are points in the same physical scene, and The smaller it is, the more correlated the two pixels are. Represents the weighted covariance. The reference image is realized in the above way corresponding level of the feature map and the remaining source image reconstruct the pixel in pair-wise matching order depth estimation value of the pixel. That is, the multi-view aggregation cost can be obtained by view weight, and further updated the depth estimation value of the pixel by propagation and optimization to minimize the multi-view aggregation cost . Wherein, the multi-view aggregation cost is defined as:

[0057] , (2)

[0058] wherein, represents the reliable pixel corresponding multi-view aggregation cost, represents the total number of images, represents the image number, represents the reliable pixel the visibility weight relative to the first source image . Wherein, the visibility weight can be generated by view selection strategy. Finally, by propagation and refinement, the aggregation cost is calculated based on multiple depth hypotheses for each reliable pixel, and the depth hypothesis corresponding to the minimum aggregation cost is selected as the final estimation result of the reliable pixel.

[0059] Step S23: reconstruct the depth estimation value of each unreliable pixel by using a preset deformation algorithm.

[0060] According to an embodiment of the present application, referring to Figure 2 which is a flowchart of the preset deformation algorithm, comprising the following steps S231, S232, S233 and S234:

[0061] Step S231: divide the reference image into different regions, and each region is estimated as a depth continuous region.

[0062] According to one embodiment of the present application, the manner of dividing the reference image into a plurality of different regions comprises: performing depth estimation on the reference image by using a depth estimation model to obtain an initial depth map; performing edge detection on the reference image by using an edge detection algorithm to obtain an initial edge map; and performing alignment processing on the initial depth map and the initial edge map to generate a boundary image, which includes a plurality of different regions, and all pixels in each region are continuous in depth in the corresponding initial depth map. The depth estimation module can use a monocular depth estimation model Depth Anything V2, which can achieve zero-shot depth generalization in different scenes. The depth estimation module can also use a Depth Pro model. The edge detection algorithm can use a Roberts operator extractor for edge detection, or a Sobel operator, etc. The present application is not limited in this regard. The technical solution of this embodiment can achieve at least the following beneficial technical effects: initializing the corresponding depth map and edge map, and aligning the depth map and the edge map to make each region depth-continuous, so as to increase the depth edge constraint, avoid edge jumping of the subsequent deformed patch, cover the heterogeneous region with depth discontinuity, and improve the stability of patch deformation.

[0063] For illustration, see Figure 3 , which is a schematic diagram of the deformation comparison results between the existing patch deformation-based method a and the method b of the present application. When the method a performs patch deformation based on the source image, edge jumping deformation occurs, which leads to the error of selecting the reliable but depth discontinuous blue pixels to perform patch deformation on the center unreliable red pixels. The method b of the present application uses the initial depth map, and aligns with the initial edge map first, and then performs robust deformation based on the depth edge alignment, i.e. limits the patch deformation within the same region to make the deformed patch located in the depth-continuous region as much as possible.

[0064] According to one embodiment of the present application, the initial edge map includes information of each initial scattered region. For each scattered region with a size exceeding a preset threshold , the RANSAC algorithm (Random Sample Consensus) is used to perform region planarization on the scattered region in combination with the initial depth map and the region information to obtain an estimated plane and the number of inlier pixels. Wherein, represents the normal, represents the depth, and the ratio of the number of inlier pixels to the total pixels of the scattered region is defined as the inlier ratio . In this way, the information of each scattered region is obtained, and the total number of scattered regions is represented by

[0065] According to one embodiment of the present invention, alignment processing is performed based on the initial depth map and the information of each initial scattered area. The alignment processing methods include: 1) intra-region erosion processing, that is, further dividing the area where heterogeneous areas (i.e., depth discontinuous areas) may exist; 2) inter-region expansion processing, merging areas that may belong to homogeneous areas (i.e., depth continuous areas). The technical solution of this embodiment can at least achieve the following beneficial technical effects: in the erosion stage, the areas belonging to heterogeneous areas are separated, and in the expansion stage, the areas belonging to homogeneous areas are merged to achieve depth edge alignment, and then the aligned map is used to restrict patch deformation within the same area, thereby increasing the probability that the deformed patch is located within the homogeneous area.

[0066] The following describes the above two alignment processing methods:

[0067] 1) Erosion treatment in the area

[0068] According to one embodiment of the present invention, the initial scattered areas are divided according to a preset division rule to obtain multiple areas, wherein the preset division rule is as follows:

[0069] , (3)

[0070] in, Indicates the Initial dispersion area, Indicates the first regions, Indicates the first regions, , represents the plane similarity function, And it means The estimated plane after the region is flattened, Indicates the The normal of the center pixel of the region, Indicates the The depth of the center pixel of the region, And it means The estimated plane after the region is flattened, Indicates the The normal of the center pixel of the region, Indicates the The depth of the center pixel of the region, It means taking the minimum of two values. Indicates taking the absolute value, Indicates the preset similarity threshold, Indicates the The ratio of the inlier pixels of a region to the total pixels of the region, the ratio of the inlier pixels of the first initial dispersed region to the total pixels of the region, the ratio of the inlier pixels of the first initial dispersed region to the total pixels of the region, the ratio of the inlier pixels of the first initial dispersed region to the total pixels of the region, the ratio of the inlier pixels of the first initial dispersed region to the total pixels of the region, the preset first ratio threshold. Similarly, the RANSAC algorithm is used to planarize the divided regions.

[0071] According to an embodiment of the present application, in the division rule, the higher the inlier ratio of each of the two divided regions, the better the planarization result after erosion, and the lower the plane similarity of the estimated plane, indicating that the two divided regions are located in the same homogeneous region to which each of them belongs. Illustratively, the region pre-divided into two regions by erosion and . Then, the two regions and are re-planarized to obtain the estimated planes and the inlier ratios , , and the inlier ratios and . Finally, when the inlier ratios and the plane similarities of the two divided regions both satisfy the conditions set in the above division rule formula, it is considered that the effective division is better, and then the region division is performed.

[0072] 2) Inter-region expansion processing

[0073] According to an embodiment of the present application, for any two initial dispersed regions, the two initial dispersed regions that satisfy the preset merging rule are merged into one region according to the initial depth map and the information of the two initial dispersed regions, wherein the merging manner is as follows:

[0074] , (4)

[0075] wherein, represents the first initial dispersed region, represents the first initial dispersed region, represents the first initial dispersed region, represents the first initial dispersed region, represents the first initial dispersed region, represents the first initial dispersed region, represents the first initial dispersed region, represents the first initial dispersed region, represents the first initial dispersed region, represents the first initial dispersed region, represents the first initial dispersed region, the ratio of the inlier pixels of the first initial dispersed region to the total pixels of the region, Indicates the The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, In the merging method of the present invention, if the plane similarity of the estimated planes of the two regions is greater than the set similarity threshold and the inlier ratio is greater than the set second ratio threshold, the regions are considered to be and In the same homogeneous area, they are merged into one area.

[0076] According to one embodiment of the present invention, after the initial depth map and the initial edge map are aligned, the aligned boundary image is further subjected to edge refinement. The edge refinement process includes: performing pixel filtering on the boundary image to obtain a fine-grained uniform boundary image. The pixel filtering method is as follows:

[0077] (5)

[0078] in, Represents pixels Belong to the edge graph the edge of Represents pixels Belong to the region , Indicates calculation The distance to the estimated plane, Indicates area The normal corresponding to the estimated plane, Represents pixels 3D position information including horizontal position , vertical position and depth , Indicates area The depth corresponding to the estimated plane, Indicates the preset distance threshold, Indicates area The corresponding interior point ratio. In the filtering method of the present invention, if The 3D position of is close to the estimated plane, i.e. If the distance to the estimated plane is less than the preset distance threshold, the Treated as a region part of.

[0079] For illustration, the principle of dividing an image into multiple regions is now fully described in conjunction with the above embodiments. Figure 4 , which is a schematic diagram of the principle of dividing an image into multiple different regions. Figure 4 a in is the source image, Figure 4b is the initial depth map corresponding to the source image , Figure 4 The c in the figure is the initial edge map corresponding to the source image. ,Edge map The edges of the α are highlighted in white in c, and the scattered areas are shown in black. Figure 4 The d in the figure is the edge graph The scattered area map related to the edge information and the information of each initial scattered area is included in the figure. Different colors in the figure represent different initial scattered areas, and they are the initial scattered areas 、 、 Etc. The initial depth map and the initial edge graph Perform alignment processing, that is, use the method of the above embodiment to perform the following four steps on each initial scattered area in sequence:

[0080] 1. Planar operation to obtain information of each scattered area ;

[0081] 2. Erosion processing, such as dividing the area that meets the division rules Divided into and , ;

[0082] 3. Expansion processing, such as the area that meets the merging conditions and Merge into : ;

[0083] 4. Pixel filtering (also called edge thinning), for example, pixel Refine to area middle: ; After all operations are completed, the Figure 4 e in which is the depth edge alignment prior (i.e., boundary image), which includes multiple regions that are finally divided 、 、 and wait;

[0084] Finally, after the above steps are completed, the multiple regions obtained are used as depth edge alignment priors to restrict patch deformation within the same region, so that the deformed patches are located in the depth continuous region as much as possible. There are unreliable pixels in Therefore, for this unreliable pixel, the conventional deformation PM algorithm is used to obtain Figure 4The unstable patch deformation f in the present invention uses the depth edge alignment prior to constrain the patch deformation, and obtains Figure 4 The robust patch deformation results in g are Figure 4 In f and g, the green window in the middle, the blue window outside the green window, and the red window represent the center pixel, the patch in the traditional PM algorithm, and the deformed patch, respectively. The edges of the region are cyan and green to indicate that they are two areas of depth discontinuity, and the boundary is gray. Figure 5 From the deformation results in f and g, it can be seen that the deformed patches obtained by performing patch deformation based on the uniform boundary image are located in the same depth continuous area, thereby improving the stability and accuracy of the deformation.

[0085] Step S232: Evaluate whether all pixels in the reference image are visible in each source image, and obtain visible pixels and invisible pixels of the reference image relative to each source image.

[0086] In existing methods, for the reference image Each pixel in , its corresponding visibility weight in the source image It is determined by evaluating the aggregation cost calculated by the above embodiment with a fixed threshold. Therefore, low aggregation costs are more likely to be visible, while high aggregation costs are usually considered invisible. Therefore, in areas with high aggregation costs and no texture, the original visible areas are often wrongly judged as invisible, that is, determining visibility only by cost is unreliable because it is difficult to strike a balance between areas with good texture and areas without texture. Not only is the cost aggregation used in the above embodiment, it can also be restated as the visibility of the pixel, where a visibility weight of 0 indicates that the corresponding pixel is invisible, and a visibility weight greater than 0 indicates that the corresponding pixel is visible, so as to provide patch deformation with cross-view visibility awareness. Therefore, according to one embodiment of the present invention, for the reference image Each pixel in , pre-initialized in the source image by the view selection strategy The corresponding visibility weight , and then use the cross-view depth reprojection method to re-determine whether the pixel is visible in the source image. If it is re-determined to be visible, the visibility weight of the pixel is updated to a value greater than 0.

[0087] According to one embodiment of the present invention, in S232, the method of evaluating whether all pixels in the reference image are visible in each source image includes: initializing the visibility weights of all pixels in the reference image in the source image using a view selection strategy; for pixels initially determined to be invisible, re-determining whether the pixel is visible in the source image by using a cross-view depth reprojection method; if it is re-determined to be visible, the visibility weight of the pixel is updated to be greater than 0.

[0088] According to one embodiment of the present invention, a cross-view depth reprojection method is used to re-determine whether the pixel is visible in the source image, including: projecting the pixel into the source image to obtain another pixel corresponding to the pixel, and projecting the other pixel into the reference image to obtain the re-projected pixel; calculating the distance between the invisible pixel and the re-projected pixel, and determining that the pixel is visible in the source image when the distance is less than or equal to a preset distance threshold. Wherein, a visibility map of the reference image relative to each source image is obtained based on the visible pixels and invisible pixels of the reference image relative to each source image. Schematically, for the reference image The depth of the invisible area is Invisible pixels , first, use the homography mapping principle to project it onto the source image Get pixels Then, and in The corresponding depth Reproject back To get the pixel after reprojection Finally, define ,when hour, In the source image is visible, thus updating the pixel Relative source image Visibility weight , thus effectively restoring the visibility of textureless areas.

[0089] Indicatively, see Figure 5 , which is a schematic diagram comparing the existing deformation patch principle and the patch deformation principle of the present invention based on pixel visibility judgment. Figure 5 a in is the reference image, Figure 5 Where b is the source image. Compare the reference image with the source image and divide the visible area and invisible area of ​​the reference image relative to the source image by red lines. The present invention adopts the view selection strategy of the above embodiment to initialize all pixels in the reference image relative to the source image. The corresponding visibility weight is obtained Figure 5 The initial visibility map of c in , where black represents pixels judged to be invisible and white represents pixels judged to be visible. Figure 5 As can be seen from c, in the high-cost and textureless area, there are many pixels in the original visible area that are mistakenly judged as invisible. Therefore, the present invention further adopts the cross-view depth reprojection method of the above embodiment to re-determine whether the pixel is visible in the source image, and obtains Figure 5The restored visibility map of d in , which is the visibility map of the final reference image relative to the source image. Figure 5 As can be seen from d, the present invention greatly improves the accuracy of pixel visibility judgment and strikes a balance between textured and textureless areas. Finally, based on the final visibility map, the present invention performs patch deformation in the visibility area to obtain the following: Figure 5 The result of f in the example. The existing deformation patch is as follows Figure 6 As shown in Figure e, the deformed patch contains a large portion of invisible areas. Therefore, the visibility constraint of the present invention effectively improves the accuracy of the deformed patch.

[0090] The technical solution of the embodiment corresponding to step S232 above can achieve at least the following beneficial technical effects: Because some areas of the reference image may be invisible in the corresponding source image, the deformed patch often includes these invisible areas, increasing the matching cost and potentially leading to distortion. The method of the present invention can determine the visibility of each pixel relative to each source image for each source image, effectively improving the accuracy of pixel visibility determination, thereby facilitating subsequent visibility-aware patch deformation, reducing deformed patches in invisible areas, and lowering matching costs.

[0091] Step S233: For each unreliable pixel and each source image, multiple anchor points are selected only from the specified pixel set in the same area as the unreliable pixel for patch deformation, so as to obtain the anchor point set corresponding to each unreliable pixel relative to each source image, wherein the specified pixel set refers to a set of pixels that are visible and reliable relative to the source image.

[0092] According to one embodiment of the present invention, a set of pixels that are visible and reliable relative to the source image refers to: each pixel in the set is a reliable pixel in the same area as the unreliable pixel, and the reliable pixel is a visible pixel in the source image. The anchor point set corresponding to each unreliable pixel relative to each source image is as follows:

[0093] , (6)

[0094] in, Indicates unreliable pixels Relative The anchor point set of source images, Represents the anchor point set No. Anchor points, Indicates that only the pixels with unreliable An initial set of anchor points selected from reliable pixels in the same region, Indicates unreliable pixels The area where you are located, Indicates anchor point Relative The visibility weight of each source image.

[0095] Step S234: reconstructing a depth estimate value of each unreliable pixel based on the anchor point set corresponding to each unreliable pixel relative to each source image;

[0096] According to one embodiment of the present invention, a method for reconstructing the depth estimation value of each unreliable pixel includes minimizing a multi-view aggregation cost to reconstruct the depth estimation value of each unreliable pixel, wherein the multi-view aggregation cost is calculated as follows:

[0097] , (7)

[0098] in, represents the multi-view aggregation cost, Indicates unreliable pixels, represents the set of anchor points corresponding to all source images, represents the number of source images, Indicates the number of the source image, Indicates the Unreliable pixels in the reference image Relative Visibility weight of Zhang Yuan image, express and The matching cost of the deformation between Relative The anchor point set of the source image.

[0099] According to one embodiment of the present invention, the deformation matching cost in the APD-MVS algorithm is calculated as follows:

[0100] , (8)

[0101] in, Represents the reference image Unreliable pixels With the source image The matching cost between pixels in , The method of the present invention obtains the relative The anchor point set of source images, represents the weight parameter, Represents the reference image Unreliable pixels With the source image The correlation between pixels in The calculation principle is the same as the above formula (1). Represents the reference image Anchor The matching cost between pixels in the source image ,The calculation principle of the matching cost between pixels in the source image is the same as the above formula (1), represents the sum of the anchor point numbers in the anchor point set of the reference image relative to the first source image.

[0102] Step S24: obtaining the depth map of this time based on the depth estimation values of all pixels, and taking the depth map obtained in the last iteration as the depth map of the reference image.

[0103] According to an embodiment of the present application, the S24 further comprises setting the normal range value and the depth range value of each unreliable pixel for propagating and refining the depth map of this time. Referring to FIG. 8, which is a schematic diagram of the process principle of propagation and refinement. Figure 6 The propagation and refinement mode comprises propagation and refinement in terms of normal and depth, which will be described below:

[0104] 1) Propagation and refinement in terms of normal

[0105] According to an embodiment of the present application, the multiple visible source images relative to the unreliable pixel are obtained according to the visibility map of the reference image relative to each source image, wherein the source image is taken as a visible source image when the unreliable pixel is a visible pixel relative to the source image. The unreliable pixel is mapped to the multiple visible source images to obtain the mapping pixels of the multiple visible source images, the camera centers of the multiple visible source images are respectively connected with the corresponding mapping pixels to obtain the view cone directions of each visible source image relative to the pixel, and the normal range value of the pixel is set according to all the view cone directions.

[0106] According to an embodiment of the present application, as shown in a of FIG. 9, in order to ensure the multi-view geometry consistency, all the visible source images relative to the unreliable pixel are identified according to the visibility map of the reference image relative to each source image, and the visible source images are represented by 3 light blue diagrams in a of FIG. 9, which are respectively denoted as Figure 6 , , and Figure 6 The unreliable pixel is mapped to the 3 visible source images , and to obtain the mapping pixels , and of the 3 visible source images respectively, and is taken as the mapping pixel of the reference image obtained by mapping the unreliable pixel . ​​ Considered as ). Another example Figure 6 As shown in a and b, the camera centers of multiple visible source images are connected to the corresponding mapping pixels to obtain the corresponding mapping pixels of each visible source image. The viewing cone direction, that is 、 and Each pixel The viewing cone direction is 、 and , reference image For this pixel The viewing cone direction is recorded as .like Figure 6 The normal aggregation principle shown in b, first, since each of all visible source images has a visible hemisphere in its viewing cone direction, the visible hemisphere is represented by the viewing cone, and the blue viewing cone corresponds to the reference image , while the green, red, and yellow cones correspond to the visible source image 、 and , therefore, all visible hemispheres can be aggregated to constrain the 3D points Secondly, for any visible source image , considering its viewing cone direction and Normal Angle between Should exceed (Right now ), i.e. the visible source image 、 and The respective viewing cone directions and The angle between the normals of 、 and Finally, by The normal range of the image is re-projected back to the reference image Pixels in , effectively limiting The aggregated normal range values ​​of the present invention act as visibility constraints during the propagation and refinement process, thereby enhancing the reliability of the normal assumption.

[0107] 2) Deep propagation and refinement

[0108] According to one embodiment of the present invention, based on the maximum disturbance value and the minimum disturbance value preset for the mapping pixel of each visible source image, pixels are selected along the pole line of the source image where the mapping pixel is located and symmetrically about the mapping pixel, and the selected pixels are mapped to the reference image to obtain the depth range value of the pixel assumed in each visible source image, and the corresponding depth range values ​​in all visible source images are aggregated to obtain the set depth range value.

[0109] According to one embodiment of the present invention, a method for obtaining the depth range value of the pixel assumed in each visible source image includes: first, the preset maximum disturbance value is and the minimum perturbation value is , fixed-length sampling is back-projected and aggregated along the multi-view epipolar lines to fine-tune the depth interval. Specifically, Figure 7 As shown in c, for the reference image Each pixel in , as described in the above embodiment, in the visible source image mid-polar line Determine its mapping pixel Then, four pixels are selected according to the maximum disturbance value and the minimum disturbance value. The four pixels are 、 、 and , located in pixels On both sides of the polar line superior, and exist to the left of They are and Location, and And distance They are and Finally, these four pixels are further inversely projected onto the reference image The corresponding depths are obtained as follows: 、 、 and , thus forming two refinement intervals: and . and Assuming that pixel under each visible source image Depth The depth range value. Since the existing method first calculates the depth of the pixels in the fixed reference image The re-projection of the depth range value of the assumed mapping pixel to the source image results in no displacement or over-displacement of the assumed mapping pixel in the source image, while the present application assumes a mapping pixel range with a fixed length in the visible source image and projects it into the reference image to refine the depth range, and the adaptive depth range value ensures that the assumed mapping pixel can achieve proper displacement along the epipolar line without being limited by the current depth, thereby improving the reliability of depth assumption.

[0110] According to an embodiment of the present application, in order to ensure that the mapping pixel experiences significant displacement in as many source images as possible, the depth range values of all visible source images are aggregated in the following manner:

[0111] , (9)

[0112] wherein, represents the number of visible source images, represents the total number of visible source images, represents the minimum depth corresponding to the maximum disturbance value from all visible source images, , represents the maximum depth corresponding to the minimum disturbance value from all visible source images, , represents the maximum depth corresponding to the minimum disturbance value from all visible source images, , represents the minimum depth corresponding to the maximum disturbance value from all visible source images, .

[0113] In step S3, the depth maps of all reference images are fused to obtain three-dimensional point cloud data of the scene.

[0114] According to an embodiment of the present application, referring to Figure 6 which is a schematic diagram of the complete processing principle of the multi-view stereo reconstruction method of the present application. The method of the present application performs multiple downsampling and multiple iterations on a series of input images to estimate the depth map of each image, and fuses the depth maps of all reference images to obtain three-dimensional point cloud data of the scene. In each iteration estimation process, first, for reliable pixels, the depth value of the pixel is estimated using the traditional PM algorithm, and for unreliable pixels, the anchor point set obtained by the traditional deformation PM algorithm is used to estimate the depth value of the pixel. Based on the depth edge alignment prior and visibility prior, patch deformation is further performed to estimate the depth value of the pixel, and the depth map of this time is obtained based on the depth values ​​of all pixels estimated this time. The following is an overall description of the depth value estimation process of unreliable pixels:

[0115] 1) The generation process of deep edge alignment prior includes:

[0116] The monocular depth estimation model Depth Anything V2 is used to estimate the depth of the reference image to obtain the initial depth map , use Roberts operator extractor to perform edge detection and get the initial edge map , align the initial depth map and the initial edge map. The alignment process includes: using the RANSAC algorithm to obtain the estimated plane of each scattered area and obtain the scattered area map, performing intra-region erosion processing on each scattered area to obtain the eroded map, performing inter-region dilation processing on each scattered area to obtain the dilated map, and refining the edges of each scattered area to obtain the final uniform boundary image, which is used as the depth edge alignment prior.

[0117] Among them, the present invention is based on the anchor point set obtained by the traditional deformation PM algorithm Based on the depth edge alignment prior, we further select anchor points from the reliable pixels in the same area as the unreliable pixels for patch deformation to obtain the initial anchor point set. .

[0118] 2) The process of generating visibility priors, including:

[0119] The view selection strategy is used to initialize the visibility weights of all pixels in the reference image in the source image. For example, the initialization results are the 0th pixel, ..., the The visibility weights of each pixel are 、……、 For the initially determined invisible pixels, cross-view depth reprojection is used to re-determine whether the pixel is visible in the source image. If it is re-determined to be visible, the visibility weight of the pixel is updated to be greater than 0. For example, the initialized 0th pixel, ..., The visibility weights of each pixel are 、……、 , and the restored visibility graph is obtained.

[0120] Among them, the initial anchor point set based on unreliable pixels In the view selection process, it is also necessary to determine whether the anchor point is visible in the source image. For each source image, according to its corresponding restored visibility map, the initial anchor point set The anchor point set relative to the 0th source image is denoted as , the anchor point set relative to the 1th source image is denoted as , and the anchor point set relative to the 2th source image is denoted as . The set of anchor point sets of all source images is denoted as , which is the anchor point set obtained by the final patch deformation of the present application. The process of part 2) is repeatedly executed to realize multiple iterations of the visibility weight of each pixel, and a more accurate visibility map is obtained.

[0121] 3) Propagation refinement is performed based on the anchor point set obtained by the final patch deformation , including:

[0122] For example, for each unreliable pixel, the normal range value and the depth range value are set to propagate and refine the depth map this time. The setting process of the normal range value and the depth range value is the same as that of the above embodiment, and will not be repeated here. Figure 8

[0123] To verify the beneficial effects of the present application, the inventors conducted the following experiments for comparison:

[0124] Experimental platform parameters:

[0125] CPU: Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz

[0126] GPU: NVIDIA GeForce 3080 RTX

[0127] Operating system: Windows 10

[0128] Code running environment: C++ with CUDA

[0129] Experimental process:

[0130] 1. Comparative experiment

[0131] ​The present application adopts ETH3D dataset and TNT (Tanks and Temples) dataset, and compares the present application method (referred to as DVP-MVS) with prior art on the above experimental platform. The prior art includes the application of more advanced learning-based methods, including PatchMatchNet method, IterMVS-LS method, MVSTER method, EPP-MVSNet method, EPNet method, and traditional methods, including PCF-MVS method, ACMM method, ACMMP algorithm, SD-MVS method, APD-MVS algorithm and HPM-MVS++ method. Regarding parameter settings: the corresponding preset threshold value when performing regional planarization ; in formulas (3) and (4) of the above embodiment: 、 ; in formulas (4) and (5) of the above embodiment: 、 、 , the preset maximum disturbance value and the minimum disturbance value: 、 ; the total number of source images visible in formula (9) of the above embodiment .

[0132] (1) Experiment under ETH3D dataset

[0133] Using ETH3D dataset, the dataset is divided into training set (Train) and test set (Test), and the present application method (referred to as DVP-MVS) is compared with prior art under the threshold value of 2cm, and the performance results of each method in F1 score, completeness (Completeness, referred to as Comp.) and accuracy (Accuracy, referred to as Acc.) are evaluated during the experiment. The experimental results are shown in Table 1 as follows:

[0134] Table 1

[0135]

[0136] In Table 1, the optimal result is marked in bold, and the suboptimal result is marked with an underline. According to Table 1, it can be seen that the present application method achieves the highest F1 score and completeness.

[0137] (2) Experiment under TNT dataset

[0138] Using the TNT dataset, it is divided into TNT intermediate and TNT advanced datasets, and under a given threshold, the method of the present application is compared with the prior art. In the experiment, the performance of each method in F1 score, recall (Rec.) and precision (Pre.) is evaluated. The experimental results are shown in Table 2 as follows:

[0139] Table 2

[0140]

[0141] In Table 2, the best result is marked in bold, and the suboptimal result is marked with an underscore. According to Table 2, the method of the present application achieves the highest F1 score and completeness in the TNT intermediate dataset. In the TNT advanced dataset, it achieves the second best result, with an F1 score only 0.3% lower than EPNet.

[0142] Overall, the method of the present application achieves the highest F1 score and completeness in the ETH3D dataset and the TNT intermediate dataset, verifying its most advanced performance and strong generalization ability.

[0143] Finally, see Figure 8 , which is a result diagram of testing the method of the present application and other prior arts in combination with the ETH3D dataset. Figure 8 Part a of Figure 8 , b, c, d, e and f parts are the results of the existing IterMVS LS method, the existing ACMMP method, the existing HPM-MVS method, the existing APD-MVS algorithm and the method of the present application on ​ The three scene images in part a of , the red box shows some challenging areas. Obviously, the method of the present application performs particularly well in large-area textureless areas (such as floors and walls), generating the most complete and realistic reconstructed point cloud without introducing detail distortion.

[0144] 2、Ablation experiment

[0145] For the preset deformation algorithm in the above embodiment, only the way of step S231: dividing the reference image into different multiple regions is as follows:

[0146] Delete the whole part of this step, and the obtained preset deformation algorithm is denoted as w / o.Agn;

[0147] Only delete the inter-region erosion processing part of this step, and the obtained preset deformation algorithm is denoted as w / o.Ero;

[0148] Only delete the inter-region dilation processing part of this step, and the obtained preset deformation algorithm is denoted as w / o.Dil;

[0149] Only delete the edge thinning part of this step, and the obtained preset deformation algorithm is denoted as w / o.Fil.

[0150] For the preset deformation algorithm in the above embodiment, only the way of step S232: evaluating whether all pixels in the reference image are visible in each source image is ablated as follows:

[0151] Delete the entire step S232 part, and the obtained preset deformation algorithm is denoted as w / o.Cro;

[0152] Only delete the part of re-determining whether the pixel is visible in the source image by using cross-view depth re-projection (restore visibility map), and the obtained preset deformation algorithm is denoted as w / o.Res;

[0153] Delete the visibility-aware patch deformation (that is, do not limit the anchor point to be a visible pixel when selecting the anchor point), and the obtained preset deformation algorithm is denoted as w / o.Vis.

[0154] For the propagation refinement process in the above embodiment, the following ablation is performed:

[0155] Delete the constraint of the set normal range value and the depth range value in the entire propagation refinement process, and the obtained preset deformation algorithm is denoted as w / o.Geo;

[0156] Only delete the propagation part based on the set normal range value, and the obtained preset deformation algorithm is denoted as w / o.Pro;

[0157] Only delete the part of refinement based on the set normal range value, and the obtained preset deformation algorithm is denoted as w / o.Ref;

[0158] Only delete the setting part of the depth range value, and the obtained preset deformation algorithm is denoted as w / o.Dep.

[0159] On the ETH3 D dataset, and under the given thresholds of 2cm and 10cm, the above 12 ablation cases are respectively experimented, and the effectiveness of each part of the preset deformation algorithm in the method is shown by evaluating the F1 score, Comp. and Acc. three indicators. The ablation study quantitative results are shown in Table 3 as follows:

[0160] Table 3

[0161]

[0162] From Table 3, for step S231, the F1 score of w / o.Agn. is the worst, highlighting the importance of edge constraints on patch deformation. The F1 score of w / o.Ero. is slightly better than that of w / o.Dil., meaning that the impact of region expansion is greater than that of region erosion. In addition, the F1 scores of w / o.Dil. and w / o.Fil. are basically the same, indicating that region erosion and pixel-by-pixel filtering are equally important. For step S232, in contrast, the F1 score of w / o.Cro. is the lowest, verifying the importance of visibility priors for visibility-aware patch deformation. In addition, the F1 score of w / o.Res. is higher than that of w / o.Vis., emphasizing that visibility-aware patch deformation is more important than restoring the visibility map. For the propagation refinement part: w / o.Geo. without normal range values and depth range values as constraints gets the worst F1 score. The F1 score of w / o.Pro. is better than that of w / o.Ref., indicating that the normal constraint is more beneficial to refinement than to propagation. In addition, the F1 scores of w / o.Ref. and w / o.Dep. are similar, indicating that the normal constraint and the depth constraint have equal contributions. Finally, it can be seen that the overall scheme of the DVP-MVS method of the present application has the best effect.

[0163] Prior art as follows:

[0164] PatchMatchNet method: Wang F, Galliani S, Vogel C, et al. Patchmatchnet: Learned multi-view patchmatch stereo [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 14194-14203.

[0165] IterMVS-LS method: Wang, F.; Galliani, S.; Vogel, C.; and Pollefeys, M. 2022a IterMVS: Iterative probability estimation for efficient multi-view stereo. In Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 8606-8615.

[0166] MVSTER method: Wang, X.; Zhu, Z.; Huang, G.; Qin, F.; Ye, Y.; He, Y.; Chi, X.; and Wang, X. 2022b. MVSTER: Epipolar Transformer for Efficient Multi-View Stereo. In European Conference on Computer Vision, 573-591. Springer.

[0167] EPP-MVSNet method: X. Ma, Y. Gong, Q. Wang, J. Huang, L. Chen, F. Yu, EPP-MVSNet: Epipolar assembling based depth prediction for multi-view stereo, in: Proc. IEEE / CVF Int. Conf. Comput. Vis., ICCV, 2021, pp. 5712-5720.

[0168] EPNet method: Su, W.; and Tao, W. 2023. Efficient Edge-Preserving Multi-View Stereo Network for Depth Estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2348-2356.

[0169] PCF-MVS method: A. Kuhn, S. Lin, O. Erdler, Plane completion and filtering for multi-view stereo reconstruction, in: Pattern Recognition, 2019, pp. 18-32.

[0170] ACMM method: Q. Xu, W. Tao, Multi-scale geometric consistency guided multi-view stereo, in: Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit., CVPR, 2019.

[0171] ACMMP algorithm: Xu Q, Kong W, Tao W, et al. Multi-scale geometric consistency guided and planar prior assisted multi-view stereo [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4): 4945-4963.

[0172] SD-MVS method: Yuan, Z.; Cao, J.; Li, Z.; Jiang, H.; and Wang, Z. 2024a. SD MVS: Segmentation-driven Deformation Multi-View Stereo with Spherical Refinement and EM Optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 38(7): 6871-6880.

[0173] APD-MVS algorithm: Wang Y, Zeng Z, Guan T, et al. Adaptive patch deformation for textureless-resilient multi-view stereo [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 1621-1630.

[0174] HPM-MVS++ method: Ren, C.; Xu, Q.; Zhang, S.; and Yang, J. 2023. Hierarchical Prior Mining for Non-Local Multi-View Stereo. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 3611-3620.

[0175] It is noted that while the above describes a particular order of steps, this is not meant to imply that the steps must be performed in the particular order described, as one skilled in the art will understand that steps can be performed in different orders, or even concurrently, and that some steps can be changed, omitted, or even replaced, without departing from the scope and spirit of the invention.

[0176] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0177] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0178] Embodiments of the application have been described above, and the description is intended to be illustrative of the embodiments of the application and not exhaustive, and is not limited to the embodiments disclosed. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of words in the description is intended to best explain the principles of the embodiments, the practical application, or technical improvement over the existing technology, or to enable others skilled in the art to best utilize the embodiments disclosed herein.

Claims

1. A multi-view stereo reconstruction method, characterized in that: Methods include: S1. Acquire multiple images of the same scene captured from different perspectives, use each of the multiple images as a reference image and the remaining images as source images corresponding to the reference image; S2. Perform multiple iterations based on the reference image and each source image to estimate a depth map of each reference image, wherein each iteration process includes: S21, evaluating whether the depth estimation value of each pixel in the depth map obtained this time is reliable, and obtaining reliable pixels and unreliable pixels, wherein the initialized depth map is used for the first time, and the depth map obtained by upsampling the previous depth map is used each time thereafter; S22, using the PatchMatch algorithm to update the depth estimation value of each reliable pixel; S23, reconstructing the depth estimation value of each unreliable pixel using a preset deformation algorithm, including: S231, dividing the reference image into a plurality of different regions, each region being estimated to be a depth-continuous region, wherein the method of dividing the reference image into the plurality of different regions comprises: performing depth estimation on the reference image using a depth estimation model to obtain an initial depth map; performing edge detection on the reference image using an edge detection algorithm to obtain an initial edge map; and aligning the initial depth map and the initial edge map to generate a boundary image, wherein the image includes the plurality of different regions, and the depths of all pixels in each region in the corresponding initial depth map are continuous; S232, evaluating whether all pixels in the reference image are visible in each source image, and obtaining visible pixels and invisible pixels of the reference image relative to each source image; S233. For each unreliable pixel and each source image, select multiple anchor points from only a specified pixel set in the same area as the unreliable pixel to perform patch deformation, thereby obtaining a corresponding anchor point set for each unreliable pixel relative to each source image, wherein the specified pixel set refers to a set of pixels that are visible and reliable relative to the source image. S234, reconstructing a depth estimation value of each unreliable pixel according to the anchor point set corresponding to each unreliable pixel relative to each source image; S24, obtaining a depth map of this time based on the depth estimation values ​​of all pixels, and using the depth map obtained in the last iteration as the depth map of the reference image; S3, fuse the depth maps of all reference images to obtain the 3D point cloud data of the scene; The initial edge map includes information of each initial scattered area, and the alignment processing method includes: dividing the initial scattered area according to the initial depth map and the information of the initial scattered area according to a preset division rule to obtain multiple areas, wherein the preset division rule is as follows: ， in, Indicates the Initial dispersion area, Indicates the first regions, Indicates the first regions, , represents the plane similarity function, And it means The estimated plane after the region is flattened, Indicates the The normal of the center pixel of the region, Indicates the The depth of the center pixel of the region, And it means The estimated plane after the region is flattened, Indicates the The normal of the center pixel of the region, Indicates the The depth of the center pixel of the region, It means taking the minimum of two values. Indicates taking the absolute value, Indicates the preset similarity threshold, Indicates the The ratio of the inlier pixels of a region to the total pixels of the region, Indicates the The ratio of the inlier pixels of a region to the total pixels of the region, Indicates the The ratio of the inlier pixels of the initial scattered area to the total pixels of the area, Indicates the preset first ratio threshold.

2. The method according to claim 1, characterized in that The alignment processing method further includes: for any two initially scattered areas, merging the two initially scattered areas that meet a preset merging rule into one area according to the initial depth map and the information of the two initially scattered areas, wherein the merging method is as follows: ， in, Indicates the Initial dispersion area, Indicates the Initial dispersion area, Indicates the merged regions, Indicates the The estimated plane after the initial scattered area is flattened, Indicates the The estimated plane after the initial scattered area is flattened, Indicates the The ratio of the inlier pixels of the initial scattered area to the total pixels of the area, Indicates the The ratio of the inlier pixels of the initial scattered area to the total pixels of the area, Indicates the preset second ratio threshold.

3. The method according to claim 1, characterized in that In S232, the method of evaluating whether all pixels in the reference image are visible in each source image includes: Initialize the visibility weights of all pixels in the reference image in the source image using the view selection strategy, where a visibility weight of 0 indicates that the corresponding pixel is invisible, and a visibility weight greater than 0 indicates that the corresponding pixel is visible; For a pixel initially determined to be invisible, cross-view depth reprojection is used to re-determine whether the pixel is visible in the source image. If it is re-determined to be visible, the visibility weight of the pixel is updated to be greater than 0.

4. The method according to claim 3, characterized in that The method of re-determining whether the pixel is visible in the source image by adopting the cross-view depth reprojection method includes: Projecting the pixel into the source image to obtain another pixel corresponding to the pixel, and projecting the other pixel into the reference image to obtain a re-projected pixel; A distance between the invisible pixel and the re-projected pixel is calculated, and when the distance is less than or equal to a preset distance threshold, it is determined that the pixel is visible in the source image.

5. The method according to any one of claims 1 to 4, characterized in that: In S234, the depth estimation value of each unreliable pixel is reconstructed by minimizing a multi-view aggregation cost to reconstruct the depth estimation value of each unreliable pixel, wherein the multi-view aggregation cost is calculated as follows: ， in, represents the multi-view aggregation cost, Indicates unreliable pixels, represents the set of anchor points corresponding to all source images, represents the number of source images, Indicates the number of the source image, Indicates the Unreliable pixels in the reference image Relative Visibility weight of Zhang Yuan image, express and The matching cost of the deformation between Relative The anchor point set of source images, , Represents the anchor point set No. Anchor points, Indicates that only the pixels with unreliable An initial set of anchor points selected from reliable pixels in the same region, Indicates unreliable pixels The area where you are located, Indicates anchor point Relative The visibility weight of each source image.

6. The method according to any one of claims 1 to 4, characterized in that The step S24 further includes setting the normal range value and the depth range value of each unreliable pixel in the following manner to propagate and refine the depth map of this time: Obtaining a visibility map of the reference image relative to each source image based on visible pixels and invisible pixels of the reference image relative to each source image, and obtaining a plurality of visible source images relative to the unreliable pixel based on the visibility map, wherein when the unreliable pixel is a visible pixel relative to the source image, the source image is regarded as a visible source image; Mapping the unreliable pixel to a plurality of visible source images to obtain mapped pixels of the plurality of visible source images, connecting camera centers of the plurality of visible source images to corresponding mapped pixels, obtaining viewing cone directions of the respective source images for the pixel, and setting a normal range value of the pixel according to all viewing cone directions; According to the maximum disturbance value and the minimum disturbance value preset for the mapping pixel of each visible source image, pixels are selected along the epipolar line of the source image where the mapping pixel is located and symmetrically about the mapping pixel, and the selected pixels are mapped to the reference image to obtain the depth range value of the pixel assumed in each visible source image. The corresponding depth range values ​​in all visible source images are aggregated to obtain the set depth range value.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 6 by executing the executable instructions.

Images