Multi-view three-dimensional reconstruction method cooperating with depth edge and visibility prior

Abstract

The invention provides a multi-view three-dimensional reconstruction method cooperating with depth edge and visibility prior, which comprises the following steps: acquiring a plurality of images acquired from different views for the same scene, taking each image as a reference image and taking other images as source images, carrying out iteration for multiple times, estimating a depth map of each reference image, and obtaining a depth map of each reference image; each time comprises the following steps: updating the depth estimation value of the reliable pixel; reconstructing the depth estimation value of the current unreliable pixel, comprising the following steps of: dividing the reference image into a plurality of areas; evaluating whether all pixels in the reference image are visible in the source image; for the unreliable pixels, a plurality of anchor points are only selected from a specified pixel set in the same area with the unreliable pixels for patch deformation, an anchor point set corresponding to the source image is obtained, and the specified pixel set refers to a pixel set which is visible and reliable relative to the source image; according to the anchor point set corresponding to each source image, reconstructing a depth estimation value of an unreliable pixel; and obtaining a current depth map based on the depth estimation values of all the pixels.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, in particular to the field of three-dimensional reconstruction and multi-view stereo matching, and more particularly to a multi-view stereo reconstruction method that coordinates depth edge and visibility priors. Background Art

[0002] Multi-View Stereo (MVS) is a core task in computer vision, which aims to densify the geometric representation of scenes or objects through certain overlapping photos taken from different perspectives. It has been widely used in various fields such as virtual reality, autonomous driving, object detection, etc. In recent years, a large number of imaginative ideas have greatly improved its performance in various benchmarks such as the ETH3D dataset. These innovations can be roughly divided into learning-based MVS and traditional MVS.

[0003] Learning-based MVS utilizes convolutional neural networks to extract high-dimensional features for reconstruction, but they require large training datasets and have poor generalization ability. In contrast, traditional MVS is extended from the PatchMatch algorithm, which propagates appropriate hypotheses from neighboring pixels and generates refined hypotheses to construct the solution space, and then selects the best hypothesis through a criterion defined by the multi-view matching cost. However, when the patch is located in a textureless area, its matching cost becomes unreliable due to the lack of distinguishable features within the receptive field.

[0004] To reconstruct textureless regions, two main types of traditional MVS have been proposed: planarization-based and patch deformation-based methods. Planarization-based methods connect reliable pixels in well-textured regions to form regions that demarcate textureless regions, and then planarize these regions for reconstruction. Various strategies have been adopted, such as superpixel TSAR-MVS, triangulation ACMMP, and KD-tree HPM-MVS, but these methods are usually affected by the size limit of the connected region and are prone to deviations during the planarization process.

[0005] Differently, patch deformation-based methods deform fixed blocks to introduce sufficient features to reduce the matching cost. For example, API-MVS introduces entropy to increase the block size and sampling interval, while the APD-MVS algorithm adaptively searches for relevant reliable pixels around each unreliable pixel and then builds multiple sub-blocks centered on them to reduce the matching cost. However, this type of method mainly focuses on exploring advanced reliable pixel search strategies to alleviate matching ambiguity. When reconstructing unreliable red pixels (i.e., pixels with matching ambiguity), traditional patch deformation-based methods usually segment their surrounding areas into multiple fixed-angle sectors and then search for reliable gray pixels in each sector to form blocks. However, in the process of searching for reliable pixels, unexpected edge jumps caused by shadows or occlusions destroy the depth continuity principle, which may lead to potential matching distortions.

[0006] Therefore, in the existing block deformation methods, on the one hand, especially in textureless areas, reliable pixels that are depth discontinuous with unreliable pixels are selected as anchor points for patch deformation, which makes the deformed patches inaccurate and leads to potential estimation bias problems. On the other hand, due to the pixel differences and occlusions caused by complex perspective changes, the deformed patches often contain pixels that do not exist in images from other perspectives, increasing the matching cost and thus leading to potential reconstruction distortion problems.

[0007] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0008] Therefore, the object of the present invention is to overcome the defects of the above-mentioned prior art and provide a multi-view stereoscopic reconstruction method that cooperates with depth edge and visibility prior.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] According to a first aspect of the present invention, a multi-view stereo reconstruction method is provided, the method comprising: S1, acquiring multiple images of the same scene captured from different viewpoints, taking each of the multiple images as a reference image and the remaining images as source images corresponding to the reference image; S2, performing multiple iterations based on the reference image and each source image to estimate a depth map of each reference image, wherein each iteration process comprises: S21, evaluating whether the depth estimation value of each pixel in the depth map acquired this time is reliable, and obtaining reliable pixels and unreliable pixels, wherein an initialized depth map is used for the first time, and then each time a depth map after upsampling the depth map obtained last time is used; S22, updating the depth estimation value of each reliable pixel by using a PatchMatch algorithm; S23, reconstructing the depth estimation value of each unreliable pixel by using a preset deformation algorithm, comprising: S231, dividing the reference image into different multi-view stereo images; S232 regions, each of which is estimated to be a depth-continuous region; S232, evaluate whether all pixels in the reference image are visible in each source image, and obtain the visible pixels and invisible pixels of the reference image relative to each source image; S233, for each unreliable pixel and each source image, select multiple anchor points only from the specified pixel set in the same area as the unreliable pixel for patch deformation, and obtain the anchor point set corresponding to each unreliable pixel relative to each source image, wherein the specified pixel set refers to a set of pixels that are visible and reliable relative to the source image; S234, reconstruct the depth estimation value of each unreliable pixel according to the anchor point set corresponding to each unreliable pixel relative to each source image; S24, obtain the depth map of this time based on the depth estimation value of all pixels, and use the depth map obtained in the last iteration as the depth map of the reference image; S3, fuse the depth maps of all reference images to obtain the three-dimensional point cloud data of the scene.

[0011] In some embodiments of the present invention, in S231, the method of dividing the reference image into multiple different areas includes: using a depth estimation model to perform depth estimation on the reference image to obtain an initial depth map; using an edge detection algorithm to perform edge detection on the reference image to obtain an initial edge map; aligning the initial depth map and the initial edge map to generate a boundary image, which includes multiple different areas, and all pixels in each area have continuous depths in the corresponding initial depth map.

[0012] In some embodiments of the present invention, the initial edge map includes information of each initial scattered area, and the alignment processing method includes: dividing the initial scattered area according to the initial depth map and the information of the initial scattered area according to a preset division rule to obtain a plurality of areas, wherein the preset division rule is as follows:

[0013] ,

[0014] in, Indicates Initial dispersion area, Indicates the first Regions, Indicates the first Regions, , represents the plane similarity function, And it means The estimated plane after the region is planarized, Indicates The normal of the pixel at the center of the region, Indicates The depth of the center pixel of the region, And it means The estimated plane after the region is planarized, Indicates The normal of the pixel at the center of the region, Indicates The depth of the center pixel of the region, It means taking the minimum of two values. Indicates taking the absolute value, Indicates the preset similarity threshold, Indicates The ratio of the internal point pixels of a region to the total pixels of the region, Indicates The ratio of the internal point pixels of a region to the total pixels of the region, Indicates The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, Indicates a preset first ratio threshold.

[0015] In some embodiments of the present invention, the alignment processing method further includes: for any two initial scattered areas, according to the initial depth map and the respective information of the two initial scattered areas, merging the two initial scattered areas that meet the preset merging rule into one area, wherein the merging method is as follows:

[0016] ,

[0017] in, Indicates Initial dispersion area, Indicates Initial dispersion area, The merged Regions, Indicates The estimated plane after the initial scattered area is planarized, Indicates The estimated plane after the initial scattered area is planarized, Indicates The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, Indicates The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, Indicates a preset second ratio threshold.

[0018] In some embodiments of the present invention, in S232, the method of evaluating whether all pixels in the reference image are visible in each source image includes:

[0019] The view selection strategy is used to initialize the visibility weights of all pixels in the reference image in the source image, where a visibility weight of 0 indicates that the corresponding pixel is invisible, and a visibility weight greater than 0 indicates that the corresponding pixel is visible. For the initially determined invisible pixels, cross-view depth reprojection is used to re-determine whether the pixel is visible in the source image. If it is re-determined to be visible, the visibility weight of the pixel is updated to be greater than 0.

[0020] In some embodiments of the present invention, the method of re-determining whether the pixel is visible in the source image by cross-view depth reprojection includes:

[0021] The pixel is projected into the source image to obtain another pixel corresponding to the pixel, and the other pixel is projected into the reference image to obtain a re-projected pixel; a distance between the invisible pixel and the re-projected pixel is calculated, and when the distance is less than or equal to a preset distance threshold, it is determined that the pixel is visible in the source image.

[0022] In some embodiments of the present invention, in S234, the method of reconstructing the depth estimation value of each unreliable pixel includes minimizing the multi-view aggregation cost to reconstruct the depth estimation value of each unreliable pixel, wherein the multi-view aggregation cost is calculated as follows:

[0023] ,

[0024] in, represents the multi-view aggregation cost, Indicates unreliable pixels. represents the set of anchor points corresponding to all source images. represents the number of source images, Indicates the source image number, Indicates Unreliable pixels of the reference image Relative The visibility weight of Zhang Yuan image, express and The matching cost of the deformation between Relative The anchor point set of source images, , Represents the anchor point set No. Anchor points, Indicates that only the pixels with unreliable The initial set of anchor points selected among reliable pixels in the same region, Indicates unreliable pixels The area where you are located, Indicates anchor point Relative The visibility weight of each source image.

[0025] In some embodiments of the present invention, the step S24 further includes setting the normal range value and the depth range value of each unreliable pixel in the following manner to propagate and refine the depth map of this time:

[0026] Obtaining a visibility map of the reference image relative to each source image according to visible pixels and invisible pixels of the reference image relative to each source image, and acquiring a plurality of visible source images relative to the unreliable pixel according to the visibility map, wherein when the unreliable pixel is a visible pixel relative to the source image, the source image is used as a visible source image;

[0027] Mapping the unreliable pixel to multiple visible source images to obtain mapping pixels of the multiple visible source images, connecting the camera centers of the multiple visible source images to the corresponding mapping pixels respectively, obtaining the viewing cone direction of each source image for the pixel, and setting the normal range value of the pixel according to all viewing cone directions;

[0028] According to the maximum disturbance value and the minimum disturbance value preset for the mapping pixel of each visible source image, pixels are selected along the pole line of the source image where the mapping pixel is located and symmetrically about the mapping pixel, and the selected pixels are mapped to the reference image to obtain the depth range value of the pixel assumed in each visible source image, and the corresponding depth range values ​​in all visible source images are aggregated to obtain the set depth range value.

[0029] According to a second aspect of the present invention, there is provided an electronic device, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of any one of the methods described in the first aspect of the present invention by executing the executable instructions.

[0030] Compared with the prior art, the advantages of the present invention are:

[0031] When the present invention uses the preset deformation algorithm to perform depth estimation of unreliable pixels, the judgment result of whether the divided multiple regions and the reference image pixels are visible in the source image is used as prior information to limit subsequent patch deformation; when performing patch deformation, for each unreliable pixel and each source image, an anchor point set corresponding to each unreliable pixel relative to each source image is constructed, wherein the conditions satisfied by the anchor points in the anchor point set include: 1) the anchor point is a reliable pixel in the same region as the unreliable pixel, and 2) the anchor point is visible in the source image. The anchor points selected under this condition make the deformed patches of the present invention more likely to be in the same region with continuous depth. At the same time, the pixels that do not exist in other source images are reduced to a greater extent, the matching cost is reduced, and the accuracy of patch deformation is greatly improved, thereby more accurately estimating the depth value of the unreliable pixel and improving the probability of successful three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0033] Figure 1 A schematic diagram of a multi-view stereoscopic reconstruction method according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of a flow chart of a preset deformation algorithm according to an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of deformation comparison results between the existing patch-based deformation method a and the method b of the present invention according to an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of the principle of dividing an image into a plurality of different regions according to an embodiment of the present invention;

[0037] Figure 5 A schematic diagram comparing an existing deformation patch principle according to an embodiment of the present invention and a patch deformation principle according to the present invention based on whether a pixel is visible;

[0038] Figure 6 A schematic diagram of the process principle of propagation and refinement according to an embodiment of the present invention;

[0039] Figure 7 A schematic diagram of the complete processing principle of the multi-view stereoscopic reconstruction method according to an embodiment of the present invention;

[0040] Figure 8 It is a schematic diagram of the test results of the method of the present invention and other prior arts in combination with the ETH3D dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] As mentioned in the background technology section, in existing block deformation methods, on the one hand, especially in textureless areas, reliable pixels that are depth discontinuous with unreliable pixels are selected as anchor points for patch deformation, which makes the deformed patches inaccurate and leads to potential estimation bias problems. On the other hand, due to pixel differences and occlusions caused by complex perspective changes, the deformed patches often contain pixels that do not exist in images from other perspectives, increasing the matching cost and thus leading to potential reconstruction distortion problems.

[0043] Based on the above problems, the inventors conducted the following analysis: For the first aspect of the problem, the inventors found that the insufficient alignment of depth information and edge information in textureless areas causes the patch to fail to accurately follow the principle of depth continuity during deformation, and it is easy to produce inaccurate deformation results in areas with discontinuous depth. The inventors found that when the patch is deformed, the depth continuity within the area can be guaranteed as much as possible by dividing the area, and the anchor point selection can be performed in the depth continuous area to improve the accuracy of reconstruction. For the second aspect of the problem, the inventors found that the existing methods usually rely on a fixed perspective to select anchor points, so that the subsequent occlusion and perspective differences may not be accurately handled when facing perspective changes, resulting in instability of patch deformation. The inventors found that by evaluating whether all pixels in the reference image are visible in each source image, for each source image, the visible pixels are selected as anchor points, so that the anchor point set after the patch deformation at different perspectives includes more visible pixels, reducing the matching cost, ensuring the stability and accuracy of the patch deformation, and thus improving the adaptability of three-dimensional construction in diverse scenes.

[0044] Based on the above analysis process, according to an embodiment of the present invention, a multi-view stereo reconstruction method that cooperates with depth edge and visibility prior is proposed. Based on the existing APD-MVS algorithm, the method of the present invention improves the deformation PM algorithm used in the existing APD-MVS algorithm to obtain the preset deformation algorithm of the present invention, and then uses the preset deformation algorithm to estimate the depth value of each unreliable pixel. Among them, when using the preset deformation algorithm to perform depth estimation of unreliable pixels, it is not only necessary to divide the reference image into multiple different regions, each region is estimated as a depth continuous region, but also necessary to evaluate whether all pixels in the reference image are visible in each source image, and use the judgment results of whether the divided multiple regions and the reference image pixels are visible in the source image as prior information to limit subsequent patch deformation, effectively helping the stability and accuracy of patch deformation; when performing patch deformation, for each unreliable pixel and each source image, a set of anchor points corresponding to each unreliable pixel relative to each source image is constructed, wherein when selecting anchor points from the reference image to construct the anchor point set corresponding to each source image, the conditions satisfied by the anchor points include: 1) the anchor point is a reliable pixel in the same area as the unreliable pixel, and 2) the anchor point is visible in the source image. The anchor point selected under this condition makes the deformed patch of the present invention more in the same area with continuous depth. At the same time, the pixels that do not exist in other source images are reduced to a greater extent, the matching cost is reduced, and the accuracy of patch deformation is greatly improved, thereby more accurately estimating the depth value of the unreliable pixel.

[0045] According to one embodiment of the present invention, see Figure 1 , which is a schematic flow chart of the multi-view stereoscopic reconstruction method of the present invention, the method includes steps S1, S2 and S3, and step S2 also includes steps S21, S22 and S23. In order to better understand the present invention, each step is described in detail below in conjunction with specific embodiments.

[0046] In step S1, a plurality of images captured from different viewing angles of the same scene are obtained, and each of the plurality of images is used as a reference image and the remaining images are used as source images corresponding to the reference image.

[0047] According to one embodiment of the present invention, given an image set formed by multiple images collected from different perspectives in the same scene, the image set is denoted as , Represents any image, Indicates the image number, Indicates the total number of images, any image The corresponding camera parameters include the intrinsic matrix , rotation matrix and Camera Center .Will Each image in is used as a reference image , the rest of the images As a reference image The source image is obtained by using the homography principle and the camera parameters of each image, and by using the reference image With the rest of the image Pairwise matching to reconstruct the reference image Depth map of .

[0048] In step S2, multiple iterations are performed based on the reference image and each source image to estimate a depth map of each reference image.

[0049] According to one embodiment of the present invention, before performing multiple iterations, each image needs to be downsampled multiple times to achieve scaling at different levels and obtain multiple feature maps at different levels. Each iteration estimates the depth map based on the feature maps of the reference image and each source image at the same level, and iterates in order of the resolution of the feature map from low to high. Schematically, for example, each image is downsampled K times, and K+1 feature maps are obtained. In the K+1 feature maps, the first feature map is the original image. After the first downsampling is performed on the original image, the second feature map is obtained. In this way, after the Kth downsampling, the K+1th feature map of different levels is obtained, and the resolution of the corresponding feature maps from the 1st to the K+1th decreases successively. Among them, each feature map is divided into the 1st level, the 2nd level, ..., and the K+1th level in the order of resolution from low to high, so that the K+1th feature map is the 1st level, the Kth feature map is the 2nd level, ..., and the 1st feature map is the K+1th level. Taking the formed K+1 feature maps of different levels as an example, the first iteration is based on the feature maps of the reference image and each source image at the 1st level to estimate the depth map of this time, and the second iteration is based on the feature maps of the reference image and each source image at the 2nd level to estimate the depth map of this time. In this way, K+1 iterations are performed, and the depth map obtained in the K+1th iteration is used as the depth map of the reference image.

[0050] According to an embodiment of the present invention, after the reference image and each source image are downsampled multiple times, multiple feature maps of different levels corresponding to each image are obtained, and multiple iterations are performed based on the multiple feature maps of all images. Each iteration process includes the following steps S21, S22 and S23:

[0051] Step S21: Evaluate whether the depth estimation value of each pixel in the depth map obtained this time is reliable, and obtain reliable pixels and unreliable pixels, wherein the initialized depth map is used for the first time, and the depth map obtained by upsampling the previous depth map is used each time thereafter.

[0052] According to an embodiment of the present invention, whether the depth estimation value of each pixel in the depth map is reliable is evaluated in the same manner as in the APD-MVS algorithm, thereby obtaining reliable pixels and unreliable pixels.

[0053] Step S22: Use the PatchMatch algorithm to update the depth estimation value of each reliable pixel.

[0054] According to an embodiment of the present invention, the PatchMatch algorithm, referred to as the PM algorithm, uses fixed patches to reconstruct and update the depth estimation value of reliable pixels. With the rest of the source images The feature maps at the corresponding level are matched pairwise to reconstruct their depth maps. Specifically, given a reference image and the source image , for the reference image Each pixel in , first randomly generate a plane hypothesis ,in, and Represents normal and depth respectively. Through the homography mapping principle, the plane hypothesis can be calculated for and Pixels between The projection matrix Then, use The reference image China-Israel Fixed size patch centered Projection onto source image In the mapping, the source image is obtained Mapping patch Finally, the matching cost can be calculated by and The NCC (Normalized Cross-Correlation) score between them is used to obtain the matching cost, which is calculated as follows:

[0055] , (1)

[0056] in, Represents the reference image Reliable pixels With source image The correlation between pixels in , to determine whether they are points in the same physical scene, and The smaller it is, the more related the two pixels are. Represents the weighted covariance. The reference image is realized in the above way Feature maps of corresponding levels and remaining source images Each source image in the corresponding level of the feature map is matched pairwise to reconstruct the pixel That is, the multi-view aggregation cost can be obtained through the view weight, and the propagation and optimization can be further used to minimize the multi-view aggregation cost to update the pixel The depth estimate of . The multi-view aggregation cost is defined as:

[0057] , (2)

[0058] in, Reliable pixels The corresponding multi-view aggregation cost, Represents the total number of images, Indicates the image number, Reliable pixels Relative Zhang Yuan Image The visibility weight can be generated by using a view selection strategy. Finally, through propagation and refinement, the aggregation cost is calculated based on multiple depth hypotheses for each reliable pixel, and the depth hypothesis corresponding to the minimum aggregation cost is selected as the final estimation result of the reliable pixel.

[0059] Step S23: Reconstruct the depth estimation value of each unreliable pixel using a preset deformation algorithm.

[0060] According to one embodiment of the present invention, see Figure 2 , which is a schematic diagram of the process of the preset deformation algorithm, including the following steps S231, S232, S233 and S234:

[0061] Step S231: Divide the reference image into a plurality of different regions, and each region is estimated to be a depth continuous region.

[0062] According to one embodiment of the present invention, the method of dividing a reference image into a plurality of different regions includes: using a depth estimation model to perform depth estimation on the reference image to obtain an initial depth map; using an edge detection algorithm to perform edge detection on the reference image to obtain an initial edge map; aligning the initial depth map and the initial edge map to generate a boundary image, which includes a plurality of different regions, and the depth of all pixels in each region in the corresponding initial depth map is continuous. Among them, the depth estimation module can adopt a monocular depth estimation model Depth Anything V2, which can achieve zero-sample depth generalization in different scenes, and the depth estimation module can also adopt a Depth Pro model. The edge detection algorithm can use a Roberts operator extractor for edge detection, or a Sobel operator, etc. The present invention is not limited to this. The technical solution of this embodiment can at least achieve the following beneficial technical effects: initialize the corresponding depth map and edge map, and align the depth map and edge map so that each region is continuous in depth, so as to increase the depth edge constraint, avoid edge jumps in the patch after subsequent deformation and cover heterogeneous regions with depth discontinuity, and improve the stability of patch deformation.

[0063] Indicatively, see Figure 3 , which is a schematic diagram of the deformation comparison results between the existing patch deformation-based method a and the method b of the present invention. When method a performs patch deformation based on the source image, edge jumping deformation occurs, causing it to mistakenly select reliable but depth-discontinuous blue pixels to perform patch deformation on the unreliable central red pixels. The method of the present invention uses the initial depth map and aligns it with the initial edge map first, and achieves robust deformation based on depth edge alignment, that is, restricting patch deformation to the same area to make the deformed patch located in the depth continuous area as much as possible.

[0064] According to one embodiment of the present invention, the initial edge map includes information about each initial scattered area. Dispersed areas , combined with the initial depth map and regional information, the RANSAC algorithm (Random Sample Consensus) is used to collect the scattered areas Perform region planarization to obtain an estimated plane and the number of inlier pixels. represents the normal line, Indicates depth and compares the number of inliers to the scattered area The ratio of the total pixels is defined as the inlier ratio In this way, information about each dispersed area is obtained. , Indicates the total number of dispersed regions.

[0065] According to one embodiment of the present invention, alignment processing is performed based on the initial depth map and the information of each initial scattered area. Among them, the alignment processing methods include: 1) intra-region erosion processing, that is, further dividing the area where heterogeneous areas (i.e., depth discontinuous areas) may exist; 2) inter-region expansion processing, merging areas that may belong to homogeneous areas (i.e., depth continuous areas). The technical solution of this embodiment can at least achieve the following beneficial technical effects: in the erosion stage, the areas belonging to heterogeneous areas are separated, and in the expansion stage, the areas belonging to homogeneous areas are merged to achieve depth edge alignment, and then the aligned map is used to restrict the patch deformation in the same area, thereby increasing the probability that the deformed patch is located in the homogeneous area.

[0066] The following are the two alignment processing methods described above:

[0067] 1) Erosion treatment in the area

[0068] According to an embodiment of the present invention, the initial scattered areas are divided according to a preset division rule to obtain multiple areas, wherein the preset division rule is as follows:

[0069] , (3)

[0070] in, Indicates Initial dispersion area, Indicates the first Regions, Indicates the first Regions, , represents the plane similarity function, And it means The estimated plane after the region is planarized, Indicates The normal of the pixel at the center of the region, Indicates The depth of the center pixel of the region, And it means The estimated plane after the region is planarized, Indicates The normal of the pixel at the center of the region, Indicates The depth of the center pixel of the region, It means taking the minimum of two values. Indicates taking the absolute value, Indicates the preset similarity threshold, Indicates The ratio of the internal point pixels of a region to the total pixels of the region, Indicates The ratio of the internal point pixels of a region to the total pixels of the region, Indicates The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, Represents a preset first ratio threshold. Similarly, the divided area is planarized using the RANSAC algorithm.

[0071] According to an embodiment of the present invention, in the division rule, the higher the corresponding inlier ratio of the two divided regions, the better the planarization result after erosion, and the lower the plane similarity of the estimated plane, the more the two divided regions are located in the homogeneous regions to which they belong. Pre-divided into two regions by erosion and Then, for the two regions and Replanarize to obtain The estimated plane and inlier ratio , The estimated plane and inlier ratio Finally, when the interior point ratio and plane similarity of the two divided regions both meet the conditions set in the above division rule formula, it is considered to be a more effective division, and the region division is performed.

[0072] 2) Inter-region expansion processing

[0073] According to an embodiment of the present invention, for any two initial scattered areas, according to the initial depth map and the respective information of the two initial scattered areas, the two initial scattered areas that meet the preset merging rule are merged into one area, wherein the merging method is as follows:

[0074] , (4)

[0075] in, Indicates Initial dispersion area, Indicates Initial dispersion area, The merged Regions, Indicates The estimated plane after the initial scattered area is flattened, Indicates The estimated plane after the initial scattered area is flattened, Indicates The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, Indicates The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, In the merging method of the present invention, if the plane similarity of the estimated planes of two regions is greater than the set similarity threshold and the inlier ratio is greater than the set second ratio threshold, the regions are considered and In the same homogeneous area, merged into one area.

[0076] According to one embodiment of the present invention, after the initial depth map and the initial edge map are aligned, the aligned boundary image is further subjected to edge thinning. The edge thinning process includes: performing pixel filtering on the boundary image to obtain a fine-grained uniform boundary image, and the pixel filtering method is as follows:

[0077] (5)

[0078] in, Represents pixels Edge Graph the edge of Represents pixels Belong to the area , Indicates the calculation The distance to the estimated plane, Indicates area The normal corresponding to the estimated plane is Represents pixels 3D position information including horizontal position , vertical position and depth , Indicates area The depth corresponding to the estimated plane is Indicates the preset distance threshold, Indicates area In the filtering method of the present invention, if The 3D position of is close to the estimated plane, i.e. If the distance to the estimated plane is less than the preset distance threshold, Treat as Region part of.

[0079] In an illustrative manner, the process principle of dividing an image into multiple different regions is now fully described in combination with the above embodiments. Figure 4 , which is a schematic diagram of the principle of dividing an image into multiple different regions. Figure 4 a in it is the source image, Figure 4b is the initial depth map corresponding to the source image. , Figure 4 The c in the figure is the initial edge map corresponding to the source image. ,Edge map The edge of is highlighted in white in c, and black indicates the scattered area. Figure 4 The d in the figure is the edge graph The scattered area map related to the edge information and the information of each initial scattered area contained in , in which different colors represent different initial scattered areas, and they are respectively the initial scattered areas , , Etc. The initial depth map and the initial edge graph Perform alignment processing, that is, use the method of the above embodiment to perform the following four steps on each initial scattered area in sequence:

[0080] 1. Planarization operation to obtain information of each scattered area ;

[0081] 2. Erosion processing, such as dividing the area that meets the division rules Divide into and , ;

[0082] 3. Expansion processing, such as the area that meets the merging conditions and Merge into : ;

[0083] 4. Pixel filtering (also called edge thinning), for example, Refine to area middle: ; After all operations are completed, the Figure 4 e in which is the deep edge alignment prior (i.e., boundary image), which includes multiple regions that are finally divided , , and wait;

[0084] Finally, after the above steps are completed, multiple regions are obtained, and multiple regions are used as depth edge alignment priors to restrict patch deformation in the same region, so that the deformed patches are located in the depth continuous region as much as possible. Schematically, There are unreliable pixels in Therefore, for this unreliable pixel, the conventional deformation PM algorithm is used to obtain Figure 4The unstable patch deformation f in the present invention constrains the patch deformation with the depth edge alignment prior, and obtains Figure 4 The robust patch deformation results in g are Figure 4 In f and g, the green window in the middle, the blue window outside the green window, and the red window represent the central pixel, the patch in the traditional PM algorithm, and the deformed patch, respectively. The edges of the regions are cyan and green to indicate that they are two regions with discontinuous depths, and the boundaries are gray. Figure 4 From the deformation results in f and g, it can be seen that the deformed patches obtained by performing patch deformation based on the uniform boundary image in the present invention are in the same depth continuous area, thereby improving the stability and accuracy of the deformation.

[0085] Step S232: Evaluate whether all pixels in the reference image are visible in each source image, and obtain visible pixels and invisible pixels of the reference image relative to each source image.

[0086] In the existing method, for the reference image Each pixel in , whose corresponding visibility weight in the source image is determined by evaluating the aggregation cost calculated by the above embodiment with a fixed threshold. Therefore, low aggregation costs are more likely to be visible, while high aggregation costs are usually considered invisible. Therefore, in areas with high aggregation costs and no texture, the originally visible areas are often wrongly judged as invisible, that is, determining visibility only by cost is unreliable because it is difficult to strike a balance between areas with good texture and areas without texture. Not only is the cost aggregation used in the above embodiment, it can also be restated as the visibility of the pixel, where a visibility weight of 0 indicates that the corresponding pixel is invisible, and a visibility weight greater than 0 indicates that the corresponding pixel is visible, so as to provide patch deformation with cross-view visibility awareness. Therefore, according to one embodiment of the present invention, for the reference image Each pixel in , pre-initialized in the source image by the view selection strategy The corresponding visibility weight in , and then use the cross-view depth reprojection method to re-determine whether the pixel is visible in the source image. If it is re-determined to be visible, the visibility weight of the pixel is updated to a value greater than 0.

[0087] According to one embodiment of the present invention, in S232, the method of evaluating whether all pixels in the reference image are visible in each source image includes: initializing the visibility weights of all pixels in the reference image in the source image using a view selection strategy; for pixels initially determined to be invisible, re-determining whether the pixel is visible in the source image by cross-view depth reprojection; if it is re-determined to be visible, the visibility weight of the pixel is updated to be greater than 0.

[0088] According to one embodiment of the present invention, a cross-view depth reprojection method is used to re-determine whether the pixel is visible in the source image, including: projecting the pixel into the source image to obtain another pixel corresponding to the pixel, and projecting the other pixel into the reference image to obtain the re-projected pixel; calculating the distance between the invisible pixel and the re-projected pixel, and determining that the pixel is visible in the source image when the distance is less than or equal to a preset distance threshold. According to the visible pixels and invisible pixels of the reference image relative to each source image, a visibility map of the reference image relative to each source image is obtained. Schematically, for the reference image The depth of the invisible area is Invisible pixels First, the homography mapping principle is used to project it onto the source image To get pixels Then, and in The corresponding depth Reproject back To get the pixel after reprojection Finally, define ,when hour, In the source image is visible, thus updating the pixel Relative source image Visibility weight , thus effectively restoring the visibility of textureless areas.

[0089] Indicatively, see Figure 5 , which is a schematic diagram comparing the existing deformation patch principle and the patch deformation principle of the present invention based on whether the pixel is visible. Figure 5 a in is the reference image, Figure 5 b is the source image. The reference image is compared with the source image, and the visible area and invisible area of ​​the reference image relative to the source image are divided by red lines. The present invention adopts the view selection strategy of the above embodiment to initialize all pixels in the reference image relative to the source image. The corresponding visibility weight is obtained Figure 5 The initial visibility map of c in , where black represents pixels judged to be invisible and white represents pixels judged to be visible. Figure 5 It can be seen from c that in the high-cost and textureless area, there are many pixels in the original visible area that are mistakenly judged as invisible. Therefore, the present invention further adopts the cross-view depth reprojection method of the above embodiment to re-determine whether the pixel is visible in the source image, and obtains Figure 5The restored visibility map of d in , which is the visibility map of the final reference image relative to the source image. Figure 5 As can be seen from d, the present invention greatly improves the accuracy of judging pixel visibility and strikes a balance between regions with good texture and regions without texture. Finally, based on the final visibility map, the present invention performs patch deformation in the visibility region to obtain Figure 5 The result of f in the figure. The existing deformation patch is as follows Figure 5 As shown in Fig. e, the deformed patch contains a large part of the invisible area. Therefore, the visibility constraint of the present invention effectively improves the accuracy of the deformed patch.

[0090] The technical solution of the embodiment corresponding to the above step S232 can at least achieve the following beneficial technical effects: since some areas of the reference image may be invisible areas in the corresponding source image, the deformed patch often contains these invisible areas to increase the matching cost, thereby causing potential distortion. The method of the present invention can judge the visibility of each pixel relative to each source image for each source image, effectively improve the accuracy of pixel visibility judgment, so as to help the subsequent visibility-perceived patch deformation, reduce the deformed patches in the invisible area, and reduce the matching cost.

[0091] Step S233: For each unreliable pixel and each source image, select multiple anchor points only from the specified pixel set in the same area as the unreliable pixel to perform patch deformation, so as to obtain the anchor point set corresponding to each unreliable pixel relative to each source image, wherein the specified pixel set refers to a set of pixels that are visible and reliable relative to the source image.

[0092] According to one embodiment of the present invention, the set of pixels that are visible and reliable relative to the source image refers to: each pixel in the set is a reliable pixel in the same area as the unreliable pixel, and the reliable pixel is a visible pixel in the source image. The anchor point set corresponding to each unreliable pixel relative to each source image is as follows:

[0093] , (6)

[0094] in, Indicates unreliable pixels Relative The anchor point set of source images, Represents the anchor point set No. Anchor points, Indicates that only the pixels with unreliable The initial set of anchor points selected among reliable pixels in the same region, Indicates unreliable pixels The area where you are located, Indicates anchor point Relative The visibility weight of each source image.

[0095] Step S234, reconstructing a depth estimation value of each unreliable pixel according to the anchor point set corresponding to each unreliable pixel relative to each source image;

[0096] According to one embodiment of the present invention, a method of reconstructing a depth estimation value of each unreliable pixel includes minimizing a multi-view aggregation cost to reconstruct a depth estimation value of each unreliable pixel, wherein the multi-view aggregation cost is calculated as follows:

[0097] , (7)

[0098] in, represents the multi-view aggregation cost, Indicates unreliable pixels. represents the set of anchor points corresponding to all source images. represents the number of source images, Indicates the source image number, Indicates Unreliable pixels of the reference image Relative The visibility weight of Zhang Yuan image, express and The matching cost of the deformation between Relative The set of anchor points for a source image.

[0099] According to one embodiment of the present invention, the deformation matching cost in the APD-MVS algorithm is calculated as follows:

[0100] , (8)

[0101] in, Represents the reference image Unreliable pixels With source image The matching cost between pixels in is The method of the present invention obtains The anchor point set of source images, represents the weight parameter, Represents the reference image Unreliable pixels With source image The correlation between pixels in The calculation principle is the same as the above formula (1). Represents the reference image Anchor point With source image The matching cost between pixels in is The calculation principle is the same as the above formula (1). Relative The sum of the number of anchor points in the anchor point set of the source images.

[0102] Step S24: Obtain a current depth map based on the depth estimation values ​​of all pixels, and use the depth map obtained in the last iteration as the depth map of the reference image.

[0103] According to an embodiment of the present invention, the step S24 further includes setting a normal range value and a depth range value for each unreliable pixel to propagate and refine the depth map of this time. Figure 6 , which is a schematic diagram of the process principle of propagation and refinement. The propagation and refinement method includes propagation and refinement for normal and depth. The following is an explanation of the propagation and refinement of the two aspects:

[0104] 1) Propagation and refinement of normals

[0105] According to one embodiment of the present invention, based on the visibility map of the reference image relative to each source image, multiple visible source images relative to the unreliable pixel are obtained, wherein when the unreliable pixel is a visible pixel relative to the source image, the source image is used as a visible source image. The unreliable pixel is mapped to multiple visible source images to obtain mapped pixels of multiple visible source images, the camera centers of the multiple visible source images are respectively connected to the corresponding mapped pixels to obtain the cone direction of each visible source image for the pixel, and the normal range value of the pixel is set according to all cone directions.

[0106] According to one embodiment of the present invention, Figure 6 As shown in a, to ensure the geometric consistency of multiple views, according to the reference image Visibility maps relative to each source image to identify the unreliable pixels All visible source images, all visible source images with Figure 6 The three light blue graphs of a are represented and recorded as , and , the unreliable pixel Mapped to 3 visible source images , and , to get the mapping pixels of the three visible source images , and , and As an unreliable pixel map reference image The resulting mapped pixels (i.e. Considered as ). Another example Figure 6 As shown in a and b, the camera centers of multiple visible source images are connected to the corresponding mapping pixels respectively to obtain the mapping of each visible source image to the pixel. The viewing cone direction is , and For each pixel The viewing cone direction is , and , reference image For this pixel The viewing cone direction is recorded as .like Figure 6 The normal aggregation principle shown in b, first, since each of the visible source images has a visible hemisphere in its viewing cone direction, the visible hemisphere is represented by the viewing cone, and the blue viewing cone corresponds to the reference image , while the green, red, and yellow cones correspond to the visible source image , and , therefore, all visible hemispheres can be aggregated to constrain the 3D points Second, for any visible source image , considering its viewing cone direction and Normal Angle between Should exceed (Right now ), i.e. the visible source image , and The respective viewing cone directions and The angle between the normals of , and Finally, by The normal range of the image is reprojected back to the reference image Pixels in , effectively limiting The normal range value aggregated by the present invention acts as a visibility constraint during the propagation and refinement process, thereby enhancing the reliability of the normal assumption.

[0107] 2) Propagation and refinement for depth

[0108] According to one embodiment of the present invention, based on the maximum disturbance value and the minimum disturbance value preset for the mapping pixel of each visible source image, pixels are selected along the pole line of the source image where the mapping pixel is located and symmetrically about the mapping pixel, the selected pixels are mapped to the reference image, and the depth range value of the pixel assumed in each visible source image is obtained, and the corresponding depth range values ​​in all visible source images are aggregated to obtain the set depth range value.

[0109] According to an embodiment of the present invention, a method for obtaining a depth range value of the pixel assumed under each visible source image includes: first, the preset maximum disturbance value is and the minimum perturbation value is , fixed-length sampling is back-projected and aggregated along the multi-view epipolar lines to fine-tune the depth interval. Specifically, Figure 6 As shown in c, for the reference image Each pixel in , as described in the above embodiment, in the visible source image Mid-polar line Determine its mapping pixel Then, four pixels are selected according to the maximum disturbance value and the minimum disturbance value. The four pixels are , , and , located in pixels On both sides of the polar line superior, and exist to the left of They are and Location, and And distance They are and Finally, these four pixels are further inversely projected onto the reference image The corresponding depths are obtained as follows: , , and , thus forming two refinement intervals: and . and Assumed to be the pixel under each visible source image Depth The depth range value of The depth range value is then projected onto the source image to obtain the assumed mapping pixel, which will cause the assumed mapping pixel in the source image to have no displacement or to have an excessive displacement. The present invention assumes the mapping pixel range with fixed-length sampling pixels in the visible source image, and projects it onto the reference image to refine the depth interval. This adaptively assumed depth range value ensures that the assumed mapping pixel can achieve appropriate displacement along the epipolar line without being restricted by the current depth, thereby improving the reliability of the depth assumption.

[0110] According to one embodiment of the present invention, in order to ensure that the mapped pixels experience significant displacement in as many source images as possible, the depth range values ​​of all visible source images are aggregated in the following manner:

[0111] , (9)

[0112] in, The number of the visible source image. Indicates the total number of visible source images, represents the selection of the image with the maximum perturbation value from all visible source images The minimum depth corresponding to , Represents the selection of the image with the minimum perturbation value from all visible source images The maximum depth corresponding to , Represents the selection of the image with the minimum perturbation value from all visible source images The maximum depth corresponding to , represents the selection of the image with the maximum perturbation value from all visible source images The minimum depth corresponding to .

[0113] In step S3, the depth maps of all reference images are fused to obtain the three-dimensional point cloud data of the scene.

[0114] According to one embodiment of the present invention, see Figure 7 , which is a schematic diagram of the complete processing principle of the multi-view stereo reconstruction method of the present invention. The method of the present invention performs multiple downsampling on a series of input images and performs multiple iterations to estimate the depth map of each image, fuse the depth maps of all reference images, and obtain the three-dimensional point cloud data of the scene. In each iterative estimation process, first, for reliable pixels, the traditional PM algorithm is used to estimate the depth value of the pixel. For unreliable pixels, the anchor point set obtained by the traditional deformation PM algorithm is used to estimate the depth value of the pixel. Based on the depth edge alignment prior and visibility prior, patch deformation is further performed to estimate the depth value of the pixel, and the depth map of this time is obtained based on the depth values ​​of all pixels estimated this time. The following is an overall description of the depth value estimation process of unreliable pixels:

[0115] 1) The generation process of deep edge alignment prior includes:

[0116] The monocular depth estimation model Depth Anything V2 is used to estimate the depth of the reference image to obtain the initial depth map , use Roberts operator extractor for edge detection to get the initial edge map , align the initial depth map and the initial edge map. The alignment process includes: using the RANSAC algorithm to obtain the estimated plane of each scattered area and obtain the scattered area map, performing intra-region erosion processing on each scattered area to obtain the eroded map, performing inter-region dilation processing on each scattered area to obtain the dilated map, and refining the edges of each scattered area to obtain the final uniform boundary image, which is used as the depth edge alignment prior.

[0117] Among them, the anchor point set obtained by the traditional deformation PM algorithm is Based on the depth edge alignment prior, we further select anchor points from reliable pixels in the same area as unreliable pixels for patch deformation to obtain the initial anchor point set. .

[0118] 2) The process of generating visibility priors includes:

[0119] The view selection strategy is used to initialize the visibility weights of all pixels in the reference image in the source image. For example, the initialization results are the 0th pixel, ..., the The visibility weights of each pixel are ,……, ; For the initially determined invisible pixel, whether the pixel is visible in the source image is re-determined by cross-view depth reprojection. If it is re-determined to be visible, the visibility weight of the pixel is updated to be greater than 0. For example, the initialized 0th pixel, ..., The visibility weights of each pixel are ,……, , and the restored visibility graph is obtained.

[0120] Among them, the initial anchor point set based on unreliable pixels In the view selection process, it is also necessary to determine whether the anchor point is visible in the source image. For each source image, according to its corresponding restored visibility map, Select the anchor points that are visible pixels in the source image to perform patch deformation, and obtain the anchor point set relative to each source image. reference images, the anchor point set relative to the 0th source image is denoted as , relative to The anchor point set of the source image is denoted as The set of anchor point sets of all source images is used as the anchor point set obtained by the final patch deformation of the present invention. The process in part 2) is repeated to implement multiple iterative updates of the visibility weight of each pixel, thereby obtaining a more accurate visibility map.

[0121] 3) Anchor point set obtained based on the final patch deformation Carry out communication refinement, including:

[0122] by Figure 6 For example, for each unreliable pixel, its normal range value and depth range value are set to propagate and refine the depth map of this time. The principle of the setting process of the normal range value and the depth range value is the same as that of the above embodiment, and will not be repeated here.

[0123] In order to verify the beneficial effects of the present invention, the inventors conducted the following experimental comparisons:

[0124] Experimental platform parameters:

[0125] CPU: Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz

[0126] GPU: NVIDIA GeForce 3080 RTX

[0127] Operating system: Windows 10

[0128] Code running environment: C++ with CUDA

[0129] Experimental process:

[0130] 1. Conduct comparative experiments

[0131] The present invention uses the ETH3D dataset and the TNT (Tanks and Temples) dataset to conduct comparative experiments on the method of the present invention (abbreviated as DVP-MVS) and the prior art on the above experimental platform. The prior art includes the application of more advanced learning-based methods, including PatchMatchNet method, IterMVS-LS method, MVSTER method, EPP-MVSNet method, EPNet method, and traditional methods, including PCF-MVS method, ACMM method, ACMMP algorithm, SD-MVS method, APD-MVS algorithm and HPM-MVS++ method. About parameter setting: corresponding to the preset threshold when performing regional planarization ; In formulas (3) and (4) of the above embodiment: , ; In formulas (4) and (5) of the above embodiment: , , , preset maximum disturbance value and minimum disturbance value: , ; The total number of source images visible in formula (9) of the above embodiment .

[0132] (1) Experiments on the ETH3D dataset

[0133] Using the ETH3D dataset, the dataset was divided into a training set (Train) and a test set (Test). At a threshold of 2 cm, a comparative experiment was conducted between the method of the present invention (abbreviated as DVP-MVS) and the prior art. During the experiment, the performance results of each method in terms of F1 score (F1 Score), completeness (Comp.) and accuracy (Acc.) were evaluated. The experimental results are shown in Table 1 below:

[0134] Table 1

[0135]

[0136] In Table 1, the best results are marked in bold, and the suboptimal results are marked in underline. According to Table 1 above, it can be seen that the method of the present invention achieves the highest F1 score and completeness.

[0137] (2) Experiments on the TNT dataset

[0138] Using the TNT dataset, divided into the TNT Intermediate dataset and the TNT Advanced dataset, the method of the present invention is compared with the prior art under a given threshold. During the experiment, the performance of each method in three aspects, namely, F1 score, recall rate (Recall, Rec.) and precision rate (Pre.) is evaluated. The experimental results are shown in Table 2 below:

[0139] Table 2

[0140]

[0141] In Table 2, the best results are marked in bold and the suboptimal results are underlined. According to Table 2, the proposed method achieves the highest F1 score and completeness in the TNT intermediate dataset. The second best result is achieved in the TNT advanced dataset, with an F1 score less than 0.3% lower than that of EPNet.

[0142] Overall, our method achieves the highest F1 score and completeness in both the ETH3D dataset and the TNT Intermediate dataset, validating its state-of-the-art performance and strong generalization ability.

[0143] Finally, see Figure 8 , which is a schematic diagram of the results of testing the method of the present invention and other prior arts in combination with the ETH3D dataset. Figure 8 Part a in the figure includes three scene images, which are office images, delivery area images, and lecture room images in the ETH3D dataset. Figure 8 Parts b, c, d, e and f are respectively the existing IterMVS LS method, the existing ACMMP method, the existing HPM-MVS method, the existing APD-MVS algorithm and the method of the present invention. Figure 8 The result image after processing the three scene images in part a. In the result image, the red box shows some challenging areas. Obviously, the proposed method performs particularly well in large texture-free areas (such as floors and walls), generating the most complete and realistic reconstructed point cloud without introducing detail distortion.

[0144] 2. Ablation experiment

[0145] With respect to the preset deformation algorithm in the above embodiment, only the method of dividing the reference image into a plurality of different regions in step S231 is subjected to the following ablation:

[0146] The entire part of this step is deleted, and the resulting preset deformation algorithm is recorded as w / o.Agn;

[0147] Only the erosion processing part in the region of this step is deleted, and the preset deformation algorithm obtained is recorded as w / o.Ero;

[0148] Only the inter-region dilation processing part of this step is deleted, and the preset deformation algorithm obtained is recorded as w / o.Dil;

[0149] Only the edge refinement part of this step is deleted, and the resulting preset deformation algorithm is recorded as w / o.Fil.

[0150] With respect to the preset deformation algorithm in the above embodiment, only the step S232: evaluating whether all pixels in the reference image are visible in each source image is ablated as follows:

[0151] Delete the entire step S232, and the obtained preset deformation algorithm is recorded as w / o.Cro;

[0152] Only the part that uses cross-view depth reprojection to re-determine whether the pixel is visible in the source image (restore the visibility map) is deleted, and the resulting preset deformation algorithm is recorded as w / o.Res;

[0153] The visibility-aware patch deformation is removed (i.e., the anchor points are selected without restricting them to be visible pixels), and the resulting preset deformation algorithm is denoted as w / o.Vis.

[0154] With respect to the propagation refinement process in the above embodiment, the following ablation is performed:

[0155] The constraints of the normal range value and the depth range value set in the entire propagation refinement process are deleted, and the preset deformation algorithm obtained is recorded as w / o.Geo;

[0156] Only the propagation part based on the set normal range value is deleted, and the preset deformation algorithm is recorded as w / o.Pro;

[0157] Only the part that is refined based on the set normal range value is deleted, and the resulting preset deformation algorithm is recorded as w / o.Ref;

[0158] Only the set part of the depth range value is deleted, and the preset deformation algorithm obtained is recorded as w / o.Dep.

[0159] On the ETH3D dataset, the above 12 ablation cases were experimented with given thresholds of 2cm and 10cm, and the effectiveness of each part of the preset deformation algorithm in the method of the present invention was demonstrated by evaluating the three indicators of F1 score, Comp. and Acc. The quantitative results of the ablation study are shown in Table 3 below:

[0160] Table 3

[0161]

[0162] As can be seen from Table 3 above, for step S231: w / o.Agn. has the worst F1 score, highlighting the importance of edge constraints on patch deformation. The F1 score of w / o.Ero. is slightly better than that of w / o.Dil., which means that the impact of region expansion is greater than that of region erosion. In addition, the F1 scores of w / o.Dil. and w / o.Fil. are basically the same, indicating that region erosion and pixel-by-pixel filtering are equally important. For step S232: In contrast, w / o.Cro. has the lowest F1 score, verifying the importance of visibility prior for visibility-aware patch deformation. In addition, the F1 score of w / o.Res. is higher than that of w / o.Vis., emphasizing that visibility-aware patch deformation is more important than restoring the visibility map. For the propagation refinement part: w / o.Geo., which does not have normal range values ​​and depth range values ​​as constraints, has the worst F1 score. The F1 score of w / o.Pro. is better than that of w / o.Ref., indicating that normal constraints are more beneficial to refinement than to propagation. In addition, the F1 scores of w / o.Ref. and w / o.Dep. are similar, indicating that the normal constraint and the depth constraint have equal contributions. Finally, it can be seen that the overall solution of the DVP-MVS method of the present invention has the best effect.

[0163] The prior art is as follows:

[0164] PatchMatchNet method: Wang F, Galliani S, Vogel C, et al. Patchmatchnet:Learned multi-view patchmatch stereo[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2021: 14194-14203.

[0165] IterMVS-LS method: Wang, F.; Galliani, S.; Vogel, C.; and Pollefeys, M. 2022a IterMVS: Iterative probability estimation for efficient multi-viewstereo. In Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 8606–8615.

[0166] MVSTER method: Wang, X.; Zhu, Z.; Huang, G.; Qin, F.; Ye, Y.; He, Y.; Chi, X.; and Wang, X. 2022b. MVSTER: Epipolar Transformer for Efficient Multi-View Stereo. In European Conference on Computer Vision, 573–591. Springer.

[0167] EPP-MVSNet method: X. Ma, Y. Gong, Q. Wang, J. Huang, L. Chen, F. Yu, EPP-MVSNet: Epipolar assembling based depth prediction for multi-view stereo, in: Proc. IEEE / CVF Int. Conf. Comput. Vis., ICCV, 2021, pp. 5712–5720.

[0168] EPNet method: Su, W.; and Tao, W. 2023. Efficient Edge-Preserving Multi-View Stereo Network for Depth Estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2348–2356.

[0169] PCF-MVS method: A. Kuhn, S. Lin, O. Erdler, Plane completion and filtering for multi-view stereo reconstruction, in: Pattern Recognition, 2019, pp. 18–32.

[0170] ACMM method: Q. Xu, W. Tao, Multi-scale geometric consistency guided multi-view stereo, in: Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit., CVPR, 2019.

[0171] ACMMP algorithm: Xu Q, Kong W, Tao W, et al. Multi-scale geometric consistency guided and planar prior assisted multi-view stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4): 4945-4963.

[0172] SD-MVS method: Yuan, Z.; Cao, J.; Li, Z.; Jiang, H.; and Wang, Z. 2024a. SD MVS: Segmentation-driven Deformation Multi-View Stereo with Spherical Refinement and EM Optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 38(7): 6871–6880.

[0173] APD-MVS algorithm: Wang Y, Zeng Z, Guan T, et al. Adaptive patch deformation for textureless-resilient multi-view stereo[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 1621-1630.

[0174] HPM-MVS++ method: Ren, C.; Xu, Q.; Zhang, S.; and Yang, J. 2023. Hierarchical Prior Mining for Non-Local Multi-View Stereo. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 3611–3620.

[0175] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0176] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0177] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0178] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A multi-view stereoscopic reconstruction method, characterized in that: Methods include: S1, obtaining multiple images of the same scene captured from different perspectives, taking each of the multiple images as a reference image and the remaining images as source images corresponding to the reference image; S2. Perform multiple iterations based on the reference image and each source image to estimate a depth map of each reference image, wherein each iteration process includes: S21, evaluating whether the depth estimation value of each pixel in the depth map obtained this time is reliable, and obtaining reliable pixels and unreliable pixels, wherein the initialized depth map is used for the first time, and the depth map obtained by upsampling the previous depth map is used each time thereafter; S22, using the PatchMatch algorithm to update the depth estimation value of each reliable pixel; S23, reconstructing the depth estimation value of each unreliable pixel using a preset deformation algorithm, including: S231, dividing the reference image into a plurality of different regions, each region being estimated as a depth continuous region; S232, evaluating whether all pixels in the reference image are visible in each source image, and obtaining visible pixels and invisible pixels of the reference image relative to each source image; S233, for each unreliable pixel and each source image, select multiple anchor points only from a specified pixel set in the same area as the unreliable pixel to perform patch deformation, so as to obtain an anchor point set corresponding to each unreliable pixel relative to each source image, wherein the specified pixel set refers to a set of pixels that are visible and reliable relative to the source image; S234, reconstructing a depth estimation value of each unreliable pixel according to an anchor point set corresponding to each unreliable pixel relative to each source image; S24, obtaining a depth map of this time based on the depth estimation values ​​of all pixels, and using the depth map obtained in the last iteration as the depth map of the reference image; S3. Fuse the depth maps of all reference images to obtain the three-dimensional point cloud data of the scene.

2. The method according to claim 1, characterized in that In the S231, the method of dividing the reference image into a plurality of different regions includes: Use the depth estimation model to estimate the depth of the reference image and obtain an initial depth map; Perform edge detection on the reference image using an edge detection algorithm to obtain an initial edge map; The initial depth map and the initial edge map are aligned to generate a boundary image, which includes a plurality of different regions, and the depths of all pixels in each region in the corresponding initial depth map are continuous.

3. The method according to claim 2, characterized in that The initial edge map includes information of each initial scattered area, and the alignment processing method includes: dividing the initial scattered area according to the initial depth map and the information of the initial scattered area according to a preset division rule to obtain multiple areas, wherein the preset division rule is as follows: ， in, Indicates Initial dispersion area, Indicates the first Regions, Indicates the first Regions, , represents the plane similarity function, And it means The estimated plane after the region is planarized, Indicates The normal of the pixel at the center of the region, Indicates The depth of the center pixel of the region, And it means The estimated plane after the region is planarized, Indicates The normal of the pixel at the center of the region, Indicates The depth of the center pixel of the region, It means taking the minimum of two values. Indicates taking the absolute value, Indicates the preset similarity threshold, Indicates The ratio of the internal point pixels of a region to the total pixels of the region, Indicates The ratio of the internal point pixels of a region to the total pixels of the region, Indicates The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, Indicates a preset first ratio threshold.

4. The method according to claim 3, characterized in that The alignment processing method further includes: for any two initial scattered areas, according to the initial depth map and the respective information of the two initial scattered areas, merging the two initial scattered areas that meet the preset merging rule into one area, wherein the merging method is as follows: ， in, Indicates Initial dispersion area, Indicates Initial dispersion area, The merged Regions, Indicates The estimated plane after the initial scattered area is flattened, Indicates The estimated plane after the initial scattered area is flattened, Indicates The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, Indicates The ratio of the inlier pixels of the initial scattered region to the total pixels of the region, Indicates a preset second ratio threshold.

5. The method according to claim 1, characterized in that In S232, the method of evaluating whether all pixels in the reference image are visible in each source image includes: Initialize the visibility weights of all pixels in the reference image in the source image using the view selection strategy, where a visibility weight of 0 indicates that the corresponding pixel is invisible, and a visibility weight greater than 0 indicates that the corresponding pixel is visible; For a pixel initially determined to be invisible, a cross-view depth reprojection method is used to re-determine whether the pixel is visible in the source image. If it is re-determined to be visible, the visibility weight of the pixel is updated to be greater than 0.

6. The method according to claim 5, characterized in that The method of re-determining whether the pixel is visible in the source image by cross-view depth reprojection includes: Projecting the pixel into the source image to obtain another pixel corresponding to the pixel, and projecting the other pixel into the reference image to obtain a re-projected pixel; The distance between the invisible pixel and the re-projected pixel is calculated, and when the distance is less than or equal to a preset distance threshold, it is determined that the pixel is visible in the source image.

7. The method according to any one of claims 1 to 6, characterized in that: In the S234, the method of reconstructing the depth estimation value of each unreliable pixel includes minimizing the multi-view aggregation cost to reconstruct the depth estimation value of each unreliable pixel, wherein the multi-view aggregation cost is calculated as follows: ， in, represents the multi-view aggregation cost, Indicates unreliable pixels. represents the set of anchor points corresponding to all source images. represents the number of source images, Indicates the number of the source image, Indicates Unreliable pixels of the reference image Relative The visibility weight of Zhang Yuan image, express and The matching cost of the deformation between Relative The anchor point set of source images, , Represents the anchor point set No. Anchor points, Indicates that only the pixels with unreliable The initial set of anchor points selected among reliable pixels in the same region, Indicates unreliable pixels The area where you are located, Indicates anchor point Relative The visibility weight of each source image.

8. The method according to any one of claims 1 to 6, characterized in that: The S24 further includes setting the normal range value and the depth range value of each unreliable pixel in the following manner to propagate and refine the depth map of this time: Obtaining a visibility map of the reference image relative to each source image according to visible pixels and invisible pixels of the reference image relative to each source image, and acquiring a plurality of visible source images relative to the unreliable pixel according to the visibility map, wherein when the unreliable pixel is a visible pixel relative to the source image, the source image is used as a visible source image; Mapping the unreliable pixel to multiple visible source images to obtain mapping pixels of the multiple visible source images, connecting the camera centers of the multiple visible source images to the corresponding mapping pixels respectively, obtaining the viewing cone direction of each source image for the pixel, and setting the normal range value of the pixel according to all viewing cone directions; According to the maximum disturbance value and the minimum disturbance value preset for the mapping pixel of each visible source image, pixels are selected along the pole line of the source image where the mapping pixel is located and symmetrically about the mapping pixel, and the selected pixels are mapped to the reference image to obtain the depth range value of the pixel assumed in each visible source image, and the corresponding depth range values ​​in all visible source images are aggregated to obtain the set depth range value.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 8 by executing the executable instructions.

Images