A multi-view stereo reconstruction method based on segmentation driven and edge-aligned deformation

Abstract

The application provides a multi-view stereoscopic reconstruction method based on segmentation driving and edge alignment deformation, comprising: acquiring multiple images of different views under the same scene, taking each image as a reference image, and taking the rest of the images as source images; fusing estimated depth maps of all reference images to obtain three-dimensional point cloud data of the scene; estimating each depth map, comprising: segmenting regions of each object instance in the reference image, and constructing occlusion relationships between the regions of each object instance; iteratively optimizing an initial recovered depth map of the reference image multiple times to obtain a depth map, each iteration comprising: acquiring a recovered depth map of this time; for each pixel of the reference image, detecting multiple boundary pixels corresponding to the region to which the pixel belongs with the pixel as the center, sequentially connecting the multiple boundary pixels to obtain a deformation block, estimating a depth value of the pixel based on a constraint condition preset based on the occlusion relationship, the deformation block, the reference image, the recovered depth map and the source image; and obtaining a depth map of this time based on the estimated depth values of all pixels.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to the technical field of three-dimensional reconstruction and multi-view stereo matching, and more particularly to a multi-view stereo reconstruction method based on segmentation driving and edge alignment deformation. BACKGROUND

[0002] Multi-view stereo (MVS for short) is a fundamental task in computer vision, which aims to reconstruct the dense three-dimensional structure of a scene or object by utilizing multiple images taken from different viewpoints. MVS has a wide range of applications in cultural heritage preservation, autonomous driving, and medical imaging, which has given rise to many related datasets (such as the ETH3D dataset) and excellent methodologies. Despite the many advances, MVS still faces many problems such as occlusion, shadow, textureless regions, and Lambertian surfaces. These problems hinder the efficiency and quality of scene or object reconstruction.

[0003] In order to achieve efficient and high-quality three-dimensional reconstruction, various MVS methods have been proposed, which can be mainly divided into two categories: learning-based MVS and traditional MVS. Learning-based MVS uses convolutional neural networks to extract high-dimensional representations of three-dimensional cost volumes for reconstruction. However, this type of method usually faces problems such as high memory consumption or limited generalization ability. In contrast, traditional MVS originated from the PatchMatch (PM for short) algorithm, which constructs a solution space by propagating neighborhood assumptions and generating optimized assumptions, and then selects the optimal solution through a criterion called multi-view matching cost. However, when facing scenes with textureless regions, due to the lack of distinguishable features within the receptive field of its patches, its matching cost inevitably becomes unreliable.

[0004] For the reconstruction of textureless regions, traditional MVS proposes three main techniques:

[0005] 1) Coarse-to-fine strategy: adopt pyramid structure, process multi-scale images from coarse to fine, so that patches can extract multi-scale features for reconstruction. However, due to the limited ability of pyramid architecture in effectively preserving high-level information, the coarse-to-fine scheme performs poorly in reconstructing large-scale textureless regions.

[0006] 2) Planarization strategy (TSAR-MVS method provided in reference [1]): then the reliable pixels in the well-textured area are stitched to form a connected region, which is considered as a non-textured region and performs planarization reconstruction. Although various strategies such as superpixels (TSAR-MVS method), triangulation (ACMMP method provided in reference [2]) and KD-tree (HPM-MVS method provided in reference [3]) are adopted, these methods are still limited by the size of the connected region and prone to planarization bias.

[0007] 3) Block deformation strategy: unlike the above two strategies, it cleverly expands the receptive field by deforming the fixed-size block to perform reconstruction. Since it is not limited by the number of pyramid layers or the size of the connected region, block deformation can flexibly handle non-textured regions of any size. For example, the API-MVS method provided in reference [4] introduces entropy and dilated convolution on the block to dynamically adjust the block size and sampling interval. In addition, the APD-MVS method provided in reference [5]: expands outward from each unreliable pixel to locate its related reliable pixel, and then builds multiple sub-blocks centered on it to calculate the matching cost.

[0008] However, these block deformation strategies mainly focus on improving the search strategy of reliable pixels to minimize matching ambiguity, but ignore the key premise of deformation stability. As in the APD-MVS method, the unreliable green pixel is centered, the block deformation method first divides the surrounding area into fixed-angle sectors, then searches for reliable red pixels within each sector, and builds sub-blocks centered on them to form a deformable PM, which is used to replace the traditional PM for reconstruction. However, it ignores the premise of depth continuity within the block, and the deformed block may mistakenly cross important depth edges, resulting in potential matching distortion in depth discontinuous regions. The SD-MVS method provided in reference [6]: adjusts the block ratio according to the distance from the pixel to the depth edge boundary in the horizontal and vertical directions, although this method can achieve feature block deformation to some extent, it is still limited by the fixed block size, which cannot achieve accurate edge alignment in complex scenes, especially when dealing with irregularly shaped objects, which is prone to edge jumping problems.

[0009] Based on the above analysis, the APD-MVS method ignores the depth edge, which causes the deformed block to be selected in the depth discontinuous region, resulting in potential matching distortion problems; the ACMMP method is difficult to reconstruct non-textured regions because its fixed-size block cannot capture enough feature points; and the SD-MVS method is still limited by the fixed block size and lacks edge alignment capability, which is only suitable for regular geometric objects, and has low reconstruction efficiency and poor quality in complex shape scenes.

[0010] Therefore, in the three-dimensional reconstruction of complex shape scenes, the existing method is limited by the fixed block size and lacks edge alignment capability, and is only suitable for regular geometric objects, and has the problems of low precision and poor reliability in the reconstruction of complex shape scenes.

[0011] It should be noted that the background art is only used to introduce the relevant information of the present application, so as to help understand the technical solutions of the present application, but does not mean that the relevant information must be the prior art. The relevant information is submitted and disclosed together with the present application scheme, and in the absence of evidence that the relevant information has been publicly disclosed before the filing date of the present application, the relevant information should not be regarded as prior art.

[0012] The references are as follows:

[0013] [1] Yuan Z, Cao J, Wang Z, et al. TSAR-MVS: Textureless-aware segmentation and correlative refinement guided multi-view stereo[C]. IEEE / CVF International Conference on Computer Vision, 2023.

[0014] [2] Xu Q, Kong W, Tao W, et al. Multi-scale geometric consistency guided and planar prior assisted multi-view stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4): 4945-4963.

[0015] [3] Ren C, Xu Q, Zhang S, et al. Hierarchical Prior Mining for Non-local Multi-View Stereo[C]. IEEE / CVF International Conference on Computer Vision, 2023, pp. 3611-3620.

[0016] [4] Sun S, Liu J, Li Y, et al. Adaptive Pixelwise Inference Multi-View Stereo[C]. Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2021, pp. 1631-1640.

[0017] [5] Wang Y, Zeng Z, Guan T, et al. Adaptive patch deformation for textureless-resilient multi-view stereo[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 1621-1630.

[0018] [6] Yuan, Z.; Cao, J.; Li, Z.; Jiang, H.; and Wang, Z. 2024a. SD MVS:Segmentation-driven Deformation Multi-View Stereo with Spherical Refinement and EM Optimization. Proceed ings of the AAAIConference onArtificialIntelligence, 38(7): 6871–6880. SUMMARY

[0019] Therefore, the purpose of the present application is to overcome the defects of the prior art described above, and to provide a multi-view stereo reconstruction method based on segmentation driving and edge alignment deformation.

[0020] The purpose of the present application is achieved by the following technical solutions:

[0021] According to a first aspect of the present application, a multi-view stereo reconstruction method is provided, the method comprising: acquiring a plurality of images collected from different perspectives under the same scene, taking each image as a reference image and taking the remaining images as source images of the reference image, estimating a respective depth map of each reference image, and fusing all the depth maps to obtain three-dimensional point cloud data of the scene. The estimation of the depth map of each reference image comprises: S1, dividing the regions of each object instance from the reference image to obtain a corresponding segmentation map, and constructing occlusion relationships between the regions of different object instances based on the segmentation map; S2, performing multiple iterations of optimization based on an initial recovered depth map of the reference image to obtain the depth map, wherein each iteration comprises: S21, obtaining a recovered depth map, wherein the initial recovered depth map is used for the first time and the depth map after the previous iteration is used for each time after the first time; S22, for each pixel of the reference image, detecting a plurality of boundary pixels corresponding to the region of the segmentation map to which the pixel belongs, connecting the plurality of boundary pixels in turn to obtain a deformed block, and estimating the depth value of the pixel based on a constraint condition preset based on the occlusion relationship, the deformed block, the reference image, the recovered depth map, and all the source images; and S23, obtaining the depth map after the current iteration based on the estimated depth values of all the pixels.

[0022] In some embodiments of the present application, in the S22, for each pixel of the reference image, the deformed block is obtained in the following manner: extracting the boundaries of the corresponding segmentation map of the reference image to obtain a boundary map comprising a plurality of boundary pixels formed between the regions of each object instance; extending in multiple directions from each pixel of the reference image until the boundary pixels of the boundary map are intersected for the first time to detect a plurality of boundary pixels intersected in multiple directions; and connecting the plurality of boundary pixels intersected in multiple directions corresponding to each pixel in turn to obtain the deformed block.

[0023] In some embodiments of the present application, in the S1, the occlusion relationship is constructed in the following manner: generating a monocular depth map of the reference image using a monocular depth estimation method, calculating the depth gradient of the boundary pixels according to the monocular depth map, generating an occlusion map of the corresponding boundary map according to the depth gradient of all the boundary pixels, the occlusion map recording whether each boundary pixel belongs to a depth-continuous category or a depth-discontinuous category with the adjacent regions thereof, and determining whether there is occlusion between the regions of each object instance according to the occlusion map to obtain the occlusion relationship.

[0024] In some embodiments of the present application, in the S22, the manner of estimating the depth value of the pixel comprises: based on the constraint condition, and according to a preset sampling strategy, sampling pixels from the deformed block corresponding to the pixel and a preset peripheral region of the pixel, to obtain a plurality of sampling pixels, wherein the constraint condition comprises: when the peripheral region belongs to a region that has an occlusion relationship with a region to which the deformed block belongs, no sampling is performed from the peripheral region, otherwise, sampling is performed from the peripheral region; based on the depth values of the plurality of sampling pixels in the recovered depth map, a depth hypothesis value of the pixel is generated, and the depth hypothesis value is optimized to estimate the depth value of the pixel.

[0025] In some embodiments of the present application, the preset sampling strategy comprises: connecting boundary pixels intersecting with a plurality of directions to the pixel respectively to obtain a plurality of trajectories, and dividing each trajectory into a corresponding number of segments according to a preset sampling number of each trajectory to obtain a plurality of trajectory segments; for each pixel of each trajectory segment of each trajectory, a window of a preset size is constructed with the pixel as the center, and a pixel with the smallest texture coefficient is selected from the window as a mapping pixel; for each trajectory segment of each trajectory, a pixel with the smallest aggregation cost is sampled from the mapping pixels of all pixels of the segment to obtain a sampling pixel, wherein the aggregation cost is obtained based on aggregating all matching costs of the source images and the reference image.

[0026] In some embodiments of the present application, the manner of generating the initial recovered depth map comprises: obtaining a monocular depth map of the reference image, and using a feature matching algorithm to extract features between the reference image and the source image to obtain feature points of the reference image relative to the source image and depth values of the feature points; based on the segmentation map, all feature points in each region of all regions of the segmentation map are processed in a triangulation manner to obtain a plurality of triangular regions, and a coarse depth map is generated based on all triangular regions; based on the monocular depth map, plane fitting based on RANSAC is performed on all pixels of each triangular region, and the coarse depth map is optimized based on the fitting result to obtain the initial recovered depth map.

[0027] In some embodiments of the present application, the manner of optimizing the coarse depth map comprises: based on the fitting result, identifying whether each triangular region is a planar region, the planar region representing a depth continuous region; if the triangular region is not a planar region, for each pixel in the triangular region, the depth value of the pixel in the triangular region is optimized according to the depth values of the vertices of the triangular region in the monocular depth map and the difference between the depth values of the vertices and the depth value of the pixel, otherwise, the depth value of each pixel in the triangular region remains unchanged.

[0028] In some embodiments of the present application, the manner of optimizing the coarse depth map comprises: for each pixel outside all triangular regions, determining a triangular region closest to the pixel and being a planar region with the segmentation map as a constraint, and calculating a distance of the pixel to the triangular region; if the distance is less than a preset distance, optimizing the depth value of the pixel by using a proportional mapping algorithm, otherwise, the depth value of the pixel remains unchanged.

[0029] According to a second aspect of the present application, a computer readable storage medium is provided, having stored thereon a computer program executable by a processor to implement the steps of the method of the first aspect of the present application.

[0030] According to a third aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is configured to store executable instructions; the one or more processors are configured to implement the steps of the method of the first aspect of the present application via execution of the executable instructions.

[0031] Compared with the prior art, the present application has the following advantages:

[0032] The present application first segments the image into regions of each object to obtain a segmentation map, and constructs an occlusion relationship between the regions of each object. Based on the segmentation map and the constraint condition preset based on the occlusion relationship, the deformed block can be prevented from crossing the border to a depth discontinuous region, and the estimated depth value of the pixel is more reliable. For each pixel of the image, a plurality of boundary pixels of the region to which the pixel belongs in the segmentation map are detected, and the plurality of boundary pixels are sequentially connected to obtain a deformed block, so that the deformed block of the present application can more closely fit the edge shape, i.e., the deformed block can more comprehensively cover the depth continuous region, so as to comprehensively consider the irregular characteristics of the edge, dynamically adapt to different edge structures, and significantly improve the edge alignment capability, especially when processing a complex shape scene, thereby improving the efficiency and quality of three-dimensional reconstruction of the complex shape scene. BRIEF DESCRIPTION OF DRAWINGS

[0033] The embodiments of the present application are further described below with reference to the accompanying drawings, in which:

[0034] Figure 1 A flowchart of estimation of a depth map according to an embodiment of the present application is shown;

[0035] Figure 2 A process of constructing an occlusion relationship according to an embodiment of the present application and a result of estimating a depth map under the guidance of an occlusion constraint are shown;

[0036] Figure 3 A process of obtaining an initial recovered depth map by using a sparse-mono collaborative recovery strategy according to an embodiment of the present application and a result of the recovered depth map are shown;

[0037] Figure 4A comparison result diagram between the method of the present application and the block deformation of the prior art according to an embodiment of the present application;

[0038] Figure 5 A principle diagram of the block deformation method of the present application according to an embodiment of the present application;

[0039] Figure 6 A complete flow diagram of the multi-view stereo reconstruction method of the present application according to an embodiment of the present application;

[0040] Figure 7 A result diagram of the present application method and other prior art methods combined with the ETH3D dataset for testing according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose of the present application clearer, the technical solutions and advantages are further described in detail below with specific embodiments combined with the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0042] As mentioned in the background section, when performing three-dimensional reconstruction in a complex shape scene, the prior art method is limited by the fixed block size and lacks edge alignment capability, and is only suitable for regular geometric objects, and has the problems of low precision and poor reliability when reconstructing complex shape scenes.

[0043] In view of the above problems, the inventors propose a multi-view stereo reconstruction method based on segmentation driving and edge alignment deformation. On the one hand, based on the existing PatchMatch algorithm, the present application introduces panoramic segmentation as a depth edge guide, i.e. first segmenting the image to obtain the regions of each object to obtain a segmentation map, and constructing the occlusion relationship between the regions of each object. Based on the segmentation map and the constraint conditions preset based on the occlusion relationship, the deformed blocks can be prevented from crossing the depth discontinuous region, and the estimated depth value of the pixel is more reliable. On the other hand, since the existing method only considers horizontal and vertical deformation, it lacks the ability to accurately align the depth edges of complex shapes. For each pixel of the image, the inventors detect a plurality of boundary pixels of the region to which the pixel belongs in the segmentation map, and sequentially connect the plurality of boundary pixels to obtain a deformed block, so that the deformed block of the present application can more closely fit the edge shape, i.e. the deformed block can more comprehensively cover the depth continuous region, realize comprehensive consideration of the irregular characteristics of the edge, dynamically adapt to different edge structures, and significantly improve the edge alignment capability, especially when processing complex shape scenes, which can improve the efficiency and quality of three-dimensional reconstruction of complex shape scenes.

[0044] According to one embodiment of the present application, the present application is based on the above-mentioned reference [6] SD-MVS method, mainly improving the block deformation algorithm (PatchMatch algorithm in the SD-MVS method) thereof. The present application method needs to obtain multiple images collected from different angles under the same scene when performing multi-view stereo reconstruction, and then takes each image as a reference image and takes the remaining images as source images of the reference image, estimates the depth map of each reference image, and fuses all the depth maps to obtain the three-dimensional point cloud data of the scene. Given a set of multiple images collected from different angles under the same scene, the image set is denoted as , wherein represents any one image, represents the number of images, represents the total number of images, and any one image corresponding camera parameters include intrinsic matrix , rotation matrix and camera center . Each image in is taken as a reference image , and the remaining images are taken as source images of the reference image . Among them, the homographic principle and the camera parameters of each image are used, and the reference image is matched with the remaining source images pair by pair to reconstruct the depth map of the reference image .

[0045] According to one embodiment of the present application, the estimation method of the depth map of each reference image can be referred to Figure 1 , which is a schematic diagram of the estimation process of the depth map. The method includes steps S1 and S2, and step S2 includes steps S21, S22 and S23. In order to better understand the present application, each step will be described in detail below in combination with specific embodiments.

[0046] Step S1, the regions of each object instance are segmented according to the reference image to obtain a corresponding segmentation map, and the occlusion relationship between the regions of different object instances is constructed based on the segmentation map.

[0047] According to one embodiment of the present application, in step S1, the segmentation map is obtained by introducing SAM2 (Segment Anything Model 2) to perform panoramic segmentation on the image, segmenting the regions of each object instance to obtain a segmentation map, and representing the segmentation map as . Wherein, SAM2 is a new generation of general segmentation model launched by Meta. Subsequently, the boundary of the segmentation map is extracted to obtain a boundary map.

[0048] According to an embodiment of the present application, the way of obtaining the boundary map comprises applying a transformation function extracting the boundary of the segmentation map , obtaining a boundary map, and representing the boundary map . Since the depth edges in are usually located between regions of different object instances, the is used as a depth edge constraint guide for the subsequent block deformation.

[0049] Further, the inventors found that using only the boundary map of the segmentation map as the depth edge constraint guide for the block deformation still has the problem of inaccurate reconstruction. To improve this situation, according to an embodiment of the present application, a monocular depth map of the reference image is first generated using DepthAnythingV2, a corresponding occlusion map is generated based on the segmentation map and the monocular depth map, and an occlusion relationship between regions of different object instances is constructed according to the occlusion map. The subsequent occlusion constraint guide of the depth edge is based on the occlusion relationship.

[0050] According to an embodiment of the present application, referring to Figure 2 , which is a schematic diagram of the construction process of the occlusion relationship and the result of estimating the depth map under the occlusion constraint guide. The schematic diagram will be described in combination with the embodiments as follows:

[0051] 1) Based on the Figure 2 , the difference between the depth map estimated under the occlusion constraint guide and the depth map using only the boundary map as the constraint guide is analyzed and compared.

[0052] First, the problem of using only the boundary map as the constraint guide is described as follows:

[0053] As shown in the a image in Figure 2 , the red box is analyzed, which has three object instances, namely instances , and . The a image in Figure 2 is processed to generate the b segmentation map in Figure 2 , in which different colors represent regions of different object instances. Then, based on the boundary map of the segmentation map as the constraint, during the block deformation, the pixel p in is only limited to the region where it is located without crossing the depth edge boundary. However, when is completely composed of textureless regions and has no well-textured pixels, it is difficult to identify enough reliable pixels from the region of the object instance where the pixel p is located to reconstruct the pixel p, which may cause matching ambiguity. Therefore, strictly using the boundary map as the depth edge constraint for the block deformation may not produce the best matching result.

[0054] Secondly, the present application is to solve the problem of only using the boundary map as a constraint guide, and after obtaining the b segmentation map in Figure 2 , the d occlusion map in Figure 2 is constructed based on the b segmentation map in Figure 2 and the e monocular depth map in Figure 2 , and the occlusion relationship of each region is obtained based on the d occlusion map. In the e monocular depth map, the lower the color temperature, the greater the depth value, and vice versa. In the d occlusion map, the red edge represents that the regions of two adjacent object instances have an occlusion relationship, and the blue edge represents that there is no occlusion relationship. The occlusion constraint guide means that reliable pixels can be obtained from the region without an occlusion relationship to reconstruct the depth value, and reliable pixels cannot be obtained from the region with an occlusion relationship to reconstruct the depth value. For example, in the a image in Figure 2 , the object instance of the pixel p has two adjacent object instances and , and the object instance has no occlusion , so the depth of the regions and is continuous near their shared boundary. Therefore, in theory, reliable pixels near the boundary between the regions and can be selected to guide the block deformation of the pixel in to reconstruct the depth value. On the contrary, since the object instance blocks , there is a depth discontinuity on their boundary, so that the reliable pixels near the boundary are not used to guide the block deformation of the pixel in .

[0055] Finally, the depth maps with and without occlusion constraints are compared. In c: the depth map without occlusion constraints in Figure 2 and f: the depth map with occlusion constraints in Figure 2 , the higher the color temperature, the greater the depth value, and vice versa. From the two depth maps, it can be seen that, compared with c in Figure 2 , f: the depth map with occlusion constraints in Figure 2 can effectively distinguish the depth edges in the textureless region.

[0056] 2) The occlusion relationship construction is described based on the Figure 2

[0057] According to an embodiment of the present application, in step S1, the occlusion relationship construction method includes the following steps a1, a2 and a3:

[0058] ​Step a1: generating a monocular depth map of the reference image by using monocular depth estimation method, and calculating the depth gradient of the boundary pixels according to the monocular depth map.

[0059] According to one embodiment of the present application, the depth gradient calculation method comprises: for each boundary pixel in the boundary map , constructing a window with as the center and as the size. Then, for each pixel in the window , calculating the depth gradient of the pixel in the monocular depth map by using Sobel operator, and selecting the maximum gradient value as the depth gradient identifier of the pixel, the depth gradient identifier being defined as:

[0060] , (1)

[0061] wherein, represents the depth gradient identifier of the pixel in the window , represents selecting the maximum gradient value from the gradient values of all the pixels in the window , represents the depth of each pixel in the window , represents the horizontal coordinate, represents the gradient relative to the horizontal coordinate, represents the vertical coordinate, represents the gradient relative to the vertical coordinate.

[0062] Step a2: generating an occlusion map of the corresponding boundary map according to the depth gradient of all the boundary pixels, the occlusion map recording each boundary pixel belonging to a depth-continuous category or a depth-discontinuous category with its adjacent regions.

[0063] According to one embodiment of the present application, if the depth gradient is less than a preset gradient threshold , i.e. when , the label of the boundary pixel is defined as the depth-continuous category with its adjacent regions, and is marked as a blue dot; otherwise, it is defined as the depth-discontinuous category with its adjacent regions, and is marked as a red dot, thereby generating the d-occlusion map in the boundary map Figure 2 . The blue boundary and the red boundary of the d-occlusion map in the boundary map Figure 3 respectively represent the adjacent regions being depth-continuous and the adjacent regions being depth-discontinuous.

[0064] Step a3: determining whether there is occlusion between the regions of each object instance according to the occlusion map, and obtaining the occlusion relationship.

[0065] According to one embodiment of the present invention, Figure 3 The occlusion map in the d occlusion map can be used to determine whether there is occlusion between the regions of each object instance. The occlusion map obtained by the present invention can effectively distinguish the depth continuity across boundaries, thereby exploring the occlusion relationship between different object instances and preparing for subsequent edge constraint guidance.

[0066] The technical solutions of the above embodiments can achieve at least the following beneficial technical effects: by identifying the occlusion relationships between different object instances, edge constraints of varying strengths can be selectively assigned to block deformations. This allows for the selection of reliable pixels near or outside the boundaries based on the occlusion map. This avoids the limitations of existing deformation strategies that employ rigid edge constraints, which often result in inaccurate results.

[0067] Step S2: Perform multiple iterations of optimization based on the initial restored depth map of the reference image to obtain a depth map.

[0068] In terms of obtaining the initial recovered depth map, most existing PatchMatch algorithms achieve depth prediction by randomly initializing depth values ​​within a predefined depth range, and then propagate and refine based on these initializations to perform depth estimation. For example, the existing Mar-MVS method uses sparse points obtained by COLMAP to constrain the pixel depth range for random initialization. However, this random initialization is heavily dependent on the reliability of the depth range and may cause the estimated depth to fall into a local minimum. Therefore, a reliable initial depth map can significantly improve the accuracy of depth estimation. Compared with the existing Mar-MVS method, monocular depth estimation (MDE) can generate geometrically reliable depth maps that are very consistent with the actual outline of each object instance. However, without inter-view projection, MDE can only produce relative depth rather than absolute depth. Therefore, directly applying the relative depth-based MDE to the absolute depth-based MVS for initialization may lead to potential scale ambiguity.

[0069] To eliminate the scale ambiguity present in some of the existing approaches listed above, according to one embodiment of the present invention, a sparse-monocular collaborative recovery strategy is proposed to synthesize high-confidence depth initialization. First, a new visual transformer-based feature matching basis model (LoFTR) is utilized to rapidly capture sufficient depth-reliable sparse points in images from different viewpoints. Second, the depth map is initialized by collaborating the depth-reliable sparse points from the LoFTR with the geometrically realistic depth map from the MDE. This yields the optimal initialization for further optimization steps. The specific implementation of this strategy involves collaboratively recovering the depth map using segmentation-driven sparse point triangulation and geometrically-aware refinement of the monocular depth map to obtain an initial recovered depth map.

[0070] According to one embodiment of the present invention, see Figure 3, which is a process of obtaining an initial recovered depth map by using a sparse-mono collaborative recovery strategy and a result diagram of the recovered depth map. The way of obtaining the initial recovered depth map includes the following steps b1, b2 and b3:

[0071] Step b1: Obtain a monocular depth map of a reference image, and use a feature matching algorithm to extract features between the reference image and a source image to obtain feature points of the reference image relative to the source image and depth values of the feature points.

[0072] According to an embodiment of the present application, a feature matching algorithm (such as LoFTR) is used to extract features between the reference image and the source image to obtain feature points (also known as sparse points) of the reference image relative to the source image and depth values of the feature points. Illustratively, as shown in Figure 3 , the image a in the figure is a reference image, and features between the image a and another source image are extracted by the feature matching algorithm to obtain Figure 3 , as shown in Figure 3 , b: feature matching result obtained by the LoFTR feature matching algorithm. The LoFTR feature matching algorithm has higher speed and robustness than Figure 3 , c: SIFT feature matching algorithm. Figure 3 , b: LoFTR feature matching algorithm has higher speed and robustness, and can extract rich feature points for depth value recovery. Among them, Figure 3 , b: LoFTR feature matching algorithm and Figure 3 , c: SIFT feature matching algorithm, the feature point pairs are connected by lines of different colors. The feature points of the reference image relative to the source image are the endpoints of the lines.

[0073] Step b2: Based on the segmentation image, all feature points in each region of the segmentation image are processed by using a triangulation method to obtain a plurality of triangular regions, and a coarse depth map is generated based on all the triangular regions.

[0074] According to an embodiment of the present application, all feature points of the reference image are processed by using a segmentation-driven triangulation method to construct a coarse depth map. It is assumed that the scene of the image is composed of triangulation planes, so the depth of the feature points with reliable depth can be recovered by triangulation, and in addition, the depth edges of the segmentation image obtained by using SAM2 are used as a constraint guide to prevent triangulation across different object instances. Illustratively, as shown in Figure 3 , a image, panoramic segmentation is performed on the image by using SAM2 to obtain Figure 3 , d: segmentation image and corresponding boundary image, and the d segmentation image and the corresponding boundary image are used as a constraint guide to triangulate all feature points of Figure 3 , a image to obtain Figure 3 , f: coarse depth map after triangulation of the feature points, and the object instance edges and the generated triangular regions are highlighted in blue and red respectively.

[0075] According to one embodiment of the present application, the triangulation method driven by segmentation is as follows:

[0076] 1) The boundary map is used to cluster all feature points of each object instance into a feature point cluster, and a plurality of clusters are obtained, denoted as , where represents the feature points of the th cluster, represents the number of feature point clusters.

[0077] 2) Triangulation is performed in each cluster to generate a series of triangular regions , where represents the th triangular region of the th cluster, represents the number of triangular regions. Then for each pixel in the triangular region whose vertices are , the calculation formula of the depth value is as follows: , (2)

[0078] where represents the depth value of the vertex

[0079] , and represents the distance from the vertex to the pixel . 3) For any pixel outside all triangular regions

[0080] , locate the nearest triangular region to the pixel in the same object instance region of the pixel using the KD tree. Then project the pixel onto the plane of its nearest triangular region , and use the above formula (2) to determine its depth . Finally, the rough depth map after triangulation of feature points is obtained as f: the rough depth map after triangulation of feature points. The triangulation method driven by segmentation proposed in the present application effectively generates a rough depth map, which not only flattens the textureless region, but also restores the approximate accurate depth for other regions, laying a solid foundation for further refinement of depth. Figure 3 ​​

[0081] Step b3: based on the monocular depth map, performing RANSAC-based plane fitting on all pixels in each triangular region, and optimizing the coarse depth map based on the fitting result to obtain an initial recovered depth map.

[0082] According to an embodiment of the present application, further based on the monocular depth map, performing geometry-aware refinement on the coarse depth map to verify the reliability of each triangular region to provide an initial recovered depth map with geometry awareness. The geometry-aware refinement process includes: based on a geometry threshold performing RANSAC-based plane fitting on its corresponding pixels in the monocular depth map to obtain an inlier ratio and an estimated plane . Then, based on the fitting result, optimizing the coarse depth map to obtain the initial recovered depth map. The optimization of the coarse depth map includes the following steps b31, b32 and b33:

[0083] Step b31: based on the fitting result, identifying whether each triangular region is a planar region, where a planar region represents a depth-continuous region;

[0084] According to an embodiment of the present application, if the inlier ratio exceeds a proportion threshold , the triangular region is classified as a planar region. Otherwise, the triangular region is regarded as a non-planar region with geometric complexity.

[0085] Step b32: if the triangular region is not a planar region, for each pixel in the triangular region, based on the depth values of the vertices of the triangular region in the monocular depth map and the difference between the depth values of the vertices and the depth value of the pixel, optimizing the depth value of the pixel in the triangular region, otherwise, the depth value of each pixel in the triangular region remains unchanged;

[0086] According to an embodiment of the present application, for the triangular region which is a planar region, the depth value of the pixel in the triangular region remains unchanged through step b2, and for the triangular region which is a non-planar region, the depth of the pixel in the triangular region is adjusted according to the relative depth difference in the monocular depth map . Wherein, for any pixel in the triangular region which is a non-planar region , and , its depth value is calculated as follows:

[0087] , (3) ​

[0088] in, Represents a monocular depth map midpoint The depth value of Indicates that from the vertex to pixels The depth value difference.

[0089] Step b33: For each pixel outside all triangular areas, determine the closest triangular area that is a planar area using the segmentation map as a constraint, and calculate the distance between it and the triangular area. If the distance is less than the preset distance, use the proportional mapping algorithm to optimize the depth value of the pixel. Otherwise, the depth value of the pixel remains unchanged.

[0090] According to one embodiment of the present invention, for a set Any pixel other than In the area of ​​the same object instance, the KD tree is used to locate the triangle area closest to the plane area. Then, calculate the pixel The area of ​​the nearest planar triangle plane If the distance is less than , then it is believed that belongs to the textureless area, its depth remains unchanged. Otherwise, the pixel is considered Located in a non-planar area, its depth value Adjustments are made using a scale mapping algorithm.

[0091] According to one embodiment of the present invention, the scale mapping algorithm includes: Determine the distance between pixels Nearest pixel , then pixel Depth value Defined as: ,in, is the pixel obtained by formula (3) The depth value of and Represents the monocular depth map respectively Medium pixels and pixels The present invention cleverly utilizes the monocular depth map through this simple proportional mapping equation. The depth variation in the convex region is used to estimate the depth of non-planar regions with geometric complexity.

[0092] Figure 3 In the monocular depth map in e, a lower color temperature is used to represent a larger depth value. Figure 3f in the formula is the rough depth map after triangulation, the initial recovered depth map obtained through the above embodiment is Figure 3 g in the formula, Figure 4 h in the formula is the final depth map. Comparative analysis shows that: Figure 4 g in the formula is compared with Figure 4 h in the formula, Figure 4 g in the formula is an initial recovered depth map which effectively recovers most of the non-texture area and the area rich in texture, without serious detail distortion, and is reliable and realistic.

[0093] The technical scheme of the above embodiment can at least achieve the following beneficial technical effects: the initial recovered depth map obtained by the above method has higher precision, making it more suitable for complex scenes. Not only can it flatten the non-texture area, but also can recover the approximate accurate depth for other areas, laying a solid foundation for further refinement.

[0094] According to one embodiment of the present application, after obtaining the initial recovered depth map, it is used for initialization and supervision guidance in the PM algorithm process. Specifically, considering that a reliable initial depth map can effectively assist in reconstruction, the initial recovered depth map is used to replace the original random initialized depth map for initialization, and multiple iteration optimizations are performed based on the initial recovered depth map, thereby effectively improving the accuracy of subsequent propagation and refinement.

[0095] According to one embodiment of the present application, before multiple iteration optimizations are performed, each image needs to be down-sampled multiple times to realize scaling on different levels and obtain multiple feature maps of different levels. In each iteration, the depth map of the reference image is estimated based on the feature maps of the reference image and each source image at the same level, and the iteration is performed in order from low to high resolution of the feature maps.

[0096] According to one embodiment of the present application, each iteration includes the following steps S21, S22 and S23:

[0097] Step S21, obtaining a recovered depth map, wherein the initial recovered depth map obtained through the above embodiment is used for the first time, and the depth map after the previous iteration is used for each time after the first time.

[0098] According to one embodiment of the present application, the initial recovered depth map obtained through the above embodiment steps b1-b3 is used for the first time, and the depth map after the up-sampling of the depth map after the previous iteration is used for each time thereafter.

[0099] Step S22, for each pixel of the reference image, detecting a plurality of boundary pixels corresponding to the region of the segmentation map to which the pixel belongs, connecting the plurality of boundary pixels in order to obtain a deformed block, and estimating the depth value of the pixel based on the constraint condition preset according to the occlusion relationship, the deformed block, the reference image, the recovered depth map and all source images.

[0100] According to one embodiment of the present application, see Figure 4 , which is a schematic diagram of the comparison results between the method of the present application and the existing method. The following compares and analyzes the existing method and the method of the present application respectively:

[0101] Figure 5 a in the ACMMP method adopts a fixed-size deformation block, which cannot capture enough feature points and is difficult to reconstruct a non-textured region;

[0102] Figure 5 b in the APD-MVS method, on the basis of the ACMMP method, patch deformation, neglect of depth edges, leads to the deformation block being selected in the depth discontinuous region, such as the red edge point corresponding to the partial region, there is a jump edge deformation instability (i.e. jump boundary deformation loses stability), thus there is a potential matching distortion problem;

[0103] And Figure 5 c in the SD-MVS method calculates the distance of each pixel to its boundary only in the vertical and horizontal directions, and uses the depth edge as a constraint guide (i.e. adopts a segmentation-driven block deformation) to deform the area within the boundary, and then scales the corresponding block according to the ratio of these distances, resulting in that it is still limited by the fixed block size, and lacks edge alignment capability, is only suitable for regular geometric objects, and has low reconstruction efficiency and poor quality in complex shape scenes;

[0104] And Figure 5 d in the method of the present application adopts a segmentation-driven and edge-aligned block deformation, i.e. taking each pixel as the center, a multi-track diffusion strategy for block deformation is adopted, the diffusion strategy includes: detecting a plurality of boundary pixels of the region to which the pixel belongs in the segmentation map, and connecting the plurality of boundary pixels in turn to obtain a deformation block. When facing a complex scene containing a large number of irregularly shaped object instances, such deformation blocks are usually more easily and accurately aligned with the boundaries of irregular shapes, so that the deformation effect is good, ensuring that the deformation block can be fully aligned with the depth edge in complex scenes.

[0105] In the block deformation, the inventors consider a phenomenon that when a light is turned on in a dark room, every beam of light emitted by the light source will diffuse through the air and reach the surface of every opaque object without leaving any gap. By analogy, if the light source, the air and the object surface are regarded as the center pixel of the deformed block (i.e. the pixel to be estimated), the textureless region and the depth edge respectively, the block deformation with edge alignment can be achieved by using a light diffusion method. However, in practice, it is computationally inefficient to enumerate the distance between the pixel in each direction and its edge for deformation. Therefore, according to one embodiment of the present application, in step S22, the multi-track diffusion strategy for block deformation described above is proposed for each pixel of the reference image, and the deformed block of each pixel is obtained, which not only achieves the overall edge alignment of the deformed block, but also achieves a compromise between computational complexity and effectiveness.

[0106] The way of obtaining the deformed block by using the multi-track diffusion strategy comprises: extracting the boundary of the segmentation image corresponding to the reference image to obtain a boundary image comprising a plurality of boundary pixels formed between the regions of each object instance; extending in a plurality of directions from each pixel of the reference image as the center until the boundary pixel of the boundary image is first intersected to detect a plurality of boundary pixels intersected in the plurality of directions; and connecting the plurality of boundary pixels intersected in the plurality of directions corresponding to each pixel in turn to obtain the deformed block.

[0107] According to one embodiment of the present application, referring to Figure 5 which is a schematic diagram of the principle of the block deformation method of the present application. In the diagram, Figure 5 , the a image is taken as an example, Figure 5 , the b and Figure 5 , the c are the segmentation image and the real depth image relative to the ground of the a image respectively, Figure 5 , the f and Figure 5 , the g are the deformation results of the ACMMP method and the SD-MVS method respectively, i.e. the red area part. Figure 5 , the h: the deformed block obtained by using the multi-track diffusion strategy, the deformed block being the red area. Figure 5 , the d and Figure 5 , the e are the depth images before and after using the deformation method of the present application respectively. Figure 5 , the i is the pixel point sampled based on the block deformation of the present application to evaluate the influence on the matching cost, i.e. the blue pixel point inside the deformed block and on the edge thereof.

[0108] According to one embodiment of the present application, taking Figure 6 as an example, the deformation process of the deformed block comprises:

[0109] Given a pixel p in the a image in Figure 7 , X rays are emitted around p as a group of tracks, denoted as L={ | =1,…,X}. Each trajectory An angle between In the implementation, X is empirically set to 16. It can also be set to 32 or 12. The present invention can adapt the track density, that is, automatically increase the number of tracks in complex edge areas (high curvature, multiple bifurcations), and set X to a larger number to improve coverage. Each track Will continue to extend until they first intersect the boundary map Boundary pixels Intersect at the edge of the grid, and detect multiple intersecting boundary pixels. Then all the intersecting boundary pixels The set is defined as:

[0110] , (4)

[0111] Connect in sequence Each intersecting boundary pixel in , forming pixels Compared with the SD-MVS method of g in the figure, Figure 7 The multi-track diffusion strategy of h in

[15] can not only effectively enhance the receptive field, but also achieve precise boundary alignment of deformed patches, which can flexibly adapt to complex scenes containing irregularly shaped objects.

[0112] According to one embodiment of the present invention, in step S22, after obtaining the deformed block based on the multi-track diffusion strategy, the depth value of the pixel is estimated by analyzing the impact of the block deformation on the matching cost and performing pixel sampling, and then propagating and refining the sampled pixels to estimate the depth value of the pixel. That is, the depth value of the pixel is estimated by the following steps c1 and c2:

[0113] Step c1: Based on the constraints and according to a preset sampling strategy, pixels are sampled from the deformed block corresponding to the pixel and from its peripheral preset area to obtain a plurality of sampled pixels, wherein the constraints include: when an area to which the peripheral preset area belongs is in an occlusion relationship with an area to which the deformed block belongs, sampling cannot be performed from the peripheral preset area; otherwise, sampling can be performed from the peripheral preset area.

[0114] According to one embodiment of the present invention, under the constraint condition, the preset sampling strategy includes steps c11, c12 and c13:

[0115] Step c11: connecting boundary pixels intersecting with multiple directions to the pixel respectively to obtain multiple tracks, and evenly dividing each track into a corresponding number of segments according to a preset number of samples for each track to obtain multiple track segments.

[0116] According to one embodiment of the present invention, Figure 7As shown in h in , 16 tracks are formed after being connected. The number of samples preset for each track is preset as follows:

[0117] = , (5)

[0118] in, Expressed as Tracks The preset number of samples, Represents trajectory length, represents the average path length of all trajectories in the deformed block corresponding to the pixel, Indicates rounding up. Then, according to Tracks Number of samples , will Tracks Divided into trajectory segments, each trajectory segment is represented as , . Each trajectory segment All pixels within can be defined as follows: ,in, Indicates the The first track The first of the trajectory segments pixels, Represents a trajectory segment The number of pixels in .

[0119] According to one embodiment of the present invention, based on the result of step c11, the preset sampling strategy may subsequently adopt the following solution 1:

[0120] First, from each trajectory segment Select the one with the minimum aggregation cost Pixels , which is defined as:

[0121] , (6)

[0122] in, Represents pixels The aggregation cost of . Then, the trajectory The set of pixels selected by each trajectory segment Expressed as: ,in, Indicates the path assigned Finally, by summing up each trajectory in L Collection A set of sampling pixels can be obtained. This sampling strategy guarantees that each path contributes at least one pixel. Longer trajectories will be assigned more sampling pixels, and vice versa, thus achieving a balanced and efficient distribution of sampling pixels.

[0123] However, when the well-textured pixels inside the warped block are not covered by the trajectories, the sampling strategy of solution one in the above embodiment will usually result in the inability to select sufficiently reliable pixels. Therefore, according to an embodiment of the present application, the following solution two is designed:

[0124] Based on the texture-aware mapping strategy, sampling is performed to ensure that the warped block obtains sufficiently reliable pixels through mapping to achieve a robust matching cost. The way of sampling based on the texture-aware mapping strategy includes: based on the result of step c11, continue to sample according to the following steps c12 and c13.

[0125] Step c12: for each pixel of each trajectory segment of each trajectory, a window of a preset size is constructed centered on the pixel, and the pixel with the smallest texture coefficient in the window is selected as the mapping pixel.

[0126] According to an embodiment of the present application, as shown in the white box in the figure indicated by i in Figure 7 , first, the texture coefficient of each pixel in the window is obtained by using the pixel-level texture coefficient calculation method proposed in the TAPA-MVS method. Then, for each pixel , its mapping pixel is defined as the pixel with the smallest texture coefficient in the window centered on , and its formula is . Wherein, represents the texture coefficient of the pixel in . According to an embodiment of the present application, when selecting the mapping pixel, it is also limited by the constraint condition preset based on the occlusion relationship, which has the same principle as the above-mentioned step S1 occlusion constraint guidance of the embodiment. That is, when the window includes a partial region of the warped block and a preset peripheral region, if there is an occlusion relationship between the region to which the peripheral preset region belongs and the region to which the warped block belongs, the mapping pixel cannot be selected from the peripheral preset region, and vice versa. In the window deformation process of the pixel , there is occlusion awareness.

[0127] According to an embodiment of the present application, when selecting the mapping pixel, it is also limited by the constraint condition preset based on the occlusion relationship, which has the same principle as the above-mentioned step S1 occlusion constraint guidance of the embodiment. That is, when the window includes a partial region of the warped block and a preset peripheral region, if there is an occlusion relationship between the region to which the peripheral preset region belongs and the region to which the warped block belongs, the mapping pixel cannot be selected from the peripheral preset region, and vice versa. In the window deformation process of the pixel , there is occlusion awareness.

[0128] ​Step c13: For each trajectory segment of each trajectory, sample the pixel that minimizes the aggregation cost from the mapped pixels of all pixels of the segment to obtain the sampled pixel, wherein the aggregation cost is obtained by aggregating the matching costs of all source images and the reference image.

[0129] According to one embodiment of the present invention, in each trajectory segment Mapping pixels for all pixels in Select the one with the minimum aggregation cost The pixels to determine the fragment , the formula is as follows:

[0130] , (7)

[0131] Finally, by aggregating the sampled pixels of all trajectory segments of each trajectory , get each trajectory Corresponding sampling pixel set , merge the sampled pixel sets of all tracks in L to obtain multiple sampled pixels. The technical solution of this embodiment can at least achieve the following beneficial technical effects: by presetting the sampling strategy to ensure that each track contributes at least one pixel, when the pixels with good texture inside the deformed block are not covered by the track, by obtaining sufficiently reliable mapping pixels, a robust matching cost is achieved. Get a more reliable set of sampled pixels . And formula (6) In contrast, the The quality of window features is significantly improved by expanding the pixel sampling range. In summary, by combining the multi-track diffusion strategy and the texture-aware mapping strategy, not only edge-aligned block deformation is achieved, but also the sampled pixels are evenly and reliably distributed in various regions of the deformed block, thus achieving a robust matching cost.

[0132] According to one embodiment of the present invention, under the constraints of the conditions, if there is an occlusion relationship between the regions of adjacent object instances, strict edge constraints need to be applied, that is, to ensure that the trajectory used for propagation and refinement and the mapped pixels for matching cost If there is no occlusion relationship between the regions of adjacent object instances, flexible edge constraints can be applied, which allows paths to be used for propagation and refinement. and the mapped pixels for matching cost In the specified threshold area Occlusion relationships between regions of adjacent object instances indicate depth discontinuity, while occlusion relationships between regions of adjacent object instances indicate depth continuity. This constraint ensures that reliable pixels from other object instances are also considered during matching, propagation, and refinement, effectively enhancing the perception of the deformed patch and the efficiency of propagation and refinement. This not only explores occlusion relationships between instances but also addresses potential deformation instabilities caused by previously applied strict edge constraints, resulting in occlusion-aware patch deformation.

[0133] Step c2: Based on the depth values ​​of the plurality of sampled pixels in the restored depth map, a depth hypothesis value of the pixel is generated, and the depth hypothesis value is optimized to estimate the depth value of the pixel.

[0134] According to one embodiment of the present invention, step c2 is a propagation and refinement process. Step c2 includes: merging each trajectory of the pixel and its diagonal trajectory to obtain multiple combined trajectories; selecting a pixel with the lowest aggregation cost from all sampled pixels corresponding to each combined trajectory of the multiple combined trajectories for propagation, thereby obtaining multiple pixels for propagation; generating a depth hypothesis value for the pixel based on the depth values ​​of the multiple pixels for propagation in the restored depth map; and optimizing the depth hypothesis value using a spherical gradient refinement strategy to estimate the depth value of the pixel.

[0135] According to one embodiment of the present invention, the propagation refinement process is similar to the SD-MVS method. ​ For example, when X is empirically set to 16, each trajectory and its diagonal trajectory are merged to obtain eight combined trajectories. Then, the pixel with the minimum aggregation cost on each combined trajectory is selected. Regarding refinement, the spherical gradient refinement strategy proposed in the SD-MVS method is used for refinement, which uses gradient descent on spherical coordinates and normals at the same time, and uses a pixel-by-pixel search interval to constrain the depth, thereby effectively enhancing the reliability of the pixel-by-pixel refinement hypothesis.

[0136] Step S23: Obtain a depth map after this iteration based on the estimated depth values ​​of all pixels.

[0137] According to one embodiment of the present invention, after the depth values ​​of all pixels of the reference image are estimated, the depth values ​​of all pixels are fused to obtain a depth map after this iteration.

[0138] In general, according to one embodiment of the present invention, the complete processing process of the method of the present invention can be found in ​ , which is a complete flowchart of the multi-view stereo reconstruction method based on segmentation-driven and edge-aligned deformation. First, before iteration, a series of tasks are performed, including the following constraint guidance process and preprocessing process:

[0139] The constraint-guided process includes:

[0140] The sparse-mono collaborative recovery strategy (see in the green box) includes: using LoFTR to capture feature points in images of different perspectives, and using DepthAnythingV2 to generate a monocular depth map of the reference image; using the segmentation map as a constraint, using the segmentation-driven triangulation method of the above embodiment to obtain a coarse depth map, and using the above step b3: the geometric perception refinement process to optimize the coarse depth map to obtain an initial recovered depth map.

[0141] The edge constraint-guided block deformation (see in the blue box) includes: using SAM2 to perform panoramic segmentation on the image to obtain a segmentation map; generating an occlusion map based on the segmentation map and the monocular depth map, and obtaining the inter-region occlusion relationship of each object instance; and limiting the sampling pixels according to the constraint conditions preset according to the occlusion relationship.

[0142] The pre-processing process (see in the purple box) includes multiple times of down-sampling on the input image, and then obtaining the recovered depth map of each iteration for multiple times of iterative optimization.

[0143] At each iteration, the process (see in the red box) includes: segmentation-driven and edge-aligned block deformation, sampling pixels, a propagation process, and a refinement process. Among them, in the block deformation, on the basis of the traditional PM method, a multi-track diffusion strategy is used to obtain a deformed block under the constraint of the segmentation map, and then the matching cost is optimized according to the constraint conditions preset according to the occlusion relationship in the manner of the above embodiment, and the pixel corresponding to the minimum aggregated cost of each track is obtained to optimize the matching cost, and then the propagation process and the refinement process are used to obtain the result of the iterative optimization, i.e., the depth map of the image.

[0144] To verify the effect of the present application, the inventors performed the following experiment:

[0145] Experimental platform parameters:

[0146] CPU: Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz

[0147] GPU: NVIDIA GeForce 3080 RTX

[0148] Operating system: Windows 10

[0149] Code running environment: C++ with CUDA

[0150] Dataset and parameter settings:

[0151] The present application is evaluated by using ETH3D dataset and TNT (Tanks and Temples) dataset, and the parameters are as follows Respectively set as , wherein, represents the current iteration number.

[0152] Experimental process:

[0153] 1. Comparative experiment

[0154] The ETH3D dataset and the TNT dataset are used to compare the present application method (referred to as SED-MVS) and the existing method on the above experimental platform. Among them, the existing method includes the application of more advanced learning-based method, including: PatchMatchNet method, IterMVS-LS method, AGG-CVCNet method, MVSTER method, EPP-MVSNet method, EPNet method, and traditional method, including TAPA-MVS method, PCF-MVS method, ACMP method, ACMMP method, ADP-MVS algorithm, HPM-MVS++ method and SD-MVS method.

[0155] 1.1) The benchmark performance comparison under the ETH3D dataset

[0156] Using the ETH3D dataset, the dataset is divided into a training set (Train) and a test set (Test), and the present application method and the existing method are compared under the threshold of 2cm and the threshold of 10cm. In the experiment, the performance of each method in F1 score, completeness (Comp.) and accuracy (Acc.) is evaluated. The experimental results are shown in Table 1 as follows:

[0157] Table 1

[0158]

[0159] In Table 1, the optimal result is marked in bold and underlined, and the suboptimal result is marked in bold. It can be seen that the present application method obtains the highest F1 score and completeness on the training set and the test set, which verifies its excellent effectiveness.

[0160] 1.2) The benchmark performance comparison under the TNT dataset

[0161] Using the TNT dataset, the TNT Intermediate dataset and the TNT Advanced dataset are divided, and the comparative experiments of the method of the present application and the existing method are carried out. In the experiment process, the performance results of each method in F1 score, recall (Rec.) and precision (Pre.) are evaluated. The experimental results are shown in Table 2 as follows:

[0162] Table 2

[0163]

[0164] In Table 2, the optimal result is marked in bold and underlined, and the suboptimal result is marked in bold. It can be seen that the method of the present application achieves the highest integrity on the advanced and intermediate datasets, and obtains the highest F1 score on the intermediate dataset, proving that it has very high robustness.

[0165] Finally, referring to ​ , which is a result diagram of testing the method of the present application and other existing methods in combination with the ETH3D dataset. ​ The a part in ​ includes 10 scene images, and respectively are office images, delivery area images, pipe images, etc. in the ETH3D dataset, ​ The b, c, d, e and f parts in are respectively the results of the existing IterMVS LS method, ACMMP method, HPM-MVS method, APD-MVS method and the method of the present application on

[0166] The 10 scene images in the a part in . In the result diagram, the red box shows some challenging areas. Obviously, the method of the present application performs particularly well in large-area textureless areas (such as floors and walls), and generates the most complete and realistic reconstructed point cloud without causing detail distortion.

[0166] 2, Ablation study

[0167] In order to verify the various strategies designed by the method of the present application, a series of ablation study experiments are carried out to illustrate the effectiveness of each proposed strategy.

[0168] 2.1) For the block deformation mode in the above embodiment, the following ablation is carried out:

[0169] deleting the whole process of the split driving and the edge-aligned block deformation in this mode, i.e. using the traditional PM to perform the block deformation (denoted as w / o.Def.); deleting the multi-track diffusion strategy in the process of the block deformation alone (denoted as w / o.Taj.); deleting the texture-aware mapping in the process of the block deformation alone, i.e. not using the scheme two realized by the steps c12 and c13 (denoted as w / o.Map.); deleting the refinement process after the block deformation alone (denoted as w / o.Ref.); deleting the propagation process after the block deformation alone (denoted as w / o.Pro.).

[0170] 2.2) For the sparse-mono collaborative recovery strategy in the above embodiment, the following ablation is performed:

[0171] deleting the whole process of the sparse-mono collaborative recovery strategy (denoted as w / o.Syn.); deleting the iterative optimization guided according to the recovered depth map alone (denoted as w / o.Sup.); deleting the initialization alone (denoted as w / o.Ini.), without using the segmentation map as a constraint when generating the coarse depth map by using the triangulation (denoted as w / o.Seg.).

[0172] 2.2) For the constraint condition in the above embodiment, the following ablation is performed:

[0173] removing the whole sampling mode of the constraint condition, i.e. not constructing the occlusion relationship to guide the sampling pixels (denoted as w / o.Occ.); deleting the way of applying strict edge constraints in the constraint condition alone (denoted as w / o.Str.).

[0174] On the ETH3D dataset, and under the given thresholds of 2cm and 10cm, the above 11 ablation cases and the complete method of the present application are respectively experimented, and the effectiveness of each part in the method of the present application is demonstrated by evaluating the F1 score (abbreviated as F1.), completeness (abbreviated as Comp.) and accuracy (abbreviated as Acc.) three aspects. The ablation study quantitative results are shown in Table 3 as follows:

[0175] Table 3

[0176]

[0177] From Table 3, for the block deformation mode: w / o.Def. produces the worst F1 score, verifying the importance of the segmentation-driven and edge-aligned block deformation strategy. The F1 score of w / o.Taj. is superior to that of w / o.Map., emphasizing that the multi-track diffusion strategy plays a more important role than the texture-aware mapping, but both have a sufficiently important contribution. Although w / o.Ref. and w / o.Pro. both achieve similar F1 scores, w / o.Ref. exhibits lower accuracy but higher completeness, which indicates that propagation performs better in reconstructing textureless regions, while refinement can effectively improve reconstruction accuracy.

[0178] For the sparse-mono collaborative recovery strategy: in contrast, the F1 score of w / o.Syn. is the lowest, verifying the importance of this strategy. The F1 score of w / o.Ini. is slightly higher than that of w / o.Sup., which indicates that using the recovered depth map for guidance during iteration is more beneficial to reconstruction than initializing with the recovered depth map before iteration. The F1 score of w / o.Seg. is only slightly lower than that of SED-MVS, indicating that using the segmentation map as a constraint during triangulation has good results.

[0179] For the constraint condition: obviously, the performance of w / o.Str is slightly better than that of w / o.Occ, but both are inferior to SED-MVS. This indicates that excluding strict edge constraints is not as good as adopting a constraint condition preset based on occlusion relationships.

[0180] Finally, the above experimental evaluation results show that the method of the present application can achieve better performance in traditional and learning-based methods, and has strong generalization ability.

[0181] The existing method is as follows:

[0182] PatchMatchNet method: Wang F, Galliani S, Vogel C, et al. Patchmatchnet: Learned multi-view patchmatch stereo [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 14194-14203.

[0183] IterMVS-LS method: Wang, F.; Galliani, S.; Vogel, C.; and Pollefeys, M. 2022a IterMVS: Iterative probability estimation for efficient multi-view stereo. In Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 8606-8615.

[0184] AGG-CVCNet method: CVCNet: Learning Cost Volume Compression for Efficient Stereo Matching.

[0185] MVSTER method: Wang, X.; Zhu, Z.; Huang, G.; Qin, F.; Ye, Y.; He, Y.; Chi, X.; and Wang, X. 2022b. MVSTER: Epipolar Transformer for Efficient Multi-View Stereo. In European Conference on Computer Vision, 573-591. Springer.

[0186] EPP-MVSNet method: X. Ma, Y. Gong, Q. Wang, J. Huang, L. Chen, F. Yu, EPP-MVSNet: Epipolar assembling based depth prediction for multi-view stereo, in: Proc. IEEE / CVF Int. Conf. Comput. Vis., ICCV, 2021, pp. 5712-5720.

[0187] EP-Net method: Su, W.; and Tao, W. 2023. Efficient Edge-Preserving MultiView Stereo Network for Depth Estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2348-2356.

[0188] TAPA-MVS method: i.e. reference [1] in the above background section.

[0189] PCF-MVS method: A. Kuhn, S. Lin, O. Erdler, Plane completion and filtering for multi-view stereo reconstruction, in: Pattern Recognition, 2019, pp. 18-32.

[0190] ACMP method: Qingshan Xu, Wenbing Tao; Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5483-5492.

[0191] ACMMP algorithm: i.e. reference [2] in the above background section.

[0192] APD-MVS algorithm: i.e. reference [5] in the above background section.

[0193] HPM-MVS++ method: Ren, C.; Xu, Q.; Zhang, S.; and Yang, J. 2023. Hierarchical Prior Mining for Non-Local Multi-View Stereo. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 3611-3620.

[0194] SD-MVS method: i.e. reference [6] in the above background section.

[0195] It is to be understood that even though various steps of the method are described in a particular order, the method can be performed in a different order, or some of the steps can be performed concurrently, or even changed, so long as the desired function is performed.

[0196] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0197] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves.

[0198] Embodiments of the application have been described above, and the description is not exhaustive, and does not limit the scope of the disclosed embodiments. Many modifications and variations of the described embodiments will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The description is intended to cover any and all modifications and variations, including such concepts contained therein of which the author is not explicitly aware. The terms used herein are to be construed in their broadest, ordinary sense, and are not to be limited by any slight changes that can be made to the specification.

Claims

1. A multi-view stereo reconstruction method, comprising: Get multiple images of the same scene from different perspectives, use each image as a reference image, and use the remaining images as source images for the reference images. Estimate the depth maps of all reference images and fuse all the depth maps to obtain the 3D point cloud data of the scene. The depth map of each reference image is estimated using the following methods: S1. Segment the region of each object instance according to the reference image to obtain a corresponding segmentation map, and construct the occlusion relationship between the regions of different object instances based on the segmentation map; S2. Perform multiple iterative optimizations based on the initial restored depth map of the reference image to obtain a depth map, wherein each iteration includes: S21, obtaining a restored depth map, wherein the initial restored depth map is used for the first time, and the depth map after the previous iteration is used each time thereafter; S22. For each pixel of the reference image, with the pixel as the center, detect multiple boundary pixels corresponding to the region of the segmentation image to which the pixel belongs, sequentially connect the multiple boundary pixels to obtain a deformed block, and estimate the depth value of the pixel based on the deformed block, the reference image, the restored depth map, and all source images based on the constraints preset by the occlusion relationship. The method of estimating the depth value of the pixel includes: Based on the constraint conditions and according to a preset sampling strategy, pixels are sampled from the deformed block corresponding to the pixel and from a peripheral preset area thereof to obtain a plurality of sampled pixels, wherein the constraint conditions include: if an area to which the peripheral preset area belongs is in an occlusion relationship with an area to which the deformed block belongs, sampling cannot be performed from the peripheral preset area; otherwise, sampling can be performed from the peripheral preset area; Based on the depth values ​​of the multiple sampled pixels in the restored depth map, a depth hypothesis value of the pixel is generated, and the depth hypothesis value is optimized to estimate the depth value of the pixel; The preset sampling strategies include: Connecting boundary pixels intersecting multiple directions to the pixel respectively to obtain multiple tracks, and evenly dividing each track into a corresponding number of segments according to a preset number of samples for each track to obtain multiple track segments; For each pixel of each trajectory segment of each trajectory, a window of preset size is constructed with the pixel as the center, and the pixel with the minimum texture coefficient in the window is selected as the mapping pixel; For each trajectory segment of each trajectory, sample the pixel that minimizes the aggregation cost from the mapped pixels of all pixels of the segment to obtain the sampled pixel, where the aggregation cost is obtained by aggregating the matching costs of all source images and the reference image; S23. Obtain a depth map after this iteration based on the estimated depth values ​​of all pixels.

2. The method according to claim 1, characterized in that In S22, for each pixel of the reference image, the deformed block is obtained in the following manner: Extracting the boundary of the segmentation image corresponding to the reference image to obtain a boundary map, which includes multiple boundary pixels formed between the regions of each object instance; Extending in multiple directions with each pixel of the reference image as the center until it intersects with the boundary pixels of the boundary image for the first time, and detecting boundary pixels intersecting in multiple directions; The boundary pixels corresponding to multiple directions of each pixel are connected in sequence to obtain the deformed block.

3. The method according to claim 2, characterized in that In S1, the occlusion relationship construction method includes: A monocular depth map of the reference image is generated using a monocular depth estimation method, and the depth gradient of the boundary pixels is calculated based on the monocular depth map; Generate an occlusion map corresponding to the boundary map based on the depth gradient of all boundary pixels. The occlusion map records whether each boundary pixel belongs to a category with continuous depth or discontinuous depth with its adjacent area. According to the occlusion map, it is determined whether there is occlusion between the regions of each object instance, and the occlusion relationship is obtained.

4. The method according to claim 3, characterized in that The initial restored depth map is generated in the following manner: Obtain a monocular depth map of the reference image, and use a feature matching algorithm to extract features between the reference image and the source image to obtain the feature points of the reference image relative to the source image and the depth values ​​of the feature points; Based on the segmentation map, all feature points in each area of ​​the segmentation map are processed by triangulation to obtain several triangular areas, and a coarse depth map is generated based on all the triangular areas; Based on the monocular depth map, RANSAC-based plane fitting is performed on all pixels in each triangular area. The coarse depth map is optimized based on the fitting results to obtain the initial restored depth map.

5. The method according to claim 4, characterized in that The method of optimizing the coarse depth map includes: Based on the fitting results, it is identified whether each triangular area is a plane area, and the plane area represents a depth-continuous area; If the triangular area is not a planar area, for each pixel in the triangular area, the depth value of the pixel in the triangular area is optimized according to the depth value of the vertex of the triangular area in the monocular depth map and the difference between the depth value of the vertex and the depth value of the pixel. Otherwise, the depth value of each pixel in the triangular area remains unchanged.

6. The method according to claim 5, characterized in that The method of optimizing the coarse depth map includes: For each pixel outside all triangular areas, determine the nearest planar triangular area with the segmentation map as a constraint, and calculate the distance between the pixel and the triangular area; If the distance is less than the preset distance, the depth value of the pixel is optimized using the proportional mapping algorithm; otherwise, the depth value of the pixel remains unchanged.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 6 by executing the executable instructions.

Images