Multi-view stereoscopic reconstruction method based on image segmentation

Abstract

The invention provides a multi-view three-dimensional reconstruction method based on image segmentation, which comprises the following steps: estimating a depth map of each image based on a plurality of acquired images of different views of the same scene, and the estimation mode comprises the following steps: down-sampling all the images for multiple times to obtain a plurality of feature maps of different hierarchies corresponding to each image; iteratively processing each image for multiple times, obtaining a depth map of each level of the image each time, and determining reliable pixels and unreliable pixels at the time; according to the feature maps of all the images at the corresponding levels, the depth estimation values of the reliable pixels are updated, the depth estimation values of the unreliable pixels are reconstructed through a preset deformation algorithm, and the algorithm comprises the steps that different objects of the images are segmented into different areas, and the depth estimation values of the unreliable pixels are obtained for the unreliable pixels; only selecting a plurality of anchor points from reliable pixels in the same area as the pixel to perform patch deformation so as to reconstruct and obtain a depth estimation value; and fusing the depth maps obtained at the last time of all the images to obtain three-dimensional point cloud data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to the field of three-dimensional reconstruction and multi-view stereo matching, and more particularly to a multi-view stereo reconstruction method based on image segmentation. BACKGROUND

[0002] Multi-View Stereo (MVS) technology plays an important role in the field of 3D computer vision, and its core goal is to reconstruct a three-dimensional scene from images captured from different perspectives. Therefore, MVS technology is widely used in the fields of autonomous driving, virtual reality, 3D printing, etc.

[0003] In recent years, MVS methods based on deep learning have introduced neural networks to create a learnable 3D cost volume, making it possible to express deep feature extraction in the perception field. Although typical methods use a deformable window or adopt a coarse-to-fine multi-layer architecture strategy to expand the receptive field, for example, by dividing a fixed 9*9 window into 9 3*3 windows, then keeping the center window unchanged and expanding the remaining windows outward until the features are found, to realize the deformable window, and the coarse-to-fine multi-layer architecture strategy is to upsample the picture to get multiple scale pictures, first reconstruct from the smallest picture, upsample the reconstruction result, and use the upsampled result and the picture of the last layer as the initial result of the next layer, until the largest scale depth map is generated, such as the most representative reference [1], referred to as PatchMatchNet algorithm, but the above-mentioned way has unbearable time and memory consumption, and is not suitable for high-resolution images.

[0004] To address the above problems, some existing methods use lightweight gated recurrent unit (GRU), which is a variant of recurrent neural network (RNN) for processing and predicting sequence data. Compared with traditional RNN, GRU introduces a gating mechanism to solve the problem of long-time dependence, and controls the flow of information through update and reset gates, making it more effective in capturing long-term dependencies in sequences. Lightweight GRU is more simplified in design and has higher computational efficiency, making it suitable for use in resource-limited embedded devices or real-time processing applications. Alternatively, attention-based feature aggregation (Attention Mechanism) is used, which is a widely used technique in deep learning that can be used to improve the model's focus on important parts of the input information. That is, by assigning different attention weights, the model can dynamically focus on the most relevant features. During feature aggregation, the attention-based feature aggregation method can adaptively adjust the weight of each feature, so as to consider the information of each dimension when predicting 3D volumes, to compress 3D volumes and improve prediction quality. For example, the most representative reference [2], referred to as AA-RMVSNet algorithm. However, when faced with scenarios different from the training dataset, these networks are often difficult to generalize, which poses a challenge to their practical application.

[0005] Further, existing MVS methods face more significant challenges in reconstructing large areas of textureless regions. This is mainly because in these regions, images lack obvious edge information, leading to inaccurate or inconsistent depth estimation. Traditional MVS methods, such as PatchMatch, predict accurate depth hypotheses by computing the minimum matching cost between fixed window deformations, and achieve fast iterative updates through propagation and refinement. Although these methods perform well in texture-rich regions, depth hypotheses are often unreliable in textureless regions. Some improved methods introduce cascaded architectures, confidence estimators, plane priors, and superpixel planarization techniques to address the reconstruction of textureless regions, such as the most representative reference [3], referred to as ACMMP algorithm. However, these methods either require complex and time-consuming post-processing or result in overly smoothed surface details. Therefore, some improved methods, such as the most representative reference [4], referred to as APD-MVS algorithm, attempt to utilize edge information for adaptive patch deformation, which greatly increases the receptive field range of patches. The core idea of this algorithm is to dynamically adjust the size and shape of the patch for pixels with uncertain matches (i.e., unreliable pixels) to ensure that a sufficient number of reliable pixels are covered, which serve as anchor points to guide depth estimation, thereby improving the accuracy and robustness of depth estimation.

[0006] Specifically, the APD-MVS algorithm effectively reconstructs the textureless region by block deformation. The algorithm takes an unreliable red pixel as the center, first divides the surrounding area into multiple sectors at a fixed angle, and then selects a reliable blue pixel in each sector. Subsequently, the original fixed block centered on the red pixel is deformed into multiple smaller blocks centered on different blue pixels, which are called anchors. Finally, each block determines a specified number of candidate points according to size and spacing for cost calculation. The way of selecting a reliable blue pixel in the surrounding area of an unreliable red pixel greatly leads to the discontinuity of depth between the anchors selected from the reliable blue pixels in the surrounding area and the unreliable red pixel, thereby affecting the accuracy of depth estimation.

[0007] Therefore, when block deformation is performed on unreliable pixels, the existing method selects reliable pixels that are not continuous in depth with the unreliable pixels as anchors for patch deformation, resulting in inaccurate deformation patches and thus inaccurate depth estimation of unreliable pixels and reconstruction failure.

[0008] It should be noted that the background technology is only used to introduce the related information of the present application, so as to help understand the technical solutions of the present application, but does not mean that the related information must be prior art. In the absence of evidence that the related information has been disclosed before the filing date of the present application, the related information should not be considered as prior art.

[0009] References:

[0010] [1] Wang F, Galliani S, Vogel C, et al. Patchmatchnet: Learned multi-view patchmatch stereo [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 14194-14203.

[0011] [2] Wei Z, Zhu Q, Min C, et al. Aa-rmvsnet: Adaptive aggregation recurrent multi-view stereo network [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 6187-6196.

[0012] [3] Xu Q, Kong W, Tao W, et al. Multi-scale geometric consistency guided and planar prior assisted multi-view stereo [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4): 4945-4963.

[0013] [4] Wang Y, Zeng Z, Guan T, et al. Adaptive patch deformation for textureless-resilient multi-view stereo [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 1621-1630. SUMMARY

[0014] Therefore, the purpose of the present application is to overcome the defects of the prior art described above, and to provide a multi-view stereo reconstruction method based on image segmentation.

[0015] The purpose of the present application is realized by the following technical solutions:

[0016] According to a first aspect of the present invention, a multi-view stereo reconstruction method based on image segmentation is provided. The method includes: acquiring multiple images collected from different perspectives in the same scene; using each image in the multiple images as a reference image and the remaining images as source images corresponding to the reference image; estimating the depth map of each reference image based on each reference image and the source images; and fusing the depth maps of all reference images to obtain three-dimensional point cloud data in the corresponding scene. The estimation method includes: S1, performing multiple downsampling operations on the reference image and each source image to obtain multiple feature maps of different levels corresponding to each image; S2, performing multiple iterative processing based on the multiple feature maps of the reference image and each source image to obtain the depth map of the reference image, wherein each iteration is based on the feature maps of the reference image and each source image at each level to obtain the depth map of each level of the reference image, and using the depth map obtained in the last iteration as the depth map of the reference image. The process includes: S21, evaluating the reliability of the depth estimate of each pixel in the acquired depth map to obtain reliable and unreliable pixels. The first iteration uses the initialized depth map, and subsequent iterations use a depth map obtained by upsampling the depth map from the previous iteration; S22, updating the depth estimate of each reliable pixel using the PatchMatch algorithm based on the feature maps of all images at their respective layers; S23, reconstructing the depth estimate of each unreliable pixel using a preset deformation algorithm based on the feature maps of all images at their respective layers. The preset deformation algorithm includes: segmenting different objects in the reference image into different regions, and for each unreliable pixel, selecting multiple anchor points from reliable pixels in the same region as the unreliable pixel for patch deformation to reconstruct the depth estimate of each unreliable pixel; S24, obtaining the depth map of the corresponding layer based on the depth estimates of all pixels.

[0017] In some embodiments of the present invention, in step S23, the method of segmenting different objects in the reference image into different regions includes:

[0018] Semantic-SAM is used to perform multi-granularity panoramic segmentation on the reference image to obtain multi-granularity mask maps. Boundary maps of each multi-granularity mask map are extracted. Each boundary map includes the boundaries of multiple segmented regions, and the region within each boundary is used as a label for an object. For each unreliable pixel, an anchor point set including multiple anchor points is constructed using each boundary map of the multiple boundary maps as constraints. Each anchor point is selected from the same region within the boundary map corresponding to the unreliable pixel. The boundary maps of the multi-granularity mask maps are aggregated based on the multiple anchor point sets to obtain an aggregated boundary map, which includes the boundaries of different regions of different objects obtained from the aggregation. The region within each of the aggregated boundaries is used as the final label for the object.

[0019] In some embodiments of the present invention, the boundary maps of the aggregated multi-granularity mask maps are configured as follows:

[0020]

[0021] Where B represents the aggregated boundary graph, τ(·) represents the transformation function, N represents the total number of mask graphs, n represents the mask graph number (n = 1, 2, ..., N), K represents the total number of objects contained, and k represents the object label number (k = 1, 2, ..., K). This represents the k-th label of the n-th mask image. This means that the k-th label of the n-th mask image is preserved during aggregation. This means that the k-th label of the n-th mask image is not retained during aggregation. This indicates the retention status of the k-th label in the l-th mask image, where l represents the mask image number and i represents the unreliable pixel number. This represents the maximum number of anchor points for the i-th unreliable pixel. This represents the set of anchor points constructed using the boundary map of the l-th mask image for the i-th unreliable pixel. Indicates an indicator function, when hour when hour This represents the number of anchor points classified as interior points after performing a planarization operation on multiple anchor point sets of multiple boundary maps of a multi-granularity mask map. |S represents the set of anchor points constructed using the boundary map of the nth mask image for the i-th unreliable pixel. n | represents the anchor set S n The number of anchor points in the code.

[0022] In some embodiments of the present invention, the method of segmenting different objects in the reference image into different regions further includes: using a preset edge correction strategy to perform edge correction on the aggregated boundary map to obtain a corrected boundary map;

[0023] The edge correction strategy is as follows:

[0024]

[0025] Where minΨ(L) represents edge correction by minimizing the overall energy function Ψ(L), K represents the total number of labels in the aggregated boundary map, i represents the label number in the aggregated boundary map, and ψ u (·) represents the single-point potential function, ψ u (l i) indicates that pixel p is segmented into label l i The probability, l i Let ψ represent the i-th label in the aggregated boundary graph, and j represent the number of another label in the aggregated boundary graph. p (·) denotes the dual potential function, ψ p (l i ,l j This indicates that two adjacent pixels, p and q, are segmented into label l. i and tag l j The joint probability, l j This represents the j-th label in the aggregated boundary graph. μ(·) represents the label compatibility function, t represents the current iteration number, and D p D represents the depth estimate of pixel p obtained in the (t-1)th iteration. q Let I represent the depth estimate of pixel q obtained in the (t-1)th iteration, where α represents the weight parameter. p I represents the normalized RGB color of pixel p. q β represents the normalized RGB color of pixel q, and β represents the weight parameter.

[0026] In some embodiments of the present invention, the method of selecting multiple anchor points for patch deformation includes:

[0027] Centered on each unreliable pixel, the region within the boundary of that pixel is divided into multiple sector regions by a preset number of pixels. The preset number of pixels is the ratio of the total number of reliable pixels within the boundary of the region to the preset number of sector regions. Multiple anchor points are selected from all reliable pixels in each sector region. Anchor point clustering is performed based on the anchor points of all sector regions to obtain multiple anchor point clusters. Each anchor point cluster includes one or more anchor points. For each anchor point cluster, a sub-patch is constructed centered on each anchor point in the cluster, and the sub-patches do not overlap. The sub-patches of all anchor points contained in all anchor point clusters are taken as the result of patch deformation.

[0028] In some embodiments of the present invention, the method of performing anchor point clustering includes: performing RANSAC-based planarization processing on the anchor points of all sector regions to obtain an optimized plane; selecting anchor points for clustering based on the optimized plane; and using the density-based DBSCAN algorithm to cluster the selected anchor points to obtain multiple anchor point clusters.

[0029] In some embodiments of the present invention, the sector equalization is performed in the following manner:

[0030]

[0031] Where, θ′ k θ represents the angle used to divide the k-th sector. ′ k-1 Let represent the angle used to divide the (k-1)th sector, where k represents the angle number, x-1 represents the number of the (x-1)th sector, and i represents the sector number. y represents the number of pre-defined sector regions. b i b represents the boundary pixel of the i-th sector region. i+1 This represents the boundary pixel of the (i+1)th sector. b i and b i+1 Let L1 distance be the distance between the i-th sector and use this distance as the reliable number of pixels in the i-th sector. Let B represent the aggregated image. b i and b i+1 None of them are located on the boundary of the aggregated boundary map.

[0032] In some embodiments of the present invention, the method of reconstructing the depth estimate for each unreliable pixel includes:

[0033] Multiple searches are performed using an iterative local search algorithm to obtain multiple sets of representative anchor points. During each search, a preset number of pixels are randomly selected from each sub-patch of all sub-patches as representative anchor points, resulting in a set of representative anchor points. Based on assigning different disparities to each anchor point in each set of representative anchor points, the minimum deformation matching cost corresponding to that set of representative anchor points is calculated. The minimum deformation matching costs corresponding to multiple sets of representative anchor points are obtained, and the set of representative anchor points corresponding to the minimum deformation matching cost is selected. Based on the set of representative anchor points and the pixels corresponding to the set of representative anchor points in the feature maps of each source image at the corresponding level, the depth estimate of the unreliable pixels is obtained.

[0034] According to a second aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method according to any one of the first aspects of the present invention by executing the executable instructions.

[0035] Compared with the prior art, the advantages of the present invention are as follows:

[0036] This invention, based on the APD-MVS algorithm, reconstructs the depth estimate of each unreliable pixel using a pre-defined deformation algorithm when estimating the depth map of each reference image. This algorithm segments different objects in the reference image into different regions, using these regions to restrict deformation patching. Specifically, for each unreliable pixel, multiple anchor points are selected from reliable pixels within the same region for patch deformation. Since reliable pixels within the same region belong to the same object in 3D space, this invention significantly reduces the probability of selecting anchor points from heterogeneous regions, thereby reducing the need for deformation patching based on heterogeneous regions, improving the accuracy of patch deformation, and ultimately more accurately estimating the depth of unreliable pixels, thus increasing the success rate of 3D reconstruction. Attached Figure Description

[0037] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0038] Figure 1 This is a schematic diagram of the multi-view stereo reconstruction method based on image segmentation according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram showing the comparison results of the APD-MVS algorithm and the method of the present invention in selecting anchor points for unreliable pixels according to an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram illustrating the complete process of obtaining the modified boundary map according to an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram illustrating the principle of reconstructing the depth estimate of each unreliable pixel using a preset deformation algorithm according to an embodiment of the present invention.

[0042] Figure 5 A schematic diagram illustrating the principle of patching and deforming the results obtained after anchor point clustering according to an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram comparing the reconstruction process of unreliable pixel depth estimates using the APD-MVS algorithm according to an embodiment of the present invention and the method of the present invention.

[0044] Figure 7 This is a schematic diagram illustrating the overall process of reconstructing unreliable pixel depth estimates using a preset deformation algorithm according to an embodiment of the present invention.

[0045] Figure 8 This is a schematic diagram illustrating the results of testing the method of the present invention and other prior art using the ETH3D dataset according to an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0047] As mentioned in the background section, existing methods, when performing block deformation with unreliable pixels, select reliable pixels whose depth is discontinuous with that of the unreliable pixels as anchor points for patch deformation. This makes the deformation patch inaccurate, resulting in inaccurate depth estimation of unreliable pixels and reconstruction failure.

[0048] To address the aforementioned issues, the inventors analyzed the problems with existing technologies: They assumed all anchor points shared the same depth and normal with the center pixel, a concept known as homogeneity. Homogeneous regions are characterized by depth continuity, making them distinguishable at depth boundaries. However, existing methods that select a reliable blue pixel from the surrounding area centered on an unreliable red pixel suffer from depth discontinuity between the anchor points selected from these reliable blue pixels and the unreliable red pixel. This is because the various reliable blue pixels in the surrounding area may belong to different objects in 3D space, and the positions of these objects vary significantly in actual 3D space. In other words, existing technologies suffer from selecting anchor points from heterogeneous regions, thus affecting the accuracy of depth estimation. Particularly in complex scenes, shadows and occlusion can cause anchor points to cross depth edges and be selected from heterogeneous regions, potentially leading to reconstruction failures.

[0049] Based on the above analysis, the inventors propose a multi-view stereo reconstruction method based on image segmentation. This method includes: acquiring multiple images from different perspectives within the same scene; using each image as a reference image and the remaining images as source images corresponding to that reference image; estimating the depth map of each reference image based on each reference image and the source images; and fusing the depth maps of all reference images to obtain 3D point cloud data for the corresponding scene. Specifically, this invention improves upon the APD-MVS algorithm by modifying its block deformation method when estimating the depth values ​​of unreliable pixels. That is, when estimating the depth map of each reference image, this invention utilizes a preset deformation algorithm to reconstruct the depth estimate value of each unreliable pixel. In this preset deformation algorithm, different objects in the reference image are divided into different regions. The deformation of the patch is restricted by different regions. That is, for each unreliable pixel, multiple anchor points are selected from reliable pixels in the same region as the unreliable pixel for patch deformation. Since the reliable pixels in the same region belong to the same object in the three-dimensional space, the method of this invention can greatly reduce the probability of selecting anchor points from heterogeneous regions, thereby reducing the situation of patch deformation based on heterogeneous regions, improving the accuracy of patch deformation, and thus making the depth estimation of unreliable pixels more accurate and improving the success rate of three-dimensional reconstruction.

[0050] According to one embodiment of the present invention, since the multi-view stereo reconstruction method of the present invention is an improvement on the existing APD-MVS algorithm, and the improvement lies in the block deformation method performed when estimating the depth values ​​of unreliable pixels, the present invention also requires acquiring multiple images captured from different viewpoints in the same scene and the camera parameters corresponding to each image to estimate the depth map of each image. During the estimation process, only the estimation method for the depth estimates of unreliable pixels differs from the APD-MVS algorithm; other processing steps can adopt the processing methods in the APD-MVS algorithm. Illustratively, given an image set formed by multiple images captured from different viewpoints in the same scene, denoted as [image set name missing],... I n Let i represent any image, n represent the image number, and N represent the total number of images. n The corresponding camera parameters include the intrinsic parameter matrix K. n Rotation matrix R n and camera center C n Record the camera parameters as n = 1…N, from Select each image as a reference image I i The rest of the images As a reference image I i The source image. Utilizing the homography principle and camera parameters of each image, and through a reference image I... i With the rest of the images Pairwise matching to reconstruct reference image I i The depth map of the reference image is obtained only after the depth estimates of all pixels in the reference image have been reconstructed. Therefore, the method of this invention adopts the same approach as the APD-MVS algorithm: for each reliable pixel, the traditional PatchMatch algorithm is used to reconstruct the depth estimate of the reliable pixel using fixed patches; while for each unreliable pixel, the modified PatchMatch algorithm in the APD-MVS algorithm is replaced with the preset deformation algorithm in this invention to reconstruct the depth estimate of the unreliable pixel.

[0051] According to one embodiment of the present invention, see Figure 1 This is a schematic diagram of a multi-view stereo reconstruction method based on image segmentation. In the multi-view stereo reconstruction method of this invention, the process of estimating the depth map of each reference image includes steps S1 and S2, and step S2 further includes steps S21, S22, S23, and S24. To better understand this invention, each step will be described in detail below with reference to specific embodiments.

[0052] In step S1, the reference image and each source image are downsampled multiple times to obtain multiple feature maps of different levels corresponding to each image.

[0053] According to one embodiment of the present invention, for each reference image and its source image, scaling at different levels is achieved by downsampling each image multiple times, resulting in a pyramid structure. This pyramid structure includes multiple feature maps at different levels. For example, downsampling each image K times yields K+1 feature maps. In these K+1 feature maps, the original image belongs to the first feature map. The second feature map is obtained after the first downsampling of the original image, and so on, until the K+1th feature map is obtained after the Kth downsampling. Furthermore, the resolution of the feature maps from the first to the K+1th decreases sequentially. These feature maps are stacked from top to bottom to form a pyramid structure corresponding to one image. The K+1th feature map belongs to the first level of the pyramid structure, the Kth feature map belongs to the second level, and the first feature map (i.e., the original image) belongs to the K+1th level.

[0054] In step S2, multiple iterations are performed based on the reference image and multiple feature maps of each source image to obtain the depth map of the reference image. In each iteration, the depth map of each layer of the reference image is obtained based on the feature maps of the reference image and each source image at each layer, and the depth map obtained in the last iteration is used as the depth map of the reference image.

[0055] According to an embodiment of the present invention, in step S2, taking the K+1 level feature map formed in the above embodiment as an example, the first iteration is based on the feature maps of the reference image and each source image at the first level to obtain the depth map of the reference image at the first level. The second iteration is based on the feature maps of the reference image and each source image at the second level to obtain the depth map of the reference image at the second level. In this manner, K+1 iterations are performed, and the depth map obtained in the K+1th iteration is used as the depth map of the reference image.

[0056] According to an embodiment of the present invention, each iteration process includes the following steps S21, S22, S23 and S24:

[0057] Step S21: Evaluate the reliability of the depth estimate of each pixel in the acquired depth map to obtain reliable pixels and unreliable pixels. The initial depth map is used in the first iteration, and the depth map obtained after the first iteration is obtained by upsampling the depth map obtained in the previous iteration.

[0058] According to one embodiment of the present invention, the method for evaluating whether the depth estimate of each pixel in the depth map is reliable is the same as that in the APD-MVS algorithm. If the depth estimate is reliable, the corresponding pixel is a reliable pixel; otherwise, the corresponding pixel is an unreliable pixel, thereby obtaining reliable pixels and unreliable pixels.

[0059] Step S22: Based on the feature maps of all images at their respective layers, update the depth estimate of each reliable pixel using the PatchMatch algorithm.

[0060] According to one embodiment of the present invention, the PatchMatch algorithm is an existing algorithm, abbreviated as PM algorithm, which uses fixed patches to reconstruct and update the depth estimates of reliable pixels. Illustratively, using reference image I... i With the remaining source images The depth map is reconstructed by sequentially matching the feature maps of the corresponding layers. Specifically, when performing each pair of matches, given a reference image I... i In the feature map of the corresponding level and any source image I j Using the homography principle and the camera parameters of the reference and source images, based on the feature map of the corresponding level, a fixed-size block centered on a reliable pixel in the feature map of the corresponding level of the reference image can be found in any source image I. j The projected blocks on the feature map of the corresponding level. The matching cost between two blocks is usually calculated based on the NCC metric. That is, given a fixed patch (the patch involved in this invention is also called a window), the patch is the reference image I. iA square window W centered on a reliable pixel p can be used to visualize the source image I using the homography principle. j Obtain its corresponding patch W ′ Then, the matching cost is calculated as follows:

[0061]

[0062] Where cov represents the weighted covariance, m i,j (p,W) represents the reference image I. i Reliable pixel p and source image I j The correlation between pixels in the image is used to determine whether they are points in the same physical scene, and m i,j The smaller (p, W) is, the more correlated the two pixels are. Finally, the reference image I is implemented in the above manner. i Feature maps of corresponding levels and residual source images In this process, the depth estimate of pixel p is reconstructed by sequentially matching the feature maps of the corresponding layer for each source image. That is, the multi-view aggregation cost can be obtained through view weights, and further propagation and optimization are used to minimize the multi-view aggregation cost to update the depth estimate of pixel p. The multi-view aggregation cost is defined as:

[0063]

[0064] Among them, w ′ *I j ) represents any source image I j The corresponding view weight parameter can be inferred through the ACMMP algorithm.

[0065] Step S23: Based on the feature maps of each image at the corresponding level, reconstruct the depth estimate of each unreliable pixel using a preset deformation algorithm. The preset deformation algorithm includes: segmenting different objects in the reference image into different regions; for each unreliable pixel, selecting multiple anchor points from reliable pixels in the same region as the unreliable pixel for patch deformation to reconstruct the depth estimate of each unreliable pixel.

[0066] According to one embodiment of the present invention, see Figure 2 This is a schematic diagram comparing the APD-MVS algorithm and the method of this invention in anchor point selection for unreliable pixels. Figure 2 In the image 'a', Figure 2In the image, b represents the segmented image obtained after dividing different objects in the image into different regions. Below are comparisons of anchor point selection in the reliable pixel map using the APD-MVS algorithm and the method of this invention, as well as the region comparison of the selected anchor point in the depth map. In the reliable pixel map, green represents reliable pixels, and white represents unreliable pixels. In both the reliable pixel map and the depth map, red represents unreliable pixels, and anchor points are selected with these unreliable pixels as the center. The selected blue points are used as anchor points. Figure 2 The anchor points in the reliable pixel map and depth map corresponding to 'c' were selected using the method described in the APD-MVS algorithm. Figure 2 In the reliable pixel map and depth map corresponding to d, the anchor points are selected using the preset deformation algorithm in the method of this invention. It can be seen that in the APD-MVS algorithm, the anchor points cross depth edges and are selected in heterogeneous regions where depth discontinuities exist, leading to inaccurate subsequent patch deformation based on these anchor points, thus reducing the accuracy of the depth estimation. In contrast, the method of this invention first segments different objects in the image into different regions, obtaining segmented images. Multiple anchor points are selected only from reliable pixels in the same region as the unreliable pixel in the segmented image, greatly reducing the probability of selecting anchor points from heterogeneous regions, thereby improving the accuracy of patch deformation and thus performing more accurate depth estimation.

[0067] The following sections will explain the methods of segmenting into different regions and selecting multiple anchor points in the preset deformation algorithm.

[0068] (a) Methods of dividing into different regions

[0069] According to an embodiment of the present invention, in step S23, the method of segmenting different objects in the reference image into different regions includes steps a1, a2, and a3:

[0070] Step a1: Use Semantic-SAM to perform multi-granularity panoramic segmentation on the reference image to obtain multi-granularity mask images. Extract the boundary maps of each multi-granularity mask image. Each boundary map includes the boundaries of multiple segmented regions. Use the region within each boundary as a label of an object.

[0071] Because the existing APD-MVS algorithm skips the selection of anchor points for depth edges in heterogeneous regions, it leads to potential inaccuracies. This invention increases the constraint on depth edges by segmenting each object into different regions. Therefore, according to one embodiment of the invention, the Semantic-SAM model is used to process the input reference image I. iMulti-granularity panoramic segmentation is performed to generate three mask images of different granularities, from coarse to fine. These three mask images capture semantic information at three different levels: scene, instance, and feature, respectively. The segmented mask images also include potential depth edges. Each mask image is denoted as M. n n = 1, 2, 3, where M n Let B represent the nth mask image. When extracting the boundary map from each mask image, a transformation function τ can be applied to obtain the boundary map, and the boundary map of the nth mask image is denoted as B. n =τ(M n If the image contains k objects, each boundary map includes the boundaries of the k segmented regions. Each region within a boundary is used as a label for an object, and... Defined as the nth mask image M n The k-th label in the model. The technical solution of this embodiment can achieve at least the following beneficial technical effects: using the Semantic-SAM model to segment and extract a mask image containing multi-granularity depth edges as a prior, so as to pre-limit the patch deformation in the uniform region, and accurately capture and utilize depth edge information at different scales, thereby enhancing the reliability of anchor point selection, improving the reliability of pixel depth estimation, and improving the reconstruction effect of textureless regions.

[0072] Step a2: For each unreliable pixel, construct an anchor point set including multiple anchor points using each boundary map in multiple boundary maps as constraints, resulting in multiple anchor point sets. Each anchor point is selected from the same region within the boundary map corresponding to the unreliable pixel.

[0073] According to one embodiment of the present invention, for each unreliable pixel, each boundary map B is used. n To constrain any corresponding anchor point s i , that is s i ∈S n This allows for the construction of multiple different anchor point sets, S n This indicates that the nth boundary map B is used. n The nth set of anchor points constructed for unreliable pixels. To keep the anchor points within the same region, when anchor points s i The connection between unreliable pixel p and B n When they do not intersect, anchor point s i Only then can it be included in the anchor set S n In this way, anchor points are selected from the same region within the boundary map corresponding to the unreliable pixel; otherwise, they are removed from S. n Remove from the middle. That is, the anchor set is constructed as follows:

[0074]

[0075] It should be noted that other methods can also be used to select anchor points from the same region within the boundary map corresponding to the unreliable pixel, and this invention is not limited to this.

[0076] According to one embodiment of the present invention, a RANSAC planeization operation is performed on all anchors in each of multiple different anchor sets to obtain the number of anchors classified as interior points, referred to as A. n Then according to A n Determine the optimal distribution S in all anchor point sets. max Specifically, it is believed that S max The number of anchor points A to be retained should be maximized first. n Then maximize the proportion of anchor points retained. The optimal distribution is determined as follows:

[0077]

[0078] According to an embodiment of the present invention, if for image I j For each unreliable pixel, determine its optimal distribution S. max You can choose to use S directly. max It is used as a deformable patch to replace the deformable patch obtained in the APD-MVS algorithm.

[0079] Step a3: Aggregate the boundary maps of the multi-granularity mask maps based on multiple anchor point sets to obtain the aggregated boundary map, which includes the boundaries of different regions of different objects obtained by aggregation. The region within each boundary obtained by aggregation is used as the final label of the object.

[0080] However, in the above embodiments, S is used directly. max Replacing the deformed patch obtained in the APD-MVS algorithm with a deformed patch may make the deformed patch more susceptible to noise interference or local optima. Therefore, according to an embodiment of the present invention, an optimal distribution is determined based on multiple anchor point sets, and the boundary maps of the multi-granularity mask maps are aggregated based on the optimal distribution to merge different mask maps M. n This forms an aggregated boundary map that segments the image into different regions. The deformation patch is then constrained based on this aggregated boundary map. Illustratively, if its mask map M... n The kth tag in This shows the optimal distribution within the internal pixels, i.e., the highest number of anchor points, which is retained only during aggregation. The k-th label in other mask images is removed. The label retention for each mask image is determined as follows:

[0081]

[0082] in, This means that the k-th label of the n-th mask image is preserved during aggregation. This means that the k-th label of the n-th mask image is not retained during aggregation. This indicates the retention status of the k-th label in the l-th mask image, where l represents the mask image number and i represents the unreliable pixel number. This represents the maximum number of anchor points for the i-th unreliable pixel. This represents the set of anchor points constructed using the boundary map of the l-th mask image for the i-th unreliable pixel. Indicates an indicator function, when hour when hour This represents the number of anchor points classified as interior points after performing a planarization operation on multiple anchor point sets of multiple boundary maps of a multi-granularity mask map. |S represents the set of anchor points constructed using the boundary map of the nth mask image for the i-th unreliable pixel. n | represents the anchor set S n The number of anchor points in the image. The technical solution of this embodiment can at least achieve the following beneficial technical effects: when merging tags from different mask images, the present invention can maximize S. max A number of reliable masks are effectively preserved. Subsequently, all preserved labels are aggregated to obtain an aggregated boundary map. The aggregated boundary map is used to constrain patch deformation. This process can not only obtain images that segment different objects into different regions, but also achieve robust patch deformation.

[0083] According to one embodiment of the present invention, the boundary map of the aggregation is obtained using the τ function. That is, the boundary maps of the aggregation of multiple granularity mask maps are obtained as follows:

[0084]

[0085] Where B represents the aggregated boundary graph, τ(·) represents the transformation function, N represents the total number of mask graphs, n represents the mask graph number (n = 1, 2, ..., N), K represents the total number of objects contained, and k represents the object label number (k = 1, 2, ..., K). This represents the k-th label of the n-th mask image.

[0086] Most panoramic segmentation algorithms may encounter problems such as edge blurring and edge overlap. Edge blurring refers to the deviation between the segmentation result and the actual depth edge, while edge overlap occurs when adjacent labels overlap. Therefore, according to an embodiment of the present invention, after performing the above steps a1-a3 to obtain the aggregated boundary map, a preset edge correction strategy is used to correct the edges of the aggregated boundary map, resulting in a corrected boundary map. Since edges mainly exhibit depth and color differences, the preset edge correction strategy is an edge correction strategy based on the Conditional Random Field (CRF) algorithm, which combines geometric and color similarity for edge correction. Finally, the corrected boundary map obtained by this strategy can be used to constrain patch deformation, enabling more accurate deformation patches.

[0087] According to one embodiment of the present invention, the preset edge correction strategy is as follows:

[0088]

[0089] Where minΨ(L) represents edge correction by minimizing the overall energy function Ψ(L), K represents the total number of labels in the aggregated boundary map, i represents the label number in the aggregated boundary map, and ψ u (·) represents the single-point potential function, ψ u (l i ) indicates that pixel p is segmented into label l i The probability, l i Let ψ represent the i-th label in the aggregated boundary graph, and j represent the number of another label in the aggregated boundary graph. p (·) denotes the dual potential function, ψ p (l i ,l j This indicates that two adjacent pixels, p and q, are segmented into label l. i and tag l j The joint probability, l j This represents the j-th label in the aggregated boundary graph. μ(·) represents the label compatibility function, t represents the current iteration number, and D p D represents the depth estimate of pixel p obtained in the (t-1)th iteration. q Let I represent the depth estimate of pixel q obtained in the (t-1)th iteration, where α represents the weight parameter. p I represents the normalized RGB color of pixel p. qLet β represent the normalized RGB color of pixel q, and β represent the weighting parameter. This embodiment achieves at least the following beneficial effects: as the estimated depth map becomes increasingly reliable with each iteration, the algorithm gradually prioritizes depth consistency over color consistency. The edge correction strategy of this invention combines depth and color, determining the optimal label allocation by minimizing the overall energy function Ψ(L) across all pixels. This further guides the details of patch deformation and improves the accuracy of patch deformation.

[0090] It should be noted that the present invention can constrain patch deformation based on the aggregated boundary map or based on the modified boundary map. The following is an illustrative description of the overall process of segmenting into different regions with the goal of obtaining the modified boundary map:

[0091] See Figure 3 This is a schematic diagram illustrating the complete process of obtaining the corrected boundary map. First, the existing technology will be explained. Figure 3 In the image, 'a' represents the patch deformation using the ADP-MVS algorithm. This image shows the corrected boundary map, divided into two homogeneous regions, A and B. Centered on pixel 'p', it is divided into eight sectors, each at a 45-degree angle. Within each sector, an anchor point is selected, resulting in multiple anchor points S1, S2, S3, S4, S5, S6, S7, and S8. Patch deformation is then performed using each anchor point, resulting in the image shown below. Figure 3 In the form of 'a', where the anchor points in the patches obtained by S2, S3, and S4 belong to heterogeneous regions. In the processing of this invention, firstly, the Semantic-SAM model of the above embodiment is used to process the input image (e.g., Figure 3 (e) Perform multi-granularity panoramic segmentation to generate three mask images of different granularities from coarse to fine, as follows: Figure 3 The middle f part is a mask image of the scene hierarchy; Figure 3 The middle g part is a mask image of the instance hierarchy; Figure 3 The h-part in the middle is a mask image of the feature level. Figure 3 In the middle b, the local boundary map is extracted from the three mask maps respectively through step a1 of the above embodiment. This boundary map is related to the left side. Figure 3 The middle 'a' corresponds to this. Figure 3 The first boundary map of b includes labels. and Figure 3 The second boundary map of b includes labels. and Figure 3 The third boundary map of b includes labels. and A, B, C, and D represent the regions corresponding to four different possible objects. Figure 3In part c, using step a2 above: for unreliable pixel p, multiple anchor point sets are constructed as S1={s1,s5,s6,s7,s8}, S2={s1,s7,s8}, and S3={s7}, therefore, S is obtained. max =S1, then and because Therefore, retain Figure 3 The first boundary map of b is removed, and the other two boundary maps are discarded to obtain the aggregated boundary map. Figure 3 In this context, 'i' represents the aggregated boundary map corresponding to the entire image. The edge correction strategy described above is then applied to the aggregated boundary map to correct its edges, resulting in... Figure 4 In the form of d. Where, Figure 4 In the image, j represents the corrected boundary map corresponding to the entire image.

[0092] (ii) Method of selecting multiple anchor points

[0093] According to one embodiment of the present invention, the method of selecting multiple anchor points for patch deformation includes steps b1, b2, and b3:

[0094] Step b1: Centered on each unreliable pixel, the region within the boundary of that pixel is divided into multiple sector regions by a preset number of pixels. The preset number of pixels is the ratio of the total number of reliable pixels within the boundary of its region to the preset number of sector regions.

[0095] Because the APD-MVS algorithm divides the sector regions by a fixed average angle, some sector regions are inevitably completely surrounded by depth edges, causing anchor points in these sector regions to be discarded, resulting in waste. Therefore, according to an embodiment of the present invention, sector equalization is performed according to a preset number of pixels (i.e., the average number of reliable pixels). The sector equalization method is as follows:

[0096]

[0097] Where, θ ′ k θ represents the angle used to divide the k-th sector. ′ k-1 Let represent the angle used to divide the (k-1)th sector, where k represents the angle number, x-1 represents the number of the (x-1)th sector, and i represents the sector number. y represents the number of pre-defined sector regions. b i b represents the boundary pixel of the i-th sector region. i+1This represents the boundary pixel of the (i+1)th sector. b i and b i+1 Let L1 distance be the distance between the i-th sector and use this distance as the reliable number of pixels in the i-th sector. Let B represent the aggregated image. b i and b i+1 None of them are located on the boundary of the aggregated boundary map. The technical solution of this embodiment can at least achieve the following beneficial technical effects: Within the same area, fan-shaped regions are uniformly divided according to the number of reliable pixels. Since reliable pixels are usually continuously distributed, b i and b i+1 The L1 distance between them can approximate the sector region V. i Number of reliable pixels n i This ensures that each sector contains approximately the same number of reliable pixels, guaranteeing a balanced division of the region without wasting resources.

[0098] According to one embodiment of the present invention, boundary pixels generally refer to pixels in an image whose grayscale values ​​change significantly. If b i and b i+1 If none of the pixels are on the boundary, or only one of the boundary pixels is on the boundary, then b is considered to be on the boundary. i and b i+1 The sector-shaped region contains a certain number of reliable pixels. Conversely, and b i and b i+1 All are located on the boundary of the unreliable pixel p, which means b i and b i+1 If both are located at the depth edge, then b is considered... i and b i+1 Some sectors lack reliable pixels. Based on this principle, sector equalization ensures that each sector includes continuously distributed reliable pixels. See also Figure 4 This diagram illustrates the formation of multiple sector regions after sector equalization in this invention. In the diagram, the purple-framed area represents one region, and the green-framed area represents another. Within the purple-framed area, white pixels are unreliable pixels, and purple pixels are reliable pixels. The preset number of sector regions is 8; therefore, it needs to be divided 8 times. Each time, the preset number of pixels is used to divide the sector regions, resulting in the desired outcome. Figure 5The division result is shown below. Centered on the unreliable pixel p, the eight sector regions obtained by the formula in the above embodiment are divided at angles θ1, θ2, θ3, θ4, θ5, θ6, θ7, and θ8, respectively. The eight sector regions are V1, V2, V3, V4, V5, V6, V7, and V8. Here, b1, b2, b3, b4, b5, b6, b7, and b8 are the boundary pixels of the corresponding sectors, and the L1 distance between the boundary pixels is the same, ensuring that each sector region contains approximately the same number of reliable pixels. Since b1 and b9 are both located on the boundary of the unreliable pixel p, it is assumed that the portion from b1 to b9 on the right side does not contain reliable pixels and is not divided. Finally, an anchor point is selected within each of the eight sector regions, with the anchor points for each region being s1, s2, s3, s4, s5, s6, s7, and s8. After patching with each of these anchor points as the center, the result is as follows: Figure 5 The result.

[0099] Step b2: Select multiple anchor points from all reliable pixels in each sector of multiple sector regions, and perform anchor point clustering based on the anchor points of all sector regions to obtain multiple anchor point clusters. Each anchor point cluster includes one or more anchor points.

[0100] According to one embodiment of the present invention, in the multiple sector regions obtained after sector equalization, 3 or 4 anchor points are selected from the reliable pixels of each sector region. The present invention is not limited to this, and the number of anchor points can be set according to actual needs.

[0101] According to one embodiment of the present invention, the anchor point clustering method includes: performing RANSAC-based planarization processing on all anchor points in the sector region to obtain an optimized plane; selecting anchor points for clustering based on the optimized plane; and using a density-based DBSCAN algorithm to cluster the selected anchor points to obtain multiple anchor point clusters. The selection of anchor points for clustering based on the optimized plane includes setting a threshold ε to filter anchor points (i.e., abnormal anchor points), with the remaining anchor points serving as reliable anchor points for subsequent clustering. The density-based DBSCAN algorithm is used for clustering, which is robust to noise. Specifically, anchor points with η neighbors within a radius are classified as core points. If these core points are within a preset range g, they are connected. If the distance between core points is within a preset range γ, they are connected to form different clusters. Other unconnected points are integrated into existing clusters if they are within the preset range γ of any connected core point; otherwise, they form an independent cluster.

[0102] Step b3: For each anchor cluster, construct a sub-patch centered on each anchor in the cluster, with no overlap between sub-patches. The sub-patches of all anchors contained in all anchor clusters are used as the result of patch deformation.

[0103] According to one embodiment of the present invention, see Figure 6 This is a schematic diagram illustrating the principle of patching and deforming the results obtained after anchor point clustering. After anchor point clustering, N anchor point clusters are obtained, namely U1, U2, U3, U4, U5, U5…, U… N-1 U N Any anchor cluster is defined as Any anchor cluster contains Anchor points, k i Represents the i-th anchor cluster U i The number of anchor points. Specifically, for cluster U... i All anchor points are S i ′ =s ij |j=1,…,k i s ij Represents the i-th anchor cluster U i The j-th anchor point in the array has its child patch size set to 1. To balance attention, w represents the original patch size, thus avoiding patch overlap in dense areas. Illustratively, if the original patch size is 4x4, the anchor cluster U... i The number of anchor points is k i =4, then the patch size for each anchor point is set to 1*1. For example... Figure 6 As shown, U1 includes two anchor points, which are s 11 and s 12 U2 includes 3 anchor points, and they are s 21 s 22 and s 23 U3 includes 4 anchor points, which are s 31 s 32 s 33 and s 34 U4 includes 4 anchor points, and they are s 41 s 42 s 43 and s 44 U5 includes 2 anchor points, U6 includes 1 anchor point, ..., U N-1 Includes 1 anchor point, U N It includes 3 anchor points. There are no overlapping areas between the sub-patches of each anchor point. Finally, it sets all anchor points in the entire cluster. Replace the selected anchor set S in the APD-MVS algorithm with the patch deformation. That is, use the set of sub-patches of all anchors contained in all anchor clusters as the result of the patch deformation.

[0104] In summary, the technical solution of the embodiment of selecting multiple anchor points in Part (II) above can achieve at least the following beneficial technical effects: by uniformly distributing reliable pixels in each sector, the problem of uneven distribution of reliable pixels between different sectors is solved, improving the accuracy and efficiency of anchor point selection in texture-rich areas, thereby enhancing the contribution of these areas to the final 3D reconstruction result. Furthermore, the existing APD-MVS algorithm selects only one anchor point for patch deformation in each sector, and selecting only one anchor point in each sector cannot well represent all the features within that sector. This invention selects multiple anchor points for patch deformation in each sector, effectively capturing the features within the sector. Selecting more anchor points and then clustering them effectively improves the anchor points' ability to generalize about the sector. The anchor point clustering strategy further optimizes the anchor point selection process. This not only ensures the geometric and textural consistency of the anchor point pixels selected during patch deformation and depth estimation, but also improves the stability and reliability of the reconstruction process, making the 3D reconstruction more accurate and realistic in detail.

[0105] Based on the above two parts, the candidate point optimization strategy based on the iterative local search algorithm is used to reconstruct the depth estimate of each unreliable pixel. The strategy is explained below.

[0106] (III) Candidate point optimization strategy based on iterative local search algorithm

[0107] According to an embodiment of the present invention, in step S23, the method for reconstructing the depth estimate of each unreliable pixel based on the patch deformation results obtained in the above embodiment includes: performing multiple searches based on an iterative local search algorithm to obtain multiple sets of representative anchor points; during each search, randomly selecting a preset number of pixels from each sub-patch of all sub-patches as representative anchor points to obtain a set of representative anchor points; calculating the minimum deformation matching cost corresponding to the set of representative anchor points based on assigning different disparities to each anchor point in each set of representative anchor points; obtaining the minimum deformation matching cost corresponding to multiple sets of representative anchor points, selecting the set of representative anchor points corresponding to the minimum deformation matching cost from them, and obtaining the depth estimate of the unreliable pixel based on the set of representative anchor points and the pixels corresponding to the set of representative anchor points in the feature maps of each source image at the corresponding level.

[0108] According to one embodiment of the present invention, see Figure 6 This is a schematic diagram comparing the reconstruction process of unreliable pixel depth estimates using the APD-MVS algorithm and the method of this invention. The following section discusses this... Figure 6The three parts a, b, and c in the text will be explained separately:

[0109] Figure 6 In diagram 'a', the reconstruction process of the existing APD-MVS algorithm is described. Anchor points 's' are selected at pixel p. i Taking the patch centered (i.e., the red square) containing 9 sampled pixels at fixed intervals (i.e., green candidate points) as an example, cost calculation is performed. During cost calculation, for the selected 9 fixed pixels, given pixel p and its disparity d, a disparity sequence D = {d0, d1, ..., d...} of length μ is created. μ-1 ,d μ}. Subsequently, for each disparity d in the disparity sequence i Used for deformation matching cost calculation to obtain the deformation matching cost sequence. like Figure 6 The results shown in the coordinate system of 'a' are illustrated, with the horizontal axis representing disparity and the vertical axis representing the similarity index (i.e., the calculated deformable matching cost). Through multiple iterations, each iteration assigns the same disparity from the disparity sequence to the sampled pixels. Different disparities are assigned to these sampled pixels in multiple iterations, constructing a cost distribution in 2D space to help determine the minimum deformable matching cost. However, sampling pixels from a sub-patch at fixed intervals typically cannot globally represent the features of that sub-patch; that is, fixed candidate points usually cannot reflect the optimal contour of the sub-patch. Therefore, its matching cost distribution is completely different from the optimal matching cost distribution. The minimum matching cost determined by existing methods may be a local minimum, failing to find the global minimum.

[0110] Figure 6 In section b, the feature analysis process for pixels within the patch is described. Anchor point s... i The pixels within the central patch are divided into three categories, referred to as blue pixels p. blue Green pixel p green and orange pixel p orange Because p blue and s i Including elements that do not belong to the same region will increase costs. Conversely, p green and p orange Both with s i Those belonging to the same region will reduce costs by including them. However, p green Located within a region, it lacks distinction from neighboring pixels. Therefore, depth variations can lead to ambiguity, preventing contour (i.e., deformable matching cost distribution) convergence. In contrast, p... orange Located near the depth boundary, these pixels are clearly distinguishable from their neighbors. The optimal depth can be identified based on edge variations as their depth changes, thus promoting contour convergence. Therefore, p... orangeThese are ideal candidate pixels. Based on these characteristics, an iterative local search algorithm (ILS function) is designed to perform multiple searches.

[0111] Figure 6 In the diagram, 'c' represents the reconstruction process of the present invention. To maintain spatial consistency, firstly, taking the original APD-MVS algorithm as an example where pixels are sampled at 9 fixed intervals, similarly, the present invention samples 9 pixels per cluster. The sampling process includes: for each cluster U... i Any anchor point S in the assignment k candidate points (also called representative anchor points) i Represents the i-th cluster U i The total number of anchor points, i.e., from cluster U i In each sub-patch, a preset number of pixels are selected, and the preset number is... The positions of the selected pixels are then randomly initialized to obtain a set of representative anchor points, which are called x0, for use in matching cost calculation. The pixel positions are then randomly shuffled to obtain multiple solutions. For each solution x j Given a pixel p and its disparity d, also create a disparity sequence D = {d0, d1, ..., d2} of length μ. μ-1 ,d μ}, with center d and interval δ, is defined as: d in each D i Used for deformation matching cost calculation to obtain the deformation matching cost sequence. and average matching cost Then, the ILS function is defined to minimize the matching cost and the maximum variance of the contour, as shown below:

[0112] F I (x j )=υf1(x j )+ωf2(x j (9)

[0113] Finally, the evaluation That is, determining F from multiple solutions. I The solution x that corresponds to the minimum value obtained from the function calculation. j And in subsequent optimizations, solution x was used. j υ and ω both represent weight parameters. The candidate point optimization strategy of this invention, based on an iterative local search algorithm, can select from multiple solutions after multiple iterations of candidate point selection. The optimal set of representative anchor points and the globally minimum deformation matching cost are identified to achieve a globally optimal estimate of pixel depth. (Illustrative example) Figure 6 Cluster U shown in (c) i If there are 3 anchor points, then there are 3 sub-patches, which is s. i1 s i2 and s i3 From each sub-patch, three representative anchor points are randomly selected and their order is randomly shuffled to generate three solutions: x1, x2, and x3. All green pixels in x1 form the first group of representative anchor points, all red pixels in x2 form the second group, and all blue pixels in x3 form the third group. Figure 6 (c) shows three solutions x1, x2, and x2, where, given pixel p and its disparity sequence, for each group of representative anchor points, the following is applied: Figure 7 (a) Repeat the same iterative update process multiple times to obtain, as shown in the example. Figure 8 The results shown in (c) are in a coordinate system where the horizontal axis represents disparity and the vertical axis represents the similarity index. This constructs multiple cost distributions in 3D space to help determine the minimum deformation matching cost. Multiple sets of representative anchor points with the minimum deformation matching cost are obtained, and the set of representative anchor points corresponding to the minimum deformation matching cost is selected. That is, the F-values ​​corresponding to solutions x1, x2, and x3 are determined using the ILS function. I (x1), F1(x2), F1(x2), and according to F1(x2) <F1(x1)<F I (x3) Solution x2 is determined to have the minimum deformation matching cost, and the depth estimate of the pixel is reconstructed using the optimal set of representative anchor points x2. The adaptive selection method of this invention effectively expands the receptive field of each patch. Furthermore, by constructing multiple deformation matching cost distributions and adaptively selecting the optimal set of representative anchor points for optimization, not only is the true global minimum deformation matching cost determined, but computational efficiency is also improved.

[0114] According to one embodiment of the present invention, the deformation matching cost in the APD-MVS algorithm is used for calculation, and the calculation method is as follows:

[0115]

[0116] Where, m i,j (p,S) represents the reference image I. i Unreliable pixel p and source image I j The correlation between pixels in the image, S represents the set of anchor points obtained by the method of this invention, and W s Let λ represent the sub-patch centered at anchor point s, λ represent the weight parameters, and m(p,W) represent the reference image I. i Unreliable pixel p and source image I jThe correlation between pixels in the matrix, the calculation principle of m(p,W) is the same as that of formula (1) above, m(s,W) s ) represents reference image I i Anchor point s and source image I j The correlation between pixels in m(s,W) s The calculation principle of ) is the same as that of formula (1) above, that is, it is necessary to find the reference image I in the source image. i sub-patch W s Corresponding sub-patch W s ′ The calculation is performed, and |S| represents the number of anchor points selected by the method of the present invention.

[0117] Step S24: Obtain the depth map of the corresponding level based on the depth estimation values ​​of all pixels.

[0118] According to one embodiment of the present invention, after obtaining the depth estimates of all pixels in the reference image, the depth estimates of all pixels are fused to obtain a depth map of the corresponding level.

[0119] According to one embodiment of the present invention, the general framework of the process for reconstructing the depth estimate of each unreliable pixel using a preset deformation algorithm will now be described. See also Figure 8This is a schematic diagram illustrating the overall process of reconstructing unreliable pixel depth estimates using a pre-defined deformation algorithm. Given a series of images, each image is downsampled multiple times to obtain feature maps at different levels that constitute a pyramid structure. Based on the feature maps at each level of all images, the depth map at each level of each image is estimated. During estimation, the Semantic-SAM model is first used to perform multi-granularity panoramic segmentation on the image to be estimated, resulting in multi-granularity boundary maps, including boundary maps at three granularities. These boundary maps are aggregated, and the edge correction strategy CRF is used to correct the edges of the aggregated boundary maps, thus obtaining corrected boundary maps. This process segments different objects in the image into different regions, obtaining segmented images, which are then used to guide subsequent patch deformation. Compared to the original APD-MVS algorithm, which divides the image into three sectors V1, V2, and V3 by angular averaging and selects an anchor point in each sector, this invention utilizes sector equalization and anchor point clustering from the above embodiment to select multiple anchor points, ensuring wider anchor point coverage and expanding the receptive field. First, sector equalization is used to divide the image into multiple sector regions, designated V1, V2, V3, V4, and V5, based on pixel-wise averages. Multiple anchor points are selected within each sector region, and then anchor point clustering is performed to form multiple anchor point clusters, designated U1, U2, U3, and so on. Next, an iterative local search-based candidate point optimization strategy is used to adaptively select multiple representative anchor points. By iterating through each of these representative anchor points multiple times, multiple cost distributions are constructed to help determine the minimum deformation matching cost. The optimal set of representative anchor points is then selected for optimization, resulting in more efficient depth estimates for unreliable pixels. After fusing the depth estimates of all pixels in the image to obtain the corresponding level's depth map, this depth map is upsampled. Based on the upsampled depth map, the next level's depth map is estimated. The final level's depth map is used as the final depth map of the image. All images are then fused to obtain the 3D point cloud data.

[0120] To verify the beneficial effects of the present invention, the inventors conducted the following experiments:

[0121] Experimental platform parameters:

[0122] CPU: Intel(R)Core(TM)i7-10700 CPU@2.90GHz;

[0123] GPU: NVIDIA GeForce 3080RTX;

[0124] Operating system: Windows 10;

[0125] Code runtime environment: C++ with CUDA.

[0126] Experimental procedure:

[0127] 1. Comparative Experiment

[0128] This invention uses the ETH3D dataset and the TNT (Tanks and Temples) dataset to conduct comparative experiments with existing technologies on the aforementioned experimental platforms. Existing technologies include advanced learning-based methods such as PatchMatchNet, IterMVS-LS, MVSTER, EPP-MVSNet, and EPNet, as well as traditional methods such as PCF-MVS, ACMM, ACMMP, SD-MVS, HPM-MVS++, and APD-MVS. For parameter settings, α and β in formula (7) are set to 3 and 0.2 respectively; υ and ω in formula (9) are set to 0.2 and 0.1 respectively; the preset range γ in anchor point clustering is set to 8, η = 1, and ε = 5 × 10⁻⁶. -3 ε is used to filter outomaly anchors. When creating the disparity sequence, the length μ is set to 5, and the corresponding interval δ is set to 2.

[0129] (1) Experiments on the ETH3D dataset

[0130] Using the ETH3D dataset, which was divided into a training set and a test set, a comparative experiment was conducted between the proposed method (MSP-MVS) and existing technologies at a threshold of 2 cm. The performance of each method was evaluated in three aspects: F1 score, completeness (Comp.), and accuracy (Accuracy). The experimental results are shown in Table 1 below:

[0131] Table 1

[0132]

[0133] In Table 1, the best results are marked in bold, and the second-best results are marked with an underline. As can be seen from Table 1, the method of this invention achieves the highest F1 score and completeness.

[0134] (2) Experiments on the TNT dataset

[0135] Using the TNT dataset, divided into the TNT Intermediate and TNT Advanced datasets, a comparative experiment was conducted between the proposed method and existing technologies under a given threshold. The performance of each method was evaluated in terms of F1 score, recall, and precision. The experimental results are shown in Table 2 below:

[0136] Table 2

[0137]

[0138] In Table 2, the best results are marked in bold, and the second-best results are marked with an underline. As can be seen from Table 2, the method of this invention achieves the highest F1 score and completeness on the TNT intermediate dataset. It achieves the second-best results on the TNT advanced dataset, with an F1 score less than 0.3% lower than EPNet.

[0139] In summary, the method of this invention achieves the highest F1 score and completeness on the ETH3D dataset and the TNT intermediate dataset, validating its state-of-the-art performance and strong generalization ability. Furthermore, the method of this invention is comparable in runtime to methods such as APD-MVS and HPM-MVS.

[0140] Finally, see Figure 8 This is a schematic diagram showing the results of testing the method of this invention and other existing technologies using the ETH3D dataset. Figure 8 Part a of the dataset includes three scene images: an office image, a delivery area image, and a lecture hall image from the ETH3D dataset. ​ Parts b, c, d, e, and f represent the existing IterMVS LS method, the existing ACMMP method, the existing APD-MVS algorithm, the method of this invention, and the existing MSP-MVS method, respectively. ​ The image shows the results after processing the three scene images in Part a. In the results, the red boxes indicate some challenging areas. Clearly, the method of this invention performs particularly well in large, textureless areas (such as floors and walls), generating the most complete and realistic reconstructed point clouds without introducing detail distortion.

[0141] 2. Ablation test

[0142] Regarding the preset deformation algorithm in the above embodiments, only its first part: the method of segmenting into different regions, is ablated as follows: ① The aggregation part of step a3 is deleted, and the resulting preset deformation algorithm is denoted as w / o.Agr; ② Only the edge correction strategy part after step a3 is deleted, and the resulting preset deformation algorithm is denoted as w / o.Ref; ③ Both the above aggregation part and the edge correction strategy part are deleted, and the resulting preset deformation algorithm is denoted as w / o.Mul.

[0143] Regarding the preset deformation algorithm in the above embodiments, only its second part: the method of selecting multiple anchor points, is ablated as follows:

[0144] ④ Only delete the sector equalization part, and the resulting preset modified algorithm is denoted as w / o.Eqd; ⑤ Only delete the anchor point clustering part, and the resulting preset modified algorithm is denoted as w / o.Clu; ⑥ Delete both sector equalization and anchor point clustering, and the resulting preset modified algorithm is denoted as w / o.Con.

[0145] For the preset deformation algorithm in the above embodiments, only its third part: the candidate point optimization strategy based on the iterative local search algorithm, is ablated as follows:

[0146] ⑦ Only remove the variance component (i.e., f2(x)) from the ILS function. j The resulting preset deformation algorithm is denoted as w / o.Var; ⑧ Only the cost component (i.e., f1(x)) in the ILS function is deleted. j In the first part, the resulting preset deformation algorithm is denoted as w / o.Cst; in the second part, the entire ILS function is deleted, and the resulting preset deformation algorithm is denoted as w / o.Syn, with the length μ set to 3.

[0147] Experiments were conducted on the ETH3D dataset, with given thresholds of 2cm and 10cm, to examine the nine ablation scenarios. The effectiveness of each component of the pre-defined deformation algorithm in this invention was demonstrated by evaluating three aspects: F1 score, completeness (Comp.), and accuracy (Accuracy). The quantitative results of the ablation studies are shown in Table 3 below.

[0148] Table 3

[0149]

[0150] As shown in Table 3 above, for Part I of the algorithm, the result of method ③ with `w / o.Mul` is the worst, highlighting the effectiveness of Part I in patch deformation. Furthermore, method ② with `w / o.Ref` performs better than method ① with `w / o.Agr`, indicating that the aggregation part has a greater impact than the edge correction strategy. This verifies the effectiveness of extracting deep edges in patch deformation. For Part II of the algorithm, the result of method ⑥ with `w / o.Con` is the worst. `w / o.Clu` is slightly better than `w / o.Eqd`, indicating that sector equalization contributes more than anchor point clustering. For Part III of the algorithm, `w / o.Syn` achieves the worst result, while `w / o.Var` produces better results than `w / o.Cst`, indicating that cost is more important than variance in the candidate point optimization process. Therefore, the overall scheme of the MSP-MVS method of this invention has the best effect.

[0151] The existing technology is as follows:

[0152] PatchMatchNet method: see the above reference [1];

[0153] IterMVS-LS method: Wang, F.; Galliani, S.; Vogel, C.; and Pollefeys, M. 2022aIterMVS: Iterative probability estimation for efficient multi-view stereo.InProc.IEEE / CVF Conf.Comput.Vis.Pattern Recognit.(CVPR), 8606–8615.

[0154] MVSTER method: Wang, X.; Zhu, Z.; Huang, G.; Qin, F.; Ye, Y.; He, Y.; Chi, X.; and Wang,

[0155] EPP-MVSNet method: X. Ma, Y. Gong, Q. Wang, J. Huang, L. Chen, F. Yu, EPP-MVSNet: Epipolar assembling based depth prediction for multi-view stereo, in: Proc. IEEE / CVF Int. Conf. Comput. Vis., ICCV, 2021, pp. 5712–5720.

[0156] EPNet method: Su, W.; and Tao, W. 2023. Efficient Edge-Preserving Multi View Stereo Network for Depth Estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2348–2356.

[0157] PCF-MVS method: A. Kuhn, S. Lin, O. Erdler, Plane completion and filtering for multi-view stereo reconstruction, in: Pattern Recognition, 2019, pp. 18–32.

[0158] ACMM method: Q. Xu, W. Tao, Multi-scale geometric consistency guided multi-view stereo, in: Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit., CVPR, 2019.

[0159] ACMMP algorithm: for the above reference [3]

[0160] SD-MVS method: Yuan, Z.; Cao, J.; Li, Z.; Jiang, H.; and Wang, Z. 2024a. SD MVS: Segmentation-driven Deformation Multi-View Stereo with Spherical Refinement and EM Optimization. Proceedings of the AAAIConference on Artificial Intelligence, 38(7):6871–6880.

[0161] HPM-MVS++ method: Ren, C.; Xu, Q.; Zhang, S.; and Yang, J. 2023. Hierar chicalPrior Mining for Non-Local Multi-View Stereo. In Proceedings of the IEEE / CVFInternational Conference on Computer Vision, 3611–3620.

[0162] APD-MVS algorithm: as mentioned in the above reference [4]

[0163] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0164] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0165] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0166] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-view stereo reconstruction method based on image segmentation, the method comprising: Multiple images captured from different perspectives within the same scene are acquired. Each image is used as a reference image, and the remaining images are used as source images corresponding to that reference image. The depth map of each reference image is estimated based on both the reference and source images. The depth maps of all reference images are then fused to obtain the 3D point cloud data for the corresponding scene. The estimation methods include: S1. Perform multiple downsampling operations on the reference image and each source image to obtain multiple feature maps of different levels corresponding to each image; S2. Perform multiple iterative processing based on the reference image and multiple feature maps of each source image to obtain the depth map of the reference image. Each iteration, based on the feature maps of the reference image and each source image at each level, obtains the depth map of the reference image at each level. The depth map obtained in the last iteration is used as the depth map of the reference image. Each iteration includes: S21. Evaluate the reliability of the depth estimate of each pixel in the obtained depth map to obtain reliable pixels and unreliable pixels. The first iteration uses the initialized depth map, and each subsequent iteration uses the depth map obtained by upsampling the depth map obtained in the previous iteration. S22. Based on the feature maps of all images at their respective levels, update the depth estimate of each reliable pixel using the PatchMatch algorithm. S23. Based on the feature maps of each image at the corresponding level, a preset deformation algorithm is used to reconstruct the depth estimate of each unreliable pixel. The preset deformation algorithm includes: dividing different objects in the reference image into different regions, and for each unreliable pixel, selecting multiple anchor points from reliable pixels in the same region as the unreliable pixel for patch deformation to reconstruct the depth estimate of each unreliable pixel. S24. Obtain the depth map of the corresponding level based on the depth estimation values ​​of all pixels.

2. The method according to claim 1, characterized in that, In step S23, the method of segmenting different objects in the reference image into different regions includes: Semantic-SAM is used to perform multi-granularity panoramic segmentation on the reference image to obtain multi-granularity mask images. Boundary maps of each multi-granularity mask image are extracted. Each boundary map includes the boundaries of multiple segmented regions. The region within each boundary is used as a label of an object. For each unreliable pixel, an anchor point set including multiple anchor points is constructed using each boundary map in multiple boundary maps as a constraint, resulting in multiple anchor point sets. Each anchor point is selected from the same region within the boundary map corresponding to the unreliable pixel. The boundary maps of multiple granularity mask maps are aggregated from multiple anchor point sets to obtain an aggregated boundary map, which includes the boundaries of different regions of different objects obtained from the aggregation. The region within each boundary obtained from the aggregation is used as the final label of the object.

3. The method according to claim 2, characterized in that, Aggregate the boundary maps of multiple granularity mask images as follows: Where B represents the aggregated boundary graph, τ(·) represents the transformation function, N represents the total number of mask graphs, n represents the mask graph number (n = 1, 2, ..., N), K represents the total number of objects contained, and k represents the object label number (k = 1, 2, ..., K). This represents the k-th label of the n-th mask image. This means that the k-th label of the n-th mask image is preserved during aggregation. This means that the k-th label of the n-th mask image is not retained during aggregation. This indicates the retention status of the k-th label in the l-th mask image, where l represents the mask image number and i represents the unreliable pixel number. This represents the maximum number of anchor points for the i-th unreliable pixel. This represents the set of anchor points constructed using the boundary map of the l-th mask image for the i-th unreliable pixel. Indicates an indicator function, when hour when hour This represents the number of anchor points classified as interior points after performing a planarization operation on multiple anchor point sets of multiple boundary maps of a multi-granularity mask map. |S represents the set of anchor points constructed using the boundary map of the nth mask image for the i-th unreliable pixel. n | represents the anchor set S n The number of anchor points in the code.

4. The method according to claim 2, characterized in that, The method of segmenting different objects in the reference image into different regions further includes: using a preset edge correction strategy to correct the edges of the aggregated boundary map to obtain a corrected boundary map; The edge correction strategy is as follows: Where minΨ(L) represents edge correction by minimizing the overall energy function Ψ(L), K represents the total number of labels in the aggregated boundary map, i represents the label number in the aggregated boundary map, and ψ u (·) represents the single-point potential function, ψ u (l i ) indicates that pixel p is segmented into label l i The probability, l i Let ψ represent the i-th label in the aggregated boundary graph, and j represent the number of another label in the aggregated boundary graph. p (·) denotes the dual potential function, ψ p (l i ,l j This indicates that two adjacent pixels, p and q, are segmented into label l. i and tag l j The joint probability, l j This represents the j-th label in the aggregated boundary graph. μ(·) represents the label compatibility function, t represents the current iteration number, and D p D represents the depth estimate of pixel p obtained in the (t-1)th iteration. q Let I represent the depth estimate of pixel q obtained in the (t-1)th iteration, where α represents the weight parameter. p I represents the normalized RGB color of pixel p. q β represents the normalized RGB color of pixel q, and β represents the weight parameter.

5. The method according to any one of claims 2-4, characterized in that, The method of selecting multiple anchor points for patch deformation includes: Centered on each unreliable pixel, the region within the boundary of that pixel is divided into multiple sector regions by a preset number of pixels. The preset number of pixels is the ratio of the total number of reliable pixels within the boundary of its region to the preset number of sector regions. Multiple anchor points are selected from all reliable pixels within each sector of multiple sector regions. Anchor point clustering is performed based on the anchor points of all sector regions to obtain multiple anchor point clusters. Each anchor point cluster includes one or more anchor points. For each anchor cluster, construct a sub-patch centered on each anchor in the cluster, with no overlap between sub-patches. The sub-patches of all anchors contained in all anchor clusters are used as the result of patch deformation.

6. The method according to claim 5, characterized in that, The methods for performing anchor point clustering include: The anchor points of all sector regions are subjected to RANSAC-based planarization to obtain an optimized plane; Anchor points for clustering are selected based on the optimized plane, and the density-based DBSCAN algorithm is used to cluster the selected anchor points to obtain multiple anchor point clusters.

7. The method according to claim 5, characterized in that, The method for balanced sector division is as follows: Where, θ′ k θ′ represents the angle used to divide the k-th sector. k-1 Let represent the angle used to divide the (k-1)th sector, where k represents the angle number, x-1 represents the sector number, and i represents the sector number. y represents the number of pre-defined sector regions. b i b represents the boundary pixel of the i-th sector region. i+1 This represents the boundary pixel of the (i+1)th sector. b i and b i+1 Let L1 distance be the distance between the i-th sector and use this distance as the reliable number of pixels in the i-th sector. Let B represent the aggregated image. b i and b i+1 None of them are located on the boundary of the aggregated boundary map.

8. The method according to claim 5, characterized in that, The methods for reconstructing the depth estimate for each unreliable pixel include: Multiple searches are performed based on the iterative local search algorithm to obtain multiple sets of representative anchor points. During each search, a preset number of pixels are randomly selected from each sub-patch of all sub-patches as representative anchor points to obtain a set of representative anchor points. The minimum deformation matching cost corresponding to the group of representative anchors is calculated by assigning different disparities to each anchor in each group of representative anchors. Obtain the minimum deformation matching cost corresponding to multiple sets of representative anchor points, select the set of representative anchor points corresponding to the minimum deformation matching cost, and obtain the depth estimate of unreliable pixels based on the set of representative anchor points and the pixels corresponding to the set of representative anchor points in the feature maps of each source image at the corresponding level.

9. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 8 by executing the executable instructions.

Images