A depth map fusion algorithm considering pixel region prediction

Abstract

This invention discloses a depth map fusion algorithm that considers pixel region prediction, including: S1, image combination calculation: calculating image combinations and selecting a set of neighbor candidate images for each reference image in the scene; S2, pixel region prediction: constructing coarse-scale object voxels and metric estimates, and proposing an inference strategy and corrector for fusing pixel regions at a fine scale guided by the coarse scale; S3, depth value fusion: fusing depth values ​​using the inferred pixel regions at the fine scale as image-side constraints. The advantages are: incorporating prediction into the fusion process, performing binary classification on image pixels in the scene, and only performing depth map fusion on valid pixel regions in subsequent steps, can eliminate redundant calculations and significantly improve fusion efficiency.

Description

Technical Field

[0001] This invention relates to the field of depth map fusion technology, and in particular to a depth map fusion algorithm that takes into account pixel region prediction. Background Technology

[0002] Multi-view stereo reconstruction technology can generate detailed and realistic 3D models. Currently, this technology is widely used in large-scale mapping, 3D model texture acquisition, and rapid 3D modeling for digital cities. It generally includes key steps such as Structure of Motion (SfM), dense matching, depth map fusion, mesh construction, and texture reconstruction. Multi-view stereo reconstruction images (referred to as multi-view images) are typically acquired through oblique aerial photography, which involves simultaneously acquiring images from five different perspectives (one vertical and four oblique) to obtain rich, high-resolution textures of building rooftops and side views. Images acquired through oblique aerial photography (referred to as oblique images) have high overlap, which can be used to ensure the reconstruction quality of complex features. Generally, the overlap between consecutive images in the same flight direction is about 80%, and the overlap between images in adjacent flight directions is about 60%. However, as the reconstruction area increases and reconstruction accuracy improves, the high overlap of oblique images leads to increasingly severe redundant computation in multi-view stereo reconstruction, resulting in lower reconstruction efficiency. The redundant computation problem in multi-view stereo reconstruction mainly manifests in the depth map fusion stage. Therefore, reducing redundant computation in depth map fusion is an effective way to improve the efficiency of multi-view stereo reconstruction.

[0003] Depth map fusion refers to fusing depth maps from different perspectives into a unified coordinate framework and performing surface calculations to generate a dense point cloud for the entire scene. Currently, scholars both domestically and internationally have made significant progress in this field, with algorithms broadly categorized into three types: voxel-based fusion algorithms, feature point diffusion-based fusion algorithms, and depth map-based fusion algorithms. Key research in depth map fusion focuses on fusion quality and efficiency. Existing algorithms are relatively mature in terms of fusion quality, but suffer from significant redundant information computation issues, leading to low efficiency. Depth map-based fusion algorithms can adapt to depth maps acquired through different methods, exhibiting lower computational complexity and higher robustness. Therefore, this type of method has become the most commonly used depth map fusion algorithm.

[0004] Depth map-based fusion algorithms take multi-view depth maps and their camera's intrinsic and extrinsic parameters as input, and use geometric information between reference and neighboring images, object photometric features, and object visibility across multiple views as fusion conditions to generate 3D point clouds for reconstructing realistic terrain features. Researchers are continuously conducting studies on depth map-based fusion algorithms, focusing on fusion quality and redundancy elimination.

[0005] Tola et al., starting from feature extraction, filtered out noise points using DASIY features and then filtered and merged redundant points based on consistency checks between adjacent views. Schops, when checking for redundant calculations between multiple viewpoints, used homography matrices to determine the connections between corresponding points, then calculated the normal vectors of points in 3D space using depth maps, and compared the normal vectors to eliminate points with significant differences. Hernández statistically analyzed the visibility probability of each point in 3D space under different viewpoints using multi-view depth maps, distinguished the foreground and background of objects using graph cuts, and then fused the depth maps of different regions. Merrell P et al. used two real-time point cloud fusion methods based on inter-view visibility: a stability-based method and a confidence-based method, to achieve depth map fusion. Liu et al. employed a multi-scale iterative method (MSS) to further optimize the original depth map and then used a thresholding method to filter conflict points. For each newly calculated 3D point, M et al. calculated the covariance error ellipse of that point. If the covariance error ellipse of this point intersects with the covariance error ellipses of other points, then these points are fused. Liu et al. defined a finite number of virtual viewpoints to minimize redundancy and cover the entire scene, fusing depth maps on the virtual viewpoints to achieve 3D reconstruction, but the algorithm was only effective for tilted images from five lenses. To eliminate the large amount of redundancy caused by directly merging depth maps, Shen et al. tested and merged points whose depth values ​​were sufficiently close by using depth maps from adjacent images. Galliani S et al. further improved upon this by first using each image as a reference map for fusion, then performing consistency detection in weak texture and occluded areas, and balancing the accuracy and completeness of the 3D point cloud according to different application requirements.

[0006] Existing depth map-based fusion algorithms typically use consistency constraints such as relative depth difference, normal angle, and reprojection error as fusion conditions to ensure the quality of the generated 3D point cloud. However, in terms of eliminating redundant computation, they only focus on the density of the 3D point cloud and ignore redundant calculations. The higher the overlap between images contained in a scene, i.e., the more pixels of the same ground feature can be seen, the more severe the redundant computation becomes during depth map fusion. The specific reasons are analyzed as follows:

[0007] (1) Pixel-by-pixel fusion determination of the reference image:

[0008] During the fusion process, existing depth map-based fusion algorithms require calculating consistency constraints on pixels. Pixels that meet the constraints are called acceptable pixels, also known as valid pixels; pixels that do not meet the constraints are called unacceptable pixels, also known as invalid pixels. The convex hull formed by all valid pixels in the image is called the valid pixel region; the remaining region in the image is called the invalid pixel region, such as... Figure 1As shown in (a) and 1(b), existing depth map-based fusion algorithms do not distinguish between valid and invalid pixel areas, requiring the calculation of consistency constraint values ​​for all pixels in the image. Therefore, the fusion determination of invalid pixel areas in the image results in unnecessary redundant information calculation.

[0009] (2) All images in the scene need to be converted into reference images:

[0010] The depth maps generated during the dense matching stage (generally including depth map estimation and filtering) often exhibit high overlap, noise from weakly textured regions, and surface undulations. To avoid generating noisy 3D point clouds from directly merging depth maps, existing depth map-based fusion algorithms typically convert all images in the scene into reference images before fusion. However, some images in the scene are invalid, meaning they lack any valid pixel areas. These images do not require fusion evaluation or 3D point cloud generation. Therefore, calculating the consistency constraint values ​​for pixels in invalid images is redundant. Summary of the Invention

[0011] The purpose of this invention is to provide a depth map fusion algorithm that takes into account pixel region prediction, thereby solving the aforementioned problems existing in the prior art.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] A depth map fusion algorithm that takes into account pixel region prediction includes the following steps:

[0014] S1. Calculation of image combination:

[0015] Using the number of shared feature points reconstructed by SfM as the main metric, and combining the angle weight function and the scale weight function, a global view selection scoring function is jointly constructed to select a set of candidate neighbor images for each reference image in the scene, and the sum of image scores is calculated based on the global score of each image in the candidate neighbor image set.

[0016] S2, Pixel region prediction:

[0017] Construct object voxels at a coarse scale and select an aggregated image subset and associated information for each object voxel; guided by the coarse scale, infer the effective pixel region at the fine scale, and correct the erroneous estimates from the object voxels at the coarse scale based on the fine scale corrector.

[0018] S3, Depth Value Fusion:

[0019] Using the pixel regions inferred at a fine scale as image-side constraints, valid pixels are determined and filtered out. The 3D points corresponding to the depth values ​​of the valid pixels and the 3D points estimated by the normal vectors are averaged into a unified 3D point.

[0020] Preferably, step S1 specifically includes the following:

[0021] S11. Constructing a global scoring function: Using the number of shared feature points in the reference image R in the scene as a weight, construct a global scoring function g. R To calculate the global score for each image V in the candidate neighbor image set N;

[0022]

[0023] Among them, F V and F R These are the feature point sets of image V and reference image R, respectively; w N (f) and w s (f) represent the angle weighting function and the scale weighting function, respectively; the candidate neighbor set N includes the reference image R;

[0024] The angle weight function w N (f) is defined as the product of all matching pairs in N.

[0025]

[0026] Among them, w α (f,V i V j )=min((α / a max ) 2 ,1);V i V j These are the i-th and j-th images, respectively; α represents the value from V. i and V j The angle between the lines of sight to feature point f; α max This represents the maximum value of the included angle α;

[0027] Calculate s of reference image R R (f), and based on the ratio r = s R (f) / s V (f) Define the scale weight w s (f),

[0028]

[0029] Among them, s R (f) represents the diameter of the sphere centered at feature point f in the reference image R; s V (f) is the diameter of the sphere centered at feature point f in image V;

[0030] S12. Calculate the sum of image scores: Iteratively add the image V with the highest score to the set of best neighbor images one by one using a greedy method, until the size of the set of best neighbor images equals the given size |N|, and obtain the sum of image scores ∑. V∈N′ g R (v).

[0031] Preferably, step S2 specifically includes the following:

[0032] S21. Coarse-scale object voxel measurement: Using the camera intrinsic parameters, image extrinsic parameters, connection points, depth maps, and image combinations in the recovered scene, the depth maps corresponding to some images in the scene are back-projected, calculated, and sampled to generate a semi-dense point cloud in the object space to construct object voxels. The object voxel measurement is estimated using image-side visibility measure and object-side intersection measure; the connection points are the shared feature points reconstructed by SfM.

[0033] S22. Fine Dimension Corrector: Constructs a fine dimension correction formula based on image-side visibility measure and object-side intersection measure, and obtains the effective voxels of the object-side voxels relative to the view based on the fine dimension correction formula. Merges the pixel ranges of the effective voxels to obtain the effective pixel region of the image.

[0034] Preferably, step S21 specifically includes the following:

[0035] S211. Select the set of images that cover all ground features and have low overlap between images based on the number of connection points, i.e., the voxel image subset IRV; and use the completeness constraint η to make IRV remove images with high overlap as much as possible while ensuring coverage of all ground features in the scene.

[0036]

[0037] Where, ∑ j ImageTiePoint is the total number of connection points for all images contained in the IRV; |SceneTiePoint| is the total number of connection points for all images contained in the IR; IR is the set of all images contained in the scene.

[0038] S212. Perform downsampling on the depth map, calculate the three-dimensional points using back projection using formula (5), and store them in the semi-dense point cloud SemiPC.

[0039]

[0040] Where X is a 3D point in the world coordinate system of the image; λ is the depth value; It is the transpose of the rotation matrix of camera u; C is the inverse of the intrinsic parameter matrix of camera u; u Let u be the translation component of the camera; p is the pixel coordinate of the image;

[0041] S213. Use an octree structure to partition SemiPC into three-dimensional space. The leaf nodes in the octree represent SceneVoxels.

[0042] S214. Use two quantitative metrics, image-side visibility measure and object-side intersection measure, to estimate voxel measure.

[0043] Preferably, step S214 specifically includes the following:

[0044] S2141. SceneVoxels is triangulated to form a scene voxel mesh (SceneMesh). The mapping relationship between the scene voxel mesh and the triangular facets is represented as follows:

[0045] SceneMesh={TrkFace vt |v=1,...,n,t=1,...,12} (6)

[0046] Where k is the index number of the voxel, and the total number of voxels is n; t is the index number of the triangular facet, and a voxel will be divided into 12 triangular facets.

[0047] S2142. Construct the projection matrix of view l based on the camera intrinsic parameters, image extrinsic parameters and image resolution in the scene, as shown in formula (7), and project the triangular facet of voxel v onto the image space of view l.

[0048]

[0049] Where M1 is the intrinsic parameter matrix and M2 is the extrinsic parameter matrix; X and Y are the x-axis and y-axis coordinates in the view image coordinate system, respectively; Z c f' is the z-axis coordinate of the camera coordinate system; f′ is the camera focal length; μ0 and v0 are the center pixel coordinates, respectively; R c It is a 3×3 rotation matrix, representing the rotation of the camera coordinate system relative to the world coordinate system; T c It is a 3×1 translation vector, representing the offset of the camera coordinate system relative to the world coordinate system; O 1×3 X is a zero vector of size 1×3; w Y w Z w These are the components along the x-axis, y-axis, and z-axis in the world coordinate system, respectively.

[0050] The rasterization of the projected triangular facet and the acquisition of the pixel set within the facet area constitute the image-side visibility measure, which is used to measure the visibility of the object-side voxel v in the view l.

[0051] VP(v,l)=(P1(x1,y2,z1),...,P n (x n y n , z n )} (8)

[0052] Where VP(v, l) is the image-side visibility measure; x n and y n Z represents pixel coordinates; n VP(v, l) represents the depth value of the ray direction after projection; the larger VP(v, l) is, the larger the visible range of voxel v projected into view l, indicating that view l makes a positive contribution to the fusion of dense point cloud within voxel v, and vice versa.

[0053] Collect views with VP(v, l) values ​​greater than 0 to initialize the aggregated image subset IA(v) of object voxel v;

[0054] S2143. Construct a measure of the intersection of three-dimensional point sets calculated by back-projection of VP(v,l) from voxels using the number and density of such points.

[0055]

[0056] Where VF(v, l) is the object-space intersection measure; VPNum is the number of 3D point sets in each voxel; VPNum / max(VPNum) represents the normalized value of the number; V is the volume of voxel v; α and β are weighting factors.

[0057] Remove views in the aggregated image subset IA(v) where VF(v,l) is close to 0.

[0058] Preferably, step S22 specifically includes the following:

[0059] S221. Using the image-side visibility measure VP(v,l) and the image-side intersection measure VF(v,l), a fine-scale correction formula is constructed, namely formula (10), to screen out qualified object-side voxels.

[0060]

[0061] Where VCorrect(v, l) represents the degree of difference between the two values; the smaller VCorrect(v, l) is, the closer VP(v, l) and VF(v, l) are, indicating that the fusion contribution of view l to voxel v is positive.

[0062] S222. Calculate the score value of each view in the aggregated image subset IA(v) of the object voxel v according to the fine-scale correction formula, and sort them in ascending order of score value. Take the top m views as the views that have a fusion contribution to voxel v at the fine scale, i.e., the effective views FusionViews(v) = {l1, l2, ..., l...} m Each view contains the pixel range corresponding to that voxel, and that voxel is called the effective voxel relative to that view;

[0063] S223. Merge the pixel ranges of all valid voxels in each view to form the valid pixel region of the image.

[0064] Preferably, based on the prediction results of the pixel regions, the images in the scene are divided into two categories: one category contains valid pixels for fusion, which are invalid images; the other category contains valid pixels for fusion, which are valid images. Step S3 specifically includes the following:

[0065] S31, Reference Image Transformation: Only convert the valid image to the reference image, and only convert the depth values ​​corresponding to the pixels in the valid pixel area of ​​the reference image to 3D points in the world coordinate system, and project them onto the neighbor view to obtain the corresponding match;

[0066] S32. Pixel fusion judgment: Set depth fusion conditions, calculate the consistency constraint value of pixels in the effective pixel area, and determine whether it meets the fusion conditions; if there are at least two adjacent views with corresponding matches that meet the above fusion conditions, the estimated depth value will be accepted.

[0067] S33, 3D point fusion: The 3D points corresponding to the accepted depth values ​​and the 3D points estimated by the normal vectors are averaged into a unified 3D point.

[0068] Preferably, the fusion conditions in step S32 include a consistent matching of relative depth differences less than or equal to a matching threshold, an angle between normals less than or equal to an angle threshold, and a reprojection error less than or equal to an error threshold.

[0069] The beneficial effects of this invention are as follows: 1. This invention incorporates prediction into the fusion process, performing binary classification on image pixels in the scene. Subsequent steps only perform depth map fusion on effective pixel regions, eliminating redundant calculations and significantly improving fusion efficiency. 2. This invention introduces a multi-scale concept for the first time in depth map fusion, proposing an inference strategy for effective pixel regions at a fine scale guided by coarse-scale object voxels. This avoids the problem of having to calculate and determine whether the depth value corresponding to each pixel in each reference image meets the fusion conditions, eliminating a large amount of redundant calculations. 3. This invention proposes a fine-scale corrector to correct erroneous estimates from object voxels at the coarse scale. This effectively removes image-side ambiguous projection ranges caused by object voxel occlusion errors or noise in the estimated depth map. 4. This invention proposes for the first time the concepts of image-side visibility measure and object-side intersection measure, comprehensively evaluating the effectiveness of voxels for fusion from both object-side and image-side perspectives. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of pixel partitioning of an image; (a) Valid image: white pixels are acceptable pixels, black pixels are unacceptable pixels; the pixel area inside the box is the valid pixel area, and the pixel area outside the box is the invalid pixel area; (b) Invalid image: all pixels in the image are unacceptable pixels;

[0071] Figure 2 This is a schematic diagram illustrating the principle of the depth map fusion algorithm in this embodiment of the invention;

[0072] Figure 3 This is a schematic diagram of the construction of object voxels with rough dimensions in an embodiment of the present invention; (a) a semi-dense point cloud SemiPC of the scene, which contains a certain number of noise or outliers; (b) the object voxel result obtained by octree spatial partitioning of the SemiPC; (c) the triangulation result of the object voxels. Wherein, v1, v2, and v3 represent three different object voxels; the view frustum l1...ln is the view in the scene (i.e., a combination of image, camera intrinsic parameters, and image extrinsic parameters);

[0073] Figure 4 This is a schematic diagram of the object-space intersection measurement in an embodiment of the present invention; (a) is a cross-sectional view of v1, v2, v3, and l1 on the yoz coordinate plane in the object-space; (b) v'1, v'2, and v'3 are the projections of v1, v2, and v3 onto the image side of l1, respectively; (c) the figure shows the distribution and number of object-space voxel meshes v1, v2, and v3 with the depth values ​​of l1 belonging to the spatial intersection relationship.

[0074] Figure 5This is a schematic diagram of image prediction and fusion in an embodiment of the present invention; (a) the area inside the rectangle is the effective pixel area of ​​the image, and the area outside the rectangle is the invalid pixel area; (b) the dark pixels inside the rectangle are effective pixels, which are actually used to estimate and generate three-dimensional points; the light pixels are invalid pixels and cannot be used to generate three-dimensional points.

[0075] Figure 6 These are the point cloud results generated by Courtyard data fusion in this embodiment of the invention; (a) is the ground laser point cloud provided by ETH3D; (b) is the point cloud generated by the COLMAP method; (c) is the point cloud generated by the OPENMVS method; (d) is the point cloud generated by the Gipuma method; (e) is the point cloud generated by the ACMP method; (f) is the point cloud generated by ACMP-PF, and the point cloud accuracy gradually decreases from other areas to the area within the box.

[0076] Figure 7 This is a schematic diagram of the prediction accuracy analysis of the effective pixel region of the Dortmund experimental data in an embodiment of the present invention; the upper left figure shows the three-dimensional reconstruction model; the four figures in the upper right figure are the effective pixel regions of different images, the rectangles are the effective pixel regions predicted by the algorithm of the present invention, the white-marked pixels are effective pixels, and the black-marked pixels are invalid pixels; the bottom is a scatter plot of prediction accuracy;

[0077] Figure 8 This is a schematic diagram of the prediction accuracy analysis of the effective pixel region of the Garden experimental data in an embodiment of the present invention; the upper left figure shows the three-dimensional reconstruction model; the four figures in the upper right figure are the effective pixel regions of different images, the rectangles are the effective pixel regions predicted by the algorithm of the present invention, the white-marked pixels are effective pixels, and the black-marked pixels are invalid pixels; the bottom is a scatter plot of prediction accuracy.

[0078] Figure 9 This is a schematic diagram of the prediction accuracy analysis of the effective pixel region of the Central-Urban experimental data in an embodiment of the present invention; the upper left figure shows the three-dimensional reconstruction model; the four figures in the upper right figure are the effective pixel regions of different images, the rectangles are the effective pixel regions predicted by the algorithm of the present invention, the white-marked pixels are effective pixels, and the black-marked pixels are invalid pixels; the bottom is a scatter plot of prediction accuracy.

[0079] Figure 10 This is a histogram showing the time consumption statistics of the method of the present invention and the comparative method in the embodiments of the present invention; each group of five bars represents COLMAP, OPENMVS, Gipuma, ACMP, and Ours (the method of the present invention) from left to right. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0081] Example 1

[0082] This embodiment provides a depth map fusion algorithm that takes into account pixel region prediction, comprising three parts: image combination calculation, pixel region prediction, and depth value fusion. These three parts will be described in detail below.

[0083] For ease of description of the algorithm of this invention, the set of all images contained in the scene is referred to as IR = {image0...image0}. i}; The set of views consisting of camera interior orientation elements (intrinsic parameters), image exterior orientation elements (extrinsic parameters), and the image is called LR = {l0...l...} i}

[0084] I. Calculation of Image Combinations

[0085] The technical approach for this part is as follows: using the number of shared feature points reconstructed by SfM as the main metric, and combining the angle weight function and the scale weight function, a global view selection scoring function is jointly constructed to select a set of candidate neighbor images for each reference image in the scene, and the sum of image scores is calculated based on the global score of each image in the candidate neighbor image set.

[0086] Image combination computation involves selecting a set of neighboring candidate images for each reference image, which can be used in subsequent steps such as pixel region prediction and depth value fusion. This part specifically includes the following:

[0087] 1. Construct the global scoring function:

[0088] The global scoring function g is constructed by weighting the number of shared feature points in the reference image R in the scene. R To calculate the global score for each image V in the candidate neighbor image set N (including the reference image R);

[0089]

[0090] Among them, F V and F R Let w be the feature point set of image V and reference image R, respectively. N (f) and w s (f) represent the angle weighting function and the scale weighting function, respectively; the candidate neighbor set N includes the reference image R;

[0091] To avoid short baselines in matching pairs composed of neighboring and reference images and to ensure a good disparity range for the matching pairs, the angle weighting function w is... N (f) is defined as the product of all matching pairs in N.

[0092]

[0093] Among them, w α (f,V i V j )=min((α / α max ) 2 ,1);V i V j These are the i-th and j-th images, respectively; α represents the value from V. i and V j The angle α between the lines of sight to feature point f and the line of sight to feature point f max The maximum value of the included angle α is used; the quadratic weighting function is used to counteract the trend of more shared feature points appearing as the angle decreases, and at the same time, it can also avoid selecting matching pairs with excessively large angles.

[0094] Weighting function w s (f) is used to measure the resolution similarity between images R and V at feature point f. To estimate the 3D sampling rate of V near feature point f, we calculate the diameter s of a sphere centered at f. V (f) The projected diameter of the sphere is equal to the pixel spacing of V. Similarly, calculate s of the reference image R. R (f), and based on the ratio r = s R (f) / s V (f) Define the scale weight w s (f),

[0095]

[0096] Among them, s R (f) is the diameter of the sphere centered at feature point f in the reference image R, s V (f) is the diameter of the sphere centered at feature point f in image V. This weighting function tends to select neighboring images with a resolution equal to or higher than the reference image.

[0097] 2. Calculate the sum of image scores.

[0098] After defining the global score function, we can obtain the best set of neighboring images of a given size (usually |N| = 10), which is the sum of the image scores ∑ V∈N g R(v). To improve the efficiency of score calculation, a greedy method is used to iteratively add the image V with the current highest score to the set of best neighbor images one by one, until the size of the set of best neighbor images is equal to a given size |N|, and then the sum of the image scores ∑ is obtained. V∈N′ g R (v).

[0099] II. Pixel Region Prediction

[0100] The technical approach for this part is as follows: construct object voxels at a coarse scale, and select an aggregated image subset and associated information for each object voxel; guided by the coarse scale, infer the effective pixel region at the fine scale, and correct the erroneous estimates from the object voxels at the coarse scale based on the fine scale corrector.

[0101] Pixel region prediction mainly refers to accurately inferring effective pixel regions at the fine scale using strategies such as multi-scale propagation of reliable metric estimation and object-to-image transformation. This effectively solves the redundant computation problem caused by calculating consistency constraints per image and per pixel in existing methods. This part constructs object voxels at the coarse scale and selects aggregated image subsets and calculates metric estimates for them. Then, guided by the metric information at the coarse scale, it infers effective pixel regions that can be used for fusion at the fine scale (single image) and proposes a fine-scale corrector to correct erroneous estimates of object voxels from the coarse scale.

[0102] This section specifically includes the following content.

[0103] 1. Coarse-scale object voxel measurement: Using the camera intrinsic parameters, image extrinsic parameters, connection points, depth maps, and image combinations in the recovered scene, the depth maps corresponding to some images in the scene are back-projected, calculated, and sampled to generate a semi-dense point cloud in the object space to construct object voxels. The object voxel measurement is estimated using image-side visibility measure and object-side intersection measure; the connection points are the shared feature points reconstructed by SfM.

[0104] Specifically, it includes the following sub-sections:

[0105] 1.1 Constructing the Voxel Image Subset IRV: The number of connection points between images in a scene is a measure of their correlation. The more connection points, the stronger the correlation (higher overlap) between the reference image and its neighboring images. Therefore, the set of images that covers all ground features and has low overlap is selected based on the number of connection points, i.e., the voxel image subset IRV; and the completeness constraint η is used to ensure that the IRV covers all ground features in the scene while removing images with high overlap as much as possible.

[0106]

[0107] Where, ∑j ImageTiePoint is the total number of connection points for all images contained in IRV; |SceneTiePoint| is the total number of connection points for all images contained in IR. Experimentally, it is generally set to 0.85; IR is the set of all images contained in the scene.

[0108] 1.2 Calculation of Semi-Dense Point Cloud: The semi-dense point cloud is a set of three-dimensional points calculated by back-projecting the depth map corresponding to the image in the IRV, hereinafter referred to as SemiPC. Since SemiPC is mainly used to construct object voxels, and the scale of object voxels is generally tens of times or even hundreds of times the original resolution of the image, the depth map is downsampled, and the three-dimensional points are calculated by back-projection using formula (5), and stored in the semi-dense point cloud SemiPC, such as... Figure 3 As shown in (a);

[0109]

[0110] Where X is a 3D point in the world coordinate system of the image; λ is the depth value; It is the transpose of the rotation matrix of camera u. C is the inverse of the intrinsic parameter matrix of camera u. u Let u be the translation component of the camera; p is the pixel coordinate of the image.

[0111] 1.3 Creating Object Voxels: First, determine the size of the object voxels. Since the size of the object voxels affects the error estimation removal and efficiency in the subsequent fine-scale corrector, their size should not be too large or too small; generally, it can be set to a factor of ten to several tens of times the original image resolution. Use an octree structure to partition the SemiPC into three-dimensional spaces; the leaf nodes in the octree represent the object voxels, such as... Figure 3 (b) Using SceneVoxels.

[0112] 1.4 Voxel Measure Estimation and its Aggregated Image Subset: In order to measure the fusion availability of voxels, i.e. the prediction accuracy of the effective pixel region, this invention proposes voxel measure estimation, which includes two quantitative metrics: image-side visibility measure and object-side intersection measure.

[0113] When calculating the two voxels, SceneVoxels are projected onto the scene's view set LR. To obtain accurate projection results, SceneVoxels need to be triangulated into a material-square voxel mesh, SceneMesh, such as... Figure 3 As shown in (c), the mapping relationship between object voxel meshes and triangular patches in the scene can be represented as:

[0114] SceneMesh={TrkFacevt |v=1,…,n,t=1,…,12} (6)

[0115] Where k represents the index number of the voxel, and the total number of voxels is n; t represents the index number of the triangular facet, and a voxel will be divided into 12 triangular facets.

[0116] (1) Image Visibility Measure: Construct the projection matrix of view l based on the camera intrinsic parameters, image extrinsic parameters, and image resolution in the scene, as shown in formula (7). Project the triangular facets of voxel v onto the image space of view l, as shown in formula (7). Figure 4 As shown in (b).

[0117]

[0118] Where M1 is the intrinsic parameter matrix and M2 is the extrinsic parameter matrix; X and Y are the x-axis and y-axis coordinates in the view image coordinate system, respectively; Z c f' is the z-axis coordinate of the camera coordinate system; f' is the camera focal length; μ0 and v0 are the center pixel coordinates, respectively; R c It is a 3×3 rotation matrix, representing the rotation of the camera coordinate system relative to the world coordinate system; T c It is a 3×1 translation vector, representing the offset of the camera coordinate system relative to the world coordinate system; O 1×3 X is a zero vector of size 1×3; w Y w Z w These are the components along the x-axis, y-axis, and z-axis in the world coordinate system, respectively.

[0119] The rasterization of the projected triangular facet and the acquisition of the pixel set within the facet area constitute the image-side visibility measure, which is used to measure the visibility of the object-side voxel v in the view l.

[0120] VP(v,l)=(P1(x1,y2,Z1),...,P n (x n y n Z n )} (8)

[0121] Where VP(v, l) is the image-side visibility measure, x n and y n Z represents pixel coordinates. n VP(v, l) represents the depth value of the ray direction after projection; the larger VP(v, l) is, the larger the visible range of voxel v projected into view l, indicating that view l makes a positive contribution to the fusion of dense point cloud within voxel v, and vice versa.

[0122] Collect views with VP(v, l) values ​​greater than 0 to initialize the aggregated image subset IA(v) of object voxel v;

[0123] (2) Object-side intersection measure: Using only image-side visibility measure as a coarse-scale voxel measure estimation cannot accurately predict the effective pixel region. For example... Figure 4 As shown in (a), the image-side visibility measures VP(v1, l1) and VP(v3, l3) of voxels v1 and v3 are relatively large, but approximately half of the 3D space in v1 does not intersect with view l1, meaning this part of the space does not contain any depth values ​​from l1 and is therefore invalid space; all of v3 is invalid space. Therefore, it is necessary to propose the concept of an object-side intersection measure, constructing the object-side intersection measure using the number and density of the 3D point set contained in the voxels, calculated via backprojection from VP(v, l).

[0124]

[0125] Where VF(v, l) represents the object-space intersection measure; VPNum is the number of 3D points in each voxel; VPNum / max(VPNum) represents the normalized value of the number; V is the volume of voxel v; α and β are weighting factors; the formula is divided into a data term and a density term, the data term is α*(VPNum / max(VPNum)), and the density term is... The number of data items is proportional to the number of 3D point sets; the density item reflects the spatial distribution of the 3D point sets in voxel v. α and β are weighting factors. Generally, α is larger, and this invention selects α = 0.65; β is smaller, and this invention selects β = 0.35.

[0126] The smaller VF(v, l), the fewer the number of 3D points contained in voxel v, or the less ideal the spatial distribution (density), the greater the possibility of noise or outliers, and the more negative the contribution of this value to fusion. For example... Figure 4 As shown in (c), in view l1, the number of effective visible depths of object v1 is relatively small, indicating that v1 is more likely to be an invalid voxel. When VF(v, l) is close to 0, it indicates that voxel v contains 3D points of view l or the intersection between the two is extremely small. Figure 4 As shown in (c), in view l1, the effective visible depth of object v3 is 0, so v3 is an invalid v3.

[0127] Remove views in the aggregated image subset IA(v) where VF(v,l) is close to 0.

[0128] 2. Fine Dimension Corrector: A fine dimension correction formula is constructed based on the image-side visibility measure and the object-side intersection measure. Based on the fine dimension correction formula, the effective voxels of the object-side voxels relative to the view are obtained. The pixel ranges of the effective voxels are merged to obtain the effective pixel region of the image.

[0129] Coarse-scale object voxel metric estimation helps infer the effective pixel region in an image at a fine scale, but it only approximates the visibility relationship between object voxels and the view, as well as their intersection with the depth map. These relationships still contain a degree of erroneous estimation. These erroneous metric estimates mainly fall into two categories: occlusion between object voxels and invalid object voxels caused by noise or outliers in the semi-dense point cloud. To accurately predict the effective pixel region at a fine scale, a fine-grained corrector is required.

[0130] Specifically, it includes the following sub-sections:

[0131] 2.1. Construct a fine-scale correction formula, i.e. formula (10), using the image-side visibility measure VP(v,l) and the image-side intersection measure VF(v,l) to screen out qualified object-side voxels.

[0132]

[0133] Here, VCorrect(v, l) represents the degree of difference between the two values; the smaller VCorrect(v, l), the closer VP(v, l) and VF(v, l) are, indicating that view l makes a positive contribution to the fusion of voxel v; therefore, voxel v can be used for effective pixel region prediction of view l. Experiments have verified that when the value of VCorrect(v, l) is greater than 0.7, view l is considered to have no fusion contribution to voxel v.

[0134] 2.2. Calculate the score value of each view in the aggregated image subset IA(v) of object v using the fine-scale correction formula, and sort them in ascending order of score value. Take the top m views as the views that have a fusion contribution to voxel v at the fine scale, i.e., the effective views FusionViews(v) = {l1, l2, ..., l...} m Each view contains the pixel range corresponding to that voxel, and that voxel is called the effective voxel relative to that view;

[0135] 2.3. Merge the pixel ranges of all valid voxels in each view to form the valid pixel region of the image. The result is as follows: Figure 5 As shown in (a).

[0136] III. Deep Value Fusion

[0137] The technical approach for this part is as follows: using the pixel regions that have been inferred at a fine scale as image-side constraints, valid pixels are determined and screened, and the 3D points corresponding to the depth values ​​of the valid pixels and the 3D points estimated by the normal vectors are averaged into a unified 3D point.

[0138] Based on the pixel region prediction results, the images in the scene are divided into two categories: one category contains valid pixels for fusion, which are considered invalid images; the other category contains valid pixels for fusion, which are considered valid images. After pixel region prediction, the valid images are fused into a complete point cloud.

[0139] This section specifically includes the following content.

[0140] 1. Reference image conversion:

[0141] Only convert valid images to reference images, and only convert pixels within the valid pixel region of the reference image (i.e., ...) Figure 5 (a) The depth value corresponding to the pixels within the rectangle is converted into a 3D point in the world coordinate system and projected onto the neighbor view to obtain the corresponding match;

[0142] 2. Pixel fusion judgment:

[0143] Set depth fusion conditions, calculate the consistency constraint value of pixels within the effective pixel area, and determine whether it meets the fusion conditions; if there are at least two adjacent views whose corresponding matches meet the above fusion conditions, the estimated depth value will be accepted.

[0144] The fusion conditions include a consistent matching of the relative depth difference less than or equal to a matching threshold, an angle between normals less than or equal to an angle threshold, and a reprojection error less than or equal to an error threshold. Specifically, the consistent matching of the relative depth difference ∈ ≤ 0.01, the angle between normals θ ≤ 30°, and the reprojection error ψ ≤ 2.

[0145] 3. Three-dimensional point fusion:

[0146] The three-dimensional points corresponding to the accepted depth values ​​and the three-dimensional points estimated by the normal vectors are averaged into a unified three-dimensional point.

[0147] Compared to the ACMP method, the method of this invention only performs the above calculations and judgments on pixels within the predicted pixel region, eliminating a large amount of redundant information calculations, such as... Figure 5 As shown in (b).

[0148] Example 2

[0149] In this embodiment, the effectiveness of the algorithm of the present invention is verified from aspects such as the quality of fused point clouds, elimination of redundant calculations, and fusion efficiency.

[0150] 1. Achieve data and environment integration

[0151] The experimental data include: (1) the publicly available benchmark dataset ETH3D, which consists of three different scenes: Courtyard, facade, and terrace, with 38, 76, and 23 images respectively. All three datasets are high-resolution multi-view stereo datasets with a resolution of 6048×4032. In addition, this type of dataset includes image datasets and real ground laser point clouds. (2) the publicly available dataset Dortmund provided by ISPRS, which was captured by a five-lens oblique imaging system (IGI PentaCam), with 584 images and a resolution of 8176×6132. (3) the field-collected oblique photography dataset, which consists of two different scenes: Garden and Central-Urban, with 162 and 566 images respectively, and resolutions of 5472×3648 and 4864×3648 respectively. The main land cover types covered include buildings, vegetation, roads, and water surfaces, which have general significance for experimental verification.

[0152] The experimental environment consisted of a workstation with a Windows 10 64-bit operating system, an Intel Core™ i9-10900X CPU (3.70GHz), and 128GB of memory.

[0153] 2. Point cloud quality analysis

[0154] Based on generally accepted benchmarks, this embodiment will use the ETH3D public benchmark dataset to quantitatively analyze point cloud quality from three aspects: accuracy, completeness, and F1 score. To fully verify the point cloud quality fused by the algorithm of this invention, it will be compared with COLMAP (… This paper compares and analyzes several classic methods, including JL, Zheng E, Frahm JM, et al. (Pixelwise view selection for unstructured multi-view stereo), Shen (Shen S. Accurate multiple view 3d reconstruction using patch-based stereo for large-scale scenes), Gipuma (Galliani S, Lasinger K, Schindler K. Massively parallel multiview stereopsis by surface normal diffusion), and ACMP (Xu Q, Tao W. Multi-scale geometric consistency-guided multi-view stereo). The Shen method has been integrated into the open-source library OpenMVS.

[0155] Table 1 lists the accuracy, completeness, and F1 score of the fused point clouds from COLMAP, OPEMNVS, Gipuma, and ACMP. All methods show similar results in terms of accuracy. The algorithm of this invention is closest to the ACMP method in terms of accuracy, completeness, and F1 score. This is mainly because the depth map-based fusion algorithm proposed in this invention is combined with the high-performance ACMP dense matching method to form a new 3D reconstruction pipeline (hereinafter referred to as ACMP-PF). Therefore, the depth map used in the fusion process of this invention is provided by the ACMP method. Since the algorithm of this invention only performs fusion calculations on pixels within the predicted effective pixel region, if the predicted effective pixel region cannot completely contain all effective pixels in the image, the accuracy and completeness of the fused point cloud will decrease, and the F1 score will also decrease accordingly. Therefore, the quality of the point cloud generated by the algorithm of this invention may be slightly worse than that of the ACMP method.

[0156] Table 1. Point cloud quality assessment of the ETH3D benchmark dataset.

[0157] The accuracy, integrity, and F1 score (%) are shown at different thresholds (including 2cm and 10cm).

[0158]

[0159] Figure 6Qualitative results of point clouds generated by COLMAP, OPENMVS, Gipuma, ACMP, and the fusion algorithm of this invention are presented. It can be observed that the point cloud generated by ACMP-PF is almost entirely light-colored, indicating that the point cloud generated by this method has high accuracy and low noise; its performance in terms of completeness is slightly worse but close to that of the ACMP method.

[0160] 3. Prediction accuracy analysis

[0161] Accuracy is the proportion of valid pixels within the valid pixel region to the total number of valid pixels in the image; it characterizes the accuracy of the valid pixel region predicted by the algorithm of this invention. The calculation formula is as follows:

[0162] Accuracy(i)=B w / I w (11)

[0163] Where i is the image number in the scene, B w It is the number of effective pixels within the effective pixel area, I w It is the total number of valid pixels in the image.

[0164] Based on the performance of existing mainstream dense matching methods, we integrate our algorithm into the more advanced ACMP reconstruction pipeline, replacing its depth map-based fusion algorithm. First, the original fusion algorithm is executed, and the image pixels capable of generating point clouds are labeled and their counts are determined (I...). w Then, the improved fusion algorithm is executed to mark and count the number of image pixels within the effective pixel area that can generate point clouds (B). w Finally, calculate the Accuracy(i) value for each image.

[0165] Due to computer memory limitations, large datasets cannot be processed all at once. Therefore, this section selects DortmundSub, GardenSub, and Central-UrbanSub from the Dortmund, Garden, and Central-Urban datasets as experimental data, with 135, 296, and 162 images respectively. Figures 7-10 The reconstruction models, predicted effective pixel regions, and scatter plots of prediction accuracy for three sets of scenes are shown.

[0166] from Figures 7-9 It can be seen that the accuracy of the three experimental data sets—DortmundSub, GardenSub, and Central-UrbanSub—is relatively high. Specifically, the percentages of accuracy greater than 0.9 reached 81.1%, 70.0%, and 100%, respectively, and the percentages of accuracy greater than 0.8 reached 87.1%, 80.1%, and 100%, respectively.

[0167] When the effective pixels of an image are distributed at the image edges and are few in number, the prediction accuracy of the algorithm of this invention is low, such as... Figure 7 (d) and Figure 8 As shown in (d). In addition, the prediction accuracy of the algorithm of this invention is above 0.9, as... Figure 7 (a)(b)(c) Figure 8 (a)(b)(c) and Figure 9 As shown in (a), (b), (c), and (d), GardenSub contains many images with fewer effective pixels located at the image edges, resulting in the lowest prediction accuracy for this dataset. In contrast, Central-UrbanSub contains effective pixels filling the entire image, leading to prediction accuracies greater than 0.9 for all images.

[0168] Although the algorithm of this invention has low prediction accuracy for effective pixel regions located at the image edge with a small number of pixels, it does not affect the quality of point cloud fusion. The main reasons include the following two aspects: (1) These effective pixel regions are basically located at the edges or corners of the image, and the depth values ​​generated by dense matching of pixels in the region are generally of low quality. Therefore, even if these pixels can meet the fusion conditions, the quality of the fused point cloud may not be high. (2) Since there is overlap between images, there will also be overlap between the effective pixel regions of the images. Therefore, the three-dimensional points generated by the effective pixels in these effective pixel regions may be generated by the effective pixel regions in other images after depth value fusion. Therefore, even if the algorithm of this invention does not fuse and generate this part of the point cloud, it will not have a significant impact on the overall quality and quantity of the point cloud. In addition, in order to verify the degree of influence of the prediction accuracy of these effective pixel regions located at the image edge with a small number of pixels on the quality of the point cloud, we will perform a quality analysis on the point cloud generated by the algorithm of this invention. Since there is no laser point cloud in the three sets of experimental data, the point cloud generated by the ACMP method is used as the ground truth to evaluate the quality of the point cloud generated by the algorithm of this invention. As shown in Table 2, it can be seen that the point cloud generated by the algorithm of this invention is almost completely consistent with the point cloud generated by the ACMP method.

[0169] Table 2 Point Cloud Quality Assessment

[0170] The accuracy, integrity, and F1 score (%) are shown at different thresholds (including 2cm and 10cm).

[0171]

[0172] 4. Fusion efficiency analysis

[0173] Nine image subsets of different sizes were selected from the Dortmund, Garden, and Central-Urban datasets as experimental data. The fusion times of COLMAP, OpenMVS, Gipuma, ACMP, and the algorithm of this invention were statistically analyzed, as shown in Table 3. A bar chart of the time statistics is shown below. Figure 10 As shown.

[0174] from Figure 10 As can be seen, the time required for point cloud fusion by various methods increases with the number of images. The fusion times of COLMAP, OPENMVS, Gipuma, and ACMP are roughly the same, while the algorithm of this invention requires significantly less time. The fusion algorithms of the above four methods follow the idea of ​​image-by-image, pixel-by-pixel fusion calculation and judgment, resulting in a large amount of redundant information computation. In contrast, the algorithm of this invention performs effective pixel region prediction, dividing the image into invalid and valid images, and further dividing the valid images into valid and invalid pixel regions, reducing unnecessary computation. Therefore, the more images there are, the more obvious the advantage of the algorithm proposed in this invention becomes. When the number of images is 38, the algorithm of this invention improves the time by 32.2% compared to the ACMP method; when the number of images is 481, the algorithm of this invention improves the time by 59.4% compared to the ACMP method.

[0175] Table 3 shows the number of images included in the experimental data and the time (in seconds) for each method.

[0176]

[0177] 5. Conclusion

[0178] This invention proposes a depth map fusion algorithm that takes into account pixel region prediction, which is used to eliminate redundant information calculation in the fusion process and improve fusion efficiency. First, a coarse-scale object voxel and metric estimation are constructed, and the concept of multi-scale is introduced for the first time. An inference strategy and corrector for effective pixel regions at the fine scale guided by coarse-scale object voxels are proposed. Then, the effective pixel regions are used as image-side constraints for depth value fusion. Experimental verification and analysis are conducted using public datasets and field-collected oblique photogrammetry datasets. The conclusions are as follows: (1) In terms of point cloud quality: Compared with mainstream methods such as COLMAP, OPEMNVS, Gipuma and ACMP, the new reconstruction process (ACMP-PF) that combines the fusion algorithm of this invention with the ACMP dense matching method generates point clouds that are closest to the ACMP method in terms of both accuracy and F1 score. (2) In terms of prediction accuracy: Using DortmundSub, GardenSub and Central-UrbanSub as three sets of experimental data, the algorithm of this invention has very high accuracy in predicting effective pixels. For example, the percentages with an accuracy greater than 0.8 reached 87.1%, 80.1%, and 100%, respectively. In addition, using the point cloud generated by the ACMP method as the ground truth, the point cloud of the algorithm of this invention achieved an accuracy, completeness, and F1 score of over 99%. (3) In terms of fusion efficiency: using nine sets of sub-images of different sizes as experimental data for depth map fusion and statistically analyzing the time consumed, compared with mainstream methods such as COLMAP, OPENMVS, Gipuma, and ACMP, the fusion time was reduced by approximately 30% to 60%.

[0179] The algorithm of this invention predicts that a considerable number of pixels in the effective pixel region are invalid pixels. Further research can be conducted on the identification and refined prediction of these pixels to further improve the fusion efficiency.

[0180] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0181] This invention provides a depth map fusion algorithm that considers pixel region prediction. The algorithm incorporates prediction during the fusion process, performing binary classification on image pixels in the scene. Subsequent steps only perform depth map fusion on valid pixel regions, eliminating redundant computation and significantly improving fusion efficiency. This invention introduces a multi-scale concept for the first time in depth map fusion, proposing an inference strategy for valid pixel regions at a fine scale guided by coarse-scale object voxels. This avoids the problem of calculating and judging the fusion conditions for each pixel-level depth value of each reference image, eliminating a large amount of redundant computation. The algorithm proposes a fine-scale corrector to correct erroneous estimates from object voxels at the coarse scale. This effectively removes ambiguous projection ranges caused by object voxel occlusion errors or noise in the estimated depth map. This invention also introduces the concepts of image-side visibility measure and object-side intersection measure for the first time, comprehensively evaluating the effectiveness of voxels for fusion from both object-side and image-side perspectives.

[0182] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A depth map fusion algorithm that takes into account pixel region prediction, characterized in that: Includes the following steps, S1. Calculation of image combination: Using the number of shared feature points reconstructed by SfM as the main metric, and combining the angle weight function and the scale weight function, a global view selection scoring function is jointly constructed to select a set of candidate neighbor images for each reference image in the scene, and the sum of image scores is calculated based on the global score of each image in the candidate neighbor image set. S2, Pixel region prediction: Construct object voxels at a coarse scale, and select an aggregated image subset and associated information for each object voxel; Guided by a coarse scale, the effective pixel region at the fine scale is inferred, and the erroneous estimation of object voxels from the coarse scale is corrected based on the fine scale corrector. S3, Depth Value Fusion: Using the pixel regions inferred at a fine scale as image-side constraints, valid pixels are determined and filtered out. The 3D points corresponding to the depth values ​​of the valid pixels and the 3D points estimated by the normal vectors are averaged into a unified 3D point. Based on the prediction results of the pixel regions, the images in the scene are divided into two categories: one category contains valid pixels for fusion, which are invalid images; the other category contains valid pixels for fusion, which are valid images. Step S3 specifically includes the following: S31, Reference Image Transformation: Only convert the valid image to the reference image, and only convert the depth values ​​corresponding to the pixels in the valid pixel area of ​​the reference image to 3D points in the world coordinate system, and project them onto the neighbor view to obtain the corresponding match; S32. Pixel fusion judgment: Set the depth fusion conditions, calculate the consistency constraint value of pixels in the effective pixel area, and determine whether it meets the fusion conditions. If there are at least two adjacent views whose corresponding matches satisfy the above fusion conditions, the estimated depth value will be accepted; S33, 3D point fusion: The 3D points corresponding to the accepted depth values ​​and the 3D points estimated by the normal vectors are averaged into a unified 3D point.

2. The depth map fusion algorithm considering pixel region prediction according to claim 1, characterized in that: Step S1 specifically includes the following: S11. Construct a global scoring function: using reference images in the scene. The number of shared feature points is used as a weight to construct the global score function. To compute the candidate neighbor image set Each image in The overall score; （1） in, and Images and reference image The set of feature points; and These are the angle weight function and the scale weight function, respectively; the candidate neighbor set Includes reference images ; Angle weight function Defined as The product of all matching pairs in the set. （2） in, ; The first The image and the first One image; From and to feature point The angle between the lines of sight; Angle The maximum value; Calculate reference image of and based on ratio Define scale weights , （3） in, For reference image China-Israel feature points The diameter of the sphere centered on it; For image China-Israel feature points The diameter of the sphere centered on it; S12. Calculate the sum of image scores: Iteratively sum the scores of the images with the highest scores using a greedy method. Add them one by one to the set of best neighbor images until the number of best neighbor images in the set equals the given size. Obtain the sum of image scores. .

3. The depth map fusion algorithm considering pixel region prediction according to claim 1, characterized in that: Step S2 specifically includes the following: S21. Coarse-scale object voxel measurement: Using the camera intrinsic parameters, image extrinsic parameters, connection points, depth maps, and image combinations in the recovered scene, the depth maps corresponding to some images in the scene are back-projected, calculated, and sampled to generate a semi-dense point cloud in the object space to construct object voxels. The object voxel measurement is estimated using image-side visibility measure and object-side intersection measure; the connection points are the shared feature points reconstructed by SfM. S22. Fine Dimension Corrector: Constructs a fine dimension correction formula based on image-side visibility measure and object-side intersection measure, and obtains the effective voxels of the object-side voxels relative to the view based on the fine dimension correction formula. Merges the pixel ranges of the effective voxels to obtain the effective pixel region of the image.

4. The depth map fusion algorithm considering pixel region prediction according to claim 3, characterized in that: Step S21 specifically includes the following: S211. Select a subset of voxel images that covers all ground features and has low overlap between images based on the number of connection points. And utilize completeness constraints make While ensuring coverage of all scene features, remove images with high overlap as much as possible; （4） in, for The total number of connection points across all included images; for The total number of connection points across all included images; It is a collection of all images contained in the scene; S212. Downsample the depth map, calculate the 3D points using back projection using formula (5), and store them in the semi-dense point cloud. middle; （5） in, λ represents a 3D point in the world coordinate system of the image; λ is the depth value. It's a camera. The transpose of the rotation matrix; For camera The inverse of the intrinsic parameter matrix; For camera The translation component; 𝑝 is the pixel coordinate of the image; S213, Using an octree structure to... In a three-dimensional spatial partitioning, the leaf nodes of the octree represent object voxels. ; S214. Use two quantitative metrics, image-side visibility measure and object-side intersection measure, to estimate voxel measure.

5. The depth map fusion algorithm considering pixel region prediction according to claim 4, characterized in that: Step S214 specifically includes the following: S2141, to Triangulation forms a cubic voxel mesh The mapping relationship between object voxel meshes and triangular faces in the scene is represented as follows: （6） in, This is the index number of the voxel; the total number of voxels is... ; The index number is the triangular facet number; a voxel is divided into 12 triangular facets. S2142. Construct a view based on the camera intrinsic parameters, image extrinsic parameters, and image resolution in the scene. The projection matrix, as shown in formula (7), will project the voxels. The triangular facet is projected onto the view. Image space; （7） Among them, among them, This is the intrinsic parameter matrix. This is the extrinsic parameter matrix; , These are the x-axis coordinates and y-axis coordinates in the view image coordinate system, respectively; The z-axis coordinate value in the camera coordinate system; The focal length of the camera; , These are the center pixel coordinates; It is a 3×3 rotation matrix, which represents the rotation of the camera coordinate system relative to the world coordinate system; It is a 3×1 translation vector, which is the offset of the camera coordinate system relative to the world coordinate system; It is a zero vector of size 1×3; These are the components along the x-axis, y-axis, and z-axis in the world coordinate system, respectively. The image-side visibility measure is obtained by rasterizing the projected triangular facet and acquiring the set of pixels within that facet. This value is used to measure the object-side voxel. In view Visibility in; （8） in, For image-side visibility measurement; and These are pixel coordinates; This represents the depth value of the direction of the projected light ray; The larger the volume, the more voxels Projected onto view The larger the visible area, the better the view. voxels The fusion contribution of dense point clouds within a certain range is positive, while the fusion contribution of smaller point clouds is negative. collect Views with values ​​greater than 0 are used to initialize object voxels. aggregated image subset ; S2143, Utilizing the pathways contained in voxels The number and density of three-dimensional point sets calculated by back projection constitute the intersection measure of the structure. （9） in, For the measure of object-space intersection; The number of three-dimensional point sets in each voxel; The normalized value representing the quantity; voxels Volume; As a weighting factor; Delete aggregated image subset middle A view close to 0.

6. The depth map fusion algorithm considering pixel region prediction according to claim 5, characterized in that: Step S22 specifically includes the following: S221, Using image-side visibility measure and image-side intersection measure A fine-scale correction formula, namely formula (10), is constructed to screen out qualified object voxels; （10） in, The degree of difference between the two values; The smaller, the more it indicates and The closer, the more likely the view is to be correct. For voxels In this respect, the contribution of integration is positive; S222. Calculate the object voxel according to the fine-scale correction formula. aggregated image subset The rating value of each view is displayed, and the views are sorted in ascending order of rating value, with the top-ranked views selected. Each view is a voxel A view that contributes to fusion at a fine scale, i.e., an effective view. Each view contains the pixel range corresponding to that voxel, and that voxel is called the effective voxel relative to that view; S223. Merge the pixel ranges of all valid voxels in each view to form the valid pixel region of the image.

7. The depth map fusion algorithm considering pixel region prediction according to claim 1, characterized in that: The fusion conditions in step S32 include a consistent matching of relative depth differences less than or equal to a matching threshold, an angle between normals less than or equal to an angle threshold, and a reprojection error less than or equal to an error threshold.

Images