General multi-view three-dimensional reconstruction method based on normal visibility and frequency domain consistency repairing strategy

Abstract

The invention discloses a universal multi-view three-dimensional reconstruction method based on a normal visibility and frequency domain consistency repairing strategy, which comprises the following steps: extracting surface normal information from a multi-view image, extracting features of the multi-view image, and obtaining feature pairs based on the extracted features; determining an initial visibility graph based on a normal according to the surface normal information and the view angle direction of the camera; calculating the correlation between the reference view features and the source view features, and fusing the initial visibility map and the correlation to obtain a final visibility map; fusing the final visibility graph and the correlation, and carrying out weighted fusion on multiple groups of fusion results to obtain a three-dimensional cost body; determining a depth map based on the three-dimensional cost body, and optimizing the depth map by using a frequency domain optimized consistency repair strategy; fusing each optimized depth map to obtain a final depth map; and generating a three-dimensional reconstruction point cloud based on the final depth map. According to the method, the robustness of shielding problems and view angle transformation can be effectively enhanced, and the integrity and accuracy of reconstruction are enhanced.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, specifically to a general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategies. Background Technology

[0002] Multi-view stereo vision aims to recover the 3D geometry of a scene from a set of 2D images with known camera parameters, and has wide applications in fields such as remote sensing and monitoring. With the development of deep learning technology, learning-based MVS methods have gradually replaced traditional methods based on handcrafted features, becoming the mainstream research approach. Typical methods, such as MVSNet and its variants, estimate depth maps by constructing a 3D cost volume and performing regularization. To reduce the enormous computational overhead of 3D convolution, a series of methods based on RNNs and coarse-to-fine cascade frameworks have emerged. The former has been gradually replaced by the latter, represented by CasMVSNet, due to its slow depth inference. However, existing methods still face the following challenges: First, when faced with complex problems such as occlusion and drastic changes in viewpoint, methods like MVSNet that do not use geometric prior guidance often result in low reconstruction completeness. This is especially true in the cost volume construction stage: occlusion makes parts of the object's surface invisible between different views, leading to a large number of incorrect correspondences during feature matching. Consequently, during cost volume construction, features in occluded regions cannot be correctly aligned and fused, thus generating errors and affecting the final reconstruction quality.

[0003] Second, the depth maps estimated at each stage contain errors in the background, edges, weak textures, or occluded areas, and staged upsampling methods like CasMVSNet continuously amplify these errors. Furthermore, in weak texture regions, relying solely on pixel color or luminance matching can easily lead to accidental correspondences, resulting in depth blurring. These problems severely impact the final reconstruction quality. Therefore, existing frameworks urgently need a filtering and patching method that can effectively remove erroneous noise without affecting the depth information in key areas.

[0004] Third, existing methods can only address the reconstruction needs of objects at fixed scales. For example, methods like MVSNet focus on the reconstruction of small indoor objects in the DTU dataset. When faced with medium-scale outdoor scenes such as tanks and sculptures, the reconstruction quality drops significantly, and the results are particularly poor when dealing with ultra-large-scale remote sensing reconstruction scenes. In addition, these methods consume excessive GPU computing resources. Summary of the Invention

[0005] This invention provides a general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy to solve the above problems.

[0006] In a first aspect, the present invention provides a general multi-view stereo 3D reconstruction method based on a normal visibility and frequency domain consistency repair strategy, the method comprising: Acquire multi-view images and corresponding camera parameters, wherein the camera parameters include the camera viewpoint direction; Surface normal information is extracted from the multi-view image, and feature extraction is performed on the multi-view image. Feature pairs are obtained based on the extracted features, and the feature pairs include reference view features and source view features. Based on the surface normal information and the camera view direction, an initial visibility map based on the normal is determined; the correlation between the reference view features and the source view features is calculated, and the initial visibility map is fused with the correlation to obtain a final visibility map; the final visibility map is fused with the correlation, and multiple fusion results are weighted and fused to obtain a three-dimensional cost volume; The depth map is determined based on the 3D cost volume, and the depth map is optimized using a frequency domain optimization consistency patching strategy; the optimized depth maps are fused to obtain the final depth map; the frequency domain optimization consistency patching strategy includes a cross-view geometric consistency pixel patching strategy and a staged Gaussian frequency domain filtering strategy. A 3D reconstructed point cloud is generated based on the final depth map.

[0007] In some embodiments of the present invention, fusing the initial visibility map with the correlation to obtain the final visibility map includes: The initial visibility map and the correlation are convolved by convolutional layers respectively to obtain the convolved visibility map and the convolved correlation. The convolved visibility map and the convolved correlation are then combined to obtain the concatenated feature map. The concatenated feature map is processed using a convolutional neural network to obtain the final visibility map; The process of fusing the final visibility map with the correlation, and weighted fusing multiple sets of fusion results to obtain a three-dimensional cost volume, includes: The final visibility map is multiplied by the correlation to obtain the cost contribution of a single set of views that has been visibility modulated after the product of each feature pair. The cost contributions of the single set of views that have been visibility modulated after the product of each feature pair are weighted and fused to obtain the three-dimensional cost volume.

[0008] In some embodiments of the present invention, the step of determining the depth map based on the three-dimensional cost volume and optimizing the depth map using a frequency-domain optimized consistency patching strategy includes: The three-dimensional cost volume is regularized to obtain the probability volume; Based on the probability volume, a coarse-to-fine depth map estimation is performed to obtain a coarse-to-fine depth map; In the depth map estimation process, a frequency domain optimization consistency patching strategy is introduced to optimize the depth map and obtain the optimized depth map.

[0009] In some embodiments of the present invention, the step of introducing a frequency-domain optimized consistency patching strategy during the depth map estimation process to optimize the depth map and obtain the optimized depth map includes: Perform a geometric consistency check across views to identify inconsistencies; Using the inconsistency point as the center, search for the nearest pixel point that meets the geometric consistency constraint along multiple directions to obtain at least one target pixel point; The target depth value is determined based on the mean of the depth values ​​of the at least one target pixel, and the inconsistencies are removed. The pixel at the position of the inconsistency is filled with the target depth value, and the preliminary optimization result of the optimized depth map is obtained based on the filled pixel.

[0010] In some embodiments of the present invention, the depth map includes a first depth map, a second depth map, and a third depth map with sequentially increasing sizes; After obtaining the preliminary optimization results of the optimized depth map, the method further includes: Based on the previous optimization results of the optimized depth map, an original depth map is obtained, which includes a first depth map, a second depth map, and a third depth map with successively increasing sizes. The first depth map, the second depth map, and the third depth map are transformed in the frequency domain using the Fast Fourier Transform algorithm to obtain the corresponding first original frequency domain information, second original frequency domain information, and third original frequency domain information. The first original frequency domain information, the second original frequency domain information, and the third original frequency domain information are filtered using a Gaussian filter to obtain the first filtered frequency domain information, the second filtered frequency domain information, and the third filtered frequency domain information, respectively. The first filtered frequency domain information and the second filtered frequency domain information are inversely transformed respectively to obtain the first filtered depth map and the second filtered depth map respectively; and the first filtered depth map and the second filtered depth map are subjected to high frequency error filtering respectively to obtain the final optimized first depth map and optimized second depth map respectively. The third filtered frequency domain information is fused with the third original frequency domain information to obtain enhanced high-frequency information, and the enhanced high-frequency information is inversely transformed to obtain the final optimized third depth map.

[0011] In some embodiments of the present invention, the step of filtering the first original frequency domain information, the second original frequency domain information, and the third original frequency domain information using a Gaussian filter includes: The first original frequency domain information is filtered using a first Gaussian low-pass filter; The second original frequency domain information is filtered using a second Gaussian low-pass filter, wherein the cutoff frequency of the second Gaussian low-pass filter is greater than the cutoff frequency of the first Gaussian low-pass filter. The third original frequency domain information is filtered using a Gaussian high-pass filter.

[0012] In some embodiments of the present invention, the step of extracting features from the multi-view image and obtaining feature pairs based on the extracted features includes: Multi-scale feature extraction is performed on the multi-view image to obtain source view features and reference view features at different scales; According to multiple preset depths, the source view features of different scales are transformed to the coordinate system of the reference view features through homography transformation, and multiple sets of feature pairs are obtained based on the transformed source view features and the reference view features. Each feature pair includes a transformed source view feature and a reference view feature.

[0013] In some embodiments of the present invention, determining the initial visibility map based on the normal according to the surface normal information and the camera viewing direction includes: Calculate the cosine similarity between the surface normal and the camera's viewing direction, and correct the calculated cosine similarity according to a preset angle constraint. Use the weighted map composed of the corrected cosine similarity set as the initial visibility map.

[0014] In the general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy provided by this invention, the aggregation process of cost volume is effectively guided by normal visibility and correlation between feature groups, which effectively enhances the robustness of the network to occlusion problems and viewpoint changes, and enhances the completeness and accuracy of the final reconstruction result in the occluded area. It effectively enhances the completeness and accuracy of the reconstruction result with minimal computational overhead. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a general multi-view stereoscopic reconstruction method based on a normal visibility and frequency domain consistency repair strategy provided in an embodiment of the present invention. Figure 2This is a schematic diagram of geometric visibility volume fusion provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the operation of the frequency domain optimized consistency filtering strategy provided in the embodiments of the present invention; Figure 4 This is one of the schematic diagrams of the reconstruction effect provided in the embodiments of the present invention; Figure 5 This is the second schematic diagram of the reconstruction effect provided in the embodiment of the present invention; Figure 6 This is the third schematic diagram of the reconstruction effect provided in the embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0019] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0020] The use of "applies to" or "configured to" in this invention implies an open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more conditions or values ​​may in practice be based on additional conditions or values ​​beyond those conditions.

[0021] In this invention, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0022] The following describes, with reference to the accompanying drawings, a general multi-view stereoscopic reconstruction method based on normal visibility and frequency domain consistency repair strategy provided by embodiments of the present invention.

[0023] like Figure 1 As shown, this embodiment of the invention provides a general multi-view stereo 3D reconstruction method based on a normal visibility and frequency domain consistency repair strategy. The method includes the following steps: S101, acquire multi-view images and corresponding camera parameters, wherein the camera parameters include camera viewing angle direction. The multi-view images include a source view and a reference view, which are views obtained from different perspectives. The source view is the front view of the object, i.e., taken directly facing the object, while the reference view is an image of the same object from other perspectives.

[0024] S102, extract surface normal information from the multi-view image, perform feature extraction on the multi-view image, and obtain feature pairs based on the extracted features, wherein the feature pairs include reference view features and source view features.

[0025] Among them, surface normal information can be extracted through a monocular stereo normal model, which can be either the Metric3D v2-Large model or the Metric3D v2-Giant model.

[0026] S103, based on the surface normal information and the camera view direction, determine an initial visibility map based on the normal; calculate the correlation between the reference view features and the source view features, fuse the initial visibility map with the correlation to obtain a final visibility map; and fuse the final visibility map with the correlation, and weightedly fuse multiple sets of fusion results to obtain a three-dimensional cost volume.

[0027] S104, a depth map is determined based on the 3D cost volume, and the depth map is optimized using a frequency-domain optimized consistency patching strategy; the optimized depth maps are then fused to obtain the final depth map. The frequency-domain optimized consistency patching strategy includes a cross-view geometric consistency pixel patching strategy and a staged Gaussian frequency-domain filtering strategy.

[0028] S105, Generate a 3D reconstructed point cloud based on the final depth map.

[0029] The general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy provided by this invention effectively guides the cost volume aggregation process through normal visibility and correlation between feature groups, effectively enhancing the network's robustness to occlusion problems and viewpoint changes, and improving the completeness and accuracy of the final reconstruction results in the occluded area. It effectively enhances the completeness and accuracy of the reconstruction results with minimal computational overhead.

[0030] In some embodiments of the present invention, fusing the initial visibility map with the correlation to obtain the final visibility map includes: The initial visibility map is processed using convolutional layers respectively. Convolve the correlation CR to obtain the convolved visibility map and the convolved correlation, and then combine the convolved visibility map and the convolved correlation to obtain the concatenated feature map. For example... Figure 2 As shown, two 2D convolutional layers are used to refine the initial visibility map. Convolution with the aforementioned correlation CR to unify the initial visibility map. The dimensions of the correlation CR.

[0031] The concatenated feature map is processed using a convolutional neural network to obtain the final visibility map. Among them, convolutional neural networks are 2D convolutional neural networks.

[0032] The process of fusing the final visibility map with the correlation, and weighted fusing multiple sets of fusion results to obtain a three-dimensional cost volume, includes: The final visibility map is multiplied by the correlation to obtain the cost contribution of a single set of views after visibility modulation for each feature pair's product. The cost contributions of each feature pair's single set of views after visibility modulation are then weighted and fused to obtain the three-dimensional cost volume C. (Illustratively,) p is any feature pixel.

[0033] In some embodiments of the present invention, the step of determining the depth map based on the three-dimensional cost volume and optimizing the depth map using a frequency-domain optimized consistency patching strategy includes: The 3D cost volume is regularized to obtain a probability volume. Specifically, the 3D cost volume is fed into a 3DU-Net convolutional neural network for regularization to filter out some noise in the 3D cost volume for subsequent operations. Then, the regularized cost volume is subjected to a softmax operation along the depth direction to transform it into a 3D probability distribution volume. The probability volume provides a probability distribution of a depth hypothesis for each pixel.

[0034] Based on the probability volume, a coarse-to-fine depth map estimation is performed to obtain a coarse-to-fine depth map.

[0035] In the depth map estimation process, a frequency domain optimization consistency patching strategy is introduced to optimize the depth map and obtain the optimized depth map.

[0036] The general multi-view stereoscopic reconstruction method based on normal visibility and frequency domain consistency repair strategy provided by the embodiments of the present invention effectively solves the problems of occlusion, edge noise and blurring of weak texture areas by selectively suppressing noise or enhancing high-frequency edge details through multi-directional consistency repair and staged frequency domain filtering operations.

[0037] In some embodiments of the present invention, the step of introducing a frequency-domain optimized consistency patching strategy during the depth map estimation process to optimize the depth map and obtain the optimized depth map includes: Perform a geometric consistency check across views to identify inconsistencies. A cross-view refers to a set of source views and reference views.

[0038] Specifically, for a point on the reference view According to depth Project it onto each source view, and let the corresponding projection point be... Next, Projecting back onto the reference view, let this point be denoted as . If pixel reprojection error No more than pixel threshold Depth reprojection error Not exceeding the depth threshold And that point is at least If a point meets the above threshold on the Zhangyuan view, it is considered to satisfy geometric consistency and is retained; otherwise, it is considered to be an inconsistent point.

[0039] Using the inconsistency point as the center, search for the nearest pixel point that meets the geometric consistency constraint along multiple directions to obtain at least one target pixel point.

[0040] The target depth value is determined based on the mean of the depth values ​​of the at least one target pixel, and the inconsistencies are removed. The pixel at the position of the inconsistency is filled with the target depth value, and the preliminary optimization result of the optimized depth map is obtained based on the filled pixel.

[0041] Indicatively, such as Figure 3 As shown, geometric consistency is checked for each pixel in a cross-view using a cross-view geometric consistency check and its constraint formula, thereby determining the geometric consistency of each pixel. Figure 3 The inconsistencies between black and gray are identified. Then, using these inconsistencies as centers, the nearest pixels conforming to geometric consistency constraints are searched along eight directions as target pixels, such as... Figure 3 The yellow dots in the image. By summing the depth values ​​of these eight yellow dots and calculating the average as the target depth value, this target depth value is filled into the inconsistencies to form the filled pixels, which serve as the initial optimization result of the optimized depth map.

[0042] In some embodiments of the present invention, such as Figure 3 As shown, the depth map includes a first depth map, a second depth map, and a third depth map with successively increasing sizes. The first depth map has k=1 and a size of 1 / 4; the second depth map has k=2 and a size of 1 / 2; and the third depth map has k=3 and a size of 1.

[0043] After obtaining the preliminary optimization results of the optimized depth map, the method further includes: Based on the previous optimization results of the optimized depth map, an original depth map is obtained, which includes a first depth map, a second depth map, and a third depth map with successively increasing sizes.

[0044] The first depth map, the second depth map, and the third depth map are transformed in the frequency domain using the Fast Fourier Transform algorithm to obtain the corresponding first original frequency domain information, second original frequency domain information, and third original frequency domain information.

[0045] The first original frequency domain information, the second original frequency domain information, and the third original frequency domain information are filtered using Gaussian filters to obtain the first filtered frequency domain information, the second filtered frequency domain information, and the third filtered frequency domain information, respectively.

[0046] Inverse transforms are performed on the first and second filtered frequency domain information, respectively, to obtain a first filtered depth map and a second filtered depth map. High-frequency error filtering is then applied to both the first and second filtered depth maps to obtain the final optimized first and second depth maps. It can be understood that high-frequency error filtering involves subtracting one depth map from another and vice versa, thereby suppressing high-frequency erroneous information, such as cluttered textures and noise caused by areas like the sky in the background.

[0047] The third filtered frequency domain information is fused with the third original frequency domain information to obtain enhanced high-frequency information, and the enhanced high-frequency information is inversely transformed to obtain the final optimized third depth map.

[0048] Schematic, the third filtered frequency domain information is multiplied element-wise with the third original frequency domain information, thereby effectively enhancing the high-frequency edge information and obtaining enhanced high-frequency information. The calculation process for denoising and edge enhancement operations is shown below, where... Depth maps representing different time periods: Specifically, in the stage At that time, for the original depth map The frequency domain form obtained after Fast Fourier Transform is applied to a Gaussian low-pass filter, and the original depth map is compared with the filtered depth map. The subtraction operation yields the stage. The final optimized depth map at that time This suppresses high-frequency erroneous information. To compensate for the loss of high-frequency information in edge detail regions, during the stage... At that time, the original depth map of the current stage is converted into the original frequency domain information by performing a Fast Fourier Transform. and to Obtaining filtered frequency domain information using a Gaussian high-pass filter The original frequency domain information With filtered frequency domain information Perform element-wise multiplication to obtain enhanced high-frequency information. The enhanced high-frequency information is inversely transformed to obtain the optimized third depth map in the final spatial domain form.

[0049] In some embodiments of the present invention, the step of filtering the first original frequency domain information, the second original frequency domain information, and the third original frequency domain information using a Gaussian filter includes: The first original frequency domain information is filtered using a first Gaussian low-pass filter.

[0050] The second original frequency domain information is filtered using a second Gaussian low-pass filter. The cutoff frequency of the second Gaussian low-pass filter is higher than that of the first Gaussian low-pass filter. This is understandable, referencing the learning approach in a course where the first Gaussian low-pass filter has a relatively small cutoff frequency, and the cutoff frequency of the second Gaussian low-pass filter is increased, thereby gradually introducing more challenging deep learning samples to the network.

[0051] The third original frequency domain information is filtered using a Gaussian high-pass filter. Unlike GeoMVSNet, this invention uses a smoother, more natural Gaussian filter instead of the original ideal filter. Furthermore, while GeoMVSNet repeatedly filters high-frequency noise, it inevitably loses high-frequency information such as edges. This invention further explores the possibility of using a Gaussian high-pass filter to enhance edge detail information.

[0052] In some embodiments of the present invention, the step of extracting features from the multi-view image and obtaining feature pairs based on the extracted features includes: Multi-scale feature extraction is performed on the multi-view image to obtain source view features and reference view features at different scales.

[0053] In some examples, a three-layer feature pyramid network is used as a feature extractor to perform multi-scale feature extraction on multi-view images, obtaining source view features and reference view features at different scales. The three downsampling ratios in the three-layer feature pyramid network are 1 / 4, 1 / 2, and 1. Downsampling of the multi-view images based on these three downsampling ratios yields source view features at the corresponding scales. And the reference view features, where i is the i-th source view, and k is the k-th layer feature pyramid network. It is the j-th preset depth, s is the source view, emphasizing that the current F refers to the feature of the source view, and n is the total number of feature pairs.

[0054] Based on multiple preset depths, homography transformation is used to transform the source view features of different scales to reference view features. The coordinate system in which it is located, and based on the features of the transformed source view. Features of the reference view Multiple sets of feature pairs are obtained, each feature pair including a transformed source view feature. and a reference view feature as described above 'r' represents the reference view, emphasizing that the current reference F is a feature of the reference view. Here, the homography transformation is a differentiable homography transformation; through the differentiable homography transformation, the network's... Phase 1 Features extracted from Zhangyuan View According to the Hypothesis depth Transform to the coordinate system of the reference view to obtain the transformed source view features corresponding to the reference view. .

[0055] In some embodiments of the present invention, determining the initial visibility map based on the normal according to the surface normal information and the camera viewing direction includes: Calculate the cosine similarity between the surface normal and the camera's viewing direction, and correct the calculated cosine similarity according to a preset angle constraint. Use the weighted map composed of the corrected cosine similarity set as the initial visibility map.

[0056] In some examples, inspired by the geometric definition of normals, it can be observed that the visibility of a surface changes with variations in the object's surface and the camera's shooting angle. Specifically, the more similar the surface normals and the camera direction, the higher the confidence level of the visibility of the captured area. Therefore, an initial visibility map based on normals can be obtained by calculating the cosine similarity between the two, thus better guiding the cost volume fusion process. Figure 2 The normal geometry prior process in the image, where the blue reference camera is directly facing the surface normal. It has maximum visibility, but as the shooting direction of the source camera gradually shifts, such as the red source camera at the farthest distance... Visibility has decreased significantly.

[0057] The unit vector of the surface normal is denoted as The unit direction vector of the camera's view direction is Unit direction vector It is the unit direction vector of the source camera along the central optical axis of the reference camera, determined based on the camera parameters. Furthermore, to eliminate the difference in positive and negative values ​​caused by the relative directions of the camera vector and the normal vector, and The cosine similarity, i.e., the inner product result, is negatively signed. Furthermore, because the included angle exceeds... This would cause the source view to become invisible. Therefore, the following preset angle constraint is set: the weight of the invisible region is reset to 0, and the cosine similarity of the inner product is retained as the weight in other cases. All weight sets constitute the initial visibility map guided by the normal. .

[0058] In some embodiments of this invention, the aforementioned feature extractor, convolutional neural network, etc., are trained on the DTU dataset, Tanks and Temples dataset, and WHU-MVS dataset. The DTU dataset, Tanks and Temples dataset, and WHU-MVS dataset are three datasets of different scales. The DTU dataset corresponds to small indoor objects, including cardboard boxes, fruit, bricks, etc., containing 124 scenes. Each scene is scanned with structured light from 49 or 64 perspectives, following a fixed camera movement trajectory and seven illumination intensities. The Tanks and Temples dataset corresponds to medium-scale outdoor scenes, including larger real-world objects such as tanks and sculptures. The WHU-MVS dataset corresponds to ultra-large-scale remote sensing scenes, covering diverse areas such as plains and cities. This invention inputs images from the DTU dataset into the Metric3D v2-Large monocular stereo normal model to predict surface normal information, as its structure is relatively simple and does not require an overly fitted prediction model. For the Tanks and Temples dataset and the WHU-MVS dataset, the Metric3D v2-Giant monocular stereo normal model is used to predict surface normal information. The reconstruction results on the DTU dataset are as follows: Figure 4 As shown, the reconstruction results on the Tanks and Temples dataset are as follows: Figure 5 As shown, the reconstruction results on the WHU-MVS dataset are as follows: Figure 6 As shown.

[0059] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0060] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0061] The foregoing has provided a detailed description of a general multi-view stereoscopic reconstruction method based on normal visibility and frequency domain consistency repair strategy provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy, characterized in that, The method includes: Acquire multi-view images and corresponding camera parameters, wherein the camera parameters include the camera viewpoint direction; Surface normal information is extracted from the multi-view image, and feature extraction is performed on the multi-view image. Feature pairs are obtained based on the extracted features, and the feature pairs include reference view features and source view features. Based on the surface normal information and the camera view direction, an initial visibility map based on the normal is determined; the correlation between the reference view features and the source view features is calculated, and the initial visibility map is fused with the correlation to obtain a final visibility map; the final visibility map is fused with the correlation, and multiple fusion results are weighted and fused to obtain a three-dimensional cost volume; The depth map is determined based on the 3D cost volume, and then optimized using a frequency domain optimization consistency patching strategy. The frequency domain optimization consistency patching strategy includes a cross-view geometric consistency pixel patching strategy and a staged Gaussian frequency domain filtering strategy. The optimized depth maps are then fused to obtain the final depth map. A 3D reconstructed point cloud is generated based on the final depth map.

2. The general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy according to claim 1, characterized in that, The process of fusing the initial visibility map with the correlation to obtain the final visibility map includes: The initial visibility map and the correlation are convolved by convolutional layers respectively to obtain the convolved visibility map and the convolved correlation. The convolved visibility map and the convolved correlation are then combined to obtain the concatenated feature map. The concatenated feature map is processed using a convolutional neural network to obtain the final visibility map; The process of fusing the final visibility map with the correlation, and weighted fusing multiple sets of fusion results to obtain a three-dimensional cost volume, includes: The final visibility map is multiplied by the correlation to obtain the cost contribution of a single set of views that has been visibility modulated after the product of each feature pair. The cost contributions of the single set of views that have been visibility modulated after the product of each feature pair are weighted and fused to obtain the three-dimensional cost volume.

3. The general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy according to claim 1, characterized in that, The process of determining the depth map based on the three-dimensional cost volume and optimizing the depth map using a frequency-domain optimized consistency patching strategy includes: The three-dimensional cost volume is regularized to obtain the probability volume; Based on the probability volume, a coarse-to-fine depth map estimation is performed to obtain a coarse-to-fine depth map; In the depth map estimation process, a frequency domain optimization consistency patching strategy is introduced to optimize the depth map and obtain the optimized depth map.

4. The general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy according to claim 3, characterized in that, The process of estimating the depth map involves introducing a frequency-domain optimized consistency patching strategy to optimize the depth map and obtain the optimized depth map, including: Perform a geometric consistency check across views to identify inconsistencies; Using the inconsistency point as the center, search for the nearest pixel point that meets the geometric consistency constraint along multiple directions to obtain at least one target pixel point; The target depth value is determined based on the mean of the depth values ​​of the at least one target pixel, and the inconsistencies are removed. The pixel at the position of the inconsistency is filled with the target depth value, and the preliminary optimization result of the optimized depth map is obtained based on the filled pixel.

5. The general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy according to claim 4, characterized in that, The depth map includes a first depth map, a second depth map, and a third depth map with sequentially increasing sizes; After obtaining the preliminary optimization results of the optimized depth map, the method further includes: Based on the previous optimization results of the optimized depth map, an original depth map is obtained, which includes a first depth map, a second depth map, and a third depth map with successively increasing sizes. The first depth map, the second depth map, and the third depth map are transformed in the frequency domain using the Fast Fourier Transform algorithm to obtain the corresponding first original frequency domain information, second original frequency domain information, and third original frequency domain information. The first original frequency domain information, the second original frequency domain information, and the third original frequency domain information are filtered using a Gaussian filter to obtain the first filtered frequency domain information, the second filtered frequency domain information, and the third filtered frequency domain information, respectively. The first filtered frequency domain information and the second filtered frequency domain information are inversely transformed respectively to obtain the first filtered depth map and the second filtered depth map respectively; and the first filtered depth map and the second filtered depth map are subjected to high frequency error filtering respectively to obtain the final optimized first depth map and optimized second depth map respectively. The third filtered frequency domain information is fused with the third original frequency domain information to obtain enhanced high-frequency information, and the enhanced high-frequency information is inversely transformed to obtain the final optimized third depth map.

6. The general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy according to claim 5, characterized in that, The step of filtering the first, second, and third original frequency domain information using a Gaussian filter includes: The first original frequency domain information is filtered using a first Gaussian low-pass filter; The second original frequency domain information is filtered using a second Gaussian low-pass filter, wherein the cutoff frequency of the second Gaussian low-pass filter is greater than the cutoff frequency of the first Gaussian low-pass filter. The third original frequency domain information is filtered using a Gaussian high-pass filter.

7. The general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy according to claim 1, characterized in that, The step of extracting features from the multi-view image and obtaining feature pairs based on the extracted features includes: Multi-scale feature extraction is performed on the multi-view image to obtain source view features and reference view features at different scales; According to multiple preset depths, the source view features of different scales are transformed to the coordinate system of the reference view features through homography transformation, and multiple sets of feature pairs are obtained based on the transformed source view features and the reference view features. Each feature pair includes a transformed source view feature and a reference view feature.

8. The general multi-view stereo 3D reconstruction method based on normal visibility and frequency domain consistency repair strategy according to claim 1, characterized in that, The step of determining the initial visibility map based on the surface normal information and the camera view direction includes: Calculate the cosine similarity between the surface normal and the camera's viewing direction, and correct the calculated cosine similarity according to a preset angle constraint. Use the weighted map composed of the corrected cosine similarity set as the initial visibility map.

Images