Application of non-local hierarchical adaptive propagation three-dimensional reconstruction method in cultural relic heritage protection

Abstract

The invention relates to the technical field of three-dimensional reconstruction, in particular to application of a non-local hierarchical adaptive propagation three-dimensional reconstruction method in cultural relic heritage protection. Through bilateral Gaussian weighted luminosity consistency cost calculation, a robust solution is provided for a low texture region, a shielding problem and an illumination change problem in multi-view matching, and technical support is provided for high-quality three-dimensional reconstruction of a complex scene. The hierarchical sampling strategy combines the advantages of coarse sampling and fine sampling, the optimal direction is screened in the coarse sampling stage, invalid calculation is reduced, precise propagation is performed in the high-confidence direction in the fine sampling stage, checkerboard division ensures the calculation independence of adjacent pixels, and the algorithm efficiency is improved; by introducing non-local operation and flexible sampling radius setting, hypothesis propagation efficiency and depth estimation accuracy are effectively improved in a low-texture area, a shielding scene and a complex scene, and a more reliable technical guarantee is provided for high-quality 3D reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to the application of a three-dimensional reconstruction method of non-local hierarchical adaptive propagation in the protection of cultural relics and heritage. Background Art

[0002] There is an important demand for 3D reconstruction technology in the field of cultural relics protection. Cultural relics are facing the risk of being damaged or disappeared due to natural weathering, human destruction, etc. Using 3D reconstruction technology for digital archiving can provide a permanent solution for the protection, restoration and inheritance of cultural relics.

[0003] Among them, in order to cope with the reconstruction challenges of weak texture areas, a series of new 3D reconstruction methods have emerged in recent years, including the SD-MVS method, which solves the 3D reconstruction problem in areas with missing texture information through segmentation-driven deformable multi-view stereo reconstruction. The core of the SD-MVS method is to optimize the reconstruction quality through pixel-level patch deformation and expectation maximization (EM) algorithm. However, although 3D reconstruction technology has made some progress in weak texture areas, traditional methods still have the following shortcomings, as follows:

[0004] Strong dependence on feature extraction: Traditional algorithms rely on significant feature points, but this method may fail in areas with scarce textures, resulting in reduced reconstruction accuracy;

[0005] Environmental sensitivity: Lighting changes, occlusion, and background interference can easily affect the extraction and matching of image features, thereby reducing the robustness of reconstruction;

[0006] Insufficient ability to capture details: Traditional methods are difficult to achieve high-precision reconstruction of subtle textures and complex structures on the surface of cultural relics;

[0007] High computational complexity: In weakly textured scenes, additional feature matching and optimization steps increase computational time and cost;

[0008] In summary, the existing 3D reconstruction technology still has a lot of room for improvement in weak texture areas and cultural relics protection fields. To this end, the present invention proposes a 3D reconstruction method of non-local hierarchical adaptive propagation to better meet the current high-precision reconstruction needs in cultural relics protection and complex scenes. Summary of the invention

[0009] In view of the above-mentioned shortcomings of the prior art, the present invention provides an application of a non-local hierarchical adaptive propagation three-dimensional reconstruction method in the protection of cultural relics and heritage, which can effectively solve the problems of insufficient accuracy and efficiency of the three-dimensional reconstruction technology in the prior art when reconstructing the Jincheng in weak texture areas and cultural relics protection fields.

[0010] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0011] The present invention provides an application of a non-local hierarchical adaptive propagation three-dimensional reconstruction method in cultural heritage protection, comprising the following steps:

[0012] Perform bilateral Gaussian weighted photometric consistency cost calculation, specifically:

[0013] Obtain the artifact image and calculate the total Gaussian weight including the spatial Gaussian weight and the range Gaussian weight, optimize the image matching process, and based on the assumed depth of multi-view geometric consistency, the pixels of the reference view are projected to the source view, and the photometric consistency cost is calculated to generate a preliminary depth estimate;

[0014] A hierarchical chessboard propagation strategy is used to optimize the depth estimation of cultural relic images, specifically:

[0015] Through pixel division and odd-even frame processing, combined with coarse sampling and fine sampling stages, the depth estimation is optimized step by step to fill the holes in the depth estimation and refine the depth estimation results;

[0016] Non-local far-point sampling strategy, specifically:

[0017] Through non-local operations and scalable sampling strategies, the sampling range is changed to avoid falling into the local optimal solution, helping to obtain globally consistent 3D reconstruction results.

[0018] Furthermore, the calculation method of the spatial Gaussian weight is:

[0019] For each pixel p(x,y), define its neighborhood window as W p , the window size is k×k;

[0020] Spatial Gaussian weight G s (p,q), calculated based on the spatial distance between pixels:

[0021]

[0022] Where q∈W p , and σ s Represents the standard deviation of the spatial Gaussian weights.

[0023] Furthermore, the range Gaussian weight G r The calculation method of (p,q) is:

[0024] Calculated from the intensity difference between pixels:

[0025]

[0026] Among them, I r (p) represents the grayscale value of the reference image at pixel p, I r(q) represents the gray value of pixel q in the source view, σ r Indicates the standard deviation of the range Gaussian weights.

[0027] Furthermore, the total Gaussian weight is determined as follows:

[0028] The total Gaussian weight ωj is obtained by multiplying the spatial Gaussian weight and the range Gaussian weight:

[0029] ωj=G s (p,q)·G r (p,q).

[0030] Furthermore, the photometric consistency cost is calculated, and we have:

[0031] Based on multi-view geometric consistency, under the assumption that the depth d r Next, the pixel p of the reference view is projected onto the source view S to obtain the matching pixel p′ and calculate the photometric consistency cost C(p):

[0032] C(p)=Σq∈W p G s (p,q)·G r (p,q)·||I s (q)-I r (p′)||

[0033] Therefore, the cost is normalized to obtain the final photometric consistency cost of each pixel.

[0034] Furthermore, the method for optimizing depth estimation is:

[0035] Pixel division: The image pixels are divided into two types: red and black. The red pixels and black pixels are assumed to be updated independently. The parallel computing capability of the GPU is used to process the red and black pixels separately.

[0036] Even-odd frame processing: In each iteration, the hypothetical updates of red and black pixels are processed alternately;

[0037] Coarse sampling stage:

[0038] Direction division: With the current pixel as the center, the surrounding area is divided into 16 directions, each direction covers a 22.5° sector area;

[0039] Direction hypothesis sampling: select sampling points in each direction and calculate the weighted bilateral Gaussian photometric consistency cost for the sampling points;

[0040] Direction screening: Count the number of sampling points in each direction that meet the preset photometric consistency threshold, and select the direction with the most matching points as the optimal direction;

[0041] Fine sampling stage:

[0042] High-density sampling in the optimal direction: Hypothetical sampling is performed in the optimal direction selected in the coarse sampling stage;

[0043] Hypothesis update: recalculate the photometric consistency cost for fine sampling points, select the hypothesis with the lowest cost as the best depth hypothesis for the current pixel, and use checkerboard grouping to propagate the best hypothesis to neighboring pixels to optimize the overall depth field;

[0044] Iterative propagation: Coarse sampling and fine sampling are performed alternately, and the depth field is optimized by step-by-step iterative propagation until convergence or the preset number of iterations is reached.

[0045] Furthermore, the non-local far-point sampling strategy includes:

[0046] Set the sampling area;

[0047] Selection of sampling points;

[0048] Nonlocal hypothesis propagation.

[0049] Furthermore, the method for setting the sampling area is:

[0050] In the hypothetical propagation, for a pixel point p, a minimum sampling radius R is defined min and the maximum sampling radius R max , and the selection of sampling points satisfies:

[0051] R min ≤||pq||≤R max

[0052] Here, ||pq|| represents the Euclidean distance between pixel points p and q.

[0053] Furthermore, the sampling points are selected as follows:

[0054] In the set annular area, N sampling points are randomly selected as candidate points for hypothesis propagation, and the selection of sampling points is dynamically adjusted through a pseudo-random number generator or based on pixel gradient information.

[0055] Furthermore, the non-local hypothesis propagation is specifically:

[0056] The depth hypothesis of the sampling point is screened, and the hypothesis with the best cost of consistency with the current pixel light intensity is propagated and added to the candidate set. In the candidate set, the best hypothesis is selected for update according to the principle of minimum cost to ensure that the local optimal solution is skipped.

[0057] Compared with the known prior art, the technical solution provided by the present invention has the following beneficial effects:

[0058] Through bilateral Gaussian weighted photometric consistency cost calculation, a robust solution is provided for low-texture areas, occlusion problems, and illumination change problems in multi-view matching. This not only improves the accuracy and robustness of depth estimation, but also expands the application scenarios of 3D reconstruction technology, providing technical support for high-quality 3D reconstruction of complex scenes.

[0059] The hierarchical sampling strategy combines the advantages of coarse sampling and fine sampling. The optimal direction is selected in the coarse sampling stage to reduce invalid calculations. The fine sampling stage focuses on high-confidence directions for precise propagation. The checkerboard division ensures the independence of adjacent pixel calculations. Combined with GPU parallel processing, the algorithm efficiency is improved.

[0060] By introducing non-local operations and flexible sampling radius settings, the efficiency of hypothesis propagation and the accuracy of depth estimation are effectively improved in low-texture areas, occluded scenes and complex scenes, providing a more reliable technical guarantee for high-quality 3D reconstruction, while also expanding its application potential in various task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 It is a schematic diagram of the overall method flow of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The present invention will be further described below in conjunction with the embodiments.

[0065] Example 1 (see Figure 1 ): The application of the three-dimensional reconstruction method of non-local hierarchical adaptive propagation in the protection of cultural relics and heritage includes the following steps:

[0066] Specifically, in this solution, considering that the matching reliability of traditional methods is low in low-texture areas or occluded scenes, it is easily affected by illumination changes, noise and occlusion, resulting in a decrease in the accuracy of depth estimation. In order to solve this problem, a bilateral Gaussian weighted photometric consistency cost calculation method (BGWP) is introduced. By using a bilateral Gaussian filter, spatial information and photometric information are organically combined to improve the matching accuracy in low-texture areas and complex scenes while maintaining edge sharpness, including:

[0067] Input data:

[0068] Get the reference view image I r and a set of source view images I s (artifact image), where S = 1, 2, 3, ... n;

[0069] Get the initial depth hypothesis d r and the corresponding normal vector n r .

[0070] Calculation of bilateral weights:

[0071] For each pixel p(x,y), define its neighborhood window as W p , the window size is k×k;

[0072] Spatial Gaussian weight G s (p,q), calculated based on the spatial distance between pixels:

[0073]

[0074] Where q∈W p , and σ s Represents the standard deviation of the spatial Gaussian weight, usually set to 1.5;

[0075] Range Gaussian weight G r (p,q), calculated from the intensity difference between pixels:

[0076]

[0077] Among them, I r (p) represents the grayscale value of the reference image at pixel p, I r (q) represents the gray value of pixel q in the source view, σ r Indicates the standard deviation of the range Gaussian weight, usually set to 10;

[0078] Define the total Gaussian weight, which is obtained by multiplying the spatial Gaussian weight and the range Gaussian weight:

[0079] ωj=G s (p,q)·Gr (p,q)

[0080] The photometric consistency cost is defined based on the multi-view geometric consistency, assuming the depth d r Next, the pixel p of the reference view is projected onto the source view S to obtain the matching pixel p′ and calculate the photometric consistency cost C(p):

[0081] C(p)=Σq∈W p G s (p,q)·G r (p,q)·||I s (q)-I r (p′)||

[0082] Therefore, the cost is normalized to obtain the final photometric consistency cost of each pixel.

[0083] By optimizing the photometric consistency cost function, a preliminary depth estimation of the cultural relic image is generated. By combining the weighted pixel color and spatial distance, the matching error caused by uneven lighting and weak texture areas is solved.

[0084] Therefore, in this embodiment, the bilateral Gaussian filter combines spatial and photometric information, is less sensitive to intensity differences, and is suitable for processing complex lighting scenes;

[0085] The introduction of spatial Gaussian effectively maintains the sharpness of depth edges and avoids mismatching in depth discontinuity areas;

[0086] Low-texture area optimization: Through weight adjustment, bilateral Gaussian can improve the matching accuracy of low-texture areas and improve the quality of 3D reconstruction;

[0087] In summary, the bilateral Gaussian weighted photometric consistency cost is to combine bilateral filtering technology and traditional photometric consistency calculation methods to provide a robust solution to low-texture areas, occlusion problems and illumination changes in multi-view matching. This method not only improves the accuracy and robustness of depth estimation, but also expands the application scenarios of 3D reconstruction technology, providing technical support for high-quality 3D reconstruction of complex scenes.

[0088] Furthermore, by combining the hierarchical chessboard propagation strategy for depth estimation optimization, we have:

[0089] Pixel division: The image pixels are divided into two types, red and black, similar to a checkerboard pattern. The red pixels and black pixels are assumed to be updated independently of each other. The parallel computing capability of the GPU is used to process the red and black pixels separately to ensure update efficiency.

[0090] Even-odd frame processing: In each iteration, the hypothetical updates of red and black pixels are processed alternately to ensure that adjacent pixels do not interfere with each other;

[0091] Coarse sampling stage:

[0092] Direction division: With the current pixel as the center, the surrounding area is divided into 16 directions, each direction covers a 22.5° sector area;

[0093] Direction hypothesis sampling: select a certain number of sampling points in each direction and calculate the weighted bilateral Gaussian photometric consistency cost for these sampling points;

[0094] Direction screening: Count the number of sampling points in each direction that meet the preset photometric consistency threshold, and select the direction with the most matching points as the optimal direction;

[0095] Fine sampling stage:

[0096] High-density sampling in the optimal direction: more dense hypothetical sampling is performed in the optimal direction selected in the coarse sampling stage;

[0097] Hypothesis update: recalculate the photometric consistency cost for fine sampling points, select the hypothesis with the lowest cost as the best depth hypothesis for the current pixel, and use checkerboard grouping to propagate the best hypothesis to neighboring pixels to optimize the overall depth field;

[0098] Iterative propagation: Coarse sampling and fine sampling are performed alternately, and the depth field is optimized by step-by-step iterative propagation until convergence or the preset number of iterations is reached.

[0099] In the above, the preliminary depth map is optimized step by step to fill some of the depth estimation holes and refine the depth estimation results. This process captures the global structure through coarse sampling and restores local details through fine sampling. It is a key step in refining the depth map.

[0100] In this embodiment, the optimal direction is screened out in the coarse sampling stage, and resources are concentrated to perform fine sampling on the effective direction, which significantly reduces the amount of invalid calculations and optimizes resource allocation;

[0101] By screening out representative hypothesis points through a weighted photometric consistency cost function, the propagation process of the present invention can adaptively target the feature distribution of different scenes, and especially exhibits higher robustness in weak texture areas.

[0102] Under the red and black checkerboard division, the powerful parallel computing capability of GPU is used to simultaneously process the hypothetical updates of the red and black groups of pixels; hierarchical sampling is used to reduce the computational complexity of each iteration, thereby further improving the overall computing speed;

[0103] In weak texture areas, information is usually concentrated in a few directions. The coarse sampling stage of the present invention effectively locates these directions, and focuses on high-confidence areas for deep propagation in the fine sampling stage, which fully reflects the original design intention of achieving high-quality reconstruction with a small amount of resources.

[0104] Furthermore, in low-texture areas, traditional sampling methods usually focus on the local area around the center point. This limitation may cause hypothesis propagation to easily fall into the local optimal solution, thereby reducing the accuracy of depth estimation. In order to solve this problem, an improved method of non-local operation and scalable sampling strategy is introduced. By skipping the sampling points around the center point and focusing on the more distant non-local areas, the efficiency and robustness of hypothesis propagation are improved. Then:

[0105] Set the sampling area:

[0106] In the hypothetical propagation, for a pixel point p, a minimum sampling radius R is defined min and the maximum sampling radius R max , and the selection of sampling points satisfies:

[0107] R min ≤||pq||≤R max

[0108] Where ||pq|| represents the Euclidean distance between pixels p and q;

[0109] Selection of sampling points:

[0110] In the set annular area, N sampling points are randomly selected as candidate points for hypothetical propagation. The selection of sampling points can be dynamically adjusted through a pseudo-random number generator or based on pixel gradient information to meet the needs of different scenarios.

[0111] Non-local hypothesis propagation:

[0112] The depth hypothesis of the sampling point is screened, and the hypothesis with the best cost of consistency with the current pixel light intensity is added to the candidate set. In the candidate set, the best hypothesis is selected for update according to the principle of minimum cost to ensure that the local optimal solution is skipped.

[0113] In the above, it should be noted that:

[0114] The sampling range is expanded through non-local operations, which increases the possibility of selecting high-quality pixels at a long distance and can effectively avoid falling into the local optimal solution;

[0115] Depth estimation in low-texture areas relies less on local information, while non-local sampling strategies provide richer support for depth estimation by integrating more long-range global information;

[0116] In summary, the innovation of sampling distant points to prevent falling into the local optimal solution is to break through the limitations of traditional local sampling methods. By introducing non-local operations and flexible sampling radius settings, the efficiency of hypothesis propagation and the accuracy of depth estimation are effectively improved in low-texture areas, occluded scenes and complex scenes. This method provides a more reliable technical guarantee for high-quality 3D reconstruction, and also expands its application potential in various task scenarios.

[0117] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. Application of the 3D reconstruction method based on non-local hierarchical adaptive propagation in the protection of cultural relics, characterized in that: The steps include: Perform bilateral Gaussian weighted photometric consistency cost calculation, specifically: Obtain the artifact image and calculate the total Gaussian weight including the spatial Gaussian weight and the range Gaussian weight, optimize the image matching process, and based on the assumed depth of multi-view geometric consistency, the pixels of the reference view are projected to the source view, and the photometric consistency cost is calculated to generate a preliminary depth estimate; A hierarchical chessboard propagation strategy is used to optimize the depth estimation of cultural relic images, specifically: Through pixel division and odd-even frame processing, combined with coarse sampling and fine sampling stages, the depth estimation is optimized step by step to fill the holes in the depth estimation and refine the depth estimation results; Execute the non-local far-point sampling strategy, specifically: Through non-local operations and scalable sampling strategies, the sampling range is changed to avoid falling into the local optimal solution, helping to obtain globally consistent 3D reconstruction results.

2. The application of the non-local hierarchical adaptive propagation 3D reconstruction method in cultural heritage protection according to claim 1 is characterized in that: The calculation method of the spatial Gaussian weight is: For each pixel p(x,y), define its neighborhood window as W p , the window size is k×k; Spatial Gaussian weight G s (p,q), calculated based on the spatial distance between pixels: Where q∈W p , and σ s Represents the standard deviation of the spatial Gaussian weights.

3. The application of the non-local hierarchical adaptive propagation 3D reconstruction method in cultural heritage protection according to claim 2 is characterized in that: The range of Gaussian weights G r The calculation method of (p,q) is: Calculated from the intensity difference between pixels: Among them, I r(p) Represents the gray value of the reference image at pixel p, I r(q) represents the gray value of pixel q in the source view, σ r Indicates the standard deviation of the range Gaussian weights.

4. The application of the non-local hierarchical adaptive propagation 3D reconstruction method in cultural heritage protection according to claim 3 is characterized in that: The total Gaussian weight is determined as follows: The total Gaussian weight ωj is obtained by multiplying the spatial Gaussian weight and the range Gaussian weight: ωj=G s (p,q)·G r (p,q).

5. The application of the non-local hierarchical adaptive propagation 3D reconstruction method in cultural heritage protection according to claim 4 is characterized in that: To calculate the photometric consistency cost, we have: Based on multi-view geometric consistency, under the assumption that the depth d r Next, the pixel p of the reference view is projected onto the source view S to obtain the matching pixel p′ and calculate the photometric consistency cost C(p): C(p)=Σq∈W p G s (p,q)·G r (p,q)·||I s (q)-I r (p′)|| Therefore, the cost is normalized to obtain the final photometric consistency cost of each pixel.

6. The application of the non-local hierarchical adaptive propagation 3D reconstruction method in cultural heritage protection according to claim 1 is characterized in that: The method for optimizing the depth estimation is: Pixel division: The image pixels are divided into two types: red and black. The red pixels and black pixels are assumed to be updated independently. The parallel computing capability of the GPU is used to process the red and black pixels separately. Even-odd frame processing: In each iteration, the hypothetical updates of red and black pixels are processed alternately; Coarse sampling stage: Direction division: With the current pixel as the center, the surrounding area is divided into 16 directions, each direction covers a 22.5° sector area; Direction hypothesis sampling: select sampling points in each direction and calculate the weighted bilateral Gaussian photometric consistency cost for the sampling points; Direction screening: Count the number of sampling points in each direction that meet the preset photometric consistency threshold, and select the direction with the most matching points as the optimal direction; Fine sampling stage: High-density sampling in the optimal direction: Hypothetical sampling is performed in the optimal direction selected in the coarse sampling stage; Hypothesis update: recalculate the photometric consistency cost for fine sampling points, select the hypothesis with the lowest cost as the best depth hypothesis for the current pixel, and use checkerboard grouping to propagate the best hypothesis to neighboring pixels to optimize the overall depth field; Iterative propagation: Coarse sampling and fine sampling are performed alternately, and the depth field is optimized by step-by-step iterative propagation until convergence or the preset number of iterations is reached.

7. The application of the non-local hierarchical adaptive propagation 3D reconstruction method in cultural heritage protection according to claim 5 is characterized in that: The non-local far-point sampling strategy includes: Set the sampling area; Selection of sampling points; Nonlocal hypothesis propagation.

8. The application of the non-local hierarchical adaptive propagation 3D reconstruction method in cultural heritage protection according to claim 7 is characterized in that: The method for setting the sampling area is: In the hypothetical propagation, for a pixel point p, the minimum sampling radius R is defined min and the maximum sampling radius R max , and the selection of sampling points satisfies: R min ≤||p-q||≤R max Here, ||pq|| represents the Euclidean distance between pixel points p and q.

9. The application of the non-local hierarchical adaptive propagation 3D reconstruction method in cultural heritage protection according to claim 7, characterized in that: The selection of the sampling points is specifically as follows: In the set annular area, N sampling points are randomly selected as candidate points for hypothesis propagation, and the selection of sampling points is dynamically adjusted through a pseudo-random number generator or based on pixel gradient information.

10. The application of the non-local hierarchical adaptive propagation 3D reconstruction method in cultural heritage protection according to claim 7, characterized in that: The non-local hypothesis propagation is specifically: The depth hypothesis of the sampling point is screened, and the hypothesis with the best cost of consistency with the current pixel light intensity is propagated and added to the candidate set. In the candidate set, the best hypothesis is selected for update according to the principle of minimum cost to ensure that the local optimal solution is skipped.

Images