Cultural relic handicraft three-dimensional reconstruction method based on multiple views and deep learning

Abstract

The invention discloses a cultural relic handicraft three-dimensional reconstruction method based on multiple views and deep learning, and relates to the technical field of cultural relic handicraft three-dimensional reconstruction methods, and the method comprises the steps: carrying out the image collection of cultural relic handicrafts through employing a multi-view shooting method, and obtaining a structured image data set; performing feature matching and camera parameter estimation on the obtained multi-view image set by adopting a sparse reconstruction method to obtain camera internal parameters, external parameters and sparse point cloud data; performing fine processing on the obtained sparse point cloud data by adopting a dense reconstruction method to obtain a 3D dense point cloud model with a texture map; analyzing the obtained 3D dense point cloud model by adopting a depth map generation method to obtain a depth map and a rendering image corresponding to each image; image acquisition is performed on the cultural relic handicraft by adopting a multi-view shooting method, so that information such as geometric shapes and texture details of the cultural relic handicraft is comprehensively captured, and the problem of information loss caused by a single view angle is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction methods for cultural relics and handicrafts, and in particular to a three-dimensional reconstruction method for cultural relics and handicrafts based on multi-view and deep learning. Background Art

[0002] The three-dimensional reconstruction method technology of cultural relics and handicrafts is a technology specifically used to restore the three-dimensional structure of cultural relics and handicrafts from two-dimensional images. This technology is mainly based on computer vision and deep learning algorithms, and reconstructs the three-dimensional model of the object by processing images from multiple perspectives.

[0003] In the field of 3D reconstruction methods for cultural relics and artifacts, existing datasets for multi-view stereoscopic 3D reconstruction tasks primarily focus on large outdoor scenes and indoor household items. There is a lack of high-quality, diverse datasets specifically for cultural relics and artifacts. Due to the complex and diverse features of cultural relics and artifacts, such as shapes, materials, and textures, the inadequacy of existing datasets limits the generalization and practicality of 3D reconstruction algorithms in these areas. Furthermore, when faced with cultural relics and artifacts with complex textures and shapes, existing methods are unable to fully capture the multi-layered features of the objects, resulting in inaccurate and incomplete reconstruction results. Furthermore, existing frameworks can suffer from overfitting or underfitting in practical applications, further limiting their effectiveness. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a three-dimensional reconstruction method for cultural relics and crafts based on multi-view and deep learning to solve the problem that the shortcomings of existing data sets have led to the limited generalization ability and practicality of three-dimensional reconstruction algorithms in these fields, and when faced with cultural relics and crafts with complex textures and shapes, the existing methods are unable to fully capture the multi-level features of the objects, resulting in the reconstruction results being not accurate and complete enough.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning, comprising:

[0008] A multi-view shooting method is used to collect images of cultural relics and handicrafts to obtain a structured image dataset;

[0009] The sparse reconstruction method is used to perform feature matching and camera parameter estimation on the obtained multi-view image set to obtain camera intrinsic parameters, extrinsic parameters and sparse point cloud data;

[0010] The sparse point cloud data is refined using a dense reconstruction method to obtain a 3D dense point cloud model with texture mapping;

[0011] The obtained 3D dense point cloud model is analyzed using the depth map generation method to obtain the depth map and rendered image corresponding to each image;

[0012] The feature pyramid network FPN combined with the Transformer method is used to extract features from the input multi-view image to obtain a multi-scale feature map;

[0013] The cost volume of the obtained multi-scale feature map is constructed using differentiable homography transformation and variance cost aggregation method to obtain a probability volume;

[0014] The obtained probability volume is processed using depth map estimation and optimization methods to obtain a high-precision 3D reconstruction model.

[0015] As a preferred solution of the method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning of the present invention, wherein: the multi-view shooting method is used to capture images of cultural relics and handicrafts to obtain a structured image dataset, and the specific steps are as follows:

[0016] Five categories of cultural relics and handicrafts, including Buddha statues, clothing, ornaments, ceramics, and daily necessities, were selected as the subjects for the photography;

[0017] The subject is placed on a rotatable platform, and a fixed camera is used to photograph each subject from multiple angles in a 180-degree surround manner, with no less than 20 original images collected for each subject;

[0018] Screen the collected original images and remove invalid images that are blurred, blocked or have abnormal exposure;

[0019] The screened valid images were subjected to resolution standardization and uniformly adjusted to high-resolution images of 768 × 576 pixels;

[0020] Perform color correction and white balance processing on the standardized image;

[0021] The processed multi-view images are classified and stored according to the photographed objects to form a structured image dataset.

[0022] As a preferred solution of the method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning described in the present invention, wherein: the sparse reconstruction method is used to perform feature matching and camera parameter estimation on the obtained multi-view image set to obtain camera intrinsic parameters, extrinsic parameters and sparse point cloud data, the specific steps are as follows:

[0023] Input the structured image dataset into the COLMAP software and start the sparse reconstruction process;

[0024] The SIFT feature extraction algorithm is used to detect feature points in each image and extract scale-invariant feature descriptors;

[0025] Perform feature matching between images based on feature descriptors and establish feature correspondence between multi-view images;

[0026] The feature matching includes using a kd tree to accelerate the nearest neighbor search, adopting a bidirectional matching strategy to eliminate mismatches, and applying a RANSAC algorithm to eliminate abnormal matching point pairs;

[0027] Execute an incremental structure-from-motion (SfM) algorithm, which sequentially includes selecting an initial image pair for relative pose estimation, computing an initial 3D point cloud through triangulation, and gradually adding new images for bundle adjustment optimization.

[0028] Output camera parameters, including an intrinsic parameter matrix and an extrinsic parameter matrix, and generate sparse point cloud data;

[0029] The sparse point cloud data includes three-dimensional space point coordinates, point cloud color information, and point cloud visibility information.

[0030] As a preferred solution of the method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning described in the present invention, wherein: the dense reconstruction method is used to refine the obtained sparse point cloud data to obtain a 3D dense point cloud model with texture mapping, and the specific steps are as follows:

[0031] Use OpenMVS for dense reconstruction processing, input multi-view images and camera parameter files, and obtain the initial dense point cloud;

[0032] The initial dense point cloud is meshed and the Delaunay triangulation algorithm is applied to obtain a fine triangular mesh model.

[0033] Calculate texture maps based on multi-view images and use the best view selection method to obtain high-resolution texture maps;

[0034] UV mapping of the texture map to the triangular mesh model, performing seam optimization processing, and obtaining a complete texture-mapped 3D model;

[0035] Perform multi-view rendering on the optimized 3D model, set fixed view parameters, and obtain a rendered image with a resolution of 768×576. Then extract depth information from the 3D model, calculate the depth value of each pixel, and obtain a depth map of the corresponding view.

[0036] The texture maps, rendered images and depth maps are packaged and stored in a standard format to obtain a 3D reconstruction dataset.

[0037] As a preferred solution of the method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning of the present invention, the depth map generation method is used to analyze the obtained 3D dense point cloud model to obtain the depth map and rendered image corresponding to each image. The specific steps are as follows:

[0038] Perform perspective analysis on the textured 3D point cloud, select a specific observation perspective, and obtain the visible surface point set under that perspective;

[0039] Project the visible surface points onto the virtual imaging plane, calculate the depth value of each point, and obtain the initial depth map;

[0040] Fill the holes in the initial depth map and use the neighboring point interpolation algorithm to obtain a complete depth map;

[0041] Perform perspective rendering based on 3D point cloud texture information, set lighting parameters, and obtain realistic rendered images;

[0042] Align the depth map with the rendered image, check the correspondence, and obtain a matching image pair;

[0043] The generated depth map is filtered using a bilateral filtering algorithm to obtain a smooth depth map;

[0044] The processed depth map and rendered image are packaged and the camera parameter information is added to obtain the final training data sample.

[0045] As a preferred solution of the method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning of the present invention, wherein: the feature pyramid network FPN combined with the Transformer method is used to extract features from the input multi-view image to obtain a multi-scale feature map;

[0046] The input image is fed into the ResNet backbone network and processed sequentially through five convolutional layers to obtain five feature maps C1 to C5 of different scales.

[0047] Perform 1×1 convolution on the C5 feature map to obtain the initial P5 feature map;

[0048] The P5 feature map is upsampled by 2 times and then concatenated with the C4 feature map, and then processed by 3×3 convolution to obtain the P4 feature map;

[0049] The P4 feature map is upsampled by 2 times and then concatenated with the C3 feature map, and then processed by 3×3 convolution to obtain the P3 feature map;

[0050] The P3 feature map is upsampled by 2 times and then concatenated with the C2 feature map, and then processed by 3×3 convolution to obtain the P2 feature map;

[0051] Divide the P2 to P4 feature maps into 7×7 image blocks and construct the input matrix X;

[0052] The input matrix X is input to the Transformer encoder and processed by 6 sub-encoders to obtain the optimized multi-scale feature map.

[0053] As a preferred solution of the method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning described in the present invention, wherein: the cost volume is constructed by using differentiable homography transformation and variance cost aggregation method to obtain the multi-scale feature map to obtain a probability volume, and the specific steps are as follows:

[0054] Establish a depth hypothesis plane in the direction of the reference camera cone, set the depth range and resolution, and obtain D depth planes;

[0055] For each depth plane, calculate the homography transformation matrix between the source image and the reference image to obtain the transformation matrix;

[0056] Project the source image feature map onto each depth plane through homography transformation to obtain the feature volume;

[0057] Calculate the variance of each feature body and the reference feature body, and aggregate them to obtain the initial cost body;

[0058] The initial cost volume is input into the 3D convolutional U-Net network, and after 4 layers of downsampling and upsampling, the regularized cost volume is obtained;

[0059] Apply the softmax operation in the depth direction to the regularized cost volume to obtain the probability volume.

[0060] As a preferred solution of the method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning of the present invention, the method of using depth map estimation and optimization method to process the obtained probability volume to obtain a high-precision 3D reconstruction model is specifically performed as follows:

[0061] Perform weighted sum calculation on the probability volume along the depth direction to obtain the initial depth map;

[0062] Concatenate the initial depth map and the reference image feature map channel by channel to construct the optimized input;

[0063] The optimized input is processed through three 32-channel 2D convolutional layers to obtain intermediate features, and the intermediate features are subjected to 1-channel convolution to obtain the depth residual map;

[0064] Add the initial depth map and the depth residual map to obtain an optimized depth map, and perform bilateral filtering on the optimized depth map to obtain the final depth map;

[0065] The multi-view depth maps are fused and Poisson surface reconstruction is performed to obtain a high-precision 3D mesh model.

[0066] In a second aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the method for three-dimensional reconstruction of cultural relics and artifacts based on multi-views and deep learning as described in the first aspect of the present invention is implemented.

[0067] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the method for three-dimensional reconstruction of cultural relics and artifacts based on multi-views and deep learning as described in the first aspect of the present invention.

[0068] The beneficial effects of the present invention are as follows: by adopting a multi-perspective shooting method to collect images of cultural relics and handicrafts, the geometric shape, texture details and other information of the cultural relics and handicrafts are fully captured, the information missing problem caused by a single perspective is effectively solved, and the authenticity and accuracy of the 3D reconstruction model are improved; by adopting a sparse reconstruction method to perform feature matching and camera parameter estimation on a multi-perspective image set, an accurate 3D spatial position description of the cultural relics and handicrafts is achieved, the robustness of the 3D reconstruction process is enhanced, and errors caused by factors such as lighting changes and occlusion are reduced, thereby obtaining more accurate sparse point cloud data, which is crucial to the geometric accuracy of the final model; by adopting a dense reconstruction method to perform fine processing on the sparse point cloud data, a 3D dense point cloud model with high-resolution texture mapping is obtained, which not only increases the detail richness of the model, but also improves its appearance texture, so that the final 3D model has both high geometric accuracy and true color reproduction capability, and is suitable for various application scenarios such as cultural heritage protection and digital museums. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 This is a flowchart of the method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning in Example 1. DETAILED DESCRIPTION

[0071] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0072] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0073] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0074] Example 1, with reference to Figure 1 , which is the first embodiment of the present invention, provides a method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning, comprising the following steps:

[0075] S1. Use multi-view shooting method to collect images of cultural relics and handicrafts to obtain a structured image dataset;

[0076] Furthermore, five categories of cultural relics and handicrafts, including Buddha statues, clothing, ornaments, ceramics and daily necessities, were selected as the subjects for the photography;

[0077] The subject is placed on a rotatable platform, and a fixed camera is used to photograph each subject from multiple angles in a 180-degree surround manner, with no less than 20 original images collected for each subject;

[0078] Screen the collected original images and remove invalid images that are blurred, blocked or have abnormal exposure;

[0079] The screened valid images were subjected to resolution standardization and uniformly adjusted to high-resolution images of 768 × 576 pixels;

[0080] Perform color correction and white balance processing on the standardized image;

[0081] The processed multi-view images are classified and stored according to the shooting objects to form a structured image dataset;

[0082] It should be noted that the multi-view shooting method can not only fully capture the appearance characteristics of cultural relics and artifacts, but also minimize the impact of external factors on image quality by controlling the shooting environment and parameter settings, thereby providing high-quality data support for subsequent three-dimensional reconstruction. In addition, the use of a rotatable platform combined with a fixed camera for image acquisition helps to ensure consistency and coherence between images and improve reconstruction accuracy.

[0083] S2. Use sparse reconstruction method to perform feature matching and camera parameter estimation on the obtained multi-view image set to obtain camera intrinsic parameters, extrinsic parameters and sparse point cloud data;

[0084] Furthermore, the structured image dataset was fed into the COLMAP software to start the sparse reconstruction process;

[0085] The SIFT feature extraction algorithm is used to detect feature points in each image and extract scale-invariant feature descriptors;

[0086] Perform feature matching between images based on feature descriptors and establish feature correspondence between multi-view images;

[0087] The feature matching includes using a kd tree to accelerate the nearest neighbor search, adopting a bidirectional matching strategy to eliminate mismatches, and applying a RANSAC algorithm to eliminate abnormal matching point pairs;

[0088] Execute an incremental structure-from-motion (SfM) algorithm, which sequentially includes selecting an initial image pair for relative pose estimation, computing an initial 3D point cloud through triangulation, and gradually adding new images for bundle adjustment optimization.

[0089] Output camera parameters, including an intrinsic parameter matrix and an extrinsic parameter matrix, and generate sparse point cloud data;

[0090] The sparse point cloud data includes three-dimensional space point coordinates, point cloud color information and point cloud visibility information;

[0091] It should be noted that sparse reconstruction is a crucial step in the 3D reconstruction process. By performing feature matching and camera parameter estimation on multi-view image sets, basic data is provided for subsequent dense reconstruction. COLMAP software, with its powerful automated processing capabilities and high-precision reconstruction effects, has become an ideal choice for implementing this process, and can effectively improve reconstruction efficiency and accuracy.

[0092] S3, using a dense reconstruction method to refine the obtained sparse point cloud data to obtain a 3D dense point cloud model with texture mapping;

[0093] Furthermore, OpenMVS is used for dense reconstruction processing, and multi-view images and camera parameter files are input to obtain the initial dense point cloud;

[0094] The initial dense point cloud is meshed and the Delaunay triangulation algorithm is applied to obtain a fine triangular mesh model.

[0095] Calculate texture maps based on multi-view images and use the best view selection method to obtain high-resolution texture maps;

[0096] UV mapping of the texture map to the triangular mesh model, performing seam optimization processing, and obtaining a complete texture-mapped 3D model;

[0097] Perform multi-view rendering on the optimized 3D model, set fixed view parameters, and obtain a rendered image with a resolution of 768×576. Then extract depth information from the 3D model, calculate the depth value of each pixel, and obtain a depth map of the corresponding view.

[0098] Pack the texture map, rendered image and depth map into data and store them in a standard format to obtain a 3D reconstruction data set;

[0099] It should be noted that dense reconstruction technology aims to transform sparse point clouds into 3D models with rich details and smooth surfaces, and enhance visual realism through texture mapping. As an efficient dense reconstruction tool, OpenMVS performs well in generating high-quality dense point clouds. It refines the mesh through the Delaunay triangulation algorithm and generates high-resolution texture maps through the optimal view selection method, which not only improves the geometric accuracy of the model, but also ensures the true reproduction of its color information.

[0100] S4, analyzing the obtained 3D dense point cloud model using a depth map generation method to obtain a depth map and a rendered image corresponding to each image;

[0101] Furthermore, the textured 3D point cloud is subjected to perspective analysis, a specific observation perspective is selected, and the visible surface point set under that perspective is obtained;

[0102] Project the visible surface points onto the virtual imaging plane, calculate the depth value of each point, and obtain the initial depth map;

[0103] Fill the holes in the initial depth map and use the neighboring point interpolation algorithm to obtain a complete depth map;

[0104] Perform perspective rendering based on 3D point cloud texture information, set lighting parameters, and obtain realistic rendered images;

[0105] Align the depth map with the rendered image, check the correspondence, and obtain a matching image pair;

[0106] The generated depth map is filtered using a bilateral filtering algorithm to obtain a smooth depth map;

[0107] The processed depth map and rendered image are packaged and the camera parameter information is added to obtain the final training data sample;

[0108] It should be noted that depth map generation, as one of the important links in three-dimensional reconstruction, directly affects the quality and application value of the final 3D model. By selecting a specific observation perspective and calculating the depth value of the visible surface point set, and then undergoing a series of post-processing, an accurate and smooth depth map can be obtained. The depth map, combined with the rendered image, provides rich data resources for subsequent machine learning training.

[0109] S5. Use the feature pyramid network FPN combined with the Transformer method to extract features from the input multi-view image and obtain a multi-scale feature map;

[0110] Furthermore, the input image is fed into the ResNet backbone network and processed through five convolutional layers in sequence to obtain five feature maps C1 to C5 of different scales;

[0111] Perform 1×1 convolution on the C5 feature map to obtain the initial P5 feature map;

[0112] The P5 feature map is upsampled by 2 times and then concatenated with the C4 feature map, and then processed by 3×3 convolution to obtain the P4 feature map;

[0113] The P4 feature map is upsampled by 2 times and then concatenated with the C3 feature map, and then processed by 3×3 convolution to obtain the P3 feature map;

[0114] The P3 feature map is upsampled by 2 times and then concatenated with the C2 feature map, and then processed by 3×3 convolution to obtain the P2 feature map;

[0115] Divide the P2 to P4 feature maps into 7×7 image blocks and construct the input matrix X;

[0116] The input matrix X is input to the Transformer encoder and processed by 6 sub-encoders to obtain the optimized multi-scale feature map;

[0117] The Transformer encoder’s attention calculation uses the formula:

[0118]

[0119] Among them, Q, K, V represent the query matrix, key matrix and value matrix respectively, d k is the dimension of the key vector;

[0120] It should be noted that the method of combining the feature pyramid network FPN with the Transformer architecture effectively solves the problem of multi-scale feature extraction, so that detailed information at different scales can be fully utilized. The method strengthens the local feature expression ability while maintaining global information, laying a solid foundation for cost volume construction and depth map estimation in subsequent steps.

[0121] S6. Use differentiable homography transformation and variance cost aggregation method to construct the cost volume of the obtained multi-scale feature map to obtain the probability volume;

[0122] Furthermore, a depth hypothesis plane is established in the direction of the reference camera cone, and the depth interval [d2, d1] and resolution Δd are set to obtain D = (d2-d1) / Δd depth planes;

[0123] For each depth plane, calculate the homography transformation matrix between the source image and the reference image to obtain the transformation matrix, which is expressed as:

[0124]

[0125] Among them, K i 、R i , t i represents the intrinsic and extrinsic parameters of the source camera, K N 、R N , t N represents the intrinsic and extrinsic parameters of the reference camera, n N Indicates the main axis direction of the reference camera;

[0126] The source image feature map is projected onto each depth plane through homography transformation to obtain the feature volume V i ;

[0127] Calculate the variance of each feature body and the reference feature body, and aggregate them to obtain the initial cost body C, which is expressed as:

[0128]

[0129] Where N is the total number of views, V i Represents the feature body, V i Represents the mean of the feature volume

[0130] The initial cost volume is input into the 3D convolutional U-Net network, and after 4 layers of downsampling and upsampling, the regularized cost volume is obtained;

[0131] Apply the softmax operation to the regularized cost volume in the depth direction to obtain the probability volume P;

[0132] It should be noted that the process of constructing probability volumes through differentiable homography transformation and variance cost aggregation cleverly combines the advantages of geometric transformation and feature comparison, and can accurately measure the similarity between features while considering the depth changes of the scene. The method significantly improves the accuracy and robustness of multi-view stereo matching, and provides a guarantee for obtaining high-quality probability volumes and then generating fine three-dimensional reconstruction models.

[0133] S7, using a depth map estimation and optimization method to process the obtained probability volume to obtain a high-precision three-dimensional reconstruction model;

[0134] Furthermore, the probability volume is weighted and summed along the depth direction to obtain the initial depth map D, which is expressed as:

[0135]

[0136] Where D is the predicted depth map, d is the depth hypothesis value, and P(d) is the probability at depth d;

[0137] Concatenate the initial depth map D with the reference image feature map to construct the optimized input;

[0138] The optimized input is processed through three 32-channel 2D convolutional layers to obtain intermediate features, and the intermediate features are subjected to 1-channel convolution to obtain the depth residual map;

[0139] Add the initial depth map and the depth residual map to obtain an optimized depth map, and perform bilateral filtering on the optimized depth map to obtain the final depth map;

[0140] Fuse multi-view depth maps and perform Poisson surface reconstruction to obtain a high-precision 3D mesh model;

[0141] Among them, the loss function expression is:

[0142]

[0143] Among them, p is the pixel coordinate, p valid The effective pixel set, d(p) is the true depth value of pixel p, The initial depth estimate for pixel p, The optimized depth estimate of pixel p, the λ balance parameter;

[0144] It should be noted that the depth map estimation and optimization stage adopts a series of advanced algorithms and technologies to improve the accuracy and detail of the reconstruction results. The initial depth map is calculated by weighted summation, and is optimized using the depth residual map. Finally, the multi-view depth maps are fused to perform Poisson surface reconstruction. This can not only correct the errors in the initial estimation, but also further improve the model details to achieve the purpose of high-precision three-dimensional reconstruction.

[0145] This embodiment also provides a computer device, which is suitable for the three-dimensional reconstruction method of cultural relics and handicrafts based on multi-views and deep learning, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the three-dimensional reconstruction method of cultural relics and handicrafts based on multi-views and deep learning proposed in the above embodiment.

[0146] The computer device may be a terminal, comprising a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be achieved through Wi-Fi, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0147] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for three-dimensional reconstruction of cultural relics and crafts based on multi-view and deep learning as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0148] In summary, the present invention uses a multi-perspective shooting method to collect images of cultural relics and handicrafts, thereby fully capturing the geometric shape, texture details and other information of cultural relics and handicrafts, effectively solving the problem of information missing due to a single perspective, and improving the authenticity and accuracy of the 3D reconstruction model. By adopting a sparse reconstruction method to perform feature matching and camera parameter estimation on a multi-perspective image set, an accurate 3D spatial position description of the cultural relics and handicrafts is achieved, the robustness of the 3D reconstruction process is enhanced, and errors caused by factors such as lighting changes and occlusion are reduced, thereby obtaining more accurate sparse point cloud data, which is crucial to the geometric accuracy of the final model. By adopting a dense reconstruction method to perform fine processing on the sparse point cloud data, a 3D dense point cloud model with high-resolution texture mapping is obtained, which not only increases the detail richness of the model, but also improves its appearance texture, so that the final 3D model has both high geometric accuracy and true color reproduction capability, and is suitable for various application scenarios such as cultural heritage protection and digital museums.

[0149] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning, characterized by: include: A multi-view shooting method is used to capture images of cultural relics and artifacts to obtain a structured image dataset; The sparse reconstruction method is used to perform feature matching and camera parameter estimation on the obtained multi-view image set to obtain camera intrinsic parameters, extrinsic parameters and sparse point cloud data; The sparse point cloud data is refined using a dense reconstruction method to obtain a 3D dense point cloud model with texture mapping; The obtained 3D dense point cloud model is analyzed using the depth map generation method to obtain the depth map and rendered image corresponding to each image; The feature pyramid network FPN combined with the Transformer method is used to extract features from the input multi-view image to obtain a multi-scale feature map; The cost volume of the obtained multi-scale feature map is constructed using differentiable homography transformation and variance cost aggregation method to obtain a probability volume; The obtained probability volume is processed using depth map estimation and optimization methods to obtain a high-precision 3D reconstruction model.

2. The method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning according to claim 1, characterized in that: The multi-view shooting method is used to capture images of cultural relics and handicrafts to obtain a structured image dataset. The specific steps are as follows: Five categories of cultural relics and handicrafts, including Buddha statues, clothing, ornaments, ceramics, and daily necessities, were selected as the subjects for the photography; The subjects are placed on a rotatable platform, and a fixed camera is used to photograph each subject from multiple angles in a 180-degree surround manner, with no less than 20 original images collected for each subject; Screen the collected original images and remove invalid images that are blurred, blocked or have abnormal exposure; The screened valid images were subjected to resolution standardization and uniformly adjusted to high-resolution images of 768 × 576 pixels; Perform color correction and white balance processing on the standardized image; The processed multi-view images are classified and stored according to the photographed objects to form a structured image dataset.

3. The method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning according to claim 2, characterized in that: The sparse reconstruction method is used to perform feature matching and camera parameter estimation on the obtained multi-view image set to obtain camera intrinsic parameters, extrinsic parameters and sparse point cloud data. The specific steps are as follows: Input the structured image dataset into the COLMAP software and start the sparse reconstruction process; The SIFT feature extraction algorithm is used to detect feature points in each image and extract scale-invariant feature descriptors; Perform feature matching between images based on feature descriptors and establish feature correspondence between multi-view images; The feature matching includes using a kd tree to accelerate the nearest neighbor search, adopting a bidirectional matching strategy to eliminate mismatches, and applying a RANSAC algorithm to eliminate abnormal matching point pairs; Execute an incremental structure-from-motion (SfM) algorithm, which sequentially includes selecting an initial image pair for relative pose estimation, computing an initial 3D point cloud through triangulation, and gradually adding new images for bundle adjustment optimization. Output camera parameters, including an intrinsic parameter matrix and an extrinsic parameter matrix, and generate sparse point cloud data; The sparse point cloud data includes three-dimensional space point coordinates, point cloud color information, and point cloud visibility information.

4. The method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning according to claim 3, characterized in that: The dense reconstruction method is used to refine the obtained sparse point cloud data to obtain a 3D dense point cloud model with texture mapping. The specific steps are: Use OpenMVS for dense reconstruction processing, input multi-view images and camera parameter files, and obtain the initial dense point cloud; The initial dense point cloud is meshed and the Delaunay triangulation algorithm is applied to obtain a fine triangular mesh model. Calculate texture maps based on multi-view images and use the best view selection method to obtain high-resolution texture maps; UV mapping of the texture map to the triangular mesh model, performing seam optimization processing, and obtaining a complete texture-mapped 3D model; Perform multi-view rendering on the optimized 3D model, set fixed view parameters, and obtain a rendered image with a resolution of 768×576. Then extract depth information from the 3D model, calculate the depth value of each pixel, and obtain a depth map of the corresponding view. The texture maps, rendered images and depth maps are packaged and stored in a standard format to obtain a 3D reconstruction dataset.

5. The method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning according to claim 4, characterized in that: The depth map generation method is used to analyze the obtained 3D dense point cloud model to obtain the depth map and rendered image corresponding to each image. The specific steps are as follows: Perform perspective analysis on the textured 3D point cloud, select a specific observation perspective, and obtain the visible surface point set under that perspective; Project the visible surface points onto the virtual imaging plane, calculate the depth value of each point, and obtain the initial depth map; Fill the holes in the initial depth map and use the neighboring point interpolation algorithm to obtain a complete depth map; Perform perspective rendering based on 3D point cloud texture information, set lighting parameters, and obtain realistic rendered images; Align the depth map with the rendered image, check the correspondence, and obtain a matching image pair; The generated depth map is filtered using a bilateral filtering algorithm to obtain a smooth depth map; The processed depth map and rendered image are packaged and the camera parameter information is added to obtain the final training data sample.

6. The method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning according to claim 5, characterized in that: The method of using feature pyramid network FPN combined with Transformer to extract features from the input multi-view image obtains a multi-scale feature map; The input image is fed into the ResNet backbone network and processed sequentially through five convolutional layers to obtain five feature maps C1 to C5 of different scales. Perform 1×1 convolution on the C5 feature map to obtain the initial P5 feature map; The P5 feature map is upsampled by 2 times and then concatenated with the C4 feature map, and then processed by 3×3 convolution to obtain the P4 feature map; The P4 feature map is upsampled by 2 times and then concatenated with the C3 feature map, and then processed by 3×3 convolution to obtain the P3 feature map; The P3 feature map is upsampled by 2 times and then concatenated with the C2 feature map, and then processed by 3×3 convolution to obtain the P2 feature map; Divide the P2 to P4 feature maps into 7×7 image blocks and construct the input matrix X; The input matrix X is input to the Transformer encoder and processed by 6 sub-encoders to obtain the optimized multi-scale feature map.

7. The method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning according to claim 6, characterized in that: The differentiable homography transformation and variance cost aggregation method are used to construct a cost volume on the obtained multi-scale feature map to obtain a probability volume. The specific steps are as follows: Establish a depth hypothesis plane in the direction of the reference camera cone, set the depth range and resolution, and obtain D depth planes; For each depth plane, calculate the homography transformation matrix between the source image and the reference image to obtain the transformation matrix; Project the source image feature map onto each depth plane through homography transformation to obtain the feature volume; Calculate the variance of each feature body and the reference feature body, and aggregate them to obtain the initial cost body; The initial cost volume is input into the 3D convolutional U-Net network, and after 4 layers of downsampling and upsampling, the regularized cost volume is obtained; Apply the softmax operation in the depth direction to the regularized cost volume to obtain the probability volume.

8. The method for 3D reconstruction of cultural relics and handicrafts based on multi-view and deep learning according to claim 7, characterized in that: The depth map estimation and optimization method is used to process the obtained probability volume to obtain a high-precision 3D reconstruction model. The specific steps are as follows: Perform weighted sum calculation on the probability volume along the depth direction to obtain the initial depth map; Concatenate the initial depth map and the reference image feature map channel by channel to construct the optimized input; The optimized input is processed through three 32-channel 2D convolutional layers to obtain intermediate features, and the intermediate features are subjected to 1-channel convolution to obtain the depth residual map; Add the initial depth map and the depth residual map to obtain an optimized depth map, and perform bilateral filtering on the optimized depth map to obtain the final depth map; The multi-view depth maps are fused and Poisson surface reconstruction is performed to obtain a high-precision 3D mesh model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for three-dimensional reconstruction of cultural relics and artifacts based on multi-view and deep learning are implemented as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for three-dimensional reconstruction of cultural relics and handicrafts based on multi-view and deep learning are implemented.

Images