Cultural heritage virtual reconstruction display system based on element universe

Abstract

The invention belongs to the technical field of virtual reality and cultural heritage protection crossing, and particularly relates to a meta-universe-based cultural heritage virtual reconstruction display system, which comprises a data acquisition module, a virtual modeling module, a meta-universe engine module, an immersive display module, a block chain authentication module and a user behavior analysis module. According to the scheme, a new method of multi-source data acquisition and deep learning restoration is combined, a laser scanner, an unmanned aerial vehicle and a high-definition camera are used for performing high-precision data acquisition on cultural heritage from multiple angles, and an enhanced U-Net network is used for performing accurate positioning and preliminary restoration on an irregular incomplete area in an image; the learning and expression ability on complex visual features is enhanced; according to the scheme, the edge guide diffusion model is introduced, the edge information is used as prior guidance, the defect area in the cultural heritage is subjected to refined restoration, and higher-quality digital restoration and protection of the cultural heritage image are realized.

Description

Technical Field

[0001] The present invention belongs to the cross-technical field of virtual reality and cultural heritage protection, and in particular to a virtual reconstruction and display system of cultural heritage based on the metaverse. Background Art

[0002] With the advancement of science and technology, traditional cultural heritage is facing multiple challenges, including natural aging, human destruction, and environmental change. In recent years, the rise of virtual reality (VR), augmented reality (AR), artificial intelligence (AI), and the concept of the metaverse has opened up new avenues for the digital protection and display of cultural heritage. However, this process has also exposed some technical deficiencies: due to natural aging, environmental erosion, or human factors, many cultural heritage sites have cracks or defects, which makes high-precision three-dimensional modeling difficult; traditional image restoration methods have difficulty dealing with irregularly shaped and complexly distributed damaged areas, and cannot achieve accurate positioning and restoration, resulting in a lack of realism in virtual reconstruction models, affecting the user experience; in addition, traditional diffusion models are prone to edge blurring and structural distortion during the restoration process, making it difficult to preserve the original artistic style and line features, and may even cause fictitious or distorted phenomena in the modeling, thereby reducing the user's understanding and perception of cultural heritage. Summary of the Invention

[0003] In response to the above situation and to overcome the shortcomings of the existing technology, the present invention provides a virtual reconstruction and display system for cultural heritage based on the metaverse. To address the cracks and defects that may appear in cultural heritage due to natural aging, environmental erosion, or human factors, traditional image restoration methods are difficult to deal with irregularly shaped and complexly distributed damaged areas, and cannot achieve accurate positioning and restoration. This solution combines new methods of multi-source data acquisition and deep learning restoration, using laser scanners, drones, and high-definition cameras to perform high-precision data acquisition of cultural heritage from multiple angles, and adopts an enhanced U-Net network to accurately locate and preliminarily restore irregular damaged areas in the image, enhancing its ability to learn and express complex visual features. To address the problem that traditional diffusion models easily cause edge blurring and structural distortion during the restoration process, this solution introduces an edge-guided diffusion model, using edge information as a priori guidance to perform fine-grained restoration of damaged areas in cultural heritage. In terms of visual effects, this method maintains the artistic style and detail authenticity consistent with the original, achieving higher-quality digital restoration and protection of cultural heritage images, and providing users with a more realistic cultural heritage model display.

[0004] The present invention provides a virtual cultural heritage reconstruction and display system based on the metaverse, which includes a data acquisition module, a virtual modeling module, a metaverse engine module, an immersive display module, a blockchain authentication module, and a user behavior analysis module;

[0005] The data acquisition module uses laser scanners, drones, and high-definition camera equipment to collect multi-angle data of cultural heritage and obtain three-dimensional point cloud data and two-dimensional images;

[0006] The virtual modeling module uses Artec Studio and Agisoft Metashape to process the collected 3D point cloud data to generate a 3D digital model of the cultural heritage. For the incomplete parts of the cultural heritage, an enhanced U-Net network and edge-guided diffusion model are used to perform intelligent restoration based on historical photos and 2D images of the incomplete parts, restoring the historical appearance of the cultural heritage.

[0007] The Metaverse Engine module integrates Unity and Unreal Engine virtual engines to build a virtual exhibition hall for cultural heritage, supporting multi-person online access and interaction;

[0008] The immersive display module provides an immersive experience through VR devices, allowing users to walk freely in the virtual environment, interact with exhibits, and receive audio guides and virtual explanation services;

[0009] The blockchain authentication module uses blockchain technology to NFT cultural heritage digital assets to ensure their authenticity, uniqueness and tradability;

[0010] The user behavior analysis module collects user behavior data in the virtual space and optimizes the display content and recommendation mechanism through big data analysis.

[0011] Furthermore, the virtual modeling module uses an enhanced U-Net network and diffusion model to perform intelligent restoration based on historical photos and two-dimensional images of the damaged parts, specifically including the following steps:

[0012] Step S1: Data preprocessing: removing noise and outliers from the two-dimensional images of historical photos and incomplete parts, and performing standard data processing to obtain standard image data;

[0013] Step S2: Crack identification and location: using an enhanced U-Net network to generate a defect mask for the defective parts in the standard image data, marking the defective areas in the standard image data;

[0014] Step S3: Edge-guided repair: Use a mask covering 70% of the damaged area to mask the standard image data, use the Canny algorithm to extract the image edge, generate an edge mask, fuse the damaged mask and the edge mask, and finally generate a fused mask;

[0015] Step S4: Diffusion model repair, using the diffusion model to remove noise and repair the incomplete areas of the standard image data to obtain a repaired image;

[0016] Step S5: outputting the restored image and using it to generate a three-dimensional digital model of the cultural heritage;

[0017] Furthermore, in step S2, crack identification and location specifically include the following steps:

[0018] Step S21: Encoder processing, introduce window multi-head self-attention and shift window multi-head self-attention into the encoder of the U-Net network, gradually compress the standard image data resolution to 1 / 16 of the original size, extract multi-scale features, output bottleneck features, and pass them to the decoder. The formula used is as follows

[0019] A w (Q,K,V)=Concat(h1,h2,…,h m )W o ;

[0020] A sw =Shift(A w (Shift(Z)));

[0021] Where A w represents the window multi-head self-attention mechanism, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, Concat represents the splicing operation, h1,h2,…,h m Represents the output of each head in the multi-head self-attention mechanism, W o is the linear transformation matrix, A sw Represents the shift window multi-head self-attention mechanism, Shift is the shift operation, and Z represents the input image;

[0022] Step S22: Decoder reconstruction, using a deconvolution upsampling module to perform three times 2-fold upsampling, and constructing a decoder by linear mapping through 4-fold upsampling to generate a high-resolution defect mask;

[0023] Step S23: Deep supervision strategy, adding supervision signals to different network layers in the U-Net network encoder, calculating additional loss functions, and accelerating model convergence. The formula used is as follows:

[0024]

[0025] Where, L tot Represents the total loss function, i represents the subscript of the number of network layers in the encoder, L represents the number of network layers in the encoder, α i is the weight coefficient, S i is the supervisory signal, P i is the true label, Indicates that S is calculated using the L2 norm i and P iThe square of the error between

[0026] Furthermore, in step S4, the diffusion model is repaired, specifically including the following steps:

[0027] Step S41: adding noise to the standard image data containing the defective area to generate a noisy image. The formula used is as follows:

[0028] I noise (x,y)=I(x,y)+N(x,y);

[0029] Where (x, y) represents the coordinates, I(x, y) represents the original image, N(x, y) represents the added noise, and I noise (x,y) represents a noisy image;

[0030] Step S42: training a diffusion model, training a diffusion model, using the diffusion model to learn to remove noise from the noisy image and preliminarily repair the defective area;

[0031] Step S43: Preliminary restoration: using the trained diffusion model to process the noisy image to obtain a preliminarily restored image;

[0032] Step S44: edge-guided restoration, using an edge-guided diffusion model built based on the Canny algorithm, performs detail restoration on the fusion mask containing edge information and the preliminarily restored image to obtain a final restored image.

[0033] The beneficial effects achieved by the present invention using the above scheme are as follows:

[0034] (1) In view of the cracks and defects that may appear in cultural heritage due to natural aging, environmental erosion or human factors, traditional image restoration methods are difficult to deal with irregularly shaped and complexly distributed damaged areas, and cannot achieve accurate positioning and restoration. A new method combining multi-source data acquisition and deep learning restoration uses laser scanners, drones and high-definition cameras to collect high-precision data on cultural heritage from multiple angles to obtain comprehensive and detailed structural information; and uses an enhanced U-Net network to accurately locate and preliminarily restore irregular damaged areas in the image, effectively improving the model's ability to understand local details and overall structure, and enhancing its ability to learn and express complex visual features;

[0035] (2) In view of the problem that traditional diffusion models are prone to edge blurring and structural distortion during the restoration process, making it difficult to preserve the original artistic style and line features, and even causing fictitious or deformed phenomena in the modeling, this solution introduces an edge-guided diffusion model, using edge information as a priori guidance to perform fine-grained restoration of the damaged areas in cultural heritage. This method not only improves the structural accuracy and edge continuity of the restoration area, but also maintains the artistic style and detail authenticity consistent with the original in terms of visual effects, achieving higher-quality digital restoration and protection of cultural heritage images, and providing users with a more realistic cultural heritage model display. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of a virtual cultural heritage reconstruction and display system based on the metaverse proposed by the present invention;

[0037] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] Example 1, see Figure 1 The present invention provides a virtual cultural heritage reconstruction and display system based on the metaverse, which includes a data acquisition module, a virtual modeling module, a metaverse engine module, an immersive display module, a blockchain authentication module, and a user behavior analysis module;

[0040] The data acquisition module uses laser scanners, drones, and high-definition camera equipment to collect multi-angle data of cultural heritage and obtain three-dimensional point cloud data and two-dimensional images;

[0041] The virtual modeling module processes the collected 3D point cloud data to generate a 3D digital model of the cultural heritage. For the incomplete parts of the cultural heritage, an enhanced U-Net network and an edge-guided diffusion model are used to perform intelligent restoration based on historical photos and 2D images of the incomplete parts, thereby restoring the historical appearance of the cultural heritage.

[0042] The Metaverse Engine module integrates Unity and Unreal Engine virtual engines to build a virtual exhibition hall for cultural heritage, supporting multi-person online access and interaction;

[0043] The immersive display module provides an immersive experience through VR devices, allowing users to walk freely in the virtual environment, interact with exhibits, and receive audio guides and virtual explanation services;

[0044] The blockchain authentication module uses blockchain technology to NFT cultural heritage digital assets to ensure their authenticity, uniqueness and tradability;

[0045] The user behavior analysis module collects user behavior data in the virtual space and optimizes the display content and recommendation mechanism through big data analysis.

[0046] Example 2: This example is based on the above example. The Metaverse Engine module integrates Unity and Unreal Engine virtual engines to build a highly restored virtual exhibition hall of cultural heritage. Through the powerful graphics rendering capabilities and physical simulation systems of these two engines, the realistic lighting effects, material performance and dynamic interactive experience of the cultural heritage scene are realized. The module supports high-precision 3D modeling import and optimization to ensure the realistic reproduction of cultural relics details, integrates a multi-person online access system, and supports real-time user interaction and voice communication.

[0047] The immersive display module is combined with the cultural heritage virtual exhibition hall constructed by the Metaverse Engine module, which provides users with an immersive cultural heritage visiting experience through VR devices, allowing users to walk freely in the virtual environment, rotate the perspective, and zoom in to view the details of the exhibits. It also supports gesture recognition and handle operation to achieve touch interaction with virtual cultural relics, and adopts a low-latency network transmission mechanism to ensure the smoothness of the synchronous experience of multiple people.

[0048] Example 3: Based on the above example, the blockchain authentication module generates a unique NFT identifier for each cultural heritage digital model and stores it on a decentralized blockchain to ensure data security and transparency. It is integrated with the Metaverse exhibition hall, allowing users to purchase and display NFT collections directly in the virtual space, promoting the digital confirmation of ownership and transaction circulation of cultural assets.

[0049] The user behavior analysis module collects user behavior data in the virtual space, including browsing paths, dwell time, click hotspots, and interaction frequency, collects user behavior logs in real time and stores them in a data warehouse, uses big data analysis and artificial intelligence algorithms to mine user preferences, analyze user interest maps, identify popular exhibits and unpopular areas, thereby optimizing display content and recommendation strategies, and improving user experience and engagement.

[0050] Example 4: Based on the above example, the virtual modeling module uses an enhanced U-Net network and a diffusion model to perform intelligent restoration based on historical photos and two-dimensional images of the incomplete parts. Specifically, the virtual modeling module includes the following steps:

[0051] Step S1: Data preprocessing: removing noise and outliers from the two-dimensional images of historical photos and incomplete parts, and performing standard data processing to obtain standard image data;

[0052] Step S2: Crack identification and location: using an enhanced U-Net network to generate a defect mask for the defective parts in the standard image data, marking the defective areas in the standard image data;

[0053] Step S3: Edge-guided repair: Use a mask covering 70% of the damaged area to mask the standard image data, and use the Canny algorithm to extract the image edge, generate an edge mask, fuse the defect mask and the edge mask, and finally generate a fused mask. The formula used is as follows:

[0054]

[0055] M com (x,y)=max(M edge (x,y),M(x,y));

[0056] Where M can (x, y) represents the Canny edge detection result of the pixel at the coordinate (x, y), E(x, y) represents the edge intensity of the pixel at the coordinate (x, y), T h Indicates strong edge strength, T l Indicates weak edge strength, M com (x,y) represents the fusion mask, M edge (x,y) is the edge mask, M(x,y) is the defect mask, and max() means taking the maximum value;

[0057] Step S4: Diffusion model repair, using the diffusion model to remove noise and repair the incomplete areas of the standard image data to obtain a repaired image;

[0058] Step S5: Image output, outputting the restored image and using it to generate a three-dimensional digital model of the cultural heritage.

[0059] By performing the above operations, traditional image restoration methods are unable to deal with the problem of cracks and defects that may appear in cultural heritage due to natural aging, environmental erosion or human factors, and are unable to achieve accurate positioning and restoration. A new method combining multi-source data collection and deep learning restoration is used to use laser scanners, drones and high-definition cameras to collect high-precision data on cultural heritage from multiple angles to obtain comprehensive and detailed structural information; and an enhanced U-Net network is used to accurately locate and preliminarily restore irregular and damaged areas in the image, effectively improving the model's ability to understand local details and overall structure, and enhancing its ability to learn and express complex visual features.

[0060] Example 5: This example is based on the above example. In step S2, crack identification and location specifically include the following steps:

[0061] Step S21: Encoder processing, introduce window multi-head self-attention and shift window multi-head self-attention into the encoder of the U-Net network, gradually compress the standard image data resolution to 1 / 16 of the original size, extract multi-scale features, output bottleneck features, and pass them to the decoder. The formula used is as follows

[0062] A w (Q,K,V)=Concat(h1,h2,…,h m )W o ;

[0063] A sw =Shift(A w (Shift(Z)));

[0064] Where A w represents the window multi-head self-attention mechanism, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, Concat represents the splicing operation, h1,h2,…,h m Represents the output of each head in the multi-head self-attention mechanism, W o is the linear transformation matrix, A sw Represents the shift window multi-head self-attention mechanism, Shift is the shift operation, and Z represents the input image;

[0065] Step S22: Decoder reconstruction, using a deconvolution upsampling module to perform three times 2-fold upsampling, and constructing a decoder by linear mapping through 4-fold upsampling to generate a high-resolution defect mask;

[0066] Step S23: Deep supervision strategy, adding supervision signals to different network layers in the U-Net network encoder, calculating additional loss functions, and accelerating model convergence. The formula used is as follows:

[0067]

[0068] Where, L tot Represents the total loss function, i represents the subscript of the number of network layers in the encoder, L represents the number of network layers in the encoder, α i is the weight coefficient, S i is the supervisory signal, P i is the true label, Indicates that S is calculated using the L2 norm i and P i The square of the error between

[0069] By performing the operations described,

[0070] Example 6: This example is based on the above example. In step S4, the diffusion model is repaired, specifically including the following steps:

[0071] Step S41: adding noise to the standard image data containing the defective area to generate a noisy image. The formula used is as follows:

[0072] I noise (x,y)=I(x,y)+N(x,y);

[0073] Where (x, y) represents the coordinates, I(x, y) represents the original image, N(x, y) represents the added noise, and I noise (x,y) represents a noisy image;

[0074] Step S42: training a diffusion model, training a diffusion model, using the diffusion model to learn to remove noise from the noisy image and preliminarily repair the defective area;

[0075] Step S43: Preliminary restoration: using the trained diffusion model to process the noisy image to obtain a preliminarily restored image;

[0076] Step S44: edge-guided restoration, using an edge-guided diffusion model based on the Canny algorithm, performs detail restoration on the fusion mask containing edge information and the preliminarily restored image to obtain the final restored image. The formula used is as follows:

[0077]

[0078] Where, is the final repaired image, I(x,y) is the image after preliminary repair, and H() is the edge-guided diffusion model.

[0079] In view of the problems that traditional diffusion models are prone to causing edge blur and structural distortion during the restoration process, making it difficult to preserve the original artistic style and line features, and even causing fictitious or deformation phenomena in the modeling, this solution introduces an edge-guided diffusion model, using edge information as a priori guidance to perform fine-grained restoration of defective areas in cultural heritage. This method not only improves the structural accuracy and edge continuity of the repaired area, but also maintains the artistic style and detail authenticity consistent with the original in terms of visual effects, achieving higher-quality digital restoration and protection of cultural heritage images, and providing users with a more realistic cultural heritage model display.

[0080] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0081] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0082] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A virtual cultural heritage reconstruction and display system based on the Metaverse, characterized by: The system includes a data acquisition module, a virtual modeling module, a metaverse engine module, an immersive display module, a blockchain authentication module, and a user behavior analysis module; The data acquisition module uses laser scanners, drones, and high-definition camera equipment to collect multi-angle data of cultural heritage and obtain three-dimensional point cloud data and two-dimensional images; The virtual modeling module processes the collected 3D point cloud data to generate a 3D digital model of the cultural heritage. For the incomplete parts of the cultural heritage, an enhanced U-Net network and an edge-guided diffusion model are used to perform intelligent restoration based on historical photos and 2D images of the incomplete parts, thereby restoring the historical appearance of the cultural heritage. The Metaverse engine module constructs a virtual exhibition hall for cultural heritage, supporting multi-person online access and interaction; The immersive display module provides an immersive experience through VR devices, allowing users to walk freely in the virtual environment, interact with exhibits, and receive audio guides and virtual explanation services; The blockchain authentication module uses blockchain technology to NFT cultural heritage digital assets to ensure their authenticity, uniqueness and tradability; The user behavior analysis module collects user behavior data in the virtual space and optimizes the display content and recommendation mechanism through big data analysis.

2. The metaverse-based cultural heritage virtual reconstruction and display system according to claim 1, characterized in that: The virtual modeling module uses an enhanced U-Net network and diffusion model to perform intelligent restoration based on historical photos and two-dimensional images of the damaged parts, specifically including the following steps: Step S1: Data preprocessing: removing noise and outliers from the two-dimensional images of historical photos and incomplete parts, and performing standard data processing to obtain standard image data; Step S2: Crack identification and location: using an enhanced U-Net network to generate a defect mask for the defective parts in the standard image data, marking the defective areas in the standard image data; Step S3: Edge-guided repair: Use a mask covering 70% of the damaged area to mask the standard image data, use the Canny algorithm to extract the image edge, generate an edge mask, fuse the damaged mask and the edge mask, and finally generate a fused mask; Step S4: Diffusion model repair, using the diffusion model to remove noise and repair the incomplete areas of the standard image data to obtain a repaired image; Step S5: Image output, outputting the restored image and using it to generate a three-dimensional digital model of the cultural heritage.

3. The cultural heritage virtual reconstruction and display system based on the metaverse according to claim 2, characterized in that: In step S2, crack identification and location specifically include the following steps: Step S21: Encoder processing: Introduce window multi-head self-attention and shifted window multi-head self-attention into the encoder of the U-Net network, gradually compress the standard image data resolution to 1 / 16 of the original size, extract multi-scale features, output bottleneck features, and pass them to the decoder; Step S22: Decoder reconstruction, using a deconvolution upsampling module to perform three times 2-fold upsampling, and constructing a decoder by linear mapping through 4-fold upsampling to generate a high-resolution defect mask; Step S23: Deep supervision strategy, adding supervision signals to different network layers in the U-Net network encoder, calculating additional loss functions, and accelerating model convergence. The formula used is as follows: Where, L tot Represents the total loss function, i represents the subscript of the number of network layers in the encoder, L represents the number of network layers in the encoder, α i is the weight coefficient, S i is the supervisory signal, P i is the true label, Indicates that S is calculated using the L2 norm i and P i The square of the error between .

4. The cultural heritage virtual reconstruction and display system based on the metaverse according to claim 2, characterized in that: In step S4, the diffusion model is repaired, which specifically includes the following steps: Step S41: adding noise to the standard image data containing the defective area to generate a noisy image; Step S42: training a diffusion model, training a diffusion model, using the diffusion model to learn to remove noise from the noisy image and preliminarily repair the defective area; Step S43: Preliminary restoration: using the trained diffusion model to process the noisy image to obtain a preliminarily restored image; Step S44: edge-guided restoration, using an edge-guided diffusion model built based on the Canny algorithm, performs detail restoration on the fusion mask containing edge information and the preliminarily restored image to obtain a final restored image.

Images