A cultural heritage virtual reconstruction display system based on a meta universe

Abstract

The application belongs to the technical field of virtual reality and cultural heritage protection, in particular to a cultural heritage virtual reconstruction display system based on a meta universe, which comprises a data acquisition module, a virtual modeling module, a meta universe engine module, an immersive display module, a blockchain authentication module and a user behavior analysis module; the scheme combines a new method of multi-source data acquisition and deep learning repair, uses a laser scanner, a drone and a high-definition camera to collect high-precision data of cultural heritage from multiple angles, and adopts an enhanced U-Net network to accurately position and preliminarily restore irregular damaged areas in the image, thereby enhancing the learning and expression ability of the network to complex visual features; the scheme introduces an edge-guided diffusion model, uses edge information as prior guidance, finely repairs the damaged areas in the cultural heritage, and realizes higher-quality digital repair and protection of the cultural heritage image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of virtual reality and cultural heritage protection, and particularly relates to a cultural heritage virtual reconstruction display system based on a meta universe. BACKGROUND

[0002] With the progress of science and technology, traditional cultural heritage is facing multiple challenges such as natural aging, human damage and environmental changes. In recent years, the rise of virtual reality (VR), augmented reality (AR), artificial intelligence (AI) and the concept of meta universe has opened up new paths for the digital protection and display of cultural heritage. However, some technical deficiencies have also been exposed in this process: due to natural aging, environmental erosion or human factors, many cultural heritage have cracks or defects, making it difficult to achieve high-precision three-dimensional modeling; traditional image restoration methods are difficult to cope with irregular shape and complex distribution of damaged areas, and cannot achieve accurate positioning and restoration, resulting in a lack of realism in the virtual reconstruction model and affecting user experience; in addition, traditional diffusion models are prone to edge blurring and structural distortion during the restoration process, making it difficult to preserve the original artistic style and line features, and even leading to fabrication or deformation in modeling, thereby reducing user understanding and perception of cultural heritage. SUMMARY

[0003] In view of the above, in order to overcome the defects of the prior art, the present application provides a cultural heritage virtual reconstruction display system based on a meta universe, which is aimed at the problem of cracks and defects that may occur in cultural heritage due to natural aging, environmental erosion or human factors, and the problem that traditional image restoration methods are difficult to cope with irregular shape and complex distribution of damaged areas, and cannot achieve accurate positioning and restoration. The present application combines multi-source data acquisition and deep learning restoration, uses a laser scanner, a drone and a high-definition camera to collect high-precision data of cultural heritage from multiple angles, and uses an enhanced U-Net network to accurately position and preliminarily restore irregular damaged areas in the image, thereby enhancing its learning and expression ability of complex visual features; in view of the problem that traditional diffusion models are prone to edge blurring and structural distortion during the restoration process, the present application introduces an edge-guided diffusion model, which uses edge information as prior guidance to finely repair the damaged areas in the cultural heritage. This method maintains the artistic style and detail realism consistent with the original work in visual effect, realizes higher quality digital restoration and protection of cultural heritage images, and provides users with a more realistic cultural heritage model display.

[0004] The cultural heritage virtual reconstruction display system based on a meta universe provided by the present application comprises a data acquisition module, a virtual modeling module, a meta universe engine module, an immersive display module, a blockchain authentication module and a user behavior analysis module.

[0005] The data acquisition module uses a laser scanner, a drone, and a high-definition camera device to collect multi-angle data of cultural heritage, obtaining three-dimensional point cloud data and two-dimensional images.

[0006] The virtual modeling module processes the collected three-dimensional point cloud data using Artec Studio and Agisoft Metashape to generate a three-dimensional digital model of the cultural heritage. For incomplete parts of the cultural heritage, an enhanced U-Net network and an edge-guided diffusion model are used to intelligently repair the two-dimensional images based on historical photos and incomplete parts, achieving historical restoration of the cultural heritage.

[0007] The Meta Universe Engine module integrates Unity and Unreal Engine virtual engines to build a virtual exhibition hall for cultural heritage, supporting online access and interaction for multiple users.

[0008] The immersive display module provides immersive experiences through VR devices, allowing users to freely walk in the virtual environment, interact with exhibits, and receive voice-guided tours and virtual explanations.

[0009] The blockchain authentication module uses blockchain technology to process NFTs for cultural heritage digital assets, ensuring their authenticity, uniqueness, and tradability.

[0010] The user behavior analysis module collects user behavior data in the virtual space and optimizes display content and recommendation mechanisms through big data analysis.

[0011] Further, the virtual modeling module uses an enhanced U-Net network and a diffusion model to intelligently repair two-dimensional images based on historical photos and incomplete parts, including the following steps:

[0012] Step S1: Data preprocessing, removing noise and outliers from the two-dimensional images of historical photos and incomplete parts, and performing standardized data processing to obtain standard image data.

[0013] Step S2: Crack identification and positioning, using an enhanced U-Net network to generate a missing mask for the incomplete parts in the standard image data, marking the incomplete areas in the standard image data.

[0014] Step S3: Edge-guided repair, using a mask covering 70% of the damaged area to mask the standard image data, and using the Canny algorithm to extract image edges to generate an edge mask. Fuse the missing mask and the edge mask to finally generate a fused mask.

[0015] Step S4: Diffusion model repair, using a diffusion model to remove noise and repair the damaged areas of the standard image data to obtain the repaired image.

[0016] Step S5: image output, output the repaired image and use it to generate a three-dimensional digital model of the cultural heritage;

[0017] Further, in step S2, crack identification and positioning, specifically comprising the following steps:

[0018] Step S21: encoder processing, introducing window multi-head self-attention and shift window multi-head self-attention in the encoder of the U-Net network, gradually compressing the standard image data resolution to 1 / 16 of the original size, extracting multi-scale features, outputting bottleneck features, and passing them to the decoder, the formula used is as follows

[0019] A w (Q,K,V)=Concat(h1,h2,…,h m )W o ;

[0020] A sw =Shift(A w (Shift(Z)));

[0021] In the formula, A w represents the window multi-head self-attention mechanism, Q, K and V represent the query matrix, key matrix and value matrix respectively, Concat represents the concatenation operation, h1, h2, …, h m represent the output of each head in the multi-head self-attention mechanism, W o is a linear transformation matrix, A sw represents the shift window multi-head self-attention mechanism, Shift is the shift operation, and Z represents the input image.

[0022] Step S22: decoder reconstruction, using the deconvolution upsampling module to perform three times 2 times upsampling, and constructing the decoder through 4 times upsampling linear mapping to generate high-resolution incomplete mask.

[0023] Step S23: deep supervision strategy, adding supervision signals in different network layers in the U-Net network encoder, calculating additional loss functions, and accelerating model convergence, the formula used is as follows:

[0024]

[0025] In the formula, L tot represents the total loss function, i represents the subscript of the number of network layers in the encoder, L represents the number of network layers in the encoder, α i is the weight coefficient, S i is the supervision signal, P i is the true label, represents that S i and P iError square between them;

[0026] Further, in step S4, the diffusion model repair, specifically including the following steps:

[0027] Step S41: add noise, add noise to the standard image data containing the missing area, generate a noisy image, the formula is as follows:

[0028] I noise (x,y)=I(x,y)+N(x,y);

[0029] In the formula, (x,y) represents the coordinates, I(x,y) represents the original image, N(x,y) represents the added noise, I noise (x,y) represents the noisy image;

[0030] Step S42: train the diffusion model, train a diffusion model, use the diffusion model to learn to remove noise from the noisy image, and preliminarily repair the missing area;

[0031] Step S43: preliminary repair, using the trained diffusion model to process the noisy image, and obtaining the preliminary repaired image;

[0032] Step S44: edge-guided repair, using the edge-guided diffusion model constructed based on the Canny algorithm, the fusion mask containing edge information and the preliminary repaired image are used for detail repair, and the final repaired image is obtained.

[0033] The beneficial effects obtained by the above scheme are as follows:

[0034] (1) For the cracks and defects of cultural heritage under the influence of natural aging, environmental erosion or human factors, the traditional image repair method is difficult to deal with the irregular shape and complex distribution of the missing area, and cannot realize the accurate positioning and restoration. Combined with the new method of multi-source data acquisition and deep learning repair, the laser scanner, unmanned aerial vehicle and high-definition camera are used to collect high-precision data of cultural heritage from multiple angles, and obtain comprehensive and fine structure information; and the enhanced U-Net network is used to accurately position and preliminarily restore the irregular missing area in the image, effectively improve the understanding ability of the model to local details and overall structure, and enhance the learning and expression ability of the model to complex visual features;

[0035] (2) In order to solve the problems that the traditional diffusion model is easy to cause edge blur and structure distortion in the repair process, it is difficult to retain the original artistic style and line features, and even the phenomenon of fabrication or deformation may occur in modeling, the edge guided diffusion model is introduced, the edge information is used as prior guidance, and the fine repair is carried out on the damaged area in the cultural heritage, the method not only improves the structure accuracy and edge continuity of the repaired area, but also maintains the artistic style and detail reality consistent with the original work in visual effect, realizes the higher quality digital repair and protection of the cultural heritage image, and provides more real cultural heritage model display for users. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A schematic diagram of a cultural heritage virtual reconstruction display system based on the metaverse is provided for the present application.

[0037] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation on the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0039] Embodiment one, refer to Figure 1 The present application provides a cultural heritage virtual reconstruction display system based on the metaverse, which comprises a data acquisition module, a virtual modeling module, a metaverse engine module, an immersive display module, a blockchain authentication module and a user behavior analysis module.

[0040] The data acquisition module uses a laser scanner, a drone and a high-definition camera device to collect multi-angle data of cultural heritage and obtain three-dimensional point cloud data and two-dimensional images.

[0041] The virtual modeling module processes the collected three-dimensional point cloud data to generate a three-dimensional digital model of the cultural heritage. For the damaged parts of the cultural heritage, an enhanced U-Net network and an edge guided diffusion model are used for intelligent repair according to historical photos and two-dimensional images of damaged parts to realize the restoration of the historical appearance of the cultural heritage.

[0042] The metaverse engine module integrates Unity and Unreal Engine virtual engines to build a virtual exhibition hall of cultural heritage, supporting online access and interaction of multiple people.

[0043] The immersive display module provides an immersive experience through the VR device, allowing users to freely walk in the virtual environment, interact with exhibits, and obtain voice guide and virtual explanation services;

[0044] The blockchain authentication module uses blockchain technology to process NFT of cultural heritage digital assets, ensuring their authenticity, uniqueness, and tradability.

[0045] The user behavior analysis module collects user behavior data in the virtual space and optimizes display content and recommendation mechanisms through big data analysis.

[0046] Embodiment Two, based on the above embodiment, the Meta Universe Engine module integrates Unity and UnrealEngine virtual engines to build a highly restored virtual cultural heritage exhibition hall. Through the powerful graphics rendering capabilities and physical simulation systems of these two engines, the module achieves realistic lighting effects, material performance, and dynamic interaction experience for cultural heritage scenes. The module supports high-precision 3D modeling import and optimization to ensure the realistic reproduction of cultural relic details, integrates a multi-user online access system, and supports real-time user interaction and voice communication.

[0047] The immersive display module, combined with the cultural heritage virtual exhibition hall built by the Meta Universe Engine module, provides users with an immersive cultural heritage visit experience through VR devices, allowing them to freely walk, rotate their perspective, and zoom in to view exhibit details in the virtual environment. The module also supports gesture recognition and controller operation, enabling touch interaction with virtual cultural relics, and uses low-latency network transmission mechanisms to ensure smooth multi-user synchronous experience.

[0048] Embodiment Three, based on the above embodiment, the blockchain authentication module generates a unique NFT identifier for each cultural heritage digital model and stores it on a decentralized blockchain, ensuring data security and transparency. The module integrates with the Meta Universe exhibition hall, allowing users to directly purchase and display NFT collections in the virtual space, promoting the digital rights protection and transaction circulation of cultural assets.

[0049] The user behavior analysis module collects user behavior data in the virtual space, including browsing paths, dwell times, click hotspots, and interaction frequencies. It collects user behavior logs in real time and stores them in a data warehouse. Using big data analysis and artificial intelligence algorithms, the module mines user preferences, analyzes user interest graphs, identifies popular exhibits and cold areas, and optimizes display content and recommendation strategies to improve user experience and engagement.

[0050] Embodiment Four, based on the above embodiment, the virtual modeling module uses an enhanced U-Net network and diffusion model to intelligently repair historical photos and two-dimensional images of damaged parts, including the following steps:

[0051] Step S1: data preprocessing, removing noise and outliers from the two-dimensional images of historical photos and missing parts, and performing standardization data processing to obtain standard image data;

[0052] Step S2: crack identification and positioning, using an enhanced U-Net network to generate a missing mask for the missing part in the standard image data, and marking the missing area in the standard image data;

[0053] Step S3: edge-guided repair, using a mask covering 70% of the damaged area to mask the standard image data, and using the Canny algorithm to extract the image edge to generate an edge mask, and fusing the missing mask and the edge mask to finally generate a fused mask, the formula is as follows:

[0054]

[0055] M com (x,y)=max(M edge (x,y),M(x,y));

[0056] In the formula, M can (x,y) represents the Canny edge detection result of the pixel at coordinate (x,y), E(x,y) represents the edge intensity of the pixel at coordinate (x,y), T h represents the strong edge intensity, T l represents the weak edge intensity, M com (x,y) represents the fused mask, M edge (x,y) is the edge mask, M(x,y) is the missing mask, and max() represents the maximum value;

[0057] Step S4: diffusion model repair, using a diffusion model to remove noise and repair the missing area of the standard image data to obtain the repaired image;

[0058] Step S5: image output, outputting the repaired image and using it to generate a three-dimensional digital model of the cultural heritage.

[0059] By performing the operation, for the cracks and defects that may occur in cultural heritage under the influence of natural aging, environmental erosion or human factors, traditional image repair methods are difficult to cope with irregular shape and complex distribution of missing areas, and cannot realize accurate positioning and restoration. Combined with the new method of multi-source data collection and deep learning repair, the laser scanner, unmanned aerial vehicle and high-definition camera are used to collect high-precision data of cultural heritage from multiple angles to obtain comprehensive and fine structure information; and an enhanced U-Net network is used to accurately position and preliminarily restore irregular missing areas in the image, effectively improving the understanding ability of the model to local details and overall structure, and enhancing the learning and expression ability of the model to complex visual features.

[0060] In the fifth embodiment, based on the above-mentioned embodiments, in the crack identification and positioning in step S2, the following steps are specifically included:

[0061] Step S21: encoder processing, introducing window multi-head self-attention and shift window multi-head self-attention in the encoder of the U-Net network, gradually compressing the standard image data resolution to 1 / 16 of the original size, extracting multi-scale features, outputting bottleneck features, and passing them to the decoder, the formula used is as follows

[0062] A w (Q,K,V)=Concat(h1,h2,…,h m )W o ;

[0063] A sw =Shift(A w (Shift(Z)));

[0064] In the formula, A w represents the window multi-head self-attention mechanism, Q, K and V represent the query matrix, key matrix and value matrix respectively, Concat represents the concatenation operation, h1, h2, …, h m represent the output of each head in the multi-head self-attention mechanism, W o is a linear transformation matrix, A sw represents the shift window multi-head self-attention mechanism, Shift is the shift operation, and Z represents the input image.

[0065] Step S22: decoder reconstruction, using the deconvolution upsampling module to perform three times of 2 times upsampling, and constructing the decoder through 4 times upsampling linear mapping to generate high-resolution incomplete mask;

[0066] Step S23: deep supervision strategy, adding supervision signals in different network layers in the U-Net network encoder, calculating an additional loss function, and accelerating the model convergence, the formula used is as follows:

[0067]

[0068] In the formula, L tot represents the total loss function, i represents the subscript of the number of network layers in the encoder, L represents the number of network layers in the encoder, α i is the weight coefficient, S i is the supervision signal, P i is the true label, represents the error square between S i and P i calculated using the L2 norm;

[0069] By performing the operations,

[0070] Embodiment six, based on the above embodiment, in step S4, the diffusion model repair, specifically including the following steps:

[0071] Step S41: add noise, add noise to the standard image data containing the missing area, generate a noisy image, the formula is as follows:

[0072] I noise (x,y)=I(x,y)+N(x,y);

[0073] In the formula, (x,y) represents the coordinates, I(x,y) represents the original image, N(x,y) represents the added noise, I noise (x,y) represents the noisy image;

[0074] Step S42: train the diffusion model, train a diffusion model, use the diffusion model to learn to remove noise from the noisy image and preliminarily repair the missing area;

[0075] Step S43: preliminary repair, use the trained diffusion model to process the noisy image to obtain the preliminary repaired image;

[0076] Step S44: edge-guided repair, use the edge-guided diffusion model constructed based on the Canny algorithm to perform detail repair on the preliminary repaired image and the fusion mask containing edge information to obtain the final repaired image, the formula is as follows:

[0077]

[0078] In the formula, is the final repaired image, I(x,y) is the preliminary repaired image, and H() is the edge-guided diffusion model.

[0079] In view of the problems that the traditional diffusion model is easy to cause edge blur and structure distortion in the repair process, it is difficult to retain the original artistic style and line features, and even may appear fictitious or deformation phenomenon in modeling, the scheme introduces an edge-guided diffusion model, uses edge information as prior guidance to finely repair the missing area in cultural heritage, which not only improves the structure accuracy and edge continuity of the repaired area, but also maintains the artistic style and detail authenticity consistent with the original work in visual effect, realizes higher quality digital repair and protection of cultural heritage images, and provides more real cultural heritage model display for users.

[0080] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and implementations, it is to be understood that the terminology used is for the purpose of descriptive clarity and that it is intended to be limited only by the claims.

[0081] While the embodiments of the application have been shown and described, it is to be understood that the embodiments described are only by way of example and that changes can be made to the embodiments without departing from the spirit and scope of the application. The scope of the application is limited only by the claims and the equivalents thereof.

[0082] The above description of the application and its embodiments is not intended to limit the application, as described by the appended claims, and their equivalents.

Claims

1. A virtual reconstruction and display system for cultural heritage based on the metaverse, characterized in that: The system includes a data acquisition module, a virtual modeling module, a metaverse engine module, an immersive display module, a blockchain authentication module, and a user behavior analysis module; The data acquisition module uses laser scanners, drones, and high-definition camera equipment to collect data on cultural heritage from multiple angles, and obtain three-dimensional point cloud data and two-dimensional images. The virtual modeling module processes the collected 3D point cloud data to generate a 3D digital model of the cultural heritage. For the incomplete parts of the cultural heritage, an enhanced U-Net network and an edge-guided diffusion model are used to perform intelligent restoration based on historical photos and 2D images of the incomplete parts, so as to restore the original historical appearance of the cultural heritage. The metaverse engine module constructs a virtual exhibition hall for cultural heritage, supporting multi-user online access and interaction; The immersive display module provides an immersive experience through VR devices, allowing users to walk freely in the virtual environment, interact with exhibits, and receive audio guides and virtual explanations. The blockchain authentication module uses blockchain technology to process cultural heritage digital assets into NFTs, ensuring their authenticity, uniqueness, and tradability. The user behavior analysis module collects user behavior data in the virtual space and optimizes the displayed content and recommendation mechanism through big data analysis. The virtual modeling module employs an enhanced U-Net network and a diffusion model to perform intelligent restoration based on historical photographs and two-dimensional images of the damaged parts. Specifically, it includes the following steps: Step S1: Data preprocessing, removing noise and outliers from historical photos and 2D images of incomplete parts, and performing standardized data processing to obtain standard image data; Step S2: Crack identification and localization. Use the enhanced U-Net network to generate a defect mask for the defective parts in the standard image data, and mark the defective areas in the standard image data. Step S3: Edge-guided repair. Standard image data is masked using a mask covering 70% of the damaged area, and the image edges are extracted using the Canny algorithm to generate an edge mask. The incomplete mask and the edge mask are then fused to generate a final fused mask. Step S4: Diffusion model restoration. The diffusion model is used to remove noise and restore the missing areas of the standard image data to obtain the restored image. Step S5: Image Output. The restored image is output and used to generate a three-dimensional digital model of the cultural heritage.

2. The virtual reconstruction and display system for cultural heritage based on the metaverse as described in claim 1, characterized in that: In step S2, crack identification and location specifically includes the following steps: Step S21: Encoder processing. In the encoder of the U-Net network, window multi-head self-attention and shift window multi-head self-attention are introduced to progressively compress the resolution of the standard image data to 1 / 16 of the original size, extract multi-scale features, output bottleneck features, and pass them to the decoder. Step S22: Decoder reconstruction, using the deconvolution upsampling module to perform three 2x upsampling operations, and then using 4x upsampling to perform linear mapping to construct the decoder and generate a high-resolution incomplete mask; Step S23: Deep supervision strategy. Supervision signals are added to different network layers in the U-Net encoder to calculate additional loss functions and accelerate model convergence. The formula used is as follows: ； In the formula, Represents the total loss function. The index indicating the network layer number in the encoder. Indicates the number of network layers in the encoder. These are the weighting coefficients. For monitoring signals, For real labels, Indicates calculation using L2 norm. and The squared error between them.

3. The virtual reconstruction and display system for cultural heritage based on the metaverse according to claim 1, characterized in that: In step S4, the diffusion model is repaired, which specifically includes the following steps: Step S41: Add noise by adding noise to the standard image data containing the missing region to generate a noisy image; Step S42: Train a diffusion model. Train a diffusion model to learn how to remove noise from noisy images and perform preliminary repair of damaged areas. Step S43: Preliminary repair: The trained diffusion model is used to process the noisy image to obtain the preliminarily repaired image; Step S44: Edge-guided restoration. Using an edge-guided diffusion model based on the Canny algorithm, the fusion mask containing edge information and the initially restored image are used for detail restoration to obtain the final restored image.

Images