Digital twinning processing system and method for text travel virtual reality

Abstract

The invention discloses a digital twinning processing system and method for text travel virtual reality, and relates to the technical field of virtual reality, and the method comprises the steps: collecting historic building component data, including three-dimensional coordinates, point cloud normal vectors, reflectivity, illumination intensity, light direction, spectral intensity distribution and humidity data, and carrying out the preprocessing of the historic building component data; the method comprises the following steps: defining a time node, a space node and a semantic node, calculating a time node similarity weight, a space node distance weight, a semantic node correlation weight and a historical mode comprehensive weight, carrying out mutation detection on humidity data, and predicting an ancient building deformation quantity through a Fick diffusion law and finite element analysis. Dimensionality reduction is carried out on the historical light source parameter with the highest historical mode comprehensive weight, the historical light source parameter is spliced with the real-time light source parameter, and a virtual light source parameter is obtained by inputting into a generator. The deformation quantity is analyzed and predicted through the Fick diffusion law and the finite element, and the problem that deformation prediction of the ancient building is inaccurate due to humidity changes is solved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a digital twin processing system and method for cultural and tourism virtual reality. Background Art

[0002] With the widespread application of digital twin technology and virtual reality (VR) in the field of cultural tourism, how to accurately reproduce historical buildings and their environment has become a research hotspot. Existing methods can collect three-dimensional data of ancient building components through equipment such as lidar and hyperspectral cameras, and use computer graphics to achieve preliminary visualization. However, these methods mostly focus on static display and lack the simulation of dynamic light and shadow effects and the deformation process of ancient buildings, resulting in insufficient realism and immersion in the user experience.

[0003] Although some methods try to combine light intensity and direction for simple rendering, they fail to fully consider the characteristics of light sources in different historical periods and their impact on building materials, resulting in inaccurate historical scenes. In addition, existing methods are deficient in dealing with the impact of humidity changes on the deformation of ancient building materials, making it difficult to accurately predict and reflect the actual deformation. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a digital twin processing method for cultural tourism virtual reality to solve the problems of inaccurate historical lighting patterns and deformation prediction of ancient building materials caused by environmental changes.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a digital twin processing method for cultural tourism virtual reality, which comprises: Collect data on ancient building components, including three-dimensional coordinates, point cloud normal vectors, reflectivity, light intensity, light direction, spectral intensity distribution, and humidity data, and pre-process the data on ancient building components; Define time nodes, space nodes and semantic nodes, calculate time node similarity weights, space node distance weights, semantic node relevance weights and historical pattern comprehensive weights, perform mutation detection on humidity data, predict the shape of ancient buildings through Fick's diffusion law and finite element analysis, reduce the dimension of historical light source parameters with the highest comprehensive weight of historical patterns and splice them with real-time light source parameters, and input them into the generator to obtain virtual light source parameters; Based on the virtual light source parameters, the real channel and the historical channel are rendered, the user's sight direction is obtained to calculate the user's perspective, and virtual and real light and shadow are mixed; Optimize the generator's neural network weights via incremental training.

[0007] As a preferred solution of the digital twin processing method for cultural tourism virtual reality described in the present invention, wherein: the time nodes, space nodes and semantic nodes are defined, and the time node similarity weight, space node distance weight, semantic node relevance weight and historical pattern comprehensive weight are calculated. The specific steps are as follows: Identify component types through 3D coordinates and normal vectors, determine light source types through spectral intensity, and concatenate component type names and lighting type names into semantic nodes; Use Gaussian kernel function to calculate the time node similarity weight and space node distance weight; The semantic node relevance weights are calculated by combining the BERT model with the spectral intensity, and the comprehensive weights of the historical patterns are calculated based on the time node similarity weights, spatial node distance weights, and semantic node relevance weights.

[0008] As a preferred solution of the digital twin processing method for cultural tourism virtual reality described in the present invention, the specific steps of rendering the real channel and the historical channel, obtaining the user's line of sight direction to calculate the user's perspective, and mixing virtual and real light and shadow are as follows: Based on the virtual light source parameters, the Lumen global illumination engine is used in combination with the ray tracing denoising algorithm for real channel rendering; The roughness of the ancient building material is adjusted using a time-based sinusoidal wave, and the roughness of the ancient building material is mapped to a normal map for historical channel rendering through physically based rendering; The real channel weight is adjusted according to the user's perspective, and the edge blur algorithm is used to eliminate the virtual and real boundaries. The user's gaze area is captured based on the eye tracker, and the light and shadow of the non-gaze area are updated through GPU asynchronous calculation delay.

[0009] As a preferred solution of the digital twin processing method for cultural tourism virtual reality described in the present invention, wherein: the historical light source parameters with the highest comprehensive weight in the historical mode are reduced in dimension and spliced ​​with the real-time light source parameters, and input into the generator to obtain the virtual light source parameters. The specific steps are as follows: The real-time light source parameters refer to light intensity, light direction and spectral intensity; The virtual light source parameters refer to virtual intensity, virtual direction and virtual spectrum; The historical light source parameters with the highest comprehensive weight in the historical mode are reduced in dimension through principal component analysis, and then concatenated with the real-time light source parameters, while random noise is added to obtain the light source vector. The light source vector is input into the generator and converted into a normalized value using the Sigmoid function and the Tanh function. The virtual intensity is obtained by linearly mapping the output of the Sigmoid function. The virtual direction is obtained by linearly mapping the output of the Tanh function. The virtual spectrum is restored by inverse principal component analysis of the output of the fully connected layer.

[0010] As a preferred solution of the digital twin processing method for cultural tourism virtual reality described in the present invention, wherein: the humidity data is subjected to mutation detection, and the shape of the ancient building is predicted by Fick's diffusion law and finite element analysis. The specific steps are as follows: Perform sliding window mean filtering on humidity data, calculate the humidity change rate per minute, and determine environmental mutation events; The two-dimensional humidity grid is generated by Kriging interpolation method, the humidity grid is convolved by Sobel operator, and the gradient vector is calculated; The humidity distribution is generated based on Fick's diffusion law, the humidity change rate is calculated according to the gradient vector and the point cloud normal vector, the strain value is calculated in combination with the expansion coefficient, and the overall deformation of the material is calculated through the mechanical equilibrium equation combined with finite element analysis.

[0011] As a preferred solution of the digital twin processing method for cultural and tourism virtual reality described in the present invention, the step of optimizing the neural network weights of the generator through incremental training refers to collecting abnormal data, setting the optimizer and parameters, inputting the abnormal data into the generator to calculate the total loss, and using the back propagation method to update the neural network weights.

[0012] As a preferred solution of the digital twin processing method for cultural tourism virtual reality described in the present invention, preprocessing the ancient building component data refers to aligning the time axis and coordinates of the ancient building component data.

[0013] In a second aspect, the present invention provides a digital twin processing system for cultural tourism virtual reality, comprising: The preprocessing module collects data of ancient building components, including three-dimensional coordinates, point cloud normal vectors, reflectivity, light intensity, light direction, spectral intensity distribution and humidity data, and preprocesses the data of ancient building components; The generator module defines time nodes, space nodes and semantic nodes, calculates the time node similarity weight, space node distance weight, semantic node correlation weight and historical pattern comprehensive weight, performs mutation detection on humidity data, predicts the shape of ancient buildings through Fick's diffusion law and finite element analysis, reduces the dimension of the historical light source parameters with the highest comprehensive weight of the historical pattern and splices them with the real-time light source parameters, and inputs them into the generator to obtain virtual light source parameters; The mixing module performs real channel and historical channel rendering based on virtual light source parameters, obtains the user's line of sight direction to calculate the user's perspective, and mixes virtual and real light and shadow; The optimization module optimizes the neural network weights of the generator through incremental training.

[0014] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the digital twin processing method for cultural and tourism virtual reality as described in the first aspect of the present invention is implemented.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the digital twin processing method for cultural and tourism virtual reality as described in the first aspect of the present invention is implemented.

[0016] The beneficial effects of the present invention are as follows: by defining time nodes, space nodes and semantic nodes, combining the Gaussian kernel function to calculate the time and space weights, and the BERT model to calculate the semantic similarity, and reducing the dimension of the historical light source parameters with the highest comprehensive weight of the historical pattern and inputting them into the generator to generate virtual light source parameters, the lighting parameters in the historical documents (such as candlelight color temperature, spectral peak) are dynamically associated with the real-time environmental data, thereby solving the problem of being unable to accurately reproduce the historical lighting pattern; detecting humidity mutations through sliding window filtering, generating humidity fields through Kriging interpolation, calculating gradient vectors with the Sobel operator, and combining Fick's diffusion law with finite element analysis to predict deformation variables, thereby solving the problem of inaccurate deformation prediction of ancient buildings caused by humidity changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0018] Figure 1 This is a flow chart of the digital twin processing method for cultural tourism virtual reality in Example 1.

[0019] Figure 2 This is a schematic diagram of the digital twin processing system used for cultural tourism virtual reality in Example 1.

[0020] Figure 3 This is a flow chart for predicting humidity deformation in Example 1.

[0021] Figure 4 This is a flowchart for generating a virtual light source in Example 1. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0025] Example 1, reference Figure 1 , Figure 2 , Figure 3 and Figure 4 , this embodiment provides a digital twin processing method for cultural tourism virtual reality, comprising the following steps: S1: Collect data on ancient building components, including three-dimensional coordinates, point cloud normal vector, reflectivity, light intensity, light direction, spectral intensity distribution and humidity data, and pre-process the data on ancient building components.

[0026] The specific steps are as follows: Phase-type laser radar is used to perform phase scanning on ancient building components (such as brackets and eaves) to collect three-dimensional point cloud data of the building components, including the three-dimensional coordinates and point cloud normal vectors of the ancient building components (such as brackets and eaves), with a point density of ≥200 points / square meter.

[0027] It should also be noted that through the high-precision point cloud collection of phase-controlled laser radar (point density ≥ 200 points / square meter), the tiny structural details (such as carved textures) of ancient architectural components (such as brackets and eaves) are ensured to be fully recorded, providing a high-fidelity geometric benchmark for subsequent spatiotemporal alignment and deformation prediction.

[0028] A hyperspectral camera (400-1000nm band) is used to collect the reflectivity of the material corresponding to each point cloud (wavelength interval 1nm).

[0029] It should also be noted that the high spectral resolution of 1nm wavelength interval can accurately capture the differences in reflective properties of different materials (such as wood and glazed tiles), providing spectral-level data support for the physical rendering of the interaction between historical lighting and materials.

[0030] Use a high-precision digital light sensor (such as ams OSRAM TSL25911) to record light intensity in real time, with a range of 0-100klux and a sampling rate of 100Hz.

[0031] It should also be noted that the combination of 100Hz high-frequency sampling and ±0.1° azimuth accuracy can capture slight changes in the direction of light (such as shadow shifts caused by cloud movement) in real time, avoiding the sudden changes in light and shadow caused by traditional low-frequency sampling.

[0032] The light direction is collected through the six-axis IMU, including the measurement of the azimuth and elevation of the light, with an accuracy of ±0.1°.

[0033] A spectrometer was used to capture the spectral intensity distribution in the 380–780 nm band with a resolution of 1 nm.

[0034] Use temperature and humidity sensors to collect humidity data, sampling once per second.

[0035] It should also be noted that the 380-780nm band covers the sensitive spectral range of the human eye. Combined with humidity sampling per second, it can dynamically correlate changes in ambient humidity with material optical properties (such as the attenuation of reflectivity of wood after moisture absorption).

[0036] Use OCR to parse historical documents and extract lighting parameters (such as candlelight color temperature 1900K and lantern spectrum peak wavelength 560nm).

[0037] The RoBERTa-NLP model and the knowledge graph of "Yingzaofashi" are used to parse the construction rules in the literature (such as "the spacing between brackets ≤ 0.3m"), and the logical constraints are extracted through the breadth-first search dependency tree to build a logical constraint library.

[0038] It should also be noted that by combining the RoBERTa-NLP model with the "Yingzaofashi" knowledge graph, implicit construction rules (such as "bracket spacing ≤ 0.3m") can be automatically extracted to avoid subjective bias in manual labeling.

[0039] An atomic clock (such as a high-precision GPS-disciplined rubidium atomic clock (±10μs)) is used to add timestamps to the three-dimensional point cloud data, reflectivity, light intensity, light direction, and spectral intensity distribution of building components, and the time axis is aligned through the dynamic time warping (DTW) algorithm to ensure that the timestamp deviation between the sensor and the point cloud is ≤10ms.

[0040] The point cloud coordinates (UTM-50N coordinate system), high-precision digital light sensor coordinates, six-axis IMU coordinates, and spectrometer coordinates are aligned with the UTM-50N coordinate system through the ICP algorithm and encoded into the UTM coordinate format.

[0041] It should also be noted that the combined application of the GPS disciplined rubidium atomic clock (±10μs) and the ICP algorithm ensures that multi-source sensor data (such as point clouds and spectra) are strictly aligned in the temporal and spatial dimensions, eliminating artifacts caused by clock asynchrony.

[0042] The temporal dimensions (dynasty (NLP extraction), lunar season (GPS timestamp conversion) and time (sun altitude angle calculation)), spatial dimensions (point cloud coordinates in UTM coordinate format, high-precision digital light sensor coordinates, six-axis IMU coordinates and spectrometer coordinates) and semantic dimensions (component type (bracket / eaves corner) and lighting type (candlelight / natural light)) are fused into spatiotemporal labels.

[0043] It should also be noted that through high-precision collaborative acquisition of multimodal sensors (point cloud, spectrum, light, humidity), a full-dimensional digital twin model of the time, space, physics, and materials of the ancient building components was constructed, providing a millimeter-level geometric benchmark and spectral-level material feature library for subsequent historical scene reconstruction.

[0044] S2: Define time nodes, space nodes and semantic nodes, calculate the time node similarity weight, space node distance weight, semantic node correlation weight and historical pattern comprehensive weight, perform mutation detection on humidity data, predict the shape of ancient buildings through Fick's diffusion law and finite element analysis, reduce the dimension of historical light source parameters with the highest comprehensive weight of historical patterns and splice them with real-time light source parameters, and input them into the generator to obtain virtual light source parameters.

[0045] Time nodes are defined based on dynasties, lunar seasons, and hours. For example, "Ming Dynasty_Autumn_Youshi" means the dusk period in autumn of the Ming Dynasty.

[0046] Spatial nodes are defined based on the UTM-50N coordinates of ancient architectural components (such as brackets and eaves) in the point cloud coordinates, high-precision digital light sensor coordinates, six-axis IMU coordinates, and spectrometer coordinates. Each spatial node is labeled with the component type and sensor number.

[0047] Define semantic nodes based on component type (bracket / eaves) and lighting type (candlelight / natural light): Identify component types through three-dimensional coordinates and normal vectors, including brackets, eaves, beams and columns. The classification is based on the component shape database recorded in the "Yingzaofashi". The light source type is determined by spectral intensity. Candlelight corresponds to a peak wavelength of 560-580nm, and natural light corresponds to a peak wavelength of 450-4803. Concatenate the component type name and the lighting type name into a semantic node. For example, "bracket_candlelight" represents the semantic features of brackets in a candlelight environment.

[0048] It should also be noted that integrating dynasties, lunar seasons and times into time nodes can accurately link the lighting characteristics recorded in historical documents (such as "candlelight at dusk in the Ming Dynasty") with the physical form of ancient architectural components, avoiding the periodization errors of modern time division.

[0049] Calculate the similarity weight of time nodes: If the dynasties are different (such as "Ming Dynasty" and "Qing Dynasty"), set the similarity weight of the time nodes to zero; if the dynasties of the two time nodes are the same (such as "Ming Dynasty"), map the season to the month difference (for example, September in autumn and December in winter are 3 months apart), and use the Gaussian kernel function ( σ The similarity weight of time nodes is calculated by using the time node similarity method (the input is the month difference, which is 12 months). The smaller the month difference, the higher the time node similarity weight. For example, the time nodes "Ming Dynasty_Autumn_Youshi" (September) and "Ming Dynasty_Winter_Shenshi" (December) have a month difference of 3 months, and the time node similarity weight is 0.97.

[0050] It should also be noted that the nonlinear attenuation characteristics of the Gaussian kernel function (σ = 12 months) can distinguish the differences in lighting in different seasons of the same dynasty (such as strong direct light in summer and scattered light in winter), thereby improving the spatiotemporal continuity of historical lighting patterns.

[0051] Calculate the distance weight of spatial nodes: Use the Haversine algorithm to calculate the distance between spatial nodes (for example, the distance between the bracket and the light sensor is 30 meters), and use the Gaussian kernel function ( σ The spatial node distance weight is calculated by using the distance between spatial nodes (80 meters, and the input is the distance between spatial nodes). The spatial node distance weight decreases exponentially with the increase of distance. For example, the distance between the bracket and the light sensor is 30 meters, and the spatial node distance weight is 0.82.

[0052] It should also be noted that the Haversine algorithm combined with the Gaussian kernel function (σ = 80 meters) can quantify the impact of spatial distance on light propagation (such as the attenuation difference between eaves and ground candlelight sensors) and avoid the linear simplification of Euclidean distance.

[0053] Calculate the relevance weight of semantic nodes: Calculate the similarity of semantic nodes through the BERT model (such as the similarity of "dougong_candlelight" and "yanjiao_candlelight" is 0.6), including dividing the semantic nodes into component type and lighting type (such as "dougong" and "candlelight"), inputting them into the BERT model to generate component type vectors and lighting type vectors, and performing weighted summation of the component type vectors and lighting type vectors to obtain the semantic node vector. The weight of the component type vector is 0.6, and the weight of the lighting type vector is 0.4. The cosine similarity algorithm is used to calculate the semantic similarity between semantic node vectors; the spectral intensity is normalized to the range of 0-1 (eliminating the influence of absolute brightness), and the peak wavelength of the spectral intensity is identified through the local maximum algorithm (such as Savitzky-Golay filtering), and the peak wavelength of the spectral intensity is calculated through CIE The 1931 standard is converted into a color temperature value (unit: K) (for example, the color temperature of candlelight is 1900K, and the color temperature of an oil lamp is 1850K), and the absolute value of the color temperature difference between the two light sources is calculated (for example, the color temperature difference between candlelight and an oil lamp is 50K), and the standard reference value of the color temperature difference is set to 100K. The absolute value of the color temperature difference is divided by the standard reference value of the color temperature difference to obtain the color temperature difference ratio value, and an exponential decay function is used (the initial value is 1, the decay constant is 0.5, and the input is the square of the color temperature difference ratio value to achieve nonlinear attenuation) to calculate the attenuation coefficient. The attenuation coefficient is multiplied by the semantic similarity to obtain the semantic node relevance weight. For example, the attenuation coefficient is 0.882, and the semantic node relevance weight is 0.53.

[0054] It should also be noted that the dual constraints of the BERT model and the exponential decay of color temperature differences can distinguish subtle differences between similar semantic nodes (such as the difference in spectral peaks between "dougong_candlelight" and "dougong_oil lamp"), improving the generator's ability to distinguish historical light sources.

[0055] After multiplying the time node similarity weight with the space node distance weight, multiply it by the square root of the semantic node relevance weight (square root is used to avoid excessive attenuation) to obtain the historical pattern comprehensive weight. For example, the time node similarity weight is 0.97, the space node distance weight is 0.82, the semantic node relevance weight is 0.53, and the historical pattern comprehensive weight is 0.58.

[0056] The humidity data was filtered using a sliding window mean filter (60-second window) to calculate the humidity change rate per minute. When the change rate exceeded 5% (for example, the humidity increased from 60% to 80% within 1 minute), it was determined to be an environmental mutation event. The Kriging interpolation method was used to generate a two-dimensional humidity grid with a resolution of 0.5 meters centered on the mutation area. For example, the humidity in area A increased by 2% per meter from south to north, forming a humidity field with a gradient direction pointing to due north. The Sobel operator was used to convolve the humidity grid, and the gradient components in the east-west and north-south directions were calculated respectively, and the gradient vector was synthesized. For example, a humidity field increasing from south to north would generate a gradient vector pointing to the north.

[0057] The deformation of ancient building components is predicted by Fick's diffusion law and finite element analysis: Based on the three-dimensional point cloud data of building components, the ancient building components are discretized into tetrahedral unit grids, and each grid node is assigned an initial humidity value (obtained by mapping the Kriging interpolation result). The expansion coefficient is set according to the material of the ancient building components (such as wood and stone), and the humidity diffusion process is calculated by Fick's diffusion law (the humidity diffusion coefficient is 1.2×10⁻ 6 m² / s), simulate the propagation process of humidity inside the material, consider the exchange of humidity on the surface of the material with the environment, and predict the humidity distribution inside the material; perform dot product operation on the gradient vector and the point cloud normal vector to quantify the rate of humidity change along the normal direction; multiply the humidity change rate by the expansion coefficient to obtain the strain value caused by humidity change, use the linear elastic constitutive relationship to relate the strain value to the stress, establish the mechanical equilibrium equation, set the elastic modulus and Poisson's ratio of wood and stone, and use the Newton-Raphson method to iteratively solve the displacement field to obtain the overall deformation of the material.

[0058] Candela intensity and peak wavelength of the spectra were normalized to the range 0–1 using min–max normalization.

[0059] Select the historical light source parameters with the highest comprehensive weight in the historical mode, reduce the dimension of the historical light source parameters (such as the spectral intensity distribution of the "Ming Dynasty oil lamp") to 512 dimensions through principal component analysis (PCA) to reduce redundant information, concatenate the real-time light source parameters (256 dimensions) and the historical light source parameters (512 dimensions) into a 768-dimensional vector, add 256-dimensional random noise, and generate a 1024-dimensional light source vector; input the light source vector into the generator to obtain the virtual light source parameters. The structure of the generator is a 5-layer fully connected network and a 1-layer output layer. The number of neurons in each layer of the fully connected network is 1024, 512, 256, 128, and 64, and the activation function is LeakyReLU (α=0.2). The output layer is divided into 4 branches: virtual intensity, virtual direction, virtual spectrum, and noise. Virtual intensity: extract the first 32 dimensions from the 64-dimensional data output by the 5th layer of the fully connected network, input the Sigmoid function to generate a 0-1 range value, and map the output of the Sigmoid function (0 to 1) to 800-1000 lx range (for example: 0.8→880 lx), virtual direction includes virtual azimuth and virtual pitch angle, virtual azimuth: extract the last 32 dimensions from the 64-dimensional data output by the 5th layer of fully connected network, input the Tanh function to generate a 0-1 range value, and map the output of the Tanh function (-1 to 1) to 0-360° (for example: 0.25→45°), virtual pitch angle maps the output of the Tanh function (-1 to 1) to 0-90° (for example: 0.33→30°), virtual spectrum: restore the 64-dimensional data output by the 5th layer of fully connected network to the spectral distribution in the 400-780 nm band through inverse principal component analysis, noise: superimpose Gaussian noise with a mean of 0 and a standard deviation of 0.01 on all 64-dimensional data output by the 5th layer of fully connected network to simulate sensor errors (for example: ±2 lx).

[0060] It should also be noted that the inverse principal component analysis to restore the spectral distribution combined with Gaussian noise simulation can generate a natural spectral curve that meets the CIE standard (such as candlelight 1900K color temperature) to avoid spectral distortion.

[0061] Calculate the point-by-point difference between the virtual spectrum and the spectrum in the 380-780 nm band, with a threshold ≤ 0.05. For example, the peak of the spectrum is 560 nm (intensity 0.95) and the peak of the virtual spectrum is 558 nm (intensity 0.92), then MSE (mean square error) = 0.03 (qualified). Generate a binary image comparison between the shadow mask and the real shadow, PSNR (shadow mask quality) ≥ 40 dB. For example, the maximum pixel value of the shadow mask is 255, MSE = 6.5, then PSNR ≈ 42 dB (qualified). Embed color temperature constraints and direction constraints. If the color temperature of the virtual light source exceeds the range recorded in the literature (such as candlelight needs to be 1850-1950 K), regenerate the virtual intensity. If the virtual direction error exceeds the measured accuracy of the six-axis IMU (± 0.1°), regenerate the virtual direction.

[0062] It should also be noted that the construction of the spatiotemporal-semantic three-dimensional weight system realizes the precise spatiotemporal correlation between historical lighting patterns and architectural components. The dimensionality reduction of historical light source parameters and the dynamic constraints of the generator ensure that the virtual light source is consistent with the physical properties recorded in the literature in terms of color temperature, direction, and spectral distribution, thereby improving the restoration accuracy of historical scenes.

[0063] S3: Based on the virtual light source parameters, the real channel and the historical channel are rendered, the user's line of sight direction is obtained to calculate the user's perspective, and virtual and real light and shadow are mixed.

[0064] The specific steps are as follows: Real channel rendering: input the virtual light source parameters into the LumenGlobal Illumination Engine of Unreal Engine 5, bind them to the material node of the ancient building's 3D model, and map the virtual light source parameters to the surface shader through the material editor. For example, the candlelight color temperature is bound to the material's self-luminous channel, and the dynamic directional light angle is synchronized to the normal map offset. The ray tracing denoising algorithm is used to suppress noise and enhance details of the ancient building's brackets and eaves under high-density sampling. Set 1024 light samples per pixel, focus on optimizing the anti-aliasing of shadow edges and multiple reflection calculations of hollow structures (such as multiple reflection calculations of light at the hollow eaves), reduce light noise through multiple importance sampling (MIS), and use the geometric information of the depth buffer to accelerate light intersection calculations; adjust the material reflectivity according to humidity data. For example, when the humidity exceeds 70%, the reflectivity of the wood surface decreases by 15%, simulating the enhanced diffuse reflection caused by moisture absorption; for every 5°C increase in temperature, the highlight reflection range of metal components expands by 10% to simulate the thermal expansion effect.

[0065] It should also be noted that 1024 light samples per pixel and multiple importance sampling (MIS) can eliminate the multiple reflection noise of the hollow structure of the bracket and improve the anti-aliasing effect of the shadow edge.

[0066] Historical channel rendering: The roughness of the ancient building material is adjusted according to the historical light source intensity. For example, for every 10 lux increase in candlelight intensity, the roughness of the ancient building material is reduced by 0.03, simulating the smoothness improvement of the wood surface due to the wear of the oxide layer under weak light. The roughness of the ancient building material is adjusted using time-based sinusoidal waves. For example, the roughness of the ancient building material is reduced by 5% at noon (thermal expansion of the wood fills microcracks), and increases by 5% at midnight (cold contraction causes microcracks on the surface); the roughness of the ancient building material is mapped to the normal map through physically based rendering (PBR). For example, the roughness of the ancient building material is superimposed on the mortise and tenon joints of the brackets to simulate historical traces of use, and periodic noise textures are added to the surface of the glazed tiles at the eaves to restore the weathering and erosion effects.

[0067] It should also be noted that the time-based sinusoidal wave roughness adjustment of ancient building materials (such as noon-midnight ±5%) can simulate the changes in surface micro-cracks caused by thermal expansion and contraction of wood, thereby enhancing the dynamic authenticity of historical scenes.

[0068] The user's line of sight is obtained through the built-in sensor of the headset, and the user's viewing angle (the angle between the user's line of sight and the light source) is calculated. When the user is facing the light source (angle ≤ 30°), the real channel weight is 70%; when the angle exceeds 30°, the real channel weight decays according to an exponential curve to avoid misalignment between the virtual projection and the real scene when looking sideways; when the user's line of sight is perpendicular to the normal vector of the light source (such as looking sideways at the side of a bracket), the historical channel weight is automatically increased by 15%, and the boundary between the virtual and real shadows is blurred through the bilateral filtering algorithm to eliminate pixel-level aliasing; the user's gaze area is captured by an eye tracker, and a 4K high-resolution circular domain with a diameter of 20° is generated with the gaze point as the center. The peripheral area is downsampled in three levels according to the distance (2K→1080p→720p) to ensure that the visual focus area is full of details; the non-gaze area is divided into 32×32 pixel blocks, and the ETC2 compression algorithm is used to reduce the amount of texture data, and the lighting calculation is processed in parallel through the GPU asynchronous pipeline. For example, only low-frequency shadow information is retained in the non-gaze area, and high-frequency details are dynamically supplemented through motion vector interpolation.

[0069] It should also be noted that the exponential decay rule of the user's viewing angle weight (the weight decreases when the angle is greater than 30°) can avoid the misalignment of the virtual light source projection and the physical shadow when looking sideways, thereby enhancing the immersion of the wide-viewing angle scene.

[0070] Combine ToF depth camera data with 3D point cloud data of building components to calculate the occlusion relationship between real objects and virtual models. For example, when the user is close to the bracket (distance <1 meter), the shadow of the real light source is only projected on the physical surface, and the projection of the virtual light source extends to the damaged part of the digital repair (such as missing carved patterns). Compare the depth values ​​of the real shadow and the virtual projection, cover the far shadow with the near shadow, and achieve transition through Alpha blending. For example, when the shadow depth of the real bracket is 0.5 meters and the virtual projection is 1.2 meters, only the real shadow is rendered, and the occlusion boundary is Gaussian blurred with a width of 2 pixels to make the error of the virtual-real transition zone ≤0.1 mm to avoid hard boundaries visible to the naked eye. Render the picture at 4K resolution in the user's gaze area, and use anti-aliasing and detail enhancement technology to ensure that the processing time of each frame does not exceed 8 milliseconds to maintain a smooth visual effect. Intelligent rendering is used to render the picture at 1080 resolution in the non-gaze area, only half of the pixels are calculated, and the image is completed through interpolation of adjacent frame information, ensuring the continuity of the picture while reducing the GPU load.

[0071] It should also be noted that the user perspective adaptive rendering strategy combined with the virtual-real occlusion fusion algorithm solves the hard boundary problem of virtual-real light and shadow in AR / VR scenes. Through gaze point rendering and dynamic resolution allocation, it reduces the GPU load while ensuring 4K details in the visual focus area, achieving a balance between immersive interaction and hardware performance.

[0072] S4: Optimizing the generator’s neural network weights via incremental training.

[0073] The specific steps are as follows: The time node similarity weights of the lighting mode are adjusted proportionally according to user preferences. The specific rule is: for every 10% increase in usage rate, the time node similarity weight of the lighting mode increases by 2%, but the upper limit does not exceed 150% of the time node similarity weight of the original lighting mode. For example, the user usage rate of "Tang Dynasty Dusk Candlelight" increases by 40%, and the time node similarity weight of the lighting mode increases by 8%.

[0074] It should also be noted that the dynamic growth of time weights (a 2% increase for every 10% usage rate) can strengthen historical lighting patterns with high frequency visits (such as "Tang Dynasty dusk candlelight") and avoid overfitting of unpopular patterns due to balanced weights.

[0075] The generator is compressed from 32-bit floating point (FP32) to 8-bit integer (INT8), the amount of computation is reduced by merging fully connected layers, and the generator is deployed on the NVIDIA Jetson AGX device to support synchronous calling of AR glasses and VR helmets, ensuring that the multi-terminal rendering delay is ≤15ms.

[0076] It should also be noted that INT8 quantization compression combined with Jetson AGX edge computing can control rendering latency within 15ms and support simultaneous calls of multiple AR / VR terminals.

[0077] Every day, the virtual spectrum (380-780nm band) is compared with the spectrum band by band, and the overlap ratio of the virtual spectrum intensity distribution and the spectrum intensity distribution is calculated. For example, if the spectral intensity of a certain band (such as 560nm) is 0.95 and the virtual spectrum is 0.92, the band matching degree is 97%, and the average matching degree of the entire band must be ≥85%; if the average matching degree is lower than 85% (such as 82%) for three consecutive days, the incremental training process is started to optimize the neural network weights of the generator, collect abnormal data in the last seven days, including low-matching spectra, high deformation error point cloud coordinates, and temperature and humidity mutation records, and store them in the training pool after time alignment, lock the first three layers of the fully connected network of the generator (1024→512→256 neurons), and retain the spectral intensity of the historical light source. , directional angle and color temperature extraction capabilities, only the last two fully connected layers (128→64 neurons) and the output layer are trained to adapt to the noise pattern and abnormal data in the new environment, the AdamW optimizer is used to optimize the neural network weights of the generator, the learning rate is set to 0.0001, the batch size is 32, and a maximum of 50 rounds of training are performed. If the verification loss does not decrease for 5 consecutive rounds, the training is terminated early, the abnormal data is input, the spectral error (the difference between the spectral intensity and the virtual spectrum) and the directional error (the difference between the light direction and the virtual direction) are calculated, and the total loss is calculated in combination with the KL divergence. The unfrozen neural network weights are updated using the backpropagation method based on the total loss, the gradient clipping threshold is set to 1.0, and the single weight change is limited to 5% to ensure training stability.

[0078] It should also be noted that when the matching degree is less than 85% for three consecutive days, incremental training is started, and only the last two layers of the generator network are updated. This can quickly adapt to new environmental noise patterns (such as seasonal atmospheric scattering changes) and avoid the computational overhead of full training.

[0079] The three-dimensional coordinates of the ancient building components are compared with the virtual coordinates point by point, and the displacement differences of all points are calculated (for example, the virtual coordinates are offset by 1.2mm compared to the measured coordinates). If the average error is greater than 1mm, the wood expansion coefficient increases from 0.005mm / %RH to 0.006mm / %RH.

[0080] It should also be noted that the incremental training mechanism and dynamic parameter adjustment strategy, while maintaining the core historical features, quickly respond to long-term changes such as sudden changes in temperature and humidity or material aging through local network updates, ensuring that the geometric deformation prediction error of the digital twin model is stable at the sub-millimeter level for a long time.

[0081] This embodiment also provides a digital twin processing system for cultural tourism virtual reality, including: The preprocessing module collects data of ancient building components, including three-dimensional coordinates, point cloud normal vectors, reflectivity, light intensity, light direction, spectral intensity distribution and humidity data, and preprocesses the data of ancient building components; The generator module defines time nodes, space nodes and semantic nodes, calculates the time node similarity weight, space node distance weight, semantic node correlation weight and historical pattern comprehensive weight, performs mutation detection on humidity data, predicts the shape of ancient buildings through Fick's diffusion law and finite element analysis, reduces the dimension of the historical light source parameters with the highest comprehensive weight of the historical pattern and splices them with the real-time light source parameters, and inputs them into the generator to obtain virtual light source parameters; The mixing module performs real channel and historical channel rendering based on virtual light source parameters, obtains the user's line of sight direction to calculate the user's perspective, and mixes virtual and real light and shadow; The optimization module optimizes the neural network weights of the generator through incremental training.

[0082] This embodiment also provides a computer device, which is suitable for the digital twin processing method for cultural and tourism virtual reality, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the digital twin processing method for cultural and tourism virtual reality proposed in the above embodiment.

[0083] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0084] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the digital twin processing method for cultural and tourism virtual reality proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, disk or optical disk.

[0085] In summary, the present invention solves the problem of being unable to accurately reproduce historical lighting patterns by: defining time nodes, space nodes and semantic nodes, combining Gaussian kernel function to calculate time and space weights, BERT model to calculate semantic similarity, and reducing the dimension of historical light source parameters with the highest comprehensive weight of historical patterns and inputting them into the generator to generate virtual light source parameters, dynamically associating lighting parameters in historical documents (such as candlelight color temperature, spectral peak) with real-time environmental data; detecting humidity mutations through sliding window filtering, generating humidity fields through Kriging interpolation, calculating gradient vectors with Sobel operator, and predicting deformation variables through combining Fick's diffusion law with finite element analysis, thereby solving the problem of inaccurate deformation prediction of ancient buildings caused by humidity changes.

[0086] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A digital twin processing method for cultural tourism virtual reality, characterized by: include, Collect data on ancient building components, including three-dimensional coordinates, point cloud normal vectors, reflectivity, light intensity, light direction, spectral intensity distribution, and humidity data, and pre-process the data on ancient building components; Define time nodes, space nodes and semantic nodes, calculate time node similarity weights, space node distance weights, semantic node relevance weights and historical pattern comprehensive weights, perform mutation detection on humidity data, predict the shape of ancient buildings through Fick's diffusion law and finite element analysis, reduce the dimension of historical light source parameters with the highest comprehensive weight of historical patterns and splice them with real-time light source parameters, and input them into the generator to obtain virtual light source parameters; Based on the virtual light source parameters, the real channel and the historical channel are rendered, the user's sight direction is obtained to calculate the user's perspective, and virtual and real light and shadow are mixed; Optimize the generator's neural network weights via incremental training.

2. The digital twin processing method for cultural tourism virtual reality according to claim 1, characterized in that: The steps of defining time nodes, space nodes and semantic nodes, and calculating time node similarity weights, space node distance weights, semantic node relevance weights and historical pattern comprehensive weights are as follows: Identify component types through 3D coordinates and normal vectors, determine light source types through spectral intensity, and concatenate component type names and lighting type names into semantic nodes; Use Gaussian kernel function to calculate the time node similarity weight and space node distance weight; The semantic node relevance weights are calculated by combining the BERT model with the spectral intensity, and the comprehensive weights of the historical patterns are calculated based on the time node similarity weights, spatial node distance weights, and semantic node relevance weights.

3. The digital twin processing method for cultural tourism virtual reality according to claim 1, characterized in that: The real channel and the historical channel are rendered, the user's sight direction is obtained to calculate the user's viewing angle, and virtual and real light and shadow are mixed. The specific steps are as follows: Based on the virtual light source parameters, the Lumen global illumination engine is used in combination with the ray tracing denoising algorithm for real channel rendering; The roughness of the ancient building material is adjusted using a time-based sinusoidal wave, and the roughness of the ancient building material is mapped to a normal map for historical channel rendering through physically based rendering; The real channel weight is adjusted according to the user's perspective, and the edge blur algorithm is used to eliminate the virtual and real boundaries. The user's gaze area is captured based on the eye tracker, and the GPU asynchronously calculates the delay to update the light and shadow of the non-gaze area.

4. The digital twin processing method for cultural tourism virtual reality according to claim 1, characterized in that: The historical light source parameters with the highest comprehensive weight in the historical mode are reduced in dimension and spliced ​​with the real-time light source parameters, and input into the generator to obtain the virtual light source parameters. The specific steps are as follows: The real-time light source parameters refer to light intensity, light direction and spectral intensity; The virtual light source parameters refer to virtual intensity, virtual direction and virtual spectrum; the historical light source parameters with the highest comprehensive weight in the historical mode are reduced in dimension through principal component analysis, and spliced ​​with the real-time light source parameters, and random noise is added to obtain the light source vector; The light source vector is input into the generator and converted into a normalized value using the Sigmoid function and the Tanh function. The virtual intensity is obtained by linearly mapping the output of the Sigmoid function. The virtual direction is obtained by linearly mapping the output of the Tanh function. The virtual spectrum is restored by inverse principal component analysis of the output of the fully connected layer.

5. The digital twin processing method for cultural tourism virtual reality according to claim 1, characterized in that: The specific steps of performing mutation detection on humidity data and predicting the shape of ancient buildings through Fick's diffusion law and finite element analysis are as follows: Perform sliding window mean filtering on humidity data, calculate the humidity change rate per minute, and determine environmental mutation events; The two-dimensional humidity grid is generated by Kriging interpolation method, the humidity grid is convolved by Sobel operator, and the gradient vector is calculated; The humidity distribution is generated based on Fick's diffusion law, the humidity change rate is calculated according to the gradient vector and the point cloud normal vector, the strain value is calculated in combination with the expansion coefficient, and the overall deformation of the material is calculated through the mechanical equilibrium equation combined with finite element analysis.

6. The digital twin processing method for cultural tourism virtual reality according to claim 1, characterized in that: The method of optimizing the neural network weights of the generator through incremental training refers to collecting abnormal data, setting the optimizer and parameters, inputting the abnormal data into the generator to calculate the total loss, and updating the neural network weights using the back propagation method.

7. The digital twin processing method for cultural tourism virtual reality according to claim 1, characterized in that: The preprocessing of the ancient building component data refers to aligning the time axis and coordinates of the ancient building component data.

8. A digital twin processing system for cultural tourism virtual reality, based on the digital twin processing method for cultural tourism virtual reality according to any one of claims 1 to 7, characterized in that: include, The preprocessing module collects data of ancient building components, including three-dimensional coordinates, point cloud normal vectors, reflectivity, light intensity, light direction, spectral intensity distribution and humidity data, and preprocesses the data of ancient building components; The generator module defines time nodes, space nodes and semantic nodes, calculates the time node similarity weight, space node distance weight, semantic node correlation weight and historical pattern comprehensive weight, performs mutation detection on humidity data, predicts the shape of ancient buildings through Fick's diffusion law and finite element analysis, reduces the dimension of the historical light source parameters with the highest comprehensive weight of the historical pattern and splices them with the real-time light source parameters, and inputs them into the generator to obtain virtual light source parameters; The mixing module performs real channel and historical channel rendering based on virtual light source parameters, obtains the user's line of sight direction to calculate the user's perspective, and mixes virtual and real light and shadow; The optimization module optimizes the neural network weights of the generator through incremental training.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the digital twin processing method for cultural and tourism virtual reality described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the digital twin processing method for cultural and tourism virtual reality described in any one of claims 1 to 7 are implemented.

Images