A cultural heritage immersive experience system, method and device based on digital twinning and multi-modal interaction

Abstract

The application relates to the field of digital twinning and cultural heritage digitization, and particularly discloses a cultural heritage immersive experience system, method and device based on digital twinning and multi-modal interaction, which comprises a multi-modal interaction input module, an intention recognition and hierarchical processing module, a digital twinning scene construction module, a data acquisition module and a cultural and creative product display module; the method collects user voice and gesture signals, identifies intentions through a double-layer progressive mechanism, wherein clear intentions are quickly responded to by first-layer rule matching, and fuzzy intentions are deeply analyzed by second-layer multi-modal feature fusion, and display content is called from a digital twinning scene according to the intentions and is output through intelligent cultural and creative products; and the device is a desktop ornament, an AR postcard or an intelligent badge. The application takes into account the interactive response efficiency and intention recognition accuracy, and improves the fluency and realism of the immersive experience.

Description

Technical Field

[0001] This invention belongs to the field of digital twins and digitization of cultural heritage, specifically a system, method and device for immersive cultural heritage experience based on digital twins and multimodal interaction. Background Technology

[0002] Cultural heritage carries the memory and wisdom of human civilization, and its digital protection and transmission have become an important research direction in the field of cultural heritage. In recent years, with the maturity of digital twin technology, the digital reproduction of cultural heritage sites and traditional crafts through 3D modeling and virtual reality can break through the limitations of time and space, allowing the public to have an immersive cultural experience. At the same time, the introduction of multimodal interaction technology allows users to interact with digital content in a natural way through voice commands, gestures, and other methods, significantly reducing the operational threshold for digital experiences. Driven by the integration of the above technologies, immersive experience systems based on digital twins and multimodal interaction are receiving increasing attention, and related technical solutions have been gradually implemented in some museums and cultural exhibition halls.

[0003] However, existing immersive experience systems typically employ a uniform intent recognition processing strategy when handling multimodal user input. This means that all voice commands and gestures undergo the same feature extraction and semantic understanding process before a response is generated. This singular approach leads to a significant trade-off between efficiency and accuracy in actual operation: for clearly defined instructions such as "show a work" or "zoom in," the system still needs to go through the entire recognition process, resulting in noticeable response delays and impacting the smoothness of the interaction; while for unclear boundaries such as ambiguous speech, vague gestures, or semantic conflicts between speech and gestures, the uniform processing flow lacks an effective deep analysis mechanism, easily leading to misjudgments or failure to respond, severely weakening the coherence and realism of the immersive experience. How to balance rapid response to clear intents with accurate understanding of ambiguous intents has become a key technical challenge restricting the improvement of user experience in such systems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an immersive cultural heritage experience system, method, and apparatus based on digital twins and multimodal interaction, thereby resolving the issues in the prior art where a unified intent recognition processing strategy is used for all multimodal inputs, resulting in low efficiency in responding to explicit intents and insufficient accuracy in recognizing fuzzy intents.

[0005] This invention provides an immersive cultural heritage experience system based on digital twins and multimodal interaction, comprising:

[0006] A multimodal interactive input module is used to collect user-input voice interaction data and / or gesture interaction data to generate multimodal input signals;

[0007] The intent recognition and hierarchical processing module is used to receive the multimodal input signal and recognize the user intent according to a two-layer progressive processing mechanism: the first layer uses a rule matching method to quickly classify the multimodal input signal into intents and outputs a first intent recognition result; for ambiguous signals that cannot be clearly classified by the first layer, the ambiguous signals are passed to the second layer, and a multimodal feature fusion method is used to perform deep intent analysis and output a second intent recognition result.

[0008] The digital twin scene construction module is used to construct a digital twin 3D scene of a cultural heritage site based on historical data of the cultural heritage and oral history data provided by the data acquisition module.

[0009] The data acquisition module is used to collect oral history data, skill action data and / or process data of cultural heritage, and to store the collected data in a structured manner.

[0010] The cultural and creative product display module is used to call up corresponding display content from the digital twin 3D scene based on the intent recognition result output by the intent recognition and hierarchical processing module, and output immersive experience content to users through intelligent cultural and creative products.

[0011] Preferably, the digital twin scene construction module includes:

[0012] Scene partitioning units are used to divide the overall scene of a cultural heritage site into several sub-areas according to the spatial structure.

[0013] An independent modeling unit is used to independently perform digital twin modeling on each of the sub-regions and generate a local 3D model of each sub-region.

[0014] The boundary collaborative correction unit is used to identify the boundary coupling relationship between adjacent sub-regions and to collaboratively correct each local 3D model at the boundary, so that the 3D models between adjacent sub-regions maintain geometric continuity and style consistency at the boundary.

[0015] The model merging unit is used to merge the collaboratively corrected 3D models of each sub-region into a complete digital twin 3D scene.

[0016] Preferably, the intent recognition and hierarchical processing module employs a redundancy verification mechanism to cross-confirm key intent recognition results, including:

[0017] The first recognition path is used to perform intent recognition on the multimodal input signal based on semantic feature analysis and generate a first recognition result;

[0018] The second identification path is used to perform intent recognition on the multimodal input signal based on temporal feature analysis and generate a second identification result. The second identification path is methodologically independent of the first identification path.

[0019] The cross-confirmation unit is used to compare the first identification result with the second identification result: if they are consistent, the consistent result is used as the final intent identification result output; if they conflict, the confidence scores of the first identification result and the second identification result are calculated respectively, the identification result with the higher confidence score is selected as the final intent identification result output, and the conflict event is recorded as a potential risk marker.

[0020] This invention also provides a method for immersive cultural heritage experiences based on digital twins and multimodal interaction, comprising the following steps:

[0021] Step S1: Collect user-input voice interaction data and / or gesture interaction data to generate multimodal input signals;

[0022] Step S2: User intent is identified according to a two-layer progressive processing mechanism: the multimodal input signal is quickly classified into intents using a rule matching method, and a first intent recognition result is output; for ambiguous signals that cannot be clearly classified, a multimodal feature fusion method is used for deep intent analysis, and a second intent recognition result is output.

[0023] Step S3: Based on historical data and oral history data of the cultural heritage, construct a digital twin 3D scene of the cultural heritage site;

[0024] Step S4: Based on the intent recognition result, retrieve the corresponding display content from the digital twin 3D scene, and output immersive experience content to the user through smart cultural and creative products.

[0025] Preferably, the method also includes a step of tiered response based on the device type of the smart cultural and creative product:

[0026] Obtain the device type identifier and / or computing power parameters of the current smart cultural and creative products;

[0027] Based on the device type identifier and / or computing power parameters, the intelligent cultural and creative products are divided into three levels: high-level devices, medium-level devices, and low-level devices.

[0028] For the high-level device, full rendering and full-function multimodal interactive response of the digital twin 3D scene are triggered;

[0029] For the mid-level device, lightweight rendering and simplified multimodal interaction response of the digital twin 3D scene are triggered;

[0030] For the aforementioned low-level devices, the loading of basic display content and the response of preset interaction methods are triggered.

[0031] Preferably, it also includes steps of confidence assessment and uncertainty propagation control:

[0032] When outputting the processing result at each interactive stage, the confidence score of the processing result is calculated synchronously. The confidence score is calculated according to the following formula:

[0033]

[0034] in, Indicates the confidence score. This represents the completeness score of the input data. This indicates a score reflecting the suitability of the current processing method for the input data type. This represents the success rate score for processing similar inputs in the past. , , These are the corresponding weight coefficients, and ;

[0035] The confidence score and the processing result are passed to the next interaction stage, so that when the next interaction stage receives a processing result that is lower than the preset confidence threshold, it will automatically adopt a conservative processing strategy and reduce the confidence score of the output result of this stage accordingly.

[0036] When the confidence score of any interaction step falls below the preset minimum confidence threshold, an uncertainty warning is triggered, the current processing flow is paused, and additional input data or manual confirmation is requested.

[0037] Preferably, it also includes a condition-triggered bypass acceleration step:

[0038] Get the intermediate processing results of the current processing node;

[0039] Determine whether the intermediate result meets the preset bypass triggering condition. The bypass triggering condition is stored in the system configuration database and is derived from the statistical summary of historical processing data.

[0040] If the intermediate result meets the bypass triggering condition, then skip the subsequent standard processing steps and directly output the final experience content based on the intermediate result mapping.

[0041] If the intermediate result does not meet the bypass triggering condition, the subsequent processing steps will continue to be executed along the standard processing flow.

[0042] This invention also provides an immersive cultural heritage experience device based on digital twins and multimodal interaction. The device is a smart cultural and creative product, comprising:

[0043] case;

[0044] A processor and a memory are disposed within the housing, the memory storing computer programs;

[0045] A voice acquisition unit disposed on or inside the housing is used to collect user voice interaction data.

[0046] A gesture sensing unit disposed on or inside the housing is used to collect user gesture interaction data.

[0047] The display output unit installed on the housing is used to output immersive experience content of cultural heritage to users;

[0048] When the computer program is executed by the processor, it implements the steps of the method described above.

[0049] Preferably, the smart cultural and creative products are desktop ornaments, AR postcards, or smart badges;

[0050] When the intelligent cultural and creative product is a desktop ornament, the display output unit includes a display screen and / or a projection device;

[0051] When the smart cultural and creative product is an AR postcard, the display output unit includes an AR identification code and / or an NFC tag, which is used to cooperate with a mobile terminal to achieve augmented reality display;

[0052] When the smart cultural and creative product is a smart badge, the display output unit includes an LED display array and / or an NFC communication module.

[0053] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] By constructing an intelligent processing architecture that combines multimodal interactive input with two-layer progressive intent recognition, accurate understanding of user voice and gesture commands is achieved. The first-layer rule matching mechanism responds quickly to explicit intents, while the second-layer multimodal feature fusion mechanism performs in-depth analysis of ambiguous signals. The two work together to ensure that the interaction process has both high real-time efficiency and robustness in handling complex commands.

[0056] In terms of digital twin scene construction, this invention adopts a strategy of independent modeling of partitions and collaborative correction of boundaries. The large-scale old site is broken down into parts and reconstructed separately. Then, seamless integration is achieved through dual collaborative correction of geometry and style. At the same time, the descriptions of color, material and decoration contained in oral history materials are directly converted into texture parameters of three-dimensional models, so that the virtual scene not only resembles the original appearance of the old site, but also carries the cultural temperament and historical warmth in the subjective narrative of the inheritors.

[0057] In response to the differences in computing power among different types of intelligent cultural and creative devices, this invention introduces a hierarchical response mechanism, enabling high-end desktop ornaments to present a complete immersive three-dimensional scene, while lightweight devices can still obtain a streamlined experience that matches their capabilities, thus achieving consistent experience across multiple terminal scenarios.

[0058] Based on this, the redundancy verification and confidence propagation mechanism provides reliability assurance for key identification results, while the condition-triggered bypass acceleration path opens up a fast track for high-frequency simple requests, further improving the system's response efficiency.

[0059] By combining the above methods, this invention not only ensures the accuracy of digital presentation of cultural heritage and the smoothness of interactive experience, but also significantly enhances the system's adaptability and stability under long-term operation, multi-device adaptation, and complex input conditions, providing complete technical support for the living inheritance and intelligent display of cultural heritage. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the overall architecture of Embodiment 1 of the present invention;

[0061] Figure 2 This is a flowchart illustrating the specific implementation of the digital twin scenario construction module in Embodiment 2 of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Example 1: As Figure 1 As shown, the present invention provides an immersive cultural heritage experience system based on digital twins and multimodal interaction. Its overall architecture includes a multimodal interaction input module, an intent recognition and hierarchical processing module, a digital twin scene construction module, a data acquisition module, and a cultural and creative product display module. The above modules interact with each other through a data bus or network communication interface.

[0064] The multimodal interactive input module includes a voice acquisition unit and a gesture acquisition unit. The voice acquisition unit uses a four-microphone circular array (sampling rate 16 kHz, quantization bit depth 16 bit) to extract valid speech segments through a dual-threshold endpoint detection algorithm based on energy and zero-crossing rate, and outputs a pulse code modulation (PCM) audio stream. The gesture acquisition unit uses an Intel RealSense D435 depth camera to acquire the three-dimensional coordinates of 21 key points of the user's hand skeleton at a frame rate of 30 fps. Simultaneously, the motion velocity vectors of each key point are recorded to form a gesture trajectory sequence; after the voice audio stream and the gesture trajectory sequence are aligned with the system timestamp, they together constitute a multimodal input signal.

[0065] The intent recognition and hierarchical processing module receives the aforementioned multimodal input signals and identifies the user's intent according to a two-layer progressive processing mechanism. The first layer uses a rule matching method for rapid intent classification (see Example 3) and outputs the first intent recognition result. For ambiguous signals that cannot be clearly classified in the first layer, the ambiguous signals are passed to the second layer, where a multimodal feature fusion method based on a cross-modal attention mechanism is used for deep intent analysis (see Example 3) and outputs the second intent recognition result. In this example, the average processing latency of the first layer is 80 ms, the average processing latency of the second layer is 450 ms, and the overall intent recognition accuracy of the system reaches 94.7% on the test set.

[0066] The digital twin scene construction module constructs a digital twin three-dimensional scene of the cultural heritage site based on historical data of the cultural heritage (including historical maps, architectural drawings, old photos, etc.) and oral history data provided by the data acquisition module; see Example 2 for the specific implementation of this module.

[0067] The data acquisition module is used to collect oral history data, skill action data, and craft data of cultural heritage, and to store the collected data in a structured manner. Oral history data is collected through high-fidelity recording equipment and converted into text by a speech recognition engine. Skill action data is collected through an optical motion capture system. Craft data is obtained through text input or image scanning. All data is stored in a digital resource library according to a preset data model (including timestamps, data types, data sources, original file paths, structured fields, etc.). This module exchanges data with the digital twin scene construction module through a standardized API. When the digital twin scene construction module needs to access a certain oral history data, it can obtain the corresponding text description and associated action data by passing the project ID and time range through the API.

[0068] The cultural and creative product display module calls up the corresponding display content from the digital twin 3D scene based on the intent recognition results output by the intent recognition and hierarchical processing module, and outputs immersive experience content to users through intelligent cultural and creative products; the intelligent cultural and creative products include three forms: desktop ornaments, AR postcards and intelligent badges, and the specific structure is shown in Example 5.

[0069] The system's workflow is as follows: Users issue interactive commands to smart cultural and creative products via voice or gestures; the multimodal interactive input module collects signals and generates multimodal input; the intent recognition and hierarchical processing module identifies user intent; based on the intent, it retrieves and calls corresponding content from the digital resource library and digital twin scene; the cultural and creative product display module presents the content to the user in the form of 3D rendering, audio narration, or augmented reality overlay; the average total latency from the user issuing the command to the presentation of the experience content is less than 1.2 seconds.

[0070] Example 2: As Figure 2 As shown, this embodiment describes in detail the specific implementation of the digital twin scene construction module, focusing on solving the efficient modeling of large-scale historical site scenes and the specific application of oral history data in the modeling process.

[0071] Step 1: Scene Zoning; Taking a traditional cultural heritage village site as an example, based on the physical partitions (walls, enclosures) and functional zoning of the building complex, the overall scene is divided into five sub-areas: main hall area, side hall area, courtyard area, stele area, and surrounding environment area; the spatial boundaries of each sub-area are marked using geographic information system (GIS) coordinates to ensure that there is at least a 2-meter overlapping buffer zone at the boundary between adjacent areas, reserving space for subsequent collaborative correction.

[0072] Step Two: Independent Modeling; Perform digital twin modeling on each sub-region separately; the specific process is as follows:

[0073] (1) Image data of each sub-region was acquired by using UAV oblique photography (ground resolution better than 2 cm / pixel), combined with point cloud data (point density ≥ 100 points / m) acquired by ground laser scanner. 2 ), generate the initial point cloud model.

[0074] (2) Denoise, register and reconstruct Poisson surface of point cloud to generate triangular mesh model; the number of mesh vertices in the main hall area is controlled at about 2 million, the number of vertices in the side hall area and courtyard area is 1 million, and the number of vertices in the stele area and the surrounding environment area is 500,000, in order to optimize the allocation of computing resources.

[0075] (3) Based on historical photos and documents, the mesh model is textured to generate basic textures; the texture resolution is: 4096×4096 pixels for the main hall area and 2048×2048 pixels for other areas.

[0076] Step 3: Oral History-Driven Texture and Style Modification; This step is the key difference between this invention and traditional modeling methods; The oral history data provided by the data acquisition module is in text form (transcribed via speech recognition and verified by the inheritor); The system extracts structured attributes related to the building's appearance, decoration, and color from the oral history text through Named Entity Recognition (NER) and relation extraction; For example, the oral text "The main hall's roof is covered with dark gray glazed tiles, and the eaves brackets are painted with a predominantly blue-green color" is analyzed to extract a set of triplets: {object: roof, material: glazed tiles, color: dark gray}, {object: bracket paintings, predominantly blue-green}.

[0077] The system establishes a "describing attribute-material parameter mapping table" to map the above structured attributes to the material parameters of the 3D model; example of the mapping rule:

[0078] The “color” description (such as “dark gray”) is mapped to the diffuse color RGB value (e.g., dark gray corresponds to RGB=(0.35,0.35,0.35)).

[0079] The “material” description (such as “glazed tile”) is mapped to the material’s roughness (roughness=0.2) and metallicity (metallic=0.9).

[0080] The "texture" description (such as "cyan painting") selects the most matching texture sample from the preset painting texture library and generates a texture map that matches the UV coordinates of the model through a texture synthesis algorithm.

[0081] For qualitative descriptions that cannot be directly mapped (such as "majestic" or "rustic"), the system uses a pre-trained emotion-style association model to convert them into tone offset and contrast adjustment parameters to stylize the overall appearance of the model. Through the above process, oral history materials not only serve as background information but also directly participate in the texture generation and style shaping of the 3D model, enabling the digital twin scene to have both geometric accuracy and to carry the cultural connotations in the subjective descriptions of the inheritors.

[0082] Step 4: Boundary Co-correction; After the independent modeling of adjacent sub-regions is completed, the system identifies the boundary coupling relationship between regions; Taking the boundary between the main hall area and the courtyard area (including steps and porches) as an example, the system extracts the elevation data sequence of the top of the steps at the boundary of the main hall area. Elevation data sequence of the bottom of the steps at the boundary of the courtyard area , Number of boundary sampling points (in this embodiment) ); calculate the difference between the two. ,like Standard deviation If the value exceeds a preset threshold of 0.05 m, a geometric inconsistency is determined to exist, triggering collaborative correction; the correction amount is:

[0083]

[0084] The average elevation on both sides of the boundary is taken as the target elevation, and the grid vertices on both sides are moved towards the target value. The amount of movement decays along the direction perpendicular to the boundary using a cosine function, with a decay radius of 2 m to ensure a smooth transition. Simultaneously, for inconsistent styles (such as differences in texture tone), the distance between the Lab color space histograms of the textures on both sides of the boundary is calculated. If the distance is greater than a threshold... If the color is transferred to the texture on both sides, the adjacent areas will maintain a consistent visual style.

[0085] Step 5: Model merging; The corrected sub-region models are stitched together according to the original GIS coordinates, and a mesh fusion algorithm based on the Poisson equation is used to eliminate the stitching seams, generating a complete digital twin 3D scene; This scene supports real-time roaming, scaling and rotation operations, and the rendering frame rate can reach 60 fps on desktop devices.

[0086] Example 3: This example describes in detail the workflow of the intent recognition and hierarchical processing module, corresponding to the two-layer progressive processing mechanism and the redundant verification mechanism.

[0087] First layer: Rule matching and rapid classification; the system has a built-in display intent rule library, each rule is a triple (trigger condition, intent category, confidence threshold); trigger conditions include combinations of voice keywords, gesture category labels, or a logical combination of both. Rule example:

[0088] Trigger condition = "voice contains 'show' and includes project name" AND gesture is pointing action, intent category = "show request", confidence threshold = 0.85;

[0089] Trigger condition = "Gesture is five fingers spread and pushed forward", Intent category = "Zoom in to view", Confidence threshold = 0.80;

[0090] Trigger condition = "voice contains 'explanation' and project name", intent category = "explanation request", confidence threshold = 0.80.

[0091] For the input multimodal signal, the speech stream is converted into text by the ASR engine (using the Conformer architecture, fine-tuned on the corpus), and the gesture trajectory is converted into gesture category labels (such as "point", "zoom in", "zoom out", "rotate" etc.) by the temporal classification network (LSTM-based). Then, the trigger conditions in the rule base are matched on the text and the labels respectively. If the match is successful and the matching confidence reaches the threshold, the first intent recognition result is directly output. The average processing latency of this layer is ≤100 ms.

[0092] The second layer: multimodal feature fusion and deep analysis; for ambiguous signals that cannot be clearly classified in the first layer, such as speech text containing only words with unclear referents (e.g., "this" or "there") and ambiguous gestures, or semantic conflicts between speech and gestures (e.g., speech "enlarge" but gesture "shrink"), the system sends such signals to the second layer; the processing flow of the second layer is as follows:

[0093] (1) Speech feature extraction: Input the speech waveform into the pre-trained Wav2Vec 2.0 model to extract frame-level acoustic features, and then obtain fixed-dimensional semantic vectors through temporal pooling. .

[0094] (2) Gesture feature extraction: The gesture trajectory sequence (21 key points × 30 frames / second × 3 coordinates) is input into the Transformer-based temporal encoder, and the gesture intent vector is output. .

[0095] (3) Cross-modal fusion: The fusion features are calculated using a cross-attention mechanism.

[0096]

[0097] in , , , , , For learnable matrices, The multi-head attention output is processed through a feedforward network and layer normalization to obtain a fused vector. .

[0098] (4) Classification output: After passing through a fully connected layer (512→128) and ReLU activation, and then through an output layer (128→ , This embodiment represents the number of intent categories. The algorithm uses the Softmax function to obtain the probability distribution of each intent category. The category with the highest probability is taken as the second intent recognition result, and the highest probability value is recorded as the confidence score of the second layer. The average processing latency for the second layer is 480 ms.

[0099] Through the above two-layer design, approximately 68% of explicit intents are quickly responded to by the first layer, and approximately 32% of ambiguous intents are accurately resolved by the second layer, resulting in an overall accuracy improvement of approximately 12 percentage points compared to the single-layer model.

[0100] Redundant verification mechanism (for key intent recognition); when the intent recognition result involves the evaluation of the accuracy of skill imitation or key interactions in a digital twin scenario (such as "entering the main hall"), the system initiates a redundant verification process; this process includes two independent recognition paths:

[0101] The first recognition path (semantic path) concatenates the speech-text and gesture category sequences from the multimodal input signal, inputs them into a BERT-based semantic encoder (fine-tuned on the domain corpus), and outputs the intent category. and its confidence level (0~1).

[0102] The second recognition path (temporal path): treats the multimodal signal as a time series, directly inputs it into a temporal classifier based on a temporal convolutional network (TCN), and outputs the intent category. and its confidence level The TCN architecture consists of 5 layers of dilated convolutions (with dilation rates of 1, 2, 4, 8, and 16, respectively), each with 256 channels. The final classification result is output after global average pooling and fully connected layers. This path does not rely on semantic templates and extracts features only from temporal patterns, making it completely independent of the semantic path in terms of methodology.

[0103] The two paths are executed in parallel, and the cross-confirmation unit receives the results from both. If R1 = R2, the consistent result is used as the final output, and the final confidence level is calculated by weighted average.

[0104]

[0105] like Then compare and The system selects the path with higher confidence as the final result based on the size of the two paths, and records the conflict event as a structured risk log (including timestamp, input data hash value, predicted distribution entropy of the two paths, and the final selection). The redundant verification mechanism reduces the error rate of key intent identification from 3.2% for a single path to 1.1%, significantly improving the reliability of the system.

[0106] Example 4: This example describes the specific implementation of three aspects: equipment capability graded response, confidence assessment and uncertainty propagation control, and condition-triggered bypass acceleration. The three aspects can work together in the same system.

[0107] (a) Equipment capability classification response.

[0108] The system maintains a database of intelligent cultural and creative product capabilities, recording typical parameters for each type of equipment; the grading criteria are as follows:

[0109] High-level devices: GPU computing power ≥100 GFLOPS and memory ≥4096 MB; or device type identified as "desktop ornament - flagship type"; for high-level devices, trigger full rendering of digital twin scene (complete mesh, 4096 textures, real-time lighting and shadow, particle effects) and full-featured multimodal interaction (voice + gesture + eye tracking).

[0110] Mid-tier devices: GPU computing power ≥ 10 GFLOPS and memory ≥ 1024 MB; or device type identified as "Desktop Ornament - Standard" or "AR Postcard - Mobile" (Note: "Mobile" here refers to the user's own smartphone, which serves as the terminal for the AR postcard, and its computing power is provided by the phone); For mid-tier devices, lightweight rendering is triggered (mesh vertices are reduced to 30% of the original, texture resolution is reduced to 1024×1024, and real-time lighting and shadows are turned off) and simplified interaction (voice + basic gestures, no eye tracking).

[0111] Low-level devices: Those that do not meet the above conditions, or whose device type is identified as "Smart Badge - Basic". For low-level devices, only static graphic content is loaded, and the interaction method is limited to NFC touch trigger or button click.

[0112] When a device connects to the system for the first time, the system obtains its device type identifier and computing capability parameters (via HTTP request headers or device information reporting interface), performs a classification determination, and caches the classification during the session. If the device capability changes dynamically (e.g., network bandwidth drops from 5 Mbps to 1 Mbps), the system can re-evaluate and downgrade the response strategy to ensure a smooth experience.

[0113] (ii) Confidence assessment and uncertainty propagation control.

[0114] After each processing step (speech recognition, gesture recognition, intent recognition, content retrieval, and rendering output) is completed, the system synchronously calculates the confidence score of the output result of the current step. The calculation formula is:

[0115]

[0116] in:

[0117] (Input Completeness Score): Calculated based on the completeness of input data at each stage; for example, in the speech recognition stage, if the percentage of valid speech time is... ,but ;like , ;like , .

[0118] (Method Applicability Score): Based on the similarity between the current input features and the training data distribution; for example, in gesture recognition, if the rate distribution of the current gesture trajectory differs from the mean of the training set by ≤1 standard deviation, then... The difference is 1 to 2 standard deviations. The difference is greater than 2 standard deviations. .

[0119] (Historical Success Rate Score): This score calculates the success rate of processing this step across the past 100 similar inputs. It is directly equal to that ratio.

[0120] The weighting coefficients were determined using the Analytic Hierarchy Process (AHP). In this embodiment, we take... , , ,satisfy .

[0121] Confidence score The result is transmitted to the next stage along with the processing result; if the next stage receives... Below the preset threshold If a conservative processing strategy is automatically adopted—for example, when the intent recognition stage receives a low-confidence speech recognition result, it does not directly infer the intent, but instead initiates a clarifying interaction (such as asking in the voice, "Do you want to view the detailed model of the main hall?"), and continues only after the user confirms; at the same time, when the next stage outputs its own result, it attenuates the output confidence of this stage based on the input confidence.

[0122]

[0123] Where the attenuation coefficient This indicates that uncertainty accumulates along the processing chain; when C at any stage falls below the minimum threshold... When the system triggers an uncertainty warning, it pauses the process, displays a message to the user that "the signal is unclear, please re-enter," and automatically returns to the previous level if no valid input is received within 10 seconds, ensuring that the system does not produce erroneous output based on low-quality data.

[0124] (iii) Condition-triggered bypass acceleration.

[0125] For high-frequency and well-defined requests (such as "showcase project xx"), the system designs a simplified bypass path that skips computationally intensive steps such as deep intent recognition, digital twin scene retrieval, and 3D model loading, directly mapping from intermediate results to the final output.

[0126] Bypass trigger conditions are stored in the system configuration database and generated by statistical summarization of historical processing data. The summarization method is as follows: collect the processing records of the most recent 10,000 complete processes, with each record containing intermediate features. final result ); intermediate features This includes one-hot encoding of speech keywords and encoding of gesture categories; the classifier is trained using the CART decision tree algorithm (maximum depth 5, minimum number of samples per leaf node 100). For the feature space region corresponding to the leaf node with a classification accuracy of ≥98%, the feature combination is stored in the database as a bypass trigger condition; for example, if 1000 records in the sample with "speech containing 'display' + 'paper cutting'" and "gesture pointing" all point to the result "display paper cutting works", then this condition combination is extracted as a bypass trigger condition.

[0127] In actual operation, the system obtains intermediate results (such as preliminary keywords and gesture tags) at each processing node and matches them in the bypass condition library. If the match is successful, the subsequent steps are skipped, and the experience content is directly output through a predefined mapping table (such as directly calling a pre-rendered paper-cut 3D model to display animation), reducing the total latency from the usual 800 ms to 200 ms. If the match fails, the standard process continues. The bypass condition library is automatically updated once a month (or triggered after accumulating 1000 new records), and the decision tree training is re-executed to adapt to changes in user habits.

[0128] Example 5: This example describes three device configurations in detail.

[0129] (a) Desktop ornaments:

[0130] The desktop ornament is designed in the style of an ancient building, with an outer shell made of ABS plastic, and measures approximately 150mm × 100mm × 80mm; internal integration:

[0131] Processor: RK3588 (octa-core ARM, 2.4 GHz clock speed, NPU computing power 6 TOPS);

[0132] Storage: 8 GB LPDDR4 RAM + 128 GB eMMC ROM;

[0133] Voice acquisition unit: Dual microphone array (signal-to-noise ratio ≥65 dB), located on the top of the housing;

[0134] Gesture sensing unit: ToF depth camera (640×480 resolution, 30 fps), located on the front of the housing;

[0135] Display output unit: 4-inch IPS touchscreen (1080×720, 330 PPI), serving as both display and touch input;

[0136] Communication modules: Wi-Fi 6 (802.11ax) and Bluetooth 5.2.

[0137] When the desktop ornament is powered on for the first time, it connects to the cloud, downloads the core data of the digital resource library (containing complete scenes of at least 5 projects, each about 500 MB), and caches it in the local ROM; when the network is normal, it retrieves the latest content from the cloud in real time; when the network is interrupted, it automatically switches to the local cache to ensure a basic experience; the processor executes the computer program in the memory to implement the steps of the method described in Examples 1 to 4.

[0138] (ii) AR postcards:

[0139] The AR postcard of this invention is a smart cultural and creative product with an embedded ultra-thin flexible circuit board, comprising: a paper substrate; an MCU processor and memory disposed on the paper substrate; an NFC tag and an ultra-wideband chip disposed on the paper substrate; a miniature LED indicator light disposed on the paper substrate; an AR identification code (QR code, 5cm×5cm, high contrast color) printed on the front of the paper substrate, and the aforementioned circuit and a flexible battery (thickness <0.5 mm) embedded on the back.

[0140] AR postcards have a built-in MCU processor and memory, possessing limited computing capabilities for near-field communication and identity authentication. They are primarily responsible for data interaction with the user's mobile phone. When the user brings their phone close to the AR postcard, NFC automatically reads the postcard's unique ID and item index number. Simultaneously, UWB provides the relative position and orientation information of the phone and postcard, enabling the mobile AR application to accurately overlay the 3D scene onto the postcard's real-time image. The mobile phone, as the primary computing device (belonging to a mid-level device), retrieves AR content from the cloud and performs rendering. The program stored on the AR postcard controls the MCU for NFC data interaction and power management. When the phone successfully reads the data, the MCU drives a miniature LED indicator to flash to notify the user.

[0141] (III) Smart Badges:

[0142] The smart badge is circular, 50 mm in diameter, 10 mm thick, and weighs approximately 30 g. The casing is made of aluminum alloy. Internal integration:

[0143] Processor: ESP32-S3 (dual-core Xtensa LX7, 240 MHz);

[0144] Memory: 8 MB PSRAM + 16 MB Flash;

[0145] Display output unit: 8×8 full-color LED array (located on the front) for displaying simplified icons or text;

[0146] Communication unit: NFC module (ISO 14443A), used for communication with mobile phones;

[0147] Sensing unit: Capacitive touch sensor (located on the edge of the badge) that can recognize light touch and long press gestures (instead of a depth camera to reduce power consumption and cost).

[0148] Power supply: Rechargeable lithium battery (200 mAh capacity), providing approximately 48 hours of battery life.

[0149] The smart badge is a low-level device and does not perform local 3D rendering. Its working mode is as follows: when the user touches the edge of the badge, the touch sensor triggers the processor to read the item index number stored in Flash (one item is preset, and the user can change it via a mobile app). Then, the NFC module transmits the index number to the user's mobile phone, and the mobile phone automatically opens the display page of the corresponding item. At the same time, the LED array displays the first letter or representative symbol of the item name with a preset animation (such as line scanning), providing instant visual feedback. The smart badge can also be used as an admission ticket for visiting venues or a stamp collection and check-in device, and can be written into the visit record via NFC.

[0150] All three devices implement the steps described above; the desktop ornaments operate completely autonomously; the AR postcards work in conjunction with smartphones, with some steps (such as scene rendering) completed by the smartphones; the smart badges are mainly responsible for triggering and identity recognition, while the core processing is handled by the smartphones or the cloud; through different hardware configurations and division of labor, the three devices cover application scenarios with different computing power requirements from high to low, enabling users to experience digital twin displays of cultural heritage in various contexts.

[0151] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Any changes, modifications, substitutions and variations made by those skilled in the art to the above embodiments within the scope of the present invention should be included within the protection scope of the present invention.

Claims

1. An immersive cultural heritage experience system based on digital twins and multimodal interaction, characterized in that, include: A multimodal interactive input module is used to collect user-input voice interaction data and / or gesture interaction data to generate multimodal input signals; The intent recognition and hierarchical processing module is used to receive the multimodal input signal and recognize the user intent according to a two-layer progressive processing mechanism: the first layer uses a rule matching method to quickly classify the multimodal input signal into intents and outputs a first intent recognition result; for ambiguous signals that cannot be clearly classified by the first layer, the ambiguous signals are passed to the second layer, and a multimodal feature fusion method is used to perform deep intent analysis and output a second intent recognition result. The digital twin scene construction module is used to construct a digital twin 3D scene of a cultural heritage site based on historical data of the cultural heritage and oral history data provided by the data acquisition module. The data acquisition module is used to collect oral history data, skill action data and / or process data of cultural heritage, and to store the collected data in a structured manner. The cultural and creative product display module is used to call up corresponding display content from the digital twin 3D scene based on the intent recognition result output by the intent recognition and hierarchical processing module, and output immersive experience content to users through intelligent cultural and creative products.

2. The system according to claim 1, characterized in that, The digital twin scenario construction module includes: Scene partitioning units are used to divide the overall scene of a cultural heritage site into several sub-areas according to the spatial structure. An independent modeling unit is used to independently perform digital twin modeling on each of the sub-regions and generate a local 3D model of each sub-region. The boundary collaborative correction unit is used to identify the boundary coupling relationship between adjacent sub-regions and to collaboratively correct each local 3D model at the boundary, so that the 3D models between adjacent sub-regions maintain geometric continuity and style consistency at the boundary. The model merging unit is used to merge the collaboratively corrected 3D models of each sub-region into a complete digital twin 3D scene.

3. The system according to claim 1, characterized in that, The intent recognition and hierarchical processing module employs a redundancy verification mechanism to cross-confirm key intent recognition results, including: The first recognition path is used to perform intent recognition on the multimodal input signal based on semantic feature analysis and generate a first recognition result; The second identification path is used to perform intent recognition on the multimodal input signal based on temporal feature analysis and generate a second identification result. The second identification path is methodologically independent of the first identification path. The cross-confirmation unit is used to compare the first identification result with the second identification result: if they are consistent, the consistent result is used as the final intent identification result output; if they conflict, the confidence scores of the first identification result and the second identification result are calculated respectively, the identification result with the higher confidence score is selected as the final intent identification result output, and the conflict event is recorded as a potential risk marker.

4. A method for immersive cultural heritage experiences based on digital twins and multimodal interaction, characterized in that, Includes the following steps: Step S1: Collect user-input voice interaction data and / or gesture interaction data to generate multimodal input signals; Step S2: Identify user intent according to a two-layer progressive processing mechanism: Use a rule matching method to quickly classify the multimodal input signal into intents and output the first intent recognition result; For ambiguous signals that cannot be clearly classified, a multimodal feature fusion method is used for deep intent analysis, and a second intent recognition result is output. Step S3: Based on historical data and oral history data of the cultural heritage, construct a digital twin 3D scene of the cultural heritage site; Step S4: Based on the intent recognition result, retrieve the corresponding display content from the digital twin 3D scene, and output immersive experience content to the user through smart cultural and creative products.

5. The method according to claim 4, characterized in that, It also includes the step of tiered response based on the device type of smart cultural and creative products: Obtain the device type identifier and / or computing power parameters of the current smart cultural and creative products; Based on the device type identifier and / or computing power parameters, the intelligent cultural and creative products are divided into three levels: high-level devices, medium-level devices, and low-level devices. For the high-level device, full rendering and full-function multimodal interactive response of the digital twin 3D scene are triggered; For the mid-level device, lightweight rendering and simplified multimodal interaction response of the digital twin 3D scene are triggered; For the aforementioned low-level devices, the loading of basic display content and the response of preset interaction methods are triggered.

6. The method according to claim 4, characterized in that, It also includes steps for confidence assessment and uncertainty propagation control: When outputting the processing result at each interactive stage, the confidence score of the processing result is calculated synchronously. The confidence score is calculated according to the following formula: in, Indicates the confidence score. This represents the completeness score of the input data. This indicates a score reflecting the suitability of the current processing method for the input data type. This represents the success rate score for processing similar inputs in the past. , , These are the corresponding weight coefficients, and ; The confidence score and the processing result are passed to the next interaction stage, so that when the next interaction stage receives a processing result that is lower than the preset confidence threshold, it will automatically adopt a conservative processing strategy and reduce the confidence score of the output result of this stage accordingly. When the confidence score of any interaction step falls below the preset minimum confidence threshold, an uncertainty warning is triggered, the current processing flow is paused, and additional input data or manual confirmation is requested.

7. The method according to claim 4, characterized in that, It also includes the steps of condition-triggered bypass acceleration: Get the intermediate processing results of the current processing node; Determine whether the intermediate result meets the preset bypass triggering condition. The bypass triggering condition is stored in the system configuration database and is derived from the statistical summary of historical processing data. If the intermediate result meets the bypass triggering condition, then skip the subsequent standard processing steps and directly output the final experience content based on the intermediate result mapping. If the intermediate result does not meet the bypass triggering condition, the subsequent processing steps will continue to be executed along the standard processing flow.

8. An immersive cultural heritage experience device based on digital twins and multimodal interaction, characterized in that, The device is a smart cultural and creative product, including: case; A processor and a memory are disposed within the housing, the memory storing computer programs; A voice acquisition unit disposed on or inside the housing is used to collect user voice interaction data. A gesture sensing unit disposed on or inside the housing is used to collect user gesture interaction data. The display output unit installed on the housing is used to output immersive experience content of cultural heritage to users; When the computer program is executed by the processor, it implements the steps of the method as described in any one of claims 4 to 7.

9. The apparatus according to claim 8, characterized in that, The smart cultural and creative products are desktop ornaments, AR postcards, or smart badges. When the intelligent cultural and creative product is a desktop ornament, the display output unit includes a display screen and / or a projection device; When the smart cultural and creative product is an AR postcard, the display output unit includes an AR identification code and / or an NFC tag, which is used to cooperate with a mobile terminal to achieve augmented reality display; When the smart cultural and creative product is a smart badge, the display output unit includes an LED display array and / or an NFC communication module.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 4 to 7.

Images