Mongolian intangible cultural heritage display system based on ai and virtual reality

Abstract

The Mongolian intangible cultural heritage display system based on AI and virtual reality is provided, comprising a user intention analysis module, a behavior generation module, a pattern generation module, an exhibition content fusion module and a virtual reality display module.A unified expression and scheduling mechanism from semantic driving to exhibition execution is constructed, the key problems of disconnection between behavior and pattern generation, cultural mismatch of display timing, and difficulty in real-time feedback of user input in the existing system are solved, and the expression accuracy, generation flexibility and interactive response capability of intangible cultural heritage in a virtual environment are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of AI and virtual reality, and in particular relates to a display system for Mongolian intangible cultural heritage based on AI and virtual reality. Background Technology

[0002] With the development of the digital wave, cultural institutions and research units both domestically and internationally have attempted to use virtual reality, multimedia displays, and other means to digitally transform and disseminate these cultural resources to the public. However, existing technologies mainly focus on the acquisition of images and videos, 3D modeling of exhibition hall scenes, and the integration of standardized voice narration, and still have significant shortcomings in solving the dynamic generation and interactive understanding of complex semantic content.

[0003] Furthermore, in terms of user experience, current exhibition systems generally lack semantic adaptation mechanisms tailored to individual needs. Users' interests, knowledge structures, and cultural understanding levels are difficult to reflect in the content generation process, leading to fragmented presentation of content, a disconnect between behavior and images, and a lack of unified driving logic between cultural processes and dynamic displays. Especially in multimodal interactive environments, traditional systems lack a semantic-centric generation and scheduling structure, making it difficult to support deep integration between cultural behaviors, pattern construction, and the sequence of exhibitions. Summary of the Invention

[0004] The purpose of this invention is to propose a Mongolian intangible cultural heritage display system based on AI and virtual reality, which realizes closed-loop modeling of the entire process of user intent-content generation-performance integration-interactive control.

[0005] To achieve the above objectives, the present invention provides a Mongolian intangible cultural heritage display system based on AI and virtual reality, the system comprising the following modules:

[0006] The user intent parsing module is used to obtain the user's natural language expression text and the Mongolian intangible cultural heritage knowledge graph. The user's natural language expression text is input into a bidirectional GRU network to obtain a context semantic representation sequence. The context semantic representation sequence and the Mongolian intangible cultural heritage knowledge graph are matched with confidence to obtain a behavior demand vector and a pattern demand vector.

[0007] The behavior generation module is used to generate a multi-frame 3D skeletal motion sequence based on the behavior requirement vector.

[0008] The pattern generation module is used to generate a pattern template image based on the pattern requirement vector;

[0009] The performance content fusion module is used to combine the multi-frame 3D skeletal motion sequence and pattern template image, and generate a pattern space mounting sequence and a performance frame scheduling diagram in each frame through a motion-image fusion engine based on cultural priors; the pattern space mounting sequence is the spatial position vector of the generated pattern in each frame, and the performance frame scheduling diagram is used to represent the triggering relationship and temporal path between the motion frame and the pattern frame;

[0010] The virtual reality display module is used to deploy the three-dimensional spatial mounting position and performance frame scheduling diagram into the virtual reality system and integrate multimodal interaction;

[0011] The behavioral demand vector and the pattern demand vector are in the form of a tuple. The behavioral demand vector includes intangible cultural heritage behavioral tags and corresponding matching confidence scores, and the pattern demand vector includes pattern keywords and corresponding matching confidence scores.

[0012] Furthermore, the execution steps of the user intent parsing module specifically include:

[0013] Obtain the user's natural language expression text;

[0014] The user's natural language expression text is input into a pre-trained BERT model and embedded to generate a sequence of word vectors;

[0015] The word vector sequence is input into a bidirectional GRU network to perform context modeling on the word vector sequence, forming a context semantic representation sequence;

[0016] Obtain a knowledge map of Mongolian intangible cultural heritage;

[0017] The word vectors of the context semantic representation sequence are combined with the nodes of the Mongolian intangible cultural heritage knowledge graph and semantically aligned using vector similarity to obtain matching confidence.

[0018] The matching confidence scores are sorted, and the top few nodes with high matching scores are selected and divided into two sets according to entity type, including:

[0019] A set of behavioral entities is generated, and cultural actions that may be involved in the user's intent are extracted from the set of behavioral entities to form a corresponding behavioral demand vector.

[0020] The pattern entity set is used to extract visual symbols that users are interested in, in order to form a pattern demand vector.

[0021] Furthermore, the execution steps of the behavior generation module specifically include:

[0022] An action generation network is constructed; the structure of the action generation network includes a behavior label encoder, an action flow template selector, and an action generator; wherein...

[0023] The behavior label encoder is used to embed the intangible cultural heritage behavior labels into semantic vectors through a lookup table, and retrieves them from the semantic dictionary using a lookup table method without training; the action flow template selector is used to retrieve a predefined action template sequence based on the intangible cultural heritage behavior labels, wherein each template consists of a triplet of action stage, duration, and skeletal topology; the action generator is a 3-layer Transformer structure that accepts the semantic vector and the action template sequence to generate the skeletal keypoint coordinates for each frame;

[0024] A cultural consistency regularization term is introduced for the coordinates of key points of the skeleton in each frame to penalize stage jumps in the action sequence that do not conform to the cultural process, so as to generate a multi-frame three-dimensional skeleton action sequence.

[0025] Furthermore, the execution steps of the pattern generation module specifically include:

[0026] The pattern keywords are converted into corresponding semantic vectors and then input into the semantic encoder;

[0027] Retrieve the closest pattern image from the image library and extract its high-level style features;

[0028] If the pattern is incomplete, a mask matrix is ​​generated;

[0029] The output of the semantic encoder, high-level style features, and mask matrix are input into the pattern generator to output a complete pattern as a pattern template image.

[0030] Furthermore, the culturally prior action-image fusion engine includes a pattern behavior mapping matrix, a spatial attachment function, and a performance frame scheduling graph.

[0031] Furthermore, the execution steps of the performance content integration module specifically include:

[0032] Based on the 3D motion data of the multi-frame 3D skeletal motion sequence, a motion stage label sequence is constructed, wherein each motion stage label vector represents the cultural motion semantic label of its corresponding frame.

[0033] The pattern behavior mapping matrix introduced should display the first frame for each frame. Assign values ​​to each pattern event to obtain the values ​​assigned at the corresponding frame number;

[0034] The spatial attachment function should be used to display the first frame for each frame. The pattern event rendering location is used to obtain the pattern space mounting sequence.

[0035] By combining the action phase label sequence and the pattern behavior mapping matrix, an optimization objective function is constructed to minimize the timestamps of two frame nodes on the timeline and whether there is a cultural logic violation in the action / pattern transition, thereby generating a performance frame scheduling graph. In the performance frame scheduling graph, the nodes represent specific performance frames, and the edges represent the cultural logic relationships between frames.

[0036] Furthermore, the use of the spatial attachment function to display the first frame in each frame... The pattern space mounting sequence is obtained by determining the rendering location of each pattern event, specifically calculated as follows:

[0037]

[0038] in, For each frame Mounting location in three-dimensional space The first The three-dimensional coordinates of the left and right hands of the frame; The coordinates of the center point of the chest or head; The weight of the pattern position is controlled to determine whether it is biased towards the area between the hands or the center of the head; The symmetry regularization weight is usually set to 0.3 to 0.5, and is used to constrain the pattern presentation direction to be consistent with the axis of symmetry of the action. As a totem symmetry penalty function, if the current skeletal posture symmetry is insufficient, the pattern is shifted to the higher side to enhance visual stability; For spatial attachment function, For pattern behavior mapping matrix;

[0039] According to each frame Mounting position in three-dimensional space This constitutes the pattern space mounting sequence. .

[0040] Furthermore, the execution steps of the virtual reality display module specifically include:

[0041] Build a standardized VR exhibition environment;

[0042] The VR performance environment is rendered based on the pattern space mounting sequence to generate the rendering result of each pattern in three-dimensional space.

[0043] A behavior-pattern-time ternary controller is constructed to monitor the position of the performance timeline and check the current state node in the performance frame scheduling diagram, and then call the corresponding action fragment and pattern update.

[0044] Three types of sensors were configured to integrate multimodal interaction signals.

[0045] Furthermore, in the invocation of the corresponding action fragment and pattern update, a buffer window is set for preloading control.

[0046] Furthermore, the three types of sensors include:

[0047] Voice input: Fixed commands are recognized by the voice recognition module;

[0048] Head orientation: The coordinates of the user's gaze are returned by the VR headset;

[0049] Gestures: Gesture information is collected by hand tracking devices;

[0050] The step of setting up three types of sensors to integrate multimodal interaction signals includes:

[0051] The sensing signals of the three types of sensors are encoded into events and mapped to node jump triggers in the performance frame scheduling diagram, thereby jumping to the corresponding state in the performance frame scheduling diagram.

[0052] The beneficial technical effects of the present invention are at least as follows:

[0053] This invention focuses on cultural elements with semantic complexity and behavioral performance characteristics in Mongolian intangible cultural heritage. It proposes a virtual display method system based on the integrated linkage of semantic parsing, generative networks and temporal control, realizing closed-loop modeling of the entire process of user intent-content generation-performance integration-interactive control.

[0054] This method first extracts the user's attention to behaviors or patterns from their natural language expressions, constructs behavioral demand vectors and pattern demand vectors through cultural knowledge graphs and semantic alignment technology, and uses these as control signals to drive generative content modules to generate cultural action sequences with process control structures and pattern image templates with style evolution capabilities.

[0055] After generation, the system synchronously integrates these two types of content through action phase recognition, pattern behavior mapping and dynamic space mounting mechanism to construct a performance scheduling diagram with cultural process consistency. Then, combined with the user's multimodal input such as voice, gaze, and gesture, it realizes interactive control response and finally outputs an immersive display result with visualization, controllability and cultural expression consistency.

[0056] This invention solves the key problems in existing systems, such as the disconnect between behavior and pattern generation, the inability to culturally match the display sequence, and the difficulty in providing real-time feedback on user input, by constructing a unified expression and scheduling mechanism from semantic driving to performance execution. It significantly improves the accuracy of intangible cultural heritage expression, generation flexibility, and interactive responsiveness in the virtual environment. Attached Figure Description

[0057] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0058] Figure 1 This is a framework diagram of a Mongolian intangible cultural heritage display system based on AI and virtual reality, as disclosed in an embodiment of the present invention. Detailed Implementation

[0059] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0060] like Figure 1 As shown, this embodiment of the invention also provides a Mongolian intangible cultural heritage display system based on AI and virtual reality, the system comprising:

[0061] The user intent parsing module 101 is used to obtain the user's natural language expression text and the Mongolian intangible cultural heritage knowledge graph, input the user's natural language expression text into a bidirectional GRU network to obtain a context semantic representation sequence, and perform matching confidence calculation on the context semantic representation sequence and the Mongolian intangible cultural heritage knowledge graph to obtain a behavior demand vector and a pattern demand vector.

[0062] Behavior generation module 102 is used to generate a multi-frame three-dimensional skeletal motion sequence based on the behavior requirement vector;

[0063] Pattern generation module 103 is used to generate a pattern template image based on the pattern requirement vector;

[0064] The performance content fusion module 104 is used to combine the multi-frame three-dimensional skeleton action sequence and pattern template image, and generate a pattern space mounting sequence and a performance frame scheduling diagram in each frame through a cultural prior action-image fusion engine; the pattern space mounting sequence is the spatial position vector of the generated pattern in each frame, and the performance frame scheduling diagram is used to represent the triggering relationship and timing path between action frames and pattern frames.

[0065] The virtual reality display module 105 is used to deploy the three-dimensional space mounting position and the performance frame scheduling diagram into the virtual reality system and integrate multimodal interaction.

[0066] Specifically, the user intent parsing module 101 performs the following steps:

[0067] This module aims to transform user-input natural language expressions (potentially from speech recognition or direct text input) into structured cultural demand vectors, serving as semantic control conditions for subsequent content generation modules. The output consists of two parts: a behavioral demand vector for behavior generation and a pattern demand vector for pattern modeling. These two vectors will drive the generation process of dynamic behaviors and static patterns within intangible cultural heritage. Because Mongolian cultural content contains a large number of unique semantics and non-general entity words, such as Sulide (war god totem), Chama dance (religious dance), and Hada ceremony, this module must not only possess general language modeling capabilities but also achieve domain adaptation through knowledge graphs. To ensure system robustness, the entire parsing process is designed with a three-layer structure: a text encoding layer, a semantic representation layer, and a knowledge alignment layer.

[0068] The input for this module is the user's natural language text. The data is a UTF-8 encoded string with a maximum length of 256 characters. Data sources can be in two forms: first, users directly input text into the system terminal (such as a mobile app, web page, or VR voice control interface); second, users input text via voice, which is then transcribed into text by a speech recognition service deployed on the system front end (e.g., a recognition interface based on the DeepSpeech model). The recognition module has been adapted for Mongolian-specific vocabulary, thus correctly handling high-frequency ethnic terms such as the Naadam horse-head fiddle.

[0069] text The text is first vectorized in the embedding module. To enhance the coverage of cultural semantics, this module uses a pre-trained BERT model as the base word vector generator. It is then fine-tuned on a large-scale dataset of Mongolian intangible cultural heritage-related texts (including intangible cultural heritage encyclopedias, cultural tourism materials, ritual texts, etc., totaling approximately 870,000 texts) to better handle ethnic-specific semantics. The embedding process converts the text into a sequence of word vectors. ,in , for the first The vector representation of each word has a dimension of 768.

[0070] Next, a bidirectional GRU network is used to model the context of the word vector sequence, forming a contextual semantic representation sequence. Each The output is a bidirectional GRU, containing the comprehensive meaning of words in context. This GRU network consists of two 512-dimensional hidden layers, using a shared parameter set, and is trained using a cross-entropy loss function aligned with the knowledge graph entity annotation supervision signal.

[0071] To link the semantics understood by the model with structured cultural elements, the system introduces a knowledge graph of Mongolian intangible cultural heritage. The graph nodes include behavioral entities (such as wedding ceremonies, fire-worshiping festivals, and toasting rituals), pattern entities (such as horse head patterns and sun totems), as well as scene and context nodes. Each graph node is accompanied by a text description and structural labels, and the graph size is approximately 30,000 nodes and 120,000 relationships.

[0072] During the alignment process, the system calculates the vector similarity between each word in the user input and the graph nodes, and obtains the weights through a dot product attention mechanism:

[0073]

[0074] in, Indicates the first character in the input text. The contextual semantic representation of each word is a 512-dimensional vector output by a bidirectional GRU network; Indicating the first in the atlas The vector representations of the candidate nodes are pre-constructed and fixed using a graph encoder; To match weights, reflecting the first [item] in the text Similarity between each word and a graph entity; all vectors have been normalized to ensure dimensional consistency; To match confidence levels.

[0075] After ranking by attention matching weights, the system selects the top few nodes with high matching degrees and divides them into two sets according to entity type. The first set is the behavioral entity set, from which the system extracts cultural actions that may be involved in the user's intent, such as offering tea, offering a hada (a ceremonial scarf), and kneeling to pray for blessings, which constitute the corresponding behavioral demand vector. The second set is the pattern entity set, which extracts visual symbols that users are interested in, such as sun patterns, sulphur patterns, and double horse totems, to form a pattern demand vector. Each element of these two vectors is a tuple containing the entity text description and its matching confidence score (a floating-point number between 0 and 1), such as:

[0076]

[0077] This approach not only extracts explicitly mentioned keywords but also infers the cultural behaviors and pattern content that users may implicitly follow based on contextual semantics. The innovation of the entire modeling process lies in achieving accurate parsing of non-general cultural semantics through a joint mechanism that aligns semantic representation with a cultural knowledge graph, avoiding the expression limitations and mismatches caused by relying on keyword retrieval. Simultaneously, the output structure is directly designed as a semantic control vector for subsequent generation modules, ensuring the controllability and cultural consistency of the entire system from the outset.

[0078] Understandably, this module outputs two structured vectors: : Behavioral demand vector, which consists of a set of (behavioral entity, matching confidence) tuples representing the cultural behavior requested by the user; : Pattern request vector, which is a set of (pattern entity, matching confidence) tuples representing the image elements requested by the user.

[0079] in, : Behavioral demand vector, in the form of ,in Indicates the first A label for intangible cultural heritage behavior, such as the Hada salute. To correspond to the confidence level, the value ranges from ; Pattern demand vector, in the form of ,in Keywords for patterns, such as sun totem. , where is the confidence level.

[0080] Specifically, the behavior generation module 102 performs the following steps:

[0081] Considering that Mongolian intangible cultural heritage practices often involve specific cultural processes and ritual logic—for example, the presentation of a hada (ceremonial scarf) at a wedding must follow the order of first introductory rites, then offering the gift, and finally praying for blessings—and that patterns often involve stylistic reproduction and partial reconstruction (such as sun patterns and horseback totems, which are mostly incomplete in historical documents), this module needs to be specially designed with a structurally controllable and semantically aligned generation mechanism to provide a high degree of freedom of expression while preserving cultural logic.

[0082] First, the behavior generation module is processed. The system designs a semantic-flow fusion action generation network, with the goal of... Output a skeletal movement sequence with Mongolian ritual logic. The network consists of three parts:

[0083] Behavior label encoder: will Embedded into semantic vectors through table lookup The semantic dictionary is retrieved using a lookup table method, without any training.

[0084] Action flow template selector: based on Retrieve predefined action template sequences Each template consists of a triplet of action phase, duration, and skeletal topology;

[0085] Action generator: A 3-layer Transformer structure that accepts... and Generate the coordinates of key points in the skeleton for each frame.

[0086] During the generation process, the system introduces a cultural consistency regularization term. It is used to punish phase transitions in a sequence of actions that do not conform to the cultural process, such as not raising the hada before the kneeling action in the hada ceremony.

[0087] The entire generation process can be represented as:

[0088]

[0089] in, The output skeletal motion sequence, in the following format: (Number of frames × Number of keypoints × Spatial dimension); : Basic motion regression loss, using L2 norm, to measure the difference between generated motion and template motion; Cultural process regularization items, based on manually defined behavior phase diagrams, will be subject to high penalties if the action transition does not comply with the legal process; Regularization term weights are typically between 0.5 and 1.0 and are selected through cross-validation.

[0090] Specifically, the pattern generation module 103 executes the following modules:

[0091] Next, the pattern generation module is modeled. Since Mongolian patterns are often partially missing and have a strong style, this invention uses a conditional StyleGAN network, but modifies its input layer to incorporate pattern semantics and partial incompleteness prompts. The entire process is as follows:

[0092] Will Transform into semantic vectors The dimension is 128, and the input is a semantic encoder;

[0093] Search the image library for the closest pattern image. Extract its high-level style characteristics Dimension 256;

[0094] If the pattern is incomplete (detected by the pattern state evaluation module), a mask matrix is ​​generated. ;

[0095] Input the above three elements into the pattern generator. Output complete pattern .

[0096] This process can be formalized as follows:

[0097]

[0098] in: A multi-input channel StyleGAN network with a learnable channel mixing module added to the first layer; Incomplete image mask, obtained by a structural inference network. Perform convolutional feature analysis to generate features; Output pattern image, size The output is a pattern template image in RGB color space with 8-bit depth, which is then generated by combining semantic vectors, style features, and incomplete information. This template will be expanded into a multi-frame dynamic pattern sequence in subsequent modules to drive temporal visual displays in VR.

[0099] This pattern modeling mechanism solves the following three key problems: (1) When only the sun pattern is given as a keyword, the system can automatically restore the missing part based on the style matching map in the library; (2) When the original pattern image has structural occlusion, the local completion is achieved through the masking mechanism; (3) The semantic control vector ensures that the output pattern does not deviate from the user's instructions. For example, the sun pattern on the horse's back will automatically strengthen the central radiation and merge the horse's head curve.

[0100] This module ultimately outputs two variables: : A multi-frame 3D skeletal motion sequence in tensor format , which are the cultural behaviors and actions generated; Pattern template image, size is This image is generated by combining semantic vectors, style features, and incomplete information. It will then be used in subsequent modules in conjunction with action sequences. Linkage, generating pattern rendering sequences through pattern scheduling mapping and spatial mounting processes. These variables are used to drive frame-by-frame visual presentations in VR. They will be used in subsequent modules for driving virtual performance actions and generating visual content, ensuring that the actions and graphics of the entire presentation system have cultural semantic consistency and stylistic reproduction capabilities.

[0101] Specifically, the performance content integration module 104 performs the following steps:

[0102] This module aims to integrate the Mongolian cultural behavior sequences generated in the behavior generation module 102 and the pattern generation module, respectively. and semantically driven pattern images The content is organically integrated into the same performance scene, and the temporal sequence of the performances is scheduled with cultural logical consistency to generate dynamic display content that can be used in virtual reality. This module is the core link in the system's transition from content generation to a presentable form, determining the immersiveness, cultural integrity, and user comprehension cost of the performance content. Considering the special characteristics of Mongolian intangible cultural heritage content, such as the periodicity of actions (e.g., dances, rituals), the strong sequential order of ritual nodes (e.g., in weddings, ancestor worship cannot precede the bride's arrival), and the spatial and temporal binding relationship with the presentation of patterns, this module innovatively designs a motion-image fusion engine based on cultural priors. Multiple regularization terms and prior reasoning mechanisms are introduced into the temporal scheduling and spatial rendering control to ensure that the generated performance content has both technical stability and meets cultural correctness requirements.

[0103] The input variables of this module directly inherit the output results of module two: and :

[0104] Firstly, regarding Construct action phase label sequences from 3D motion data. ,in Indicates the first The frames contain cultural action semantic labels (such as raising hands to kneel in preparation for prayer). These labels are generated by a lightweight action recognition network. The network uses a 3-layer one-dimensional convolutional structure to perform sliding window analysis on the skeletal features of each frame, extracting subtle behavioral changes to achieve semantic segmentation of the action phase.

[0105] Based on this, a pattern behavior semantic synchronization mapping matrix is ​​introduced. ,in This represents the number of pattern events (typically 2-5, such as a background totem loading pattern that gradually appears and the totem is highlighted). If... , indicating the first The frame should display the first A pattern event. The construction method is as follows:

[0106] First of all Clustering of continuous semantic tags in the data to form action fragments;

[0107] Then, a semantic mapping table (defined by experts and containing rules for the relationship between actions and patterns) is used to map action fragments to pattern events;

[0108] Finally, the start and end time periods of each pattern event are assigned, and values ​​are assigned to them at the corresponding frame number.

[0109] Subsequently, in terms of spatial fusion, this invention designs a dynamic pattern attachment function. For each frame Output pattern rendering position To enhance visual harmony and cultural adaptability, this invention incorporates a dynamic pose weighting factor and a totem symmetry constraint term into the basic skeletal geometry calculation, as shown in the following formula:

[0110]

[0111] in, The first The three-dimensional coordinates of the left and right hands in the frame; The coordinates of the center point of the chest or head; The weight of the pattern position is controlled to determine whether it is biased towards the area between the hands or the center of the head; The symmetry regularization weight is usually set to 0.3 to 0.5, and is used to constrain the pattern presentation direction to be consistent with the axis of symmetry of the action. The totem symmetry penalty function is used to offset the pattern to the higher side if the current skeletal posture is not symmetrical enough (e.g., one hand is higher than the other) to enhance visual stability.

[0112] This mechanism can dynamically adapt the pattern rendering position to action scenarios such as dancing, spinning, and holding totems, avoiding abnormal image coverage due to posture offset.

[0113] Finally, the performance timeline was constructed. The system is based on a sequence of action stages. and pattern event frame index matrix Based on this, a performance frame scheduling diagram is generated. The nodes represent specific performance frames, and the edges represent the cultural logical relationships between frames. To ensure a natural transition between actions and patterns, the following optimization objectives are introduced:

[0114]

[0115] in, For the timestamps of the two frame nodes on the timeline; Indicates from frame arrive Does the action / pattern transfer violate cultural logic, such as being penalized for jumping immediately after praying? To transfer the weight of the logical regularization term, it is usually set to around 1.2, and then optimized through actual testing.

[0116] The above optimization process uses dynamic programming in the schedulable graph. The system searches for the path with the strongest cultural consistency, outputs the performance frame order and image display trigger points, and finally generates a performance control script that can be used in a virtual reality system.

[0117] Understandably, this module outputs two key variables: Pattern spatial mounting sequence: The three-dimensional spatial mounting position of the pattern in each frame, used to drive the real-time dynamic attachment of the pattern in the VR scene; The presentation frame scheduling diagram represents the triggering relationship and timing path between action frames and pattern frames, which is ultimately used for rendering control of the VR presentation engine.

[0118] Understandably, the pattern generation module 103 and the performance content fusion module 104 fuse the semantically driven generated action sequence with the pattern image into virtual performance content with a spatiotemporal structure. This is achieved through a pattern behavior mapping matrix. Spatial attachment function With the performance frame scheduling diagram The joint modeling achieves deep integration across modalities and semantics. Compared to the simple mechanism of decoupling or forcibly synchronizing behavior and vision in traditional VR displays, this solution not only provides higher fidelity in cultural expression, but also builds an innovative bridge between cultural processes and behavior-graphic binding, serving as the core technical support for the entire system from semantics to performance implementation.

[0119] Specifically, the virtual reality display module 105 performs the following steps:

[0120] This module, as the final part of the patent solution, has the core task of mounting the display control structure output by the previous module—that is, the pattern space sequence. With performance schedule diagram This technology is deployed into specific virtual reality (VR) systems to achieve operable, interactive, and culturally consistent terminal displays. The overall arrangement in this invention patent is as follows: Module 1 completes requirement modeling; behavior generation module 102 and pattern generation module 103 complete content generation; performance content fusion module 104 completes the fusion and scheduling of behaviors and images; and module 105 enables these structured and fused contents to dynamically run and interact within the VR scene. Therefore, this module focuses on demonstrating the real execution of scheduling logic, the visualization of pattern action binding, and the real-time response mechanism for multimodal interaction.

[0121] The input to this module is all the outputs from module three, and it fully utilizes the two output variables from the previous step:

[0122] Exhibition Frame Scheduling Diagram : Describes the temporal organization structure of action frames and pattern events, and is the dominant input controlling the VR timeline process; pattern space mounting sequence Generates a spatial position vector for the pattern in each frame, guiding the dynamic attachment and transformation of the pattern in VR space.

[0123] The system uses these two inputs together to drive character behavior, pattern visual overlay, and user interaction logic in the virtual environment.

[0124] First, a standardized VR performance environment is constructed. Taking Unity as an example, a spatial scene based on Mongolian cultural background is built, including high-fidelity 3D assets such as yurts, grasslands, and traditional ornaments. Character models are loaded based on a preset skeletal structure, and the motion sequence output by playback module two is used. The performance scheduling diagram constructed by Module 3 Control its playback sequence and action segment switching. Each frame In the middle, the system according to Controlling pattern texture The location in virtual space is dynamically rendered using a hook-point binding method, specifically:

[0125]

[0126] The system uses pattern templates Mounting position for each frame Combined, a frame-level rendering graph is generated. This constitutes the final pattern rendering frame sequence. Wherein: : No. The rendering result of the frame pattern in three-dimensional space;

[0127] : This refers to the dynamic texture rendering function in Unity. Mount to The specified model space location; The pattern position is generated in the previous step and has been adapted to specific skeletal points or spatial paths through geometric logic.

[0128] The pattern rendering employs a strategy of using alpha channel mapping combined with soft-edge masking to ensure a natural attachment effect during dynamic switching of the VR viewpoint. For example, in the Hada ceremony, the pattern is attached to the character's hands in real time and changes synchronously with their raising and lowering movements, enhancing the sense of immersion.

[0129] Secondly, the execution of the scheduling logic is handled by... The system constructs a behavior-pattern-time ternary controller to monitor the position of the performance timeline. And look up the table Current state node Then, the corresponding action fragment and pattern update function are called. To avoid temporal logic confusion, the system sets a buffer window. Perform preloading control:

[0130]

[0131] in: The scheduled trigger time for the next state node; A time buffer window of 0.5 to 1 second is used to control the pace of resource preloading. The system decides whether to jump to the next performance node. ;

[0132] If the page jumps, a new action segment will be played, and the pattern rendering position and status will be updated; otherwise, the current state will be maintained.

[0133] For example: in the fire-worship performance To display the kneeling action and flame image, the system will... Load the flame image texture 0.5 seconds before the character approaches the node and map it onto the ground in front of the character's knees. After the user's action ends, trigger the totem to rise.

[0134] Finally, multimodal interaction signals are integrated. The system uses three types of sensors:

[0135] Voice input: Fixed commands (such as displaying a sunburst pattern toggles) are recognized by a natural language speech recognition model; the speech recognition module is a component that integrates hardware and software to convert speech signals into a computer-processable form (such as text or commands) to enable voice interaction.

[0136] Head orientation (gaze point): The coordinates of the user's gaze returned by the VR headset;

[0137] Gestures (such as reaching out or clenching a fist): Gesture information is collected by hand tracking devices.

[0138] These sensory signals are encoded as events. and mapped to Middle node jump trigger. For example, when =After focusing on the totem for 3 seconds, the system interprets this as triggering a totem magnification event, and then redirects to The corresponding state .

[0139] To avoid cultural flow disruptions caused by excessively frequent jumps, the system is designed with a conflict detection mechanism. If user input conflicts with the current performance status, the system will enter an interactive ignore protection mode, blocking input events until the status transition is stable and complete.

[0140] This module outputs two structured results: VR control scripts, including motion frame scheduling tables, pattern dynamic attachment parameters, rendering trigger logic, interaction mapping tables, etc., are the main execution configurations that drive the VR engine to run. Interaction log file, recording each user's activity. The events and the resulting state transition paths are used for subsequent performance preference modeling and feedback mechanisms.

[0141] The foregoing has described specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0142] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0143] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0144] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0149] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0150] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0151] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0152] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0153] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0154] Finally, it should be noted that the Mongolian intangible cultural heritage display system based on AI and virtual reality disclosed in the embodiments of the present invention is only a preferred embodiment of the present invention and is only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Mongolian intangible cultural heritage display system based on AI and virtual reality, characterized in that: The system includes the following modules: The user intent parsing module is used to obtain the user's natural language expression text and the Mongolian intangible cultural heritage knowledge graph. The user's natural language expression text is input into a bidirectional GRU network to obtain a context semantic representation sequence. The context semantic representation sequence and the Mongolian intangible cultural heritage knowledge graph are matched with confidence to obtain a behavior demand vector and a pattern demand vector. The behavior generation module is used to generate a multi-frame 3D skeletal motion sequence based on the behavior requirement vector. The pattern generation module is used to generate a pattern template image based on the pattern requirement vector; The performance content fusion module is used to combine the multi-frame 3D skeletal motion sequence and pattern template image, and generate a pattern spatial mounting sequence and a performance frame scheduling diagram in each frame through a motion-image fusion engine based on cultural priors; the pattern spatial mounting sequence is the spatial position vector of the generated pattern in each frame, and the performance frame scheduling diagram is used to represent the triggering relationship and temporal path between the motion frame and the pattern frame; A virtual reality display module is used to deploy the pattern space mounting sequence and performance frame scheduling diagram into the virtual reality system and integrate multimodal interaction; The behavioral demand vector and the pattern demand vector are in the form of a tuple. The behavioral demand vector includes intangible cultural heritage behavioral tags and corresponding matching confidence scores, and the pattern demand vector includes pattern keywords and corresponding matching confidence scores.

2. The Mongolian intangible cultural heritage display system based on AI and virtual reality as described in claim 1, characterized in that, The execution steps of the user intent parsing module specifically include: Obtain the user's natural language expression text; The user's natural language expression text is input into a pre-trained BERT model and embedded to generate a sequence of word vectors; The word vector sequence is input into a bidirectional GRU network to perform context modeling on the word vector sequence, forming a context semantic representation sequence; Obtain a knowledge map of Mongolian intangible cultural heritage; The word vectors of the context semantic representation sequence are combined with the nodes of the Mongolian intangible cultural heritage knowledge graph and semantically aligned using vector similarity to obtain matching confidence. The matching confidence scores are sorted, and the top few nodes with high matching scores are selected and divided into two sets according to entity type, including: A set of behavioral entities is generated, and cultural actions that may be involved in the user's intent are extracted from the set of behavioral entities to form a corresponding behavioral demand vector. The pattern entity set is used to extract visual symbols that users are interested in, in order to form a pattern demand vector.

3. The Mongolian intangible cultural heritage display system based on AI and virtual reality as described in claim 1, characterized in that, The execution steps of the behavior generation module specifically include: An action generation network is constructed; the structure of the action generation network includes a behavior label encoder, an action flow template selector, and an action generator; wherein... The behavior label encoder is used to embed the intangible cultural heritage behavior labels into semantic vectors through a lookup table, and retrieves them from the semantic dictionary using a lookup table method without training; the action flow template selector is used to retrieve a predefined action template sequence based on the intangible cultural heritage behavior labels, wherein each template consists of a triplet of action stage, duration, and skeletal topology; the action generator is a 3-layer Transformer structure that accepts the semantic vector and the action template sequence to generate the skeletal keypoint coordinates for each frame; A cultural consistency regularization term is introduced for the coordinates of key points of the skeleton in each frame to penalize stage jumps in the action sequence that do not conform to the cultural process, so as to generate a multi-frame three-dimensional skeleton action sequence.

4. The Mongolian intangible cultural heritage display system based on AI and virtual reality as described in claim 1, characterized in that, The execution steps of the pattern generation module specifically include: The pattern keywords are converted into corresponding semantic vectors and then input into the semantic encoder; Retrieve the closest pattern image from the image library and extract its high-level style features; If the pattern is incomplete, a mask matrix is ​​generated; The output of the semantic encoder, high-level style features, and mask matrix are input into the pattern generator to output a complete pattern as a pattern template image.

5. The Mongolian intangible cultural heritage display system based on AI and virtual reality as described in claim 1, characterized in that, The cultural priori-based action-image fusion engine includes a pattern behavior mapping matrix, a spatial attachment function, and a performance frame scheduling graph.

6. The Mongolian intangible cultural heritage display system based on AI and virtual reality as described in claim 5, characterized in that, The execution steps of the performance content integration module specifically include: Based on the 3D motion data of the multi-frame 3D skeletal motion sequence, a motion stage label sequence is constructed, wherein each motion stage label vector represents the cultural motion semantic label of its corresponding frame. The pattern behavior mapping matrix introduced should display the first frame for each frame. Assign values ​​to each pattern event to obtain the values ​​assigned at the corresponding frame number; The spatial attachment function should be used to display the first frame for each frame. The pattern event rendering location is used to obtain the pattern space mounting sequence. By combining the action phase label sequence and the pattern behavior mapping matrix, an optimization objective function is constructed to minimize the timestamps of two frame nodes on the timeline and whether there is a cultural logic violation in the action / pattern transition, thereby generating a performance frame scheduling graph. In the performance frame scheduling graph, the nodes represent specific performance frames, and the edges represent the cultural logic relationships between frames.

7. The Mongolian intangible cultural heritage display system based on AI and virtual reality as described in claim 6, characterized in that, The spatial attachment function is used to display the first frame in each frame. The pattern space mounting sequence is obtained by determining the rendering location of each pattern event, specifically calculated as follows: ； in, For each frame Mounting location in three-dimensional space The first The three-dimensional coordinates of the left and right hands in the frame; The coordinates of the center point of the chest or head; The weight of the pattern position is controlled to determine whether it is biased towards the area between the hands or the center of the head; The symmetric regularization weight is used to constrain the pattern presentation direction to be consistent with the axis of symmetry of the action. As a totem symmetry penalty function, if the current skeletal posture symmetry is insufficient, the pattern is shifted to the higher side to enhance visual stability; For spatial attachment function, For pattern behavior mapping matrix; according to each frame Mounting position in three-dimensional space This constitutes the pattern space mounting sequence. .

8. The Mongolian intangible cultural heritage display system based on AI and virtual reality according to claim 1, characterized in that, The execution steps of the virtual reality display module specifically include: Build a standardized VR exhibition environment; The VR performance environment is rendered based on the pattern space mounting sequence to generate the rendering result of each pattern in three-dimensional space. A behavior-pattern-time ternary controller is constructed to monitor the position of the performance timeline and check the current state node in the performance frame scheduling diagram, and then call the corresponding action fragment and pattern update. Three types of sensors were configured to integrate multimodal interaction signals.

9. The Mongolian intangible cultural heritage display system based on AI and virtual reality as described in claim 8, characterized in that, In the invocation of the corresponding action fragment and pattern update, a buffer window is set for preloading control.

10. The Mongolian intangible cultural heritage display system based on AI and virtual reality according to claim 8, characterized in that, The three types of sensors include: Voice input: Fixed commands are recognized by the voice recognition module; Head orientation: The coordinates of the user's gaze are returned by the VR headset; Gestures: Gesture information is collected by hand tracking devices; The step of setting up three types of sensors to integrate multimodal interaction signals includes: The sensing signals of the three types of sensors are encoded into events and mapped to node jump triggers in the performance frame scheduling diagram, thereby jumping to the corresponding state in the performance frame scheduling diagram.

Images