A method and system for generating a virtual exhibition of intangible cultural heritage based on multi-modal interaction

Abstract

The application discloses a kind of based on multi-modal interaction's intangible cultural heritage virtual exhibition generation method, it is related to virtual reality field technical field, it includes response to the first multi-modal instruction of user captured by the sensing component of wearable virtual device, determine real-time user intention;According to the real-time user intention, call target exhibition data from structured intangible cultural heritage knowledge graph, build interactive feedback space with the target exhibition data as core;Target exhibition data is input to multi-modal generation model, and immersive exhibition interface containing virtual role action is generated and rendered, and immersive exhibition space is built;The data stream of immersive exhibition space is pushed to wearable virtual device and is output, receives the second multi-modal instruction that user makes in the immersive exhibition space, which is returned by wearable virtual device, to retrieve corresponding interactive content from interactive feedback space, and load to the immersive exhibition space. Realize high immersion and strong interaction of individualization.

Description

Technical Field

[0001] This invention relates to the field of virtual reality, and in particular to a method and system for generating virtual performances of intangible cultural heritage based on multimodal interaction. Background Technology

[0002] Intangible cultural heritage is a form of cultural transmission, but traditional methods of its protection and dissemination have significant shortcomings:

[0003] Existing virtual performance solutions for intangible cultural heritage mostly employ pre-recorded panoramic videos or 3D model roaming technology based on fixed paths. In these solutions, users browse through wearable virtual devices, and the interaction methods are usually limited to playing, pausing, or clicking and jumping within a limited visual area using a controller.

[0004] Specifically, exhibitions are limited by time and space, restricting their audience reach; existing digital dissemination is mostly static, lacking interactivity and immersive experiences; intangible cultural heritage knowledge is scattered, making it difficult to preserve in a structured manner and update in real time; multimodal interaction technology is insufficiently applied in the field of intangible cultural heritage, resulting in low accuracy in intent recognition and a lack of personalized user experience; existing intangible cultural heritage knowledge graphs are mostly static, unable to integrate new research results and user-contributed content in a timely manner; and user intent recognition lacks dynamic optimization, failing to continuously improve interactive effects. All of these factors constrain the effective dissemination and living transmission of intangible cultural heritage. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the aforementioned problems, this invention proposes a method for generating virtual performances of intangible cultural heritage based on multimodal interaction, which solves the issues of low immersion, lack of understanding of user intent, and lack of personalized user interaction during intangible cultural heritage performances.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the technical solution adopted by this invention is as follows: Firstly, this application provides a method for generating virtual performances of intangible cultural heritage based on multimodal interaction, applicable to wearable virtual devices, comprising:

[0009] In response to a first multimodal instruction from the user captured by the sensing components of a wearable virtual device, a real-time user intent is determined; wherein the first multimodal instruction includes text instructions, voice instructions, gesture instructions, and facial expression feedback instructions;

[0010] Based on the real-time user intent, target performance data is retrieved from the structured intangible cultural heritage knowledge graph, and an interactive feedback space is constructed with the target performance data as the core.

[0011] The target performance data is input into a multimodal generation model to generate and render an immersive performance interface containing virtual character actions, and an immersive performance space is built accordingly.

[0012] The data stream of the immersive performance space is pushed to the wearable virtual device for output, and the second multimodal command made by the user in the immersive performance space is received back from the wearable virtual device, so as to retrieve the corresponding interactive content from the interactive feedback space and load it into the immersive performance space.

[0013] The aforementioned solution enables the digital analysis of users' real-time performance needs and intentions during intangible cultural heritage (ICH) experiential learning. This is achieved by collecting interactive behaviors through the sensory components of wearable virtual devices. Then, a structured ICH knowledge graph is used to build a user interaction feedback space based on these real-time performance needs and intentions, identifying all potential performance elements and triggered performance events selected by the user. The performance data selected by the user is transformed into virtual decision-making and movement sequences that can be presented in real-time in three dimensions through a multimodal generation model, achieving an immersive performance where each frame is created in real-time based on the current knowledge data and user intent. Finally, the immersive performance space data stream is output to the user's wearable virtual device. Simultaneously, the device continuously receives new multimodal behaviors naturally generated by the user while experiencing the space and loads them into the immersive performance space, enabling continuous interactive dialogue with the user and allowing ICH performances to achieve a living transmission.

[0014] In one possible implementation of the first aspect, determining the real-time user intent includes:

[0015] In response to the launch command of the wearable virtual device, data acquisition is performed through the sensing components, which include an electromyography sensor, an eye tracker, a voice noise-canceling microphone, and a facial motion capture camera; wherein...

[0016] The electromyography sensor is used to collect the changes in muscle electrical signals corresponding to the gesture command;

[0017] The eye tracker is used to capture the coordinates of the user's gaze focus;

[0018] The voice noise-canceling microphone is used to eliminate ambient noise.

[0019] The facial motion capture camera collects facial motion vectors corresponding to the expression feedback commands by using multiple preset feature point markers.

[0020] Among the aforementioned possible implementations, directly capturing subtle hand gesture changes at the muscle level using electromyography (EMG) sensors avoids visual occlusion and ambient light interference, improving the robustness of gesture recognition; eye trackers accurately capture the user's attention distribution, providing gaze-based information for intent understanding; noise-canceling microphones effectively suppress environmental noise, ensuring the clarity of voice commands; and facial motion capture cameras provide fine-grained facial movement information, enabling the system to perceive the user's emotional state. Multi-sensor collaborative acquisition achieves high-precision, low-interference acquisition of user multimodal commands, facilitating high-quality data for subsequent rendering.

[0021] In conjunction with the above implementation, another possible implementation further includes determining the real-time user intent:

[0022] The text instructions are encoded into semantic vectors using a preloaded BERT model;

[0023] The voice command is converted into an acoustic feature vector using Mel spectrum analysis.

[0024] The gesture command is used to generate a spatial motion vector by fusing the changes in the electromyographic signal with the coordinates of the gaze focus.

[0025] The facial expression feedback command is mapped to an emotional tendency value through the facial motion vector;

[0026] The semantic vector, acoustic feature vector, spatial motion vector, and sentiment tendency value are input into a multimodal fusion model based on an attention mechanism, and the user intent probability distribution is calculated and output.

[0027] Among the aforementioned possible implementations, using the BERT model for deep semantic encoding of text instructions can capture the implicit intent of complex instructions; Mel-spectrum transformation simulates the characteristics of human hearing, extracting acoustic features with strong discriminative power for speech content; the spatial motion vector generated by fusing electromyography signals with gaze coordinates integrates the intensity and spatial direction information of gestures; and the emotional tendency value mapped from facial motion vectors can reflect the user's interest in and emotional state regarding the current content. Dynamically allocating the weights of each modality through an attention mechanism can adaptively fuse multi-source features, suppress misjudgments caused by single-modal noise, and improve the accuracy and comprehensiveness of user intent recognition in complex interaction scenarios.

[0028] In one possible implementation of the first aspect, the structured intangible cultural heritage knowledge graph includes a core layer, an extension layer, and an interaction layer;

[0029] The core layer is used to determine instruction keywords based on the real-time user intent, load them into a graph neural network, and determine the associated target intangible cultural heritage core entity; wherein, the core layer adopts a resource description framework triplet, which is used to store the intangible cultural heritage core entity and the attribute information of the intangible cultural heritage core entity;

[0030] The extension layer matches target audio and video materials based on the target intangible cultural heritage core entity; wherein, the extension layer binds and associates intangible cultural heritage derivative content with the intangible cultural heritage core entity through knowledge hyperlinks;

[0031] The interaction layer generates a personalized performance data sequence as the target performance data based on the target audio and video materials and user preference data; wherein, the interaction layer is used to record the user's historical interaction data and dynamically adjust the weight of the intangible cultural heritage core entity through reinforcement learning algorithm.

[0032] In the aforementioned possible implementations, the core layer uses a resource description framework triplet for structured storage, ensuring the accuracy and consistency of core intangible cultural heritage knowledge. Then, a graph neural network is used to retrieve entities within the knowledge graph, fully leveraging the semantic relationships between intangible cultural heritage entities to accurately identify core content relevant to user intent. The extension layer enriches derivative content through knowledge hyperlinks, broadening the knowledge dimension and allowing users to perceive intangible cultural heritage from multiple perspectives. The interaction layer, based on reinforcement learning, records and analyzes users' historical interaction behavior, dynamically adjusting entity weights to personalize performance content, thereby improving the accuracy of content recommendations and user engagement.

[0033] In conjunction with the above implementation, in another possible implementation, the structured intangible cultural heritage knowledge graph is also used for:

[0034] Execute periodic data scraping tasks to obtain new data;

[0035] The newly added data is associated and matched with the core entities of intangible cultural heritage through entity links;

[0036] If the matching result is a suspected new intangible cultural heritage core entity or attribute information, and the intangible cultural heritage core entity or attribute information belongs to a specific intangible cultural heritage project field, then the new data will be sent to the terminal device of the intangible cultural heritage inheritor corresponding to that field.

[0037] Receive the verification results fed back by the intangible cultural heritage inheritor through their terminal device; wherein,

[0038] If it is a confirmation instruction, the newly added data is directly written to the core layer and the extension layer; if it is a correction instruction, the data is modified according to the received adjustment suggestions and then written.

[0039] If no feedback is received within the preset time period, the newly added data will be temporarily stored in the pending queue, and a manual review reminder message will be generated for the administrator to process.

[0040] Among the aforementioned possible implementation methods, periodic data crawling and entity linking algorithms can automatically discover and associate newly added intangible cultural heritage data, continuously updating the knowledge graph incrementally and maintaining the timeliness of knowledge. Suspected new data is pushed to intangible cultural heritage inheritors in the corresponding fields for verification. Utilizing the inheritors' authoritative domain knowledge to ensure the authenticity and accuracy of the data can effectively prevent erroneous information from entering the knowledge graph. For data that is not promptly reported, a fallback strategy of temporary storage and prompting administrators for manual review is adopted to ensure that important information is not overlooked. Through these multiple mechanisms, a balance is achieved between automated updates and ensuring data accuracy, maintaining the high credibility of the knowledge graph as a fundamental data platform for intangible cultural heritage.

[0041] In one possible implementation of the first aspect, the multimodal generation model includes a 3D scene generation sub-model, an action-driven sub-model, and an emotion rendering sub-model; wherein,

[0042] The 3D scene generation sub-model is based on the neural radiation field, generates a three-dimensional point cloud model according to the description of the intangible cultural heritage scene in the target performance data, and dynamically adjusts the level of detail of the model according to the user's current viewing distance.

[0043] The motion-driven sub-model is based on a generative adversarial network and uses motion capture data of non-heritage inheritors as training samples to generate the motion sequence of the virtual character.

[0044] The emotion rendering sub-model adjusts at least one of the scene lighting parameters, background music rhythm value, and particle effect density in real time according to the emotion tendency value corresponding to the expression feedback command.

[0045] The 3D point cloud model, the virtual character action sequence, and the adjusted scene parameters are synthesized by the rendering engine and encapsulated into a real-time data stream of the wearable virtual device based on a real-time communication protocol.

[0046] Among the aforementioned possible implementations, generating 3D scenes through neural radiation fields can restore the realistic details of intangible cultural heritage scenes; dynamically adjusting the level of detail by viewing distance ensures visual quality, i.e., realistic details, while adapting to the computing power of wearable devices and avoiding rendering latency. A motion-driven sub-model based on generative adversarial networks and trained with motion capture data from inheritors can generate virtual character animations with natural movements and conforming to skill standards, enhancing the realism of interaction. An emotion rendering sub-model adjusts lighting, music, and special effects in real time based on emotional tendency values, making the performance atmosphere resonate with the user's emotions and enhancing emotional resonance. The outputs of these three elements are synthesized through a rendering engine and encapsulated and transmitted using a real-time communication protocol, ensuring the smoothness and low latency of the performance visuals and interactive feedback, effectively suppressing dizziness.

[0047] In conjunction with the above implementation, in another possible implementation, the training process of the action-driven sub-model includes:

[0048] Acquire motion capture data of the intangible cultural heritage inheritor; wherein, the motion capture data includes annotation text describing the key points of the movements;

[0049] The annotation text is converted into structured motion adjustment information, which includes changes in the force of the motion, the speed of the rhythm, and the range of joint angles.

[0050] Based on the action adjustment information, a constraint term is introduced into the loss function of the generative adversarial network. This constraint term penalizes the deviation between the generated action and the structured action adjustment information, so that the trained target action-driven sub-model fits the motion capture data while also conforming to the annotated text.

[0051] When the action-driven sub-model generates a new action sequence, it fine-tunes the generated virtual character action sequence based on the structured action adjustment information.

[0052] Among the aforementioned possible implementations, extracting structured motion adjustment information from the inheritor's annotation text and using it as a constraint term in the generative adversarial network loss function enables the generated motion to not only closely approximate the actual captured data in appearance but also conform to the inheritor's experiential norms in terms of the essence of the technique, such as force, rhythm, and joint angles. Fine-tuning the motion using the adjustment information during the inference phase can adapt to the interactive input deviations of different users, effectively improving the accuracy and standardization of digital inheritance and virtual teaching of intangible cultural heritage skills.

[0053] In one possible implementation of the first aspect, receiving the second multimodal instruction made by the user in the immersive performance space, transmitted back by the wearable virtual device, includes:

[0054] Receive the first back-transmission data and determine the target wearable virtual device corresponding to the first back-transmission data; wherein, the first back-transmission data includes user behavior data and user physiological indicator data;

[0055] Based on the target wearable virtual device, the data stream of the target wearable virtual device in the immersive performance space is adapted and encapsulated;

[0056] Based on user behavior data, determine the user's interactive operations;

[0057] The user's real-time focus level is determined based on the user's physiological indicators.

[0058] Based on the interactive operation, the performance interface to be adjusted in the immersive performance space is determined, and based on the real-time focus, the performance interface to be adjusted is adjusted to the best performance perspective.

[0059] Among the aforementioned possible implementations, by identifying the specific wearable device type corresponding to the returned data and adapting and encapsulating the data stream, it is possible to be compatible with various forms of virtual devices, ensuring a consistent and high-quality experience. By combining user behavior data analysis of interactive operations and assessing focus based on physiological indicators, the performance interface can be autonomously adjusted to the most favorable perspective for user information acquisition, reducing the burden of manual adjustment, making the interaction process more natural and coherent, and enhancing user immersion.

[0060] In one possible implementation of the first aspect, receiving the second multimodal command made by the user in the immersive performance space, transmitted back by the wearable virtual device, further includes:

[0061] Construct a user profile database, which includes: user historical interaction records, preference tags, and behavioral feature vectors;

[0062] After determining the real-time user intent, the first multimodal instruction is matched with historical data in the user profile database for similarity.

[0063] If the match is successful, historical optimization parameters are retrieved to correct the real-time user intent.

[0064] If the matching fails, a cold start learning mechanism is triggered. The user profile database is updated based on the feedback analysis of the second multimodal instruction through a reinforcement learning algorithm, so that the intent recognition accuracy converges to the target threshold after the number of times the user uses the device reaches a preset condition.

[0065] The user profile database provides a manual editing interface, and the user's preference tags are generated based on the priority of the editing instructions received through the manual editing interface.

[0066] Among the aforementioned possible implementations, historical matching of current multimodal commands based on a user profile database can reuse optimized parameters learned from past user interactions, quickly outputting intent parsing results that better match individual habits and improving response efficiency. The cold-start learning mechanism utilizes reinforcement learning to quickly adjust the user profile based on subsequent user feedback, ensuring that recognition accuracy steadily improves with the number of interactions for new users or new intent scenarios. Simultaneously, an open manual editing interface is provided and given higher priority, allowing users to actively modify preference tags, enhancing their sense of control and trust in the system, and making the personalized experience more aligned with their subjective desires.

[0067] Secondly, this application also proposes a virtual performance generation system for intangible cultural heritage based on multimodal interaction, including:

[0068] Intent recognition module: used to respond to the user's first multimodal command captured by the perception component of the wearable virtual device and determine the real-time user intent; wherein, the first multimodal command includes text command, voice command, gesture command and facial expression feedback command;

[0069] Interactive feedback module: used to call target performance data from the structured intangible cultural heritage knowledge graph according to the real-time user intent, and to construct an interactive feedback space with the target performance data as the core;

[0070] Immersive performance module: used to input the target performance data into the multimodal generation model to generate and render an immersive performance interface containing virtual character actions, and build an immersive performance space accordingly;

[0071] Interaction adjustment module: used to push the data stream of the immersive performance space to the wearable virtual device for output, and receive the second multimodal command made by the user in the immersive performance space from the wearable virtual device, so as to retrieve the corresponding interactive content from the interaction feedback space and load it into the immersive performance space.

[0072] The above solution integrates multimodal interaction recognition, knowledge graph-driven personalized content retrieval, and real-time rendering based on deep generative models. It can provide users with a highly immersive, interactive, and dynamically adaptable virtual performance experience that can adapt to personal preferences and states, effectively enhancing the dissemination of intangible cultural heritage and user participation.

[0073] (III) Beneficial Effects

[0074] The beneficial effects of this invention are as follows:

[0075] This application encodes text commands, voice commands, gesture commands, and facial expression feedback commands into semantic vectors, acoustic feature vectors, spatial motion vectors, and sentiment values, respectively, and inputs them into a multimodal fusion model based on an attention mechanism. This enables accurate understanding of the user's real-time viewing and learning intentions during the interactive performance process. Furthermore, by constructing a structured intangible cultural heritage knowledge graph with a three-layer architecture comprising a core layer, an extension layer, and an interaction layer, the application enhances the detailed representation of user intentions and enriches the learning of intangible cultural heritage. High-fidelity 3D scenes are generated through neural radiation fields, and realistic virtual inheritor movements are generated through generative adversarial networks. Lighting, music, and particle effects are adjusted in real time based on the user's sentiment values, allowing the performance content to resonate with the user's behavior and emotions. This breaks through the static limitations of traditional pre-recorded content. Finally, by collecting the user's second multimodal commands and physiological index data, the application feeds them back into the interactive feedback space to drive content adjustments, achieving real-time dynamic evolution of the performance content.

[0076] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0077] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the architecture of a virtual performance generation system for intangible cultural heritage based on multimodal interaction, as described in an embodiment of the present invention.

[0079] Figure 2 This is a diagram showing the operating system composition of a virtual performance generation system for intangible cultural heritage based on multimodal interaction, as described in an embodiment of the present invention.

[0080] Figure 3 This is a flowchart illustrating a method for generating virtual performances of intangible cultural heritage based on multimodal interaction, as described in an embodiment of the present invention.

[0081] Figure 4 This is an interactive flowchart in an embodiment of the present invention, showing a scenario where a user imitates intangible cultural heritage skills and receives corrective feedback. Detailed Implementation

[0082] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] To facilitate understanding of the technical solutions of the embodiments of this application, a brief introduction to the relevant technologies and terms of this application is given first.

[0084] Wearable virtual reality devices refer to electronic devices that users can wear and that can generate and present virtual visual content. They can take the form of an all-in-one virtual reality headset, augmented reality glasses, or mixed reality device. In this application, the core challenge facing wearable virtual reality devices is how to generate and render highly immersive and interactive intangible cultural heritage performance content in real time, based on the user's complex multimodal behavior, under conditions of limited local computing resources.

[0085] Multimodal commands refer to a set of commands issued by a user simultaneously or quasi-synchronously through different information channels under the same interactive intent. In this application, they include at least text commands, voice commands, gesture commands, and facial expression feedback commands. Existing technologies typically handle single-modal commands in isolation, while this application aims to address how to integrate these heterogeneous and asynchronous commands to accurately understand the user's complex interactive intent regarding intangible cultural heritage.

[0086] Structured Intangible Cultural Heritage Knowledge Graph: This refers to a large-scale directed graph network database composed of intangible cultural heritage projects, inheritors, techniques, tools, and cultural spaces as entity nodes, and the semantic relationships between them as directed edges. It differs from general encyclopedic knowledge graphs in that its hierarchical structure—core layer, extension layer, and interaction layer—is specifically designed to balance the authority, richness, and personalized interaction of intangible cultural heritage knowledge.

[0087] Interactive Feedback Space: This is not a physical virtual 3D space, but rather a logical data structure built into the device's memory to define the current presentation session state. The Interactive Feedback Space contains hotspots for all potentially interactive objects under the current theme, script indexes for triggerable response events, and the default content layout for the presentation route. It is a crucial middleware connecting user intent recognition and immersive content generation.

[0088] Multimodal generative model: refers to an integrated deep learning algorithm inference framework that does not directly read and decode pre-made video or 3D model asset files, but instead uses neural networks to calculate in real time the geometric information, texture, lighting, and skeletal animation data of virtual characters that constitute the virtual scene based on the input structured data.

[0089] In existing technologies, the digital display of intangible cultural heritage often uses pre-recorded panoramic videos or fixed-path 3D model tours. A common practice is that after wearing a VR headset, users can only play, pause, or select and navigate within a limited visual area using a controller.

[0090] Common practices suffer from the following significant drawbacks: First, the interaction dimension is singular, failing to perceive multimodal inputs such as user voice, gestures, and facial expressions, resulting in users remaining passive viewers. Second, the content presentation is identical for everyone, unable to dynamically adjust the performance content based on the user's real-time interests and background knowledge. For example, the same video stream might be pushed to a user wanting to delve deeper into Suzhou embroidery techniques and a user only browsing the overview. Third, there is a lack of emotional feedback loops; adjustments to ambient lighting, background music, and virtual character movements are rigid, failing to resonate with the user's emotional state and resulting in low immersion. Based on this, this application provides a method for generating virtual performances of intangible cultural heritage based on multimodal interaction. Using wearable virtual devices as the interaction terminal, it constructs a complete closed-loop system from multimodal intent perception, knowledge graph-driven content retrieval, real-time rendering by a deep generative model, to dynamic adjustment of the immersive space based on user physiological and behavioral feedback, thereby addressing the aforementioned problems.

[0091] Figure 1 This is a schematic diagram of the architecture of a virtual performance generation system for intangible cultural heritage based on multimodal interaction, provided in an embodiment of this application. The system architecture is entirely deployed on a standalone wearable virtual device 100. The wearable virtual device 100 includes at least one system-on-a-chip (SoC) 110, which integrates a central processing unit 111, a graphics processing unit 112, and a neural network processing unit 113. The wearable virtual device 100 also includes a memory 120, a sensing component array 130, a display component 140, and a communication bus 150. The sensing component array 130 further includes an electromyography (EMG) sensor 131, an eye tracker 132, a voice noise-canceling microphone 133, and a facial motion capture camera 134, used to capture the user's multimodal commands.

[0092] At the level of logical function Figure 2 This is a schematic diagram of the system composition provided in this application embodiment. The operating system 160 integrates an intent recognition module 161, an interactive feedback module 162, an immersive performance module 163, and an interactive adjustment module 164. The scheduling and calculation of each module are uniformly completed by the system-on-a-chip 110, without relying on a remote server, thereby avoiding the disruption of the immersive experience caused by network latency. The intent recognition module 161 obtains the sensor data stream from the sensing component array 130 through the communication bus 150 and parses out the real-time user intent; the interactive feedback module 162, based on the real-time user intent, retrieves and constructs the interactive feedback space from the structured intangible cultural heritage knowledge graph 121 located in the memory 120; the immersive performance module 163 calls the multimodal generation model 165 to generate real-time content and drives the display component 140 to present the immersive performance interface through the rendering engine; the interactive adjustment module 164 is responsible for receiving the user's new round of multimodal instructions and physiological indicator data, forming a feedback loop, and dynamically adjusting the performance content.

[0093] The following is combined with Figure 3 This application provides a detailed description of a method for generating virtual performances of intangible cultural heritage based on multimodal interaction, as provided in an embodiment of this application. The figure is a schematic flowchart illustrating this method. This method can be executed by a wearable virtual device 100 and includes the following steps:

[0094] Step S101: In response to a first multimodal instruction from the user captured by the perception component of the wearable virtual device, determine the real-time user intent; wherein the first multimodal instruction includes text instructions, voice instructions, gesture instructions, and facial expression feedback instructions.

[0095] The purpose of step S101 is to collect, encode, and integrate the user's interactive commands scattered across multiple channels, accurately analyze the user's intention to perform intangible cultural heritage interaction at the current moment, and determine the precise index information for subsequent content calls.

[0096] In one possible implementation, data acquisition is performed via sensing components in response to a startup command from the wearable virtual device. The sensing components include an electromyography (EMG) sensor, an eye tracker, a noise-canceling microphone, and a facial motion capture camera, collectively forming a sensing component array 130. Each sensor in the sensing components synchronously or quasi-synchronously captures raw data constituting the first multimodal command within a preset short time window (e.g., 500 milliseconds). Specifically, the EMG sensor acquires the change in electromyographic signals corresponding to the user's gesture commands; the eye tracker captures the coordinates of the user's gaze focus; the noise-canceling microphone eliminates environmental noise using a built-in adaptive filtering algorithm to obtain clear voice commands; the facial motion capture camera acquires facial motion vectors corresponding to facial expression feedback commands by marking multiple preset feature points (e.g., 68 feature points). The startup commands include a startup command for the wearable virtual device 100 and a performance startup command to start the wearable virtual device 100 for a non-heritage cultural performance; the first multimodal command is a comprehensive command composed of text commands, voice commands, gesture commands, and facial expression feedback commands. In this application, an electromyography (EMG) sensor is used to capture gesture commands, directly sensing muscle activity signals. This avoids misjudgments caused by hand occlusion or changes in ambient lighting, which are common in purely visual gesture recognition, thus improving the robustness of recognizing fine movements in intangible cultural heritage techniques, such as Suzhou embroidery thread splitting and cloisonné enamel application. An eye tracker reflects the user's attention distribution in real time, providing interference-free spatial orientation for intent understanding. Combining these two technologies with voice pointing, text commands, and facial expressions, and through the collaboration of multiple sensors, the user's intent, i.e., the specific operational behavior, is comprehensively determined.

[0097] In another possible implementation, the process of determining real-time user intent further includes the extraction and fusion of multimodal features. Specifically, text commands are encoded into semantic vectors using a pre-loaded BERT model to capture deep semantic relationships between words in the command; voice commands are transformed using Mel-spectrum transformation to obtain acoustic feature vectors that characterize their content and emotion. Mel-spectrum transformation effectively extracts key acoustic features related to content and emotion in speech by simulating the nonlinear perception characteristics of the human ear at frequencies, while suppressing interference from environmental noise. Applied in this application, this allows users' voice query commands for intangible cultural heritage projects to be accurately converted into highly discriminative feature representations even in noisy outdoor or exhibition hall environments. Gesture commands are generated by fusing the changes in electromyography signals with the coordinates of the gaze focus to create a spatial motion vector that describes both the intensity and direction of the movement; facial expression feedback commands are mapped to one or more emotion tendency values ​​by analyzing the facial motion vector. Finally, the semantic vectors, acoustic feature vectors, spatial motion vectors, and emotion tendency values ​​are input into a multimodal fusion model based on an attention mechanism. The multimodal fusion model dynamically calculates the relevance weights of each modal feature to the final intent, outputting a user intent probability distribution vector. This allows the performance to not only showcase works from intangible cultural heritage but also demonstrate the corresponding professional techniques, enhancing the immersive understanding of intangible cultural heritage products and the transmission of techniques at the time. The wearable virtual device selects the intent label with the highest score from this vector as the real-time user intent determined in this inference. BERT, Bidirectional Encoder Representations from Transformers, is a semantic vector based on Transformers.

[0098] For example, suppose a user is visiting a virtual exhibition about Suzhou embroidery, an intangible cultural heritage. The user's gaze is focused on a beautiful double-sided embroidery piece. At this moment, an eye tracker detects that the user's focus coordinates are in the area of ​​the exhibit. The user makes a subtle hand gesture mimicking splitting a thread, an electromyography (EMG) sensor detects changes in the electrical signals of the corresponding muscle group, and the user softly says, "I want to learn about this needlework technique." A noise-canceling microphone captures a clear audio stream, while the user's face displays micro-expressions of curiosity and focus. A facial motion capture camera records the motion vectors of feature points such as a slight upturn of the mouth and widening of the eyes. The wearable virtual device fuses the eye-tracking coordinates and EMG signals to generate spatial motion vectors, converts the speech into acoustic feature vectors, and maps facial expressions into emotional tendencies representing strong interest. A multimodal fusion model weights and fuses multiple behavioral features such as gaze direction, the splitting gesture, the voice question about the needlework technique, and strong interest to calculate the intent probability distribution. For example, in one possible implementation, the confidence level for querying details of Suzhou embroidery split stitch is 0.9, with the highest confidence level being 1. When this is far greater than other intentions, the device determines the user's intention accordingly.

[0099] In this step, the allocation of attention mechanism weights in the multimodal fusion model can be dynamically adjusted based on the current interaction context. For example, when the ambient noise volume exceeds a preset threshold, the attention weights corresponding to the speech modality will be automatically reduced by the model, while the weights of modalities less susceptible to noise interference, such as gestures and eye movements, will be increased accordingly to ensure the robustness of intent recognition.

[0100] Step S102: Based on the real-time user intent, retrieve the target performance data from the structured intangible cultural heritage knowledge graph, and construct an interactive feedback space with the target performance data as the core.

[0101] The purpose of step S102 is to perform precise retrieval and personalized recommendations in the structured intangible cultural heritage knowledge graph based on the determined user intent, and to organize the retrieval results into a logical context container that can be efficiently queried and triggered in subsequent steps.

[0102] In one possible implementation, the structured intangible cultural heritage knowledge graph adopts a three-layer architecture: a core layer, an extension layer, and an interaction layer. The core layer extracts keywords from commands based on real-time user intent and loads them into a graph neural network for entity retrieval. The graph neural network can propagate information along directed edges in the knowledge graph, such as "belongs to," "inherited from," and "used tools," accurately locating the target intangible cultural heritage core entity even if the command keywords do not perfectly match the entity name. Data within the core layer is stored using resource description framework triples, ensuring the standardization and consistency of knowledge representation. The extension layer, based on the located target intangible cultural heritage core entity, matches associated intangible cultural heritage derivative content through pre-defined knowledge hyperlinks. For example, intangible cultural heritage derivative content includes associated documentary clips, audio interviews with inheritors, academic paper abstracts, or high-precision 3D scan models. The interaction layer records the user's historical interaction data and runs reinforcement learning algorithms to dynamically adjust the recommendation weights of each intangible cultural heritage core entity based on the user's past clicks, pauses, skips, and other behaviors. Finally, by combining the audio and video materials retrieved from the extension layer and the user preference data generated by the interaction layer, a personalized performance data sequence is generated as the target performance data for this interaction.

[0103] In one possible implementation, the core layer uses a relational graph convolutional network (R-GCN) as the graph neural network. It converts the real-time user intent labels determined in step S101 or the keywords extracted from the text instructions into query vectors through a pre-trained word embedding model. The query vector is concatenated with the initial embedding vectors of candidate entity nodes in the knowledge graph, serving as the input node features for the R-GCN. The R-GCN updates the representation of the current node by aggregating information from neighboring nodes layer by layer. Its message passing process considers the type of directed edges in the knowledge graph, assigning different weight matrices to different types of relationships. After multiple layers of propagation, each entity node obtains a final embedding vector that incorporates the semantic information of its multi-hop neighborhood. During retrieval, the cosine similarity between the query vector and the final embedding vectors of all candidate entities is calculated, and the entity with the highest similarity exceeding a preset threshold is selected as the target intangible cultural heritage core entity. Since the R-GCN embedding can capture the structural context of entities, in actual implementation, even if the keywords in the user instruction are not exactly the same as the entity name, as long as they are adjacent in the topological structure of the knowledge graph, they can still be successfully matched through similar embedding vectors. By leveraging R-GCN's ability to explicitly model relation types, it is possible to accurately capture the semantic associations of intangible cultural heritage performances in the intangible cultural heritage field. Entity retrieval is not limited to surface string matching, but is based on deep semantic understanding of knowledge, thereby improving the recall and accuracy of queries.

[0104] After acquiring the target presentation data, the processor pre-constructs a logical interactive feedback space in memory. This interactive feedback space is a context container that defines the current presentation session state. It contains the hotspot coordinates of all potential virtual objects under the current theme that can be interacted with by the user, the indexes of response event scripts that can be triggered by gestures or voice, and the default content layout of the current presentation route. The interactive feedback space defines the boundaries and possible branching storylines of all elements that the user can touch and that can elicit feedback in the subsequent immersive experience.

[0105] For example, continuing from the previous step where the user intent was to query details of Suzhou embroidery splitting stitch techniques, the device extracts the keywords "Suzhou embroidery," "split stitch," and "stitch technique" from the user intent. The graph neural network searches in the core layer, locating the core entity "Suzhou embroidery" and further associating it with its attribute, the splitting stitch technique. Subsequently, the extension layer is invoked, matching links to derivative content such as close-up videos of Suzhou embroidery splitting stitch techniques, audio recordings of a national-level inheritor explaining the splitting techniques, and 3D capture data of the splitting action. The interaction layer analyzes the current user's history, discovering a preference for short videos and high-precision model interactions. Therefore, the reinforcement learning model increases the weight of close-up videos and 3D capture data. Ultimately, the generated target performance data sequence prioritizes motion capture data and high-definition videos. Based on this, an interactive feedback space is constructed, defining the silk threads on the virtual embroidery frame as interactive objects. When the user looks at the silk threads on the virtual embroidery frame and makes a pinching gesture, a response event is triggered to display a 3D teaching animation of the splitting action; when the user asks by voice how many threads to split, a script is triggered to play the audio recording of the inheritor explaining the technique.

[0106] As an optional embodiment, the aforementioned structured intangible cultural heritage knowledge graph also features incremental updates to resolve the contradiction between the dynamic evolution of intangible cultural heritage knowledge and the static storage of the knowledge graph. The knowledge graph periodically performs data scraping tasks, acquiring new data from official intangible cultural heritage databases, academic literature platforms, etc. This new data consists of related skill data or product data associated with different intangible cultural heritage performance techniques and products. Through entity links, the new data is matched with existing entities in the core layer in multiple dimensions. If the matching result indicates a suspected new intangible cultural heritage core entity or attribute, and the corresponding intangible cultural heritage core entity or attribute can be clearly attributed to a specific intangible cultural heritage project area, then the suspected data is automatically pushed to the terminal device of the intangible cultural heritage inheritor in that area. The inheritor can confirm the authenticity and accuracy of the data or propose correction suggestions through the terminal. If the received feedback is a confirmation instruction, the new data is directly written into the core layer or extended layer; if it is a correction instruction, it is modified according to the adjustment suggestions before being written. If the preset time limit is exceeded (for example, a preset time limit of 7 days), and no feedback is received within 7 days, to prevent data omission, the system temporarily stores the newly added data in a pending queue and generates a manual review reminder message for the administrator. This mechanism strikes a balance between the efficiency of automated knowledge updates and the authority required for intangible cultural heritage knowledge.

[0107] Step S103: Input the target performance data into the multimodal generation model to generate and render an immersive performance interface containing virtual character actions, and build an immersive performance space accordingly.

[0108] The purpose of step S103 is to use a deep generative model to transform the structured target presentation data obtained in step S102 into a virtual 3D scene with rich interactive elements that can be directly experienced by the senses in real time.

[0109] In one possible implementation, the multimodal generation model comprises three cooperating sub-models: a 3D scene generation sub-model, an action-driven sub-model, and an emotion rendering sub-model. First, the 3D scene generation sub-model, based on neural radiation field technology, receives descriptions of intangible cultural heritage scenes from the target performance data, for example, a softly lit Suzhou garden-style embroidery workshop, and generates a corresponding 3D point cloud model. Simultaneously, to adapt to the limited computing resources of wearable devices, the 3D scene generation sub-model dynamically adjusts the model's level of detail based on the user's current viewing distance. When the user approaches the embroidery frame, the highest precision silk thread texture and wood grain details are loaded; when the user moves away, the model's polygon count is simplified to ensure a high rendering frame rate. In this application, the 3D scene generation sub-model utilizes neural radiation field technology to learn the continuous volume density and color functions present in the scene from a sparse set of two-dimensional images. Then, during rendering, it can generate intangible cultural heritage scenes with highly realistic details; for example, the three-dimensionality of carved window lattices and the natural curvature of silk threads. This means that in actual implementation, the construction of virtual workshops and other scenarios no longer requires a lot of manpower for detailed 3D modeling, significantly reducing content production costs and improving the fidelity of details.

[0110] Secondly, the action-driven sub-model is based on generative adversarial networks (GANs) and uses high-precision motion capture data from intangible cultural heritage inheritors as training samples to generate action sequences for virtual characters in real time. For example, based on data from a wire-splitting performance, it generates a coherent animation of a virtual inheritor flexibly splitting wires with both hands. The action-driven sub-model can generate visually highly realistic virtual inheritor animations with natural and smooth motion details. Compared to rule-based action synthesis or simple loops of pre-recorded actions, GANs can learn the latent distribution in action data, thereby generating diverse and non-repetitive response actions during user interaction, enhancing the immersion and realism of virtual teaching.

[0111] Finally, the emotion rendering sub-model adjusts the scene's visual and auditory parameters in real time based on the emotion tendency value corresponding to the user's facial expression feedback command parsed in step S101. When the emotion tendency value indicates strong user interest, the scene's focused lighting can be brightened, the background music rhythm adjusted to a soothing pace for calm viewing, and particle effects such as falling petals or flickering fireflies can be increased to enhance the scene's appeal. Ultimately, the generated virtual character action sequence, adjusted scene parameters, and 3D point cloud model are synthesized by the rendering engine and encapsulated into a real-time data stream suitable for transmission on a local bus or display interface based on a real-time communication protocol. This forms a continuous visual display with rich interactive hotspots on the head-mounted display screen, creating a complete immersive performance space. In this application, the rendering engine uses a mobile-optimized version of Unity or Unreal Engine, and the preferred communication protocol is WebRTC. The WebRTC protocol encapsulates and transmits rendered content. By leveraging its P2P direct connection, adaptive bitrate, and hardware-accelerated encoding and decoding capabilities, it can keep the transmission latency of immersive performance content at an extremely low level, thus preventing users from experiencing dizziness during actual implementation.

[0112] In this application, the action-driven sub-model is based on a generative adversarial network and uses non-hereditary inheritor motion capture data as training samples. It consists of two parts: a generator and a discriminator. The generator employs a fully convolutional network with an encoder-decoder structure. The encoder consists of four convolutional layers, each followed by a batch normalization layer and a Leaky ReLU activation function, used to map the input random noise vector and action category label into high-dimensional features. The decoder consists of four deconvolutional layers, each followed by a batch normalization layer and a ReLU activation function, used to progressively restore the features to an output with the same dimension as the target action sequence. The matrix, where For frame number, This represents the number of key points. The discriminator uses a 3-layer convolutional neural network (PatchGAN structure). Each convolutional kernel is 3x3 in size, with a stride of 2, and the activation function is LeakyReLU. The discriminator ultimately outputs a... The matrix represents the probability that each element of a local patch in the input action sequence is a real sample. Constraints on action details are achieved by identifying local patches. The loss function of the action-driven sub-model is... ;

[0113] ;

[0114] in, The conditional adversarial loss is expressed as the standard cross-entropy loss and is used to determine whether the generated action sequence is indistinguishable from the real action sequence in terms of distribution. As a constraint term in the action specification, this example uses mean squared error (MSE) loss, calculated using the following formula: ; The motion sequence parameters (such as joint angles) output by the generator. The structured motion adjustment information extracted from the inheritor's annotation text is represented by the expected vector consisting of force, rhythm, and joint angle range; The hyperparameters are used to balance the two losses and control the weights of motion realism and skill standardization.

[0115] During the training of the action-driven sub-model, the Adam optimizer was used, with an initial learning rate of 0.0002 and momentum parameters β1=0.5 and β2=0.999. The model was trained iteratively for 500 epochs, with a batch size of 32. With these settings, ·+ was able to generate virtual character actions that both conformed to the distribution of inheritor action data and strictly adhered to the key points of intangible cultural heritage techniques.

[0116] For example, in conjunction with the embodiment of Suzhou embroidery thread splitting described above, the 3D scene generation sub-model generates a 3D point cloud scene containing carved window lattices, a solid wood embroidery frame, and scattered silk threads using neural radiation fields, based on the description of the Suzhou embroidery workshop. When the user approaches the embroidery frame, the level of detail of the embroidery frame model switches from low dynamics to high, and the texture details of the silk threads become clearly discernible. After receiving the thread splitting motion capture data, the motion-driven sub-model outputs a skeletal animation sequence of a virtual inheritor skillfully splitting a silk thread into multiple strands using their fingers. At this time, because the emotion tendency value indicates that the user is highly focused, the emotion rendering sub-model slightly lowers the volume of the background music and adds a soft spotlight effect to the embroidery frame area to guide the user's attention to the finger movements. The rendering engine synthesizes these three elements, and what the user ultimately sees is: in an antique embroidery workshop, a virtual inheritor is sitting in front of the embroidery frame, demonstrating the exquisite skill of splitting silk threads under a spotlight, with the faint sound of a guqin in the background.

[0117] As an optional implementation, the training process of the aforementioned action-driven sub-model is key to solving the problem of generated actions merely resembling the form but failing to capture the essence of the inheritor's skills. First, motion capture data from the inheritor is acquired. This data includes not only skeletal movement trajectories but also annotations from the inheritor regarding the key points of the movements. For example, when splitting a line, the wrist should be suspended in the air, the force should be light and even, and the rhythm should be slow at first and then steady. The system converts this unstructured annotation text into structured action adjustment information through natural language processing, including changes in the force of the movement, the speed of the rhythm, and the range of joint angles. Subsequently, in the loss function of the generative adversarial network, in addition to the traditional discriminator loss, a constraint term is added. This constraint term is specifically used to calculate the deviation between the various parameters of the generated action and the aforementioned structured action adjustment information, and to penalize the deviation. This not only fits the original motion capture data during training but also forces the model to learn and adhere to the inheritor's experiential rules. During the stage of generating the performance interface through reasoning, when it is necessary to generate new action variations based on user interaction, the model will still fine-tune the generated virtual character action sequence based on the internalized structured action adjustment information to ensure that any action variation is within the allowable range of the skill specifications.

[0118] S104. Push the data stream of the immersive performance space to the wearable virtual device for output, and receive the second multimodal command made by the user in the immersive performance space from the wearable virtual device, so as to retrieve the corresponding interactive content from the interactive feedback space and load it into the immersive performance space.

[0119] The purpose of step S104 is to efficiently output the generated immersive content to the user and simultaneously establish a channel for processing subsequent user feedback, forming a complete closed loop of perception-generation-feedback-adjustment, and maintaining the continuity and dynamic adaptability of the immersive experience.

[0120] In one possible implementation, the wearable virtual device encodes and compresses the rendered binocular stereo image frames, spatial audio stream, and possible haptic feedback commands into a data stream, which is then directly output to its own display screen and audio output unit. During the experience, the user generates new behaviors, forming second-modal commands. The wearable virtual device receives this feedback data. First, the type of the target wearable virtual device from which the feedback data originates is determined. Target wearable virtual device types include VR headsets and AR glasses; one or more are selected based on the actual scenario, and the data stream is adapted and encapsulated according to the device's characteristics. The feedback data includes at least user behavior data, such as the user's gestures, voice, eye movements, and physiological indicators, namely the user's heart rate and skin conductance. The device determines the user's interactive operations based on the behavioral data, including gazing at and grasping virtual objects, and determines the user's real-time focus or emotional arousal level based on the physiological indicator data. For example, when the heart rate variability is less than or equal to a first preset value or the skin conductance value is greater than or equal to a second preset value, the user is determined to be in a high-focus or high-emotional state. Subsequently, the processor uses the interaction operation vector and state determination as query signals to retrieve the corresponding pre-loaded interaction content scripts within the interaction feedback space constructed in step S102. Specifically, the interaction content scripts include rotating exhibits and displaying the bottom signature, triggering hidden storylines, and potentially adjusting the exhibition interface to the optimal viewing angle based on the user's real-time focus. Finally, this newly retrieved content and adjustment instructions are overlaid or applied in real-time to the generation model and rendering engine described in step S103, dynamically changing the current immersive exhibition space and completing one round of interactive loop.

[0121] For example, while watching a Suzhou embroidery thread-splitting demonstration, a user is drawn to a small pair of scissors on the embroidery frame, focusing their gaze on the scissors' eye-tracking data for an extended period. Simultaneously, the user reaches out and makes a grasping and rotating motion (electromyography and gesture data). The wearable virtual reality device receives this behavioral data in real time and analyzes the interactive operation of rotating the scissors. At the same time, a heart rate sensor detects a slight increase in the user's heart rate, indicating heightened focus. In the interactive feedback space, the scissors are a defined hotspot object, carrying a script that responds to the rotation gesture. The device immediately triggers this script, retrieving detailed 3D model data of the scissors and an audio introduction of their functions from the extended layer. The immersive presentation module then responds to the user's gesture rotation angle, rendering high-definition details of each side of the scissors in real time and fine-tuning the presentation view to the optimal viewing position directly above the scissors. Throughout this process, the Suzhou embroidery scene in the background does not disappear but exists as a blurred depth background, allowing the user to return to the previous presentation space at any time via gestures or voice commands.

[0122] As an optional embodiment, to achieve more accurate long-term intent optimization, the method of this application further includes constructing and utilizing a user profile database. This database contains user historical interaction records, preference tags, and behavioral feature vectors. After determining the real-time user intent in step S101, this intent is not immediately used as the final result. Instead, the current first multimodal instruction features are first matched with historical data in the user profile database for similarity. If the match is successful, it indicates that the user has had a similar interaction before, and the system directly retrieves historical optimization parameters to correct the current intent, making the result more consistent with the user's consistent habits. If the match fails, it means that this is a completely new interaction intent or a completely new user, triggering a cold-start learning mechanism. Through a reinforcement learning algorithm, based on the user's subsequent feedback from the second multimodal instruction, such as whether to delve deeper or quickly skip, as a reward signal, the preference tags and behavioral feature vectors in the user profile database are updated. Through multiple iterations, the Top-1 accuracy of intent recognition converges to the target threshold after the number of user interactions reaches a preset condition (e.g., 5 times). It is worth noting that the user profile database provides users with a manual editing interface, allowing them to directly modify their preference tags. Furthermore, manual editing commands have higher priority than tags automatically generated by the algorithm, which gives users ultimate control over their own data profile.

[0123] To support the above solution, a specific scenario example is provided below.

[0124] Example 1: A scenario where a user imitates an intangible cultural heritage skill and receives corrective feedback.

[0125] Figure 4 The diagram illustrates the interactive flow in a scenario where a user imitates an intangible cultural heritage technique and receives corrective feedback. In this scenario, the user wants to try the cloisonné enamel technique by hand in a virtual environment.

[0126] In step S401, the user looks at a virtual cloisonné copper base and issues a voice command: "I want to try enamel application." Simultaneously, the user makes a gesture resembling holding a dropper. The device, through multimodal intent recognition, determines that the user's intention is to enter an interactive simulation mode of the cloisonné enamel application technique.

[0127] Step S402: Retrieve target performance data related to cloisonné enamel-dotting from the knowledge graph. The core of this data is the motion capture data of the inheritors. Construct an interactive feedback space, which defines virtual droppers, pigment dishes, etc., as interactive tools. The user's pinching, dripping, and other gestures will trigger different responses.

[0128] In step S403, the action-driven sub-model invokes the standard blue-dots action sequence of the inheritor to generate a virtual character for demonstration. Simultaneously, the rendering engine generates an immersive interactive scene containing a copper base, tools, and pigments.

[0129] In step S404, the user mimics the virtual character's movements, picking up the virtual dropper to perform a blue dotting operation. At this moment, the user's action becomes part of the second multimodal instruction. The device captures the user's hand movements in real time and compares them with predefined standard blue dotting action adjustment information in the interactive feedback space, such as changes in force and joint angle range. If wrist tremors or excessive force are detected, the system determines a movement deviation and triggers a correction feedback script, retrieving corresponding prompts from the interactive feedback space; for example, highlighting the wrist joint and playing a voice prompt that the force should be gentler and the wrist kept stable, and superimposing the prompt information onto the immersive performance space in real time to guide the user to improve their movements.

[0130] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that, in order to achieve the above functions, the device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and method steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] When dividing each function into modules according to its corresponding function. Figure 2 A schematic diagram of a possible composition of a virtual performance generation system for intangible cultural heritage based on multimodal interaction, as described in the above embodiments, is shown. The system 400 includes:

[0132] The intent recognition module 161 is used to respond to the user's first multimodal command captured by the perception component of the wearable virtual device and determine the real-time user intent; wherein the first multimodal command includes text command, voice command, gesture command and facial expression feedback command;

[0133] The interactive feedback module 162 is used to call target performance data from the structured intangible cultural heritage knowledge graph according to the real-time user intent, and to construct an interactive feedback space with the target performance data as the core.

[0134] The immersive performance module 163 is used to input the target performance data into the multimodal generation model to generate and render an immersive performance interface containing virtual character actions, and build an immersive performance space accordingly.

[0135] The interaction adjustment module 164 is used to push the data stream of the immersive performance space to the wearable virtual device for output, and to receive the second multimodal command made by the user in the immersive performance space, which is returned by the wearable virtual device, so as to retrieve the corresponding interactive content from the interaction feedback space and load it into the immersive performance space.

[0136] In one possible implementation, the intent recognition module 161 is specifically used to execute step S101 and all its optional solutions in the above method embodiments. The interactive feedback module 162 is specifically used to execute step S102 and all its optional solutions in the above method embodiments. The immersive performance module 163 is specifically used to execute step S103 and all its optional solutions in the above method embodiments. The interactive adjustment module 164 is specifically used to execute step S104 and all its optional solutions in the above method embodiments. For the specific implementation methods, terminology explanations, and technical effects of each of the above functional modules, please refer to the detailed descriptions of the corresponding steps in the foregoing method embodiments; they will not be repeated here.

[0137] In this application, the memory 120 stores executable program code. The central processing unit 111 calls this executable program code to cause the computing device to execute all or part of the steps in the above method embodiments. Specifically, when the central processing unit 111 executes the program code, it can respectively implement the functions of the aforementioned intent recognition module 161, interactive feedback module 162, immersive performance module 163, and interactive adjustment module 164. The communication bus 150 is used for data interaction with the sensing component array 130 and the display component 140.

[0138] It should be understood that the above-mentioned modules or units, as examples of software functional units, such as an intent recognition module, may include program code running on a processor. As an example of a hardware functional unit, an intent recognition module may be implemented by one or more circuits performing specific logical functions, such as a neural network processing unit, an application-specific integrated circuit (ASIC), or a programmable logic device (PLC). Those skilled in the art can choose different implementation methods according to specific application scenarios, and these choices do not exceed the scope of protection of this application.

[0139] This application also provides a computer-readable storage medium storing computer program instructions that, when executed on a computer, cause the computer to perform the method as described in any of the foregoing method embodiments.

[0140] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method as described in any of the foregoing method embodiments.

[0141] It is understood that, in order to demonstrate the superiority of the technical solution of this application over the prior art, a quantitative and verifiable analysis of the technical effect is conducted in conjunction with the aforementioned embodiments of user imitation of intangible cultural heritage skills. The prior art uses pre-recorded videos for skill instruction, and user interaction is limited to video playback and pausing. The process from user intent to feedback is: user intent generation – recognition (voice / handheld button only) – retrieval in local or cloud video library – loading and playing pre-recorded video clips. The response latency T1 of this process mainly consists of retrieval latency and video loading latency, and the feedback content is fixed. However, the method of this application involves: multimodal intent perception and fusion – real-time retrieval from the knowledge graph – real-time rendering and synthesis based on a generative model – real-time adjustment according to the user's second instruction. Among these, due to the three-layer architecture and in-memory retrieval optimization of the knowledge graph, the retrieval latency is extremely low; the content is generated in real-time by the model rather than loading large video files, eliminating loading latency. Under the same network and device conditions, the method of this application can shorten the end-to-end latency of the core interaction indicator—from the user issuing an imitation action to seeing feedback—on average. More importantly, the second multimodal instruction feedback provided in this application is based on real-time comparison of user actions and standard data. Its feedback content is dynamically generated correction suggestions, and its information content and skill improvement are greater than that of fixed video content, effectively improving the efficiency of intangible cultural heritage skills transmission and learning experience in a virtual environment.

[0142] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0143] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating virtual performances of intangible cultural heritage based on multimodal interaction, applied to wearable virtual devices, characterized in that: The method includes: In response to a first multimodal instruction from the user captured by the sensing components of a wearable virtual device, a real-time user intent is determined; wherein the first multimodal instruction includes text instructions, voice instructions, gesture instructions, and facial expression feedback instructions; Based on the real-time user intent, target performance data is retrieved from the structured intangible cultural heritage knowledge graph, and an interactive feedback space is constructed with the target performance data as the core. The target performance data is input into a multimodal generation model to generate and render an immersive performance interface containing virtual character actions, and an immersive performance space is built accordingly. The data stream of the immersive performance space is pushed to the wearable virtual device for output, and the second multimodal command made by the user in the immersive performance space is received back from the wearable virtual device, so as to retrieve the corresponding interactive content from the interactive feedback space and load it into the immersive performance space.

2. The method for generating virtual performances of intangible cultural heritage based on multimodal interaction as described in claim 1, characterized in that, The determination of real-time user intent includes: In response to the launch command of the wearable virtual device, data acquisition is performed through the sensing components, which include an electromyography sensor, an eye tracker, a voice noise-canceling microphone, and a facial motion capture camera; wherein... The electromyography sensor is used to collect the changes in muscle electrical signals corresponding to the gesture command; The eye tracker is used to capture the coordinates of the user's gaze focus; The voice noise-canceling microphone is used to eliminate ambient noise. The facial motion capture camera collects facial motion vectors corresponding to the expression feedback commands by using multiple preset feature point markers.

3. The method for generating virtual performances of intangible cultural heritage based on multimodal interaction as described in claim 2, characterized in that, The determination of real-time user intent also includes: The text instructions are encoded into semantic vectors using a preloaded BERT model; The voice command is converted into an acoustic feature vector using Mel spectrum analysis. The gesture command is used to generate a spatial motion vector by fusing the changes in the electromyographic signal with the coordinates of the gaze focus. The facial expression feedback command is mapped to an emotional tendency value through the facial motion vector; The semantic vector, acoustic feature vector, spatial motion vector, and sentiment tendency value are input into a multimodal fusion model based on an attention mechanism, and the user intent probability distribution is calculated and output.

4. The method for generating virtual performances of intangible cultural heritage based on multimodal interaction according to claim 1, characterized in that, The structured intangible cultural heritage knowledge graph includes a core layer, an extension layer, and an interaction layer. The core layer is used to determine instruction keywords based on the real-time user intent, load them into a graph neural network, and determine the associated target intangible cultural heritage core entity; wherein, the core layer adopts a resource description framework triplet, which is used to store the intangible cultural heritage core entity and the attribute information of the intangible cultural heritage core entity; The extension layer matches target audio and video materials based on the target intangible cultural heritage core entity; wherein, the extension layer binds and associates intangible cultural heritage derivative content with the intangible cultural heritage core entity through knowledge hyperlinks; The interaction layer generates a personalized performance data sequence as the target performance data based on the target audio and video materials and user preference data; wherein, the interaction layer is used to record the user's historical interaction data and dynamically adjust the weight of the intangible cultural heritage core entity through reinforcement learning algorithm.

5. The method for generating virtual performances of intangible cultural heritage based on multimodal interaction according to claim 4, characterized in that, The structured intangible cultural heritage knowledge graph is also used for: Execute periodic data scraping tasks to obtain new data; The newly added data is associated and matched with the core entities of intangible cultural heritage through entity links; If the matching result is a suspected new intangible cultural heritage core entity or attribute information, and the intangible cultural heritage core entity or attribute information belongs to a specific intangible cultural heritage project field, then the new data will be sent to the terminal device of the intangible cultural heritage inheritor corresponding to that field. Receive the verification results fed back by the intangible cultural heritage inheritor through their terminal device; wherein, If it is a confirmation instruction, the newly added data is directly written to the core layer and the extension layer; if it is a correction instruction, the data is modified according to the received adjustment suggestions and then written. If no feedback is received within the preset time period, the newly added data will be temporarily stored in the pending queue, and a manual review reminder message will be generated for the administrator to process.

6. The method for generating virtual performances of intangible cultural heritage based on multimodal interaction according to claim 1, characterized in that, The multimodal generation model includes a 3D scene generation sub-model, an action-driven sub-model, and an emotion rendering sub-model; among which... The 3D scene generation sub-model is based on the neural radiation field, generates a three-dimensional point cloud model according to the description of the intangible cultural heritage scene in the target performance data, and dynamically adjusts the level of detail of the model according to the user's current viewing distance. The motion-driven sub-model is based on a generative adversarial network and uses motion capture data of non-heritage inheritors as training samples to generate the motion sequence of the virtual character. The emotion rendering sub-model adjusts at least one of the scene lighting parameters, background music rhythm value, and particle effect density in real time according to the emotion tendency value corresponding to the expression feedback command. The 3D point cloud model, the virtual character action sequence, and the adjusted scene parameters are synthesized by the rendering engine and encapsulated into a real-time data stream of the wearable virtual device based on a real-time communication protocol.

7. The method for generating virtual performances of intangible cultural heritage based on multimodal interaction according to claim 6, characterized in that, The training process of the action-driven sub-model includes: Acquire motion capture data of the intangible cultural heritage inheritor; wherein, the motion capture data includes annotation text describing the key points of the movements; The annotation text is converted into structured motion adjustment information, which includes changes in the force of the motion, the speed of the rhythm, and the range of joint angles. Based on the action adjustment information, a constraint term is introduced into the loss function of the generative adversarial network. This constraint term penalizes the deviation between the generated action and the structured action adjustment information, so that the trained target action-driven sub-model fits the motion capture data while also conforming to the annotated text. When the action-driven sub-model generates a new action sequence through inference, it fine-tunes the generated virtual character action sequence based on the structured action adjustment information.

8. The method for generating virtual performances of intangible cultural heritage based on multimodal interaction according to claim 1, characterized in that, The receiving of the second multimodal command, transmitted back by the wearable virtual device and made by the user in the immersive performance space, includes: Receive the first back-transmission data and determine the target wearable virtual device corresponding to the first back-transmission data; wherein, the first back-transmission data includes user behavior data and user physiological indicator data; Based on the target wearable virtual device, the data stream of the target wearable virtual device in the immersive performance space is adapted and encapsulated; Based on user behavior data, determine the user's interactive operations; The user's real-time focus level is determined based on the user's physiological indicators. Based on the interactive operation, the performance interface to be adjusted in the immersive performance space is determined, and based on the real-time focus, the performance interface to be adjusted is adjusted to the best performance perspective.

9. The method for generating virtual performances of intangible cultural heritage based on multimodal interaction according to claim 1, characterized in that, The step of receiving the second multimodal command transmitted back by the wearable virtual device and made by the user in the immersive performance space also includes: Construct a user profile database, which includes: user historical interaction records, preference tags, and behavioral feature vectors; After determining the real-time user intent, the first multimodal instruction is matched with historical data in the user profile database for similarity. If the match is successful, historical optimization parameters are retrieved to correct the real-time user intent. If the matching fails, a cold start learning mechanism is triggered. The user profile database is updated based on the feedback analysis of the second multimodal instruction through a reinforcement learning algorithm, so that the intent recognition accuracy converges to the target threshold after the number of times the user uses the device reaches a preset condition. The user profile database provides a manual editing interface, and the user's preference tags are generated based on the priority of the editing instructions received through the manual editing interface.

10. A virtual performance generation system for intangible cultural heritage based on multimodal interaction, characterized in that, include: Intent recognition module: used to respond to the user's first multimodal command captured by the perception component of the wearable virtual device and determine the real-time user intent; wherein, the first multimodal command includes text command, voice command, gesture command and facial expression feedback command; Interactive feedback module: used to call target performance data from the structured intangible cultural heritage knowledge graph according to the real-time user intent, and to construct an interactive feedback space with the target performance data as the core; Immersive performance module: used to input the target performance data into the multimodal generation model to generate and render an immersive performance interface containing virtual character actions, and build an immersive performance space accordingly; Interaction adjustment module: used to push the data stream of the immersive performance space to the wearable virtual device for output, and receive the second multimodal command made by the user in the immersive performance space from the wearable virtual device, so as to retrieve the corresponding interactive content from the interaction feedback space and load it into the immersive performance space.

Images