AI large model and mixed reality-based explanation method, device, medium and product

Abstract

An explanation method and device based on an AI large model and mixed reality. In the method, external event data is obtained, and an event type and event content are extracted from the external event data; based on the event type, a target virtual character is matched from a preset virtual character setting library; the external event data is fused with data of a preset domain knowledge base to obtain a training data set; a preset knowledge reasoning model is trained according to the training data set to obtain a target explanation model; a query request of a user is received, and the query request is parsed to determine a query intention; the query intention and the event content are associated and input into the target explanation model to obtain initial explanation content; the initial explanation content and the character setting are input into a preset stylized generation model to generate target explanation content; a three-dimensional virtual image is loaded and associated and bound with the target explanation content. The technical solution provided in the application improves the visual expressiveness and interactivity of the explanation.

Description

Technical Field

[0001] This application relates to the technical field of artificial intelligence, specifically to a teaching method, device, medium, and product based on AI large-scale models and mixed reality. Background Technology

[0002] In the current field of digital display and information services, using AI (Artificial Intelligence) to generate explanatory content has become a common technological approach. Specifically, existing technologies typically employ the following methods: First, the system retrieves basic information from a structured database or existing knowledge base; then, upon receiving a user query, it uses natural language processing technology to understand the user's intent and retrieves relevant knowledge points from the knowledge base; finally, it uses a preset text template or a simple language model to combine the retrieved knowledge points into an explanatory text, which is then converted into speech and broadcast using text-to-speech technology. This approach achieves a certain degree of automation in information retrieval and answering.

[0003] However, the aforementioned existing technologies suffer from a significant technical problem in application: the generated content and format are separate and fixed, lacking contextual adaptability and expressiveness. Specifically, the text generated by existing technologies has a uniform style and cannot be dynamically adjusted according to different event backgrounds or thematic scenarios, resulting in stiff and mechanical content that fails to resonate with users emotionally or provide an immersive experience. For example, whether explaining a serious historical event or a lighthearted cultural activity, the language style and presentation are almost identical, greatly reducing the effectiveness of information delivery and user acceptance. Therefore, how to enable the content to adaptively match specific contexts and be presented in a more expressive manner is a pressing technical challenge that needs to be addressed in this field. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a teaching method, device, medium, and product based on AI large-scale models and mixed reality.

[0005] The first aspect of this application provides a teaching method based on AI large models and mixed reality, employing the following technical solution: Acquire external event data, and extract event types and event content from the external event data; Based on the event type, a target virtual character is matched from a preset virtual character setting library, and the target virtual character has character settings associated with the event type; The external event data is fused with data from a pre-defined domain knowledge base to obtain a training dataset; The target explanation model is obtained by training a preset knowledge reasoning model based on the training dataset. The system receives user query requests through user terminal devices and parses the query requests to determine the query intent. The query intent and the event content are associated and input into the target explanation model to obtain the initial explanation content; The initial explanation content is input into a preset stylized generation model along with the character settings to generate target explanation content that conforms to the language style of the target virtual character; Load the 3D virtual image of the target virtual character and associate and bind the 3D virtual image with the target explanatory content.

[0006] By adopting the above technical solutions, combining AI large-scale models with mixed reality technology, the technical problems of separation and rigidity between content and form, as well as lack of contextual adaptability and expressiveness in explanations are effectively solved. Specifically, by matching target virtual characters based on event types and using a stylized generation model to adapt the initial explanation content to a language style consistent with the character's setting, the explanation content can dynamically adapt to different event backgrounds (such as serious history or light cultural activities), thereby breaking the limitations of a single text mode and enhancing the emotional resonance and immersion of the content. At the same time, by loading 3D virtual images and binding them to the explanation content, the visual expressiveness and interactivity of the explanation are further enhanced, providing users with a more vivid and personalized information experience, significantly improving the effectiveness of information delivery and user acceptance.

[0007] Optionally, matching a target virtual character from a preset virtual character setting library based on the event type includes: Based on the event type, construct an event feature vector; For each virtual character in the virtual character setting library, construct multiple character feature vectors; The similarity between the event feature vector and all the character feature vectors is calculated to obtain the matching score between each virtual character and the event type. The virtual character with the highest matching score is selected as the target virtual character.

[0008] By adopting the above technical solution, intelligent and precise matching of virtual characters with event types is achieved. This solution quantifies event types and character settings into feature vectors and calculates similarity, transforming the originally subjective and vague character selection process into an objective and quantifiable data matching process. This not only avoids the subjective bias of human selection but also efficiently and automatically filters out the target characters most suited to the current event's atmosphere from a massive virtual character database. By calculating matching scores and selecting the best, it ensures that the selected virtual characters are highly compatible with the event theme in terms of knowledge background, personality traits, and language style. This lays a solid foundation for generating personalized explanations that are context-integrated and stylistically consistent, significantly enhancing the professionalism and immersive experience of the explanation service.

[0009] Optionally, the step of fusing the external event data with data from a preset domain knowledge base to obtain a training dataset includes: Extract event entities and event relationships from the external event data to construct an event knowledge subgraph; Based on the event entity, retrieve the associated knowledge entity and associated knowledge path from the domain knowledge base; The event knowledge subgraph, the associated knowledge entities, and the associated knowledge paths are merged to generate an enhanced knowledge graph; The event content is converted into natural language text and fused with the enhanced knowledge graph to obtain the training dataset.

[0010] By adopting the above technical solution, the problems of limited information and weak knowledge correlation in a single data source are effectively solved, laying a solid data foundation for generating high-quality explanatory content. This method constructs an event knowledge subgraph and associates it with a domain knowledge base, placing isolated external event data into a structurally complete and semantically rich augmented knowledge graph. This deep integration not only greatly expands the breadth and semantic depth of the training data, but more importantly, it establishes logical connections between events and background knowledge. This enables the subsequent training of the target explanatory model to perform deep knowledge reasoning and association, thereby generating more comprehensive, logically coherent, and background-rich explanatory content, significantly improving the knowledge content and accuracy of the explanations.

[0011] Optionally, training a preset knowledge reasoning model based on the training dataset to obtain the target explanation model includes: Based on the content differences between the event content and the enhanced knowledge graph, the knowledge missing parts and knowledge conflict parts in the event content are determined. Based on the missing knowledge, supplementary knowledge paths are generated by reasoning in the enhanced knowledge graph, and the supplementary knowledge paths are integrated into the training dataset. Based on the knowledge conflict, the connection weights of the corresponding nodes in the enhanced knowledge graph are adjusted to generate a revised knowledge structure. During the training of the knowledge reasoning model, the training dataset and the modified knowledge structure are used alternately, and iterative optimization is performed by setting a preset hybrid training strategy until the preset convergence condition is met, thereby obtaining the target explanation model.

[0012] By adopting the above technical solution, the problems of low training data quality and limited model reasoning ability are effectively solved, significantly improving the cognitive depth and content reliability of the final explanation model. This method actively improves the knowledge structure by dynamically identifying and correcting knowledge gaps and conflicts in event content, ensuring that model training is not simply data fitting, but rather built upon a logically consistent and complete knowledge system. Furthermore, by using a hybrid training strategy that alternates between original data and the corrected knowledge structure, the model is forced to simultaneously learn factual knowledge and master logical reasoning abilities during training, thereby cultivating its internal mechanism for identifying contradictions and supplementing information. The resulting target explanation model not only has a more solid knowledge base but also generates high-quality explanation content that is logically rigorous, factually accurate, and possesses strong causal relationships, fundamentally enhancing the authority and credibility of the explanation.

[0013] Optionally, the method further includes: Receive the interactive data stream uploaded by the user terminal device, the interactive data stream including user question text and user heart rate data; The user's question text is compared with the enhanced knowledge graph to identify knowledge points outside the graph, and a graph update candidate set is generated based on the knowledge points outside the graph. Based on the user's heart rate data at the time corresponding to the user's question text, a cognitive load score is calculated for each candidate knowledge point in the map update candidate set; The candidate set for knowledge graph updates is sorted based on the cognitive load score. Candidate knowledge points with cognitive load scores below a preset update threshold are added to the domain knowledge base, and an update operation for the enhanced knowledge graph is triggered.

[0014] By adopting the above technical solution, intelligent and adaptive dynamic updates of the domain knowledge base are achieved, effectively solving the problems of lagging knowledge updates and detachment from actual user needs in traditional systems. This method identifies knowledge gaps in real time by analyzing user questions and accurately quantifies the cognitive load users experience when understanding new knowledge by combining objective physiological indicators such as user heart rate data. Based on this, candidate knowledge points are screened, prioritizing the inclusion of new knowledge that is easily understood and accepted by users, thus improving the relevance and effectiveness of knowledge base updates.

[0015] Optionally, the step of generating target explanation content that conforms to the language style of the target virtual character by combining the initial explanation content with the character setting input preset stylized generation model includes: Style instructions are extracted from the character settings, including a set of tone style parameters, terminology usage preferences, and knowledge boundary definitions. The initial explanation is broken down into multiple semantic fragments; For each semantic segment, multiple sets of candidate style prompt templates are generated using the tone style parameter set and the term usage preference; The multiple sets of candidate style suggestion templates are filtered to determine the optimal style suggestion template for each semantic segment; The final generation instruction is constructed based on the optimal style prompt template and the corresponding semantic fragment, and the final generation instruction is input into the stylized generation model to generate preliminary target explanation content; Based on the defined knowledge boundaries, the content compliance of the initial target explanation is verified to obtain the target explanation content.

[0016] By adopting the above technical solution, a deep integration of the narration content and the virtual character's style is achieved, effectively solving the core problems of a monotonous narration style and a lack of expressiveness. This solution refines the character setting and decomposes the narration content, extending the granularity of style adaptation from the chapter level to the semantic fragment level, ensuring a thorough and natural transition in language style. By generating and filtering multiple sets of candidate templates, the most appropriate expression can be precisely matched to different content fragments while preserving the original meaning, ensuring that the final generated content is highly consistent with the virtual character's professional background and personality traits in terms of tone, terminology, and knowledge.

[0017] Optionally, loading the 3D virtual image of the target virtual character and associating the 3D virtual image with the target explanatory content includes: The target explanation content is converted into a speech audio stream, and prosodic features and phoneme sequences are extracted from the speech audio stream; Based on the phoneme sequence, lip-sync animation data corresponding to each phoneme in the phoneme sequence is retrieved from a preset lip-sync primitive library, and the lip-sync animation data is combined to generate lip-sync animation keyframes. Based on the rhythmic features and the posture habit data in the character settings, a body micro-movement sequence and a head posture sequence are calculated through a preset posture generation network. Syntactic analysis and semantic entity recognition are performed on the target explanation content to obtain the timestamp and semantic category of the semantic unit. Based on the timestamp and semantic category of the semantic unit, the corresponding gesture animation clips are retrieved from the preset gesture library and then combined to generate a gesture animation sequence. The speech audio stream, the lip-sync animation keyframes, the head pose sequence, the body micro-motion sequence, and the gesture animation sequence are time-aligned and merged into a multimodal driven data stream; The multimodal driving data stream is associated and bound with the three-dimensional virtual image.

[0018] By adopting the above technical solution, a high degree of synergy and deep integration between the virtual avatar and the narration content is achieved, effectively solving the technical problems of the traditional virtual narrator's stiffness and disconnect between actions and content. This method, through multi-dimensional analysis of the narration content, from phonetic rhythm to semantic structure, drives the generation of precisely matched lip movements, postures, and gesture animations, ensuring that the virtual avatar's expressiveness covers the complete spectrum from subtle facial expressions to overall body language. In particular, associating semantic units with specific gestures allows the virtual avatar's body language to intuitively respond to and emphasize key information in the narration. Finally, through precise time alignment, all modalities are merged into a unified driving data stream, ensuring that the presentation of the 3D virtual avatar not only features precise lip-sync, but also that its expression, posture, and gestures are highly consistent with the rhythm and meaning of the narration content, thus creating a natural, engaging, and highly immersive mixed reality narration experience.

[0019] A second aspect of this application provides an electronic device including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the foregoing.

[0020] A third aspect of this application provides a computer-readable storage medium storing instructions that, when executed, perform the method described in any of the preceding descriptions.

[0021] A fourth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method as described in any of the preceding claims.

[0022] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: By constructing an event-driven, context-adaptive mechanism, a high degree of unity between the content and presentation format was achieved. Enhanced knowledge graphs and hybrid training strategies significantly improved the accuracy, logic, and depth of reasoning in the content. Fine-grained stylized generation and multimodal driving technology ensured that the virtual avatar's presentation maintained both consistent language style and natural body language. Furthermore, cognitive load analysis based on user interaction enabled the intelligent evolution of the knowledge base. Ultimately, this effectively solved the core problems of traditional explanation systems—the separation of content and form, rigid style, and insufficient expressiveness—providing an intelligent explanation experience that combines knowledge, expressiveness, and immersion. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the system architecture of an embodiment of the explanation method based on AI large model and mixed reality applied in this application; Figure 2 This is a flowchart illustrating a teaching method based on a large AI model and mixed reality disclosed in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.

[0024] Explanation of reference numerals in the attached figures: 100, System architecture; 101, First terminal device; 102, Second terminal device; 103, Third terminal device; 104, Network; 105, Server; 301, Processor; 302, Communication bus; 303, User interface; 304, Network interface; 305, Memory. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0026] Figure 1 This is a schematic diagram of the system architecture of an embodiment of the explanation method based on AI large model and mixed reality applied in this application. like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0027] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as model training applications, video recognition applications, web browser applications, social platform software, etc.

[0028] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP3 (Moving Picture Experts Group Audio Layer IV) players, laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0029] This embodiment discloses a teaching method based on AI large models and mixed reality. Figure 2 This is a flowchart illustrating a teaching method based on AI large models and mixed reality disclosed in an embodiment of this application, as shown below. Figure 2 As shown, the method includes the following steps: S201. Obtain external event data, and extract the event type and event content from the external event data; Specifically, this can be executed by a data acquisition and processing module built into the server. This module periodically or event-triggeredly acquires data in real time from multiple preset third-party data sources via network interfaces. A preferred implementation is that this module is pre-configured with multiple API (Application Programming Interface) call tasks. For example, it can obtain real-time weather data (such as temperature and weather conditions) for a specified location by calling public weather service APIs (such as Xinzhi Weather and Hefeng Weather), obtain current holiday or anniversary information by accessing public holiday calendar APIs, and crawl the latest news related to preset key figures or fields by integrating news and information APIs (such as Juhe Data and Baidu News). Another optional implementation is to deploy a web crawler program to periodically crawl pages from specific official websites (such as the National Meteorological Administration and mainstream news portals) and extract the required information according to preset parsing rules. Upon receiving raw external event data in formats such as JSON (JavaScript Object Notation) or HTML (Hypertext Markup Language), this module immediately parses and processes it. First, it categorizes the data into specific "event types" based on the data source or content keywords (e.g., weather, holiday), such as "real-time weather," "festival events," or "personal activities." Second, it extracts key fields (e.g., temperature, text, title, summary) from the data structure as structured event content, such as "Temperature: 25℃, Weather: Sunny" or "Title: A scientist received a new award." Thus, unstructured real-time external information is efficiently transformed into standardized contextual data pairs (event type, event content), providing essential and timely input for subsequent AI models to perform contextual awareness, content generation, and personalized explanations.

[0030] S202. Based on the event type, match a target virtual character from a preset virtual character setting library, wherein the target virtual character has character settings associated with the event type; Specifically, the system accesses a pre-defined virtual character profile library, which stores profiles of multiple virtual characters with different backgrounds, personalities, and areas of knowledge. Each virtual character's profile is a structured data record, crucially containing one or more event type tags associated with their identity traits. For example, the profile of "Professor Li, the historian" would be associated with tags such as "historical events" and "cultural relic exhibitions," while "Xiao Zhi, the technology pioneer," would be associated with tags such as "technology releases" and "future exploration." The matching process is as follows: the system uses the event type (e.g., historical events) obtained in the previous step as a search keyword, queries the virtual character profile library, and filters out all candidate virtual characters whose associated event type tags contain that keyword. If there is only one match, the virtual character is directly selected as the target virtual character; if multiple matches exist, the system can determine the final target virtual character based on pre-defined priority rules (e.g., based on the character's professional level rating) or by using a random selection strategy, thereby ensuring that the subsequently generated narration content is highly consistent with the event theme in style and tone.

[0031] Optionally, matching a target virtual character from a preset virtual character setting library based on the event type includes: constructing an event feature vector based on the event type; constructing multiple character feature vectors for each virtual character setting in the virtual character setting library; calculating the similarity between the event feature vector and all the character feature vectors to obtain a matching score for each virtual character and the event type; and selecting the virtual character with the highest matching score as the target virtual character.

[0032] Specifically, to perform mathematical operations, event types in text form need to be converted into numerical event feature vectors. In this embodiment, the specific technical solutions for constructing the event feature vectors may include, but are not limited to, the following two: The first solution is one-hot encoding. The system pre-maintains a global dictionary containing all possible event types, such as [historical events, art exhibitions, technology releases, cultural festivals, ...]. When a specific event type, such as a historical event, is obtained, the system generates a vector with a dimension equal to the size of the dictionary, where the element corresponding to the historical event position has a value of 1, and the element values ​​at all other positions have a value of 0. This method is simple to implement and can clearly represent the category. The second solution is word embedding. To better capture the semantic relationships between event types (for example, historical events and archaeological discoveries are semantically similar), the system can use a pre-trained language model (such as Word2Vec, GloVe, or BERT) to map the text description of the event type (such as historical events) to a high-dimensional (e.g., 300-dimensional) dense vector. This vector not only uniquely identifies the event type, but its position in the vector space also contains rich semantic information, making semantically similar event types close to each other in space.

[0033] Furthermore, the system needs to construct multiple character feature vectors for each virtual character in the virtual character setting library to comprehensively reflect their characteristics. A virtual character's setting is typically multi-dimensional, encompassing aspects such as their knowledge domain, personality traits, language style, and professional background. Therefore, the system will vectorize these different dimensions separately. For example, for the virtual character Professor Li, the system will: 1) convert his knowledge domain (e.g., ancient Chinese history, world war history) into one or more knowledge domain feature vectors using a word embedding model with the same event type; 2) vectorize his personality traits (e.g., meticulousness, erudition, composure) by consulting a pre-set personality trait vector library or using a sentiment analysis model; 3) vectorize his professional background (e.g., university professor, archaeologist). In this way, each virtual character is no longer represented by a single vector, but is defined by a set (or multiple) of feature vectors that can characterize their image from different perspectives.

[0034] Furthermore, after the event feature vector and the multi-dimensional feature vectors of all characters are constructed, the system will perform similarity calculations to generate a comprehensive matching score for each virtual character. This calculation process typically consists of two steps: First, the similarity between the event feature vector and each character feature vector of a given virtual character is calculated. Common similarity calculation methods include cosine similarity, which measures the consistency in direction by calculating the cosine of the angle between two vectors; the closer the value is to 1, the more similar they are. Alternatively, the reciprocal of Euclidean distance can be used; the closer the distance, the more similar they are. Then, the system will sum the multiple similarity scores obtained for the virtual character (e.g., similarity with the knowledge domain vector, similarity with the personality trait vector, etc.) according to preset weights to obtain a final comprehensive matching score. For example, the weight for the knowledge domain can be set to 0.6, and the weight for personality traits can be set to 0.4; the specific weight values ​​can be flexibly configured according to the emphasis of the application scenario.

[0035] Furthermore, the system selects the virtual character with the highest matching score as the target virtual character. Specifically, the system iterates through all virtual characters in the virtual character setting library and performs the matching score calculation for each one. After all calculations are completed, the system sorts all virtual characters in descending order of their matching scores. The virtual character at the top of the list, i.e., the one with the highest score, will be ultimately determined as the target virtual character for this explanation task. In an optional implementation, the system can also set a minimum matching score threshold. If the scores of all virtual characters are below this threshold, the system can select a preset general or default virtual character to avoid inappropriate explanations when there is no highly matching candidate. Through this series of quantitative steps, this application ensures that the selected virtual character achieves optimal matching with the current event context in multiple dimensions such as knowledge background and personality.

[0036] S203. The external event data is fused with the data from a preset domain knowledge base to obtain a training dataset; One specific implementation method is to use template-based data augmentation technology: the system pre-designs a large number of dialogue or explanation templates containing placeholders, for example, "[Virtual character], considering that today's [Event Type] is [Event Content], please introduce [Knowledge Base Entry] in your style," or "[Virtual character], a visitor asked what different feelings would they have appreciating [Knowledge Base Entry] in weather like [Event Content]?" The data fusion engine automatically fills the placeholders in these templates with external event data (such as event type = real-time weather, event content = light rain) and knowledge entries randomly or rule-basedly extracted from domain knowledge bases (such as museum collection databases) (such as background information on "Along the River During the Qingming Festival"), thereby programmatically generating thousands of raw data pairs in highly context-relevant instruction-input-output or question-answer formats. As an alternative advanced implementation, the system can directly utilize a powerful foundational model (such as GPT-4) as a data generator, inputting instructions such as: "You are a data generation assistant. Please combine the following real-time events: {external event data} and knowledge points: {domain knowledge base data} to generate 10 sets of consistent question-and-answer data for training the virtual character {target virtual character}." Regardless of the method used, the final output is a structured training dataset in a format such as JSONL. Each data point cleverly integrates the instantaneous changes of the real world with deep domain knowledge, providing high-quality and unique training corpus for subsequent model fine-tuning, thereby enabling the model to possess context awareness and personalized explanation capabilities.

[0037] Optionally, the step of fusing the external event data with data from a preset domain knowledge base to obtain a training dataset includes: extracting event entities and event relationships from the external event data to construct an event knowledge subgraph; retrieving related knowledge entities and related knowledge paths from the domain knowledge base based on the event entities; merging the event knowledge subgraph, the related knowledge entities, and the related knowledge paths to generate an enhanced knowledge graph; and converting the event content into natural language text and fusing it with the enhanced knowledge graph to obtain the training dataset.

[0038] Specifically, the system extracts event entities and event relationships from the external event data to construct a temporary event knowledge subgraph representing the current context. This process is typically accomplished using Natural Language Processing (NLP) techniques. Specifically, the system employs a Named Entity Recognition (NER) model to identify key entities in the external event data. For example, for the external event data "The Palace Museum is experiencing light rain today," the NER model would identify "Palace Museum" (location entity) and "light rain" (weather entity). Subsequently, the system uses Relation Extraction (RE) techniques to identify the relationships between these entities, such as (Palace Museum, current weather, light rain). These extracted entity-relationship-entity triples together constitute a small, dynamic event knowledge subgraph. To improve extraction accuracy, the NER and RE models used can be deep learning-based models (such as BERT-CRF) and pre-trained using domain-specific corpora to accurately identify entities and relationships relevant to the application scenario.

[0039] Furthermore, based on the event entities extracted in the previous step, the system retrieves related knowledge entities and knowledge paths from a pre-defined domain knowledge base. This domain knowledge base is a large-scale, pre-built, structured knowledge graph that stores static facts about a specific domain (such as cultural relics and history). For example, this knowledge base might store a massive number of triples such as (Palace Museum, alternative name, Forbidden City), (Palace Museum, includes, Hall of Supreme Harmony), and (Hall of Supreme Harmony, function, imperial ceremonies). The retrieval process starts with the event entity (e.g., "Palace Museum") as the starting node and performs a graph traversal query within the domain knowledge base. A preferred implementation is to use a depth-limited breadth-first search (BFS) or depth-first search (DFS), for example, retrieving all entities within 2-3 hops of the Palace Museum node and the paths connecting them. In this way, the system can efficiently find the relevant knowledge entities most directly related to the current event (such as the Forbidden City and the Hall of Supreme Harmony) and the relevant knowledge paths connecting them (such as the Palace Museum -> containing -> Hall of Supreme Harmony).

[0040] Furthermore, the system merges the event knowledge subgraph, related knowledge entities, and related knowledge paths to generate a richer augmented knowledge graph. This merging operation is essentially a graph fusion process. Specifically, the system uses a shared entity in the event knowledge subgraph (such as the Palace Museum) as an anchor point and attaches all related knowledge entities and paths retrieved from the domain knowledge base to this anchor point. For example, the event subgraph (Palace Museum, current weather, light rain) is merged with knowledge paths such as (Palace Museum, alternative name, Forbidden City) and (Palace Museum, contains, Hall of Supreme Harmony). The resulting augmented knowledge graph simultaneously contains dynamic, real-time contextual information (it's raining lightly today) and static, in-depth background knowledge (the Palace Museum is also called the Forbidden City, and it contains the Hall of Supreme Harmony), forming a highly information-dense local knowledge network centered around the current event.

[0041] Furthermore, the system converts the event content into natural language text and fuses it with the augmented knowledge graph to obtain the final training dataset. The purpose of this step is to convert the structured graph data back into a natural language format that is easy for large models to understand and learn—a graph-to-text conversion. One feasible implementation is to use graph linearization techniques to convert the triples in the augmented knowledge graph into fluent sentences. For example, the augmented graph can be converted into descriptive text: "Today the weather at the Palace Museum (also known as the Forbidden City) is light rain. The Palace Museum contains the Hall of Supreme Harmony." Subsequently, the system combines this graph-generated background description text with the original event content (e.g., "Please explain the Palace Museum to me") to form a complete training sample. For example, it can be constructed in the format: {instruction: Please describe the Palace Museum in light of today's weather, input: Today the weather at the Palace Museum is light rain, it is also known as the Forbidden City, and one of its important buildings is the Hall of Supreme Harmony. Output: [A high-quality answer prepared beforehand or generated by another model]}. This step enables the system to generate a large amount of training data that includes both real-time context and deep knowledge, laying a solid foundation for subsequent fine-tuning of large AI models with excellent context awareness capabilities.

[0042] S204. Train the preset knowledge reasoning model based on the training dataset to obtain the target explanation model; The "pre-defined knowledge reasoning model" mentioned here typically refers to a pre-trained large language model with powerful general language understanding and generation capabilities. Examples include, but are not limited to, open-source or proprietary models such as the GPT series, LLaMA series, or ChatGLM based on the Transformer architecture. These models provide a solid foundation as the base model. The training process does not start from scratch but involves supervised fine-tuning on top of this base model. A preferred and efficient fine-tuning technique is to use parameter-efficient fine-tuning methods, such as low-rank adaptive techniques. Specifically, the system freezes most (e.g., over 99%) of the original weight parameters of the base model, injecting only small-scale, trainable low-rank decomposition matrices (i.e., adapter modules) into key layers of the model (such as attention layers). During training, the system uses the generated dataset as training samples and continuously adjusts the parameters of these low-rank matrices using an optimization algorithm (e.g., the AdamW optimizer). The optimization objective is to minimize the difference between the model-generated answers and the standard answers in the dataset (usually measured by the cross-entropy loss function). Because only a very small number of parameters need to be updated, this method significantly reduces the computational resources and storage costs required for training, making it possible to iterate and update the model quickly and at low cost for different external events and virtual characters. After the entire fine-tuning process is completed, these trained low-rank matrices are combined with the original base model to form the final target explanation model. This model not only inherits the general knowledge and language capabilities of the base model, but more importantly, it has deeply learned and internalized specific domain knowledge, the language style of the target virtual character, and, most importantly, how to use real-time changing external events as contextual clues to dynamically associate and reason with its own knowledge base, thereby generating highly contextualized, personalized, and professional explanation content.

[0043] Optionally, training the preset knowledge reasoning model based on the training dataset to obtain the target explanation model includes: determining the missing knowledge parts and conflicting knowledge parts in the event content based on the content differences between the event content and the enhanced knowledge graph; generating supplementary knowledge paths in the enhanced knowledge graph based on the missing knowledge parts, and integrating the supplementary knowledge paths into the training dataset; adjusting the connection weights of corresponding nodes in the enhanced knowledge graph based on the conflicting knowledge parts to generate a corrected knowledge structure; and during the training of the knowledge reasoning model, alternately using the training dataset and the corrected knowledge structure, iteratively optimizing through a preset hybrid training strategy until a preset convergence condition is met to obtain the target explanation model.

[0044] As a more refined and intelligent implementation of the above-mentioned model training steps, this application also provides an adaptive, knowledge graph-based hybrid training paradigm. This paradigm not only utilizes datasets for supervised fine-tuning but also dynamically identifies and corrects inconsistencies at the knowledge level during training, thereby training a target explanation model with stronger logical reasoning capabilities and higher factual accuracy. The specific steps of this paradigm will be described in detail below.

[0045] Specifically, the system intelligently identifies knowledge gaps and knowledge conflicts implicit in the event content (usually manifested as user questions or instructions) based on the content differences between the event content and the enhanced knowledge graph. This process can be considered a kind of knowledge alignment diagnosis. In specific implementation, the system first extracts entities and relations from the input event content (for example, a user question is: Please introduce the story of the pair of bronze kylins in the Hall of Supreme Harmony in the Forbidden City), forming a temporary question knowledge subgraph, which can be represented as (Hall of Supreme Harmony, contains, bronze kylins). Subsequently, the system performs graph matching or semantic comparison with the enhanced knowledge graph generated in the previous steps. Knowledge gaps are identified when an entity or relation in the question knowledge subgraph does not exist or has no direct association in the enhanced knowledge graph (for example, the graph only has a triple represented as (Hall of Supreme Harmony, function, imperial ceremony), but no direct information about the bronze kylins). Knowledge conflicts are identified when a triple in the question's knowledge subgraph contradicts a fact in the augmented knowledge graph (e.g., the graph explicitly records a triple as (Bronze Qilin, located outside the Hall of Supreme Harmony), which conflicts with the question's mention of its location inside the hall). Through this diagnostic step, the model can anticipate potential knowledge blind spots and factual errors before training.

[0046] Furthermore, for the identified knowledge gaps, the system proactively infers within the enhanced knowledge graph to generate supplementary knowledge paths and integrates these new paths into the training dataset. This step aims to fill knowledge gaps through the inherent logic of the knowledge graph. A preferred implementation is to employ knowledge graph completion techniques. For example, the system can utilize graph embedding-based models (such as TransE, RotatE) or inference engines based on graph neural networks (GNNs). When a knowledge gap is detected between the Hall of Supreme Harmony and the Bronze Qilin, the inference engine searches for the most likely potential relationship in the graph's vector space and may infer a supplementary knowledge path, such as (Hall of Supreme Harmony, adjacent, Gate of Supreme Harmony) and (Gate of Supreme Harmony, in front, Bronze Qilin). Although this new path does not have a direct connection, it establishes a reasonable indirect association. Subsequently, the system converts this path into natural language text, the content of which is: Hint: The Hall of Supreme Harmony and the Bronze Qilin are not directly related, but the Bronze Qilin is located outside the Gate of Supreme Harmony in front of the Hall of Supreme Harmony. The system then uses this text as a new, more informative input field or context field to update or expand the original training dataset, allowing the model to learn this deeper level of relational knowledge during training.

[0047] Furthermore, for identified knowledge conflicts, the system generates a dynamic, corrected knowledge structure by adjusting the connection weights of corresponding nodes in the enhanced knowledge graph. This step does not permanently modify the original knowledge base, but rather generates a temporary, more accurate graph view for this training task. Specifically, each edge (relationship) in the knowledge graph can be assigned a credibility weight, with an initial value of one. When a conflict is detected, such as a conflict between the implicit user question (Hall of Supreme Harmony, contains a bronze unicorn) and the graph's (Hall of Supreme Harmony, does not contain a bronze unicorn), the system reduces the weight of the relevant edges on the conflicting path (e.g., lowering the weight of relationships like (Hall of Supreme Harmony, contains a bronze unicorn)) while correspondingly increasing the weight of the factually correct path (Gate of Supreme Harmony, contains a bronze unicorn). This weight adjustment can be done directly through a small attention network or based on preset rules. The resulting graph, the corrected knowledge structure, may not change in topology, but its edge weights reflect a cognitive correction to the specific context of the current event, providing a more reliable structured knowledge source for subsequent hybrid training.

[0048] Furthermore, during the training of the knowledge reasoning model, the system alternately uses text-based training datasets and graph-based corrected knowledge structures, iteratively optimizing through a pre-defined hybrid training strategy until a pre-defined convergence condition is met, ultimately obtaining the target explanation model. This is a multimodal collaborative training method. The hybrid training strategy can be designed as follows: In a training batch, the model first processes text samples from the training dataset, updating gradients using a standard language model loss function (such as cross-entropy loss); then, in the next batch or at a certain frequency, the model receives the corrected knowledge structure as input, using a Graph Attention Network (GAT) or other graph learning algorithms to update the model's internal representations of entities and relations. The optimization goal is to make the entity representations generated by the model better match the corrected graph structure. These two processes alternate, enabling the model to learn both the fluency and contextual understanding of language (from text data) and to learn and follow structured, fact-corrected knowledge logic (from the corrected knowledge graph). Training stops when the model's overall loss function no longer decreases significantly on the validation set for several consecutive cycles, or when a pre-defined convergence condition such as the number of training rounds is met. The target explanation model obtained at this time, having undergone this deep integration and correction training, will exhibit higher factual accuracy and stronger logical reasoning ability when dealing with complex or even misleading questions.

[0049] Optionally, the method further includes: receiving an interactive data stream uploaded by the user terminal device, the interactive data stream containing user question text and user heart rate data; comparing the user question text with the enhanced knowledge graph to identify knowledge points outside the graph, and generating a graph update candidate set based on the knowledge points outside the graph; calculating a cognitive load score for each candidate knowledge point in the graph update candidate set according to the user heart rate data at the time corresponding to the user question text; sorting the graph update candidate set based on the cognitive load scores, adding candidate knowledge points with cognitive load scores lower than a preset update threshold to the domain knowledge base, and triggering an update operation on the enhanced knowledge graph.

[0050] Specifically, the interactive data receiving module in the system establishes a continuous data communication link with the user terminal device (such as a smartphone, wearable device, or personal computer equipped with a camera), for example, through the WebSocket protocol or HTTP long polling. The application on the user terminal device synchronously collects the user's question text and heart rate data obtained through sensors (such as the photoplethysmography (PPG) sensor of a smartwatch or remote rPPG technology of a camera), and packages both into a data stream with precise timestamps for uploading. After receiving this interactive data stream, the system first performs natural language processing on the user's question text, such as using named entity recognition and relation extraction techniques to parse it into structured knowledge triples (subject, predicate, object) or core entities. Subsequently, the system compares these parsed entities and relations with the existing augmented knowledge graph. If an entity is found to be absent from the graph, or a new relation between two existing entities is not defined, it is identified as an off-graph knowledge point, and one or more graph update candidate sets to be reviewed are generated based on this.

[0051] Next, the system innovatively introduces users' physiological indicators to evaluate the reliability of these candidate knowledge points. Based on the timestamp corresponding to the user's question text, the system locates heart rate data within a time window (e.g., 5 seconds before and after the question) in the synchronously received heart rate data stream. Instead of directly using the absolute value of the heart rate, the system calculates heart rate variability (HRV) related indicators within that time window, such as the root mean square of the difference between adjacent heartbeat intervals or the standard deviation of all normal heartbeat intervals. In cognitive psychology, a lower HRV is often associated with higher cognitive load or mental stress. Therefore, based on this principle, the system calculates a cognitive load score for each candidate knowledge point in the map update candidate set. One specific calculation method is to compare the current window's HRV value with the user's historical baseline HRV value; the greater the deviation, the more drastic the fluctuation in physiological state, and the higher the calculated cognitive load score. This score aims to quantify the user's cognitive state when asking the question. A lower score may mean that the user asked a valuable new question in a calm and curious state, while a higher score may suggest that the user is confused, frustrated, or inattentive, and the question may be of poor quality or poorly expressed.

[0052] Then, the system makes decisions and filters all candidate knowledge points based on the calculated cognitive load score. First, the system sorts the candidate knowledge point set for graph updates in ascending order according to their corresponding cognitive load scores. After sorting, the system compares the score of each candidate knowledge point with a preset update threshold. This update threshold can be a fixed value or an adaptive threshold that is dynamically adjusted according to the overall update frequency and accuracy requirements of the system. Its core function is to act as a quality filter, automatically filtering out knowledge points generated under high cognitive load (high score) scenarios that may originate from user misunderstanding or invalid input. Only candidate knowledge points with cognitive load scores below the preset update threshold are considered high-quality, reliable potential new knowledge. These filtered candidate knowledge points are then automatically or semi-automatically added to the domain knowledge base, which serves as the system's factual source.

[0053] Finally, updates to the knowledge base trigger synchronization operations on the augmented knowledge graph. Adding candidate knowledge points to the domain knowledge base (which could be a relational database or a document database) can be considered a "commit" operation. This operation triggers a background graph update process. One possible implementation is that the commit operation sends an update notification to a message queue. The graph building service subscribes to this queue, and upon receiving the notification, reads the newly added content from the domain knowledge base and transforms it into nodes and edges supported by the graph database, thereby completing the incremental update or reconstruction of the augmented knowledge graph. This asynchronous update architecture design ensures both the accuracy and consistency of the source data in the knowledge base and guarantees that the knowledge graph of the online service can iterate and expand in near real-time, enabling the virtual character's knowledge system to continuously learn and grow from interactions with users.

[0054] S205. Receive the user's query request through the user terminal device, and parse the query request to determine the query intent; In practice, users can input their query requests through various terminal devices, such as smartphone applications, web browsers, smart speakers, in-vehicle systems, or virtual reality headsets. This request can be in the form of voice input or text input. If it is voice input, the terminal device's built-in Automatic Speech Recognition (ASR) module first converts it into a text string. Subsequently, this text string, along with other necessary contextual information such as user ID, session ID, device geolocation (with user authorization), and current timestamp, is encapsulated into a structured data packet, typically in JSON format. This data packet is then sent to the backend server via a secure network communication protocol, such as HTTPS (Hypertext Transfer Protocol Secure) or WebSocket, as an API call. Upon receiving the request, the server immediately initiates the query intent parsing process. At the core of this process is a specially trained Natural Language Understanding (NLU) module. A preferred implementation is that this NLU module employs a joint intent classification and entity extraction model based on a pre-trained language model (such as BERT or ERNIE). The model first preprocesses the request text, including Chinese word segmentation and stop word removal. Next, it performs two parallel tasks: First, it performs intent classification, mapping the user's query to a predefined set of intent categories, such as "query exhibit information," "inquire about historical background," "query opening hours," "route navigation," or "casual conversation." Second, it performs slot filling or named entity recognition, extracting the key information parameters necessary to execute the intent from the text, also known as entities. For example, in the sentence "What material is the dragon throne in the Hall of Supreme Harmony made of?", the intent is identified as "query exhibit information," and the entities are identified as "Hall of Supreme Harmony" (location) and "dragon throne" (exhibit). Finally, the parsed result (e.g., a structured object containing the intent label query_exhibit_material and the entity list {location: 'Hall of Supreme Harmony', item: 'dragon throne'}) is output, providing clear and explicit instructions for subsequent calls to the target explanation model and the generation of targeted answers.

[0055] S206. Associate the query intent with the event content and input it into the target explanation model to obtain the initial explanation content; One concrete and efficient implementation method is to use dynamic prompt word engineering technology. The system first activates a preset prompt word template, which embeds multiple placeholders corresponding to role settings, context, and user queries. During runtime, the system first serializes the current event content, such as structured data containing {weather: "light rain", current activity: "National Treasure Special Exhibition in progress"}, into a natural language description, such as "[Background Information] Today's weather is light rain, and the National Treasure Special Exhibition is being held at the Palace Museum," and fills it into the context placeholder of the template. At the same time, the system reassembles the query intent parsed from S205, such as intent "query exhibit information" and entity "Jadeite Cabbage," into a clear user instruction, such as "[User Question] Please give me a detailed introduction to the Jadeite Cabbage," and fills it into the user query placeholder. Ultimately, a complete, multi-part compound prompt is dynamically constructed, which might read: "[System Instruction] You are a knowledgeable and witty virtual guide for the Palace Museum. [Context] Background information: Today's weather is light rain, and the National Treasures Special Exhibition is being held at the Palace Museum. [User Question] Please give me a detailed introduction to the Jadeite Cabbage." This compound prompt is then encapsulated and sent to the target explanation model via the application programming interface. Upon receiving this input, the model's finely tuned neural network comprehensively understands all the information, especially utilizing its learned associative reasoning ability to creatively integrate real-time contexts such as "light rain" or "special exhibition" with static knowledge of the "Jadeite Cabbage." It then generates a coherent, vivid, and highly contextualized text word by word through autoregression. This text, directly output by the model, is the initial explanation content.

[0056] S207. Combine the initial explanation content with the character settings into a preset stylized generation model to generate target explanation content that conforms to the language style of the target virtual character. In practice, the system takes the initial explanation content generated in the previous step (a relatively objective and neutral factual text) and the predefined character setting as input and feeds them into a preset stylized generation model. This character setting is not a simple label, but a set of structured descriptive information, such as a JSON object containing key-value pairs like {"Identity": "Qing Dynasty Princess", "Personality": "Lively and playful", "Catchphrases": ["I think...", "Oh, this is really fun!"], "Tone": "Light and slightly coquettish"}. A preferred implementation is that the stylized generation model itself is a powerful, instruction-following large language model. The system dynamically constructs a new input prompt that explicitly instructs the model to play the specified role and rewrite the text, for example: "[System Instruction] You are now a lively and playful Qing Dynasty princess. Please use your identity and tone, keeping the core factual information in the following [Original Text] unchanged, and rewrite it in a language style that matches your character setting. [Original Text]: "[Initial explanation content here]" Upon receiving this instruction, the large language model leverages its powerful text generation and style transfer capabilities to output text that retains the original knowledge points while incorporating the distinctive characteristics of the specified character—the target explanation content. As another feasible technical solution, this stylized generation model can also be a dedicated model fine-tuned with a specific style corpus (e.g., a large amount of text simulating the princess's tone). When it receives the initial explanation content, it will naturally restate the text in its trained style. Through this step, the explanation content output to the user is ultimately ensured to be not only accurate but also highly personalized and immersive, thus significantly enhancing the user's interactive experience.

[0057] Optionally, the step of inputting the initial explanation content and the character setting into a preset stylized generation model to generate target explanation content that conforms to the language style of the target virtual character includes: parsing style instructions from the character setting, the style instructions including a tone style parameter set, terminology usage preferences, and knowledge boundary definitions; decomposing the initial explanation content into multiple semantic segments; for each semantic segment, generating multiple sets of candidate style prompt templates using the tone style parameter set and the terminology usage preferences; filtering the multiple sets of candidate style prompt templates to determine the optimal style prompt template for each semantic segment; constructing a final generation instruction based on the optimal style prompt template and the corresponding semantic segment, and inputting the final generation instruction into the stylized generation model to generate preliminary target explanation content; and performing content compliance verification on the preliminary target explanation content based on the knowledge boundary definitions to obtain the target explanation content.

[0058] Specifically, the system parses style instructions. This process doesn't simply transmit a textual description of the character's persona; instead, a dedicated parsing module performs in-depth analysis of the structured character profile file (such as a JSON or YAML file). For example, a character profile for a meticulous archaeologist might include: {Tone: {Formalism: 0.9, Professionalism: 0.95, Emotional Intensity: 0.2}, Vocabulary: {Preferred words: [Artifacts, Dating, Excavated], Avoided words: [Treasures, Gadgets]}, Knowledge Boundaries: {Domain: Chinese Shang and Zhou Dynasty Bronze Ware, Prohibited Topics: [Estimation, Historical Anecdotes]}}. The parsing module extracts this information and transforms it into style instructions that the machine can directly execute. These instructions include: a quantified set of tone style parameters, a terminology usage preference list containing preferences and avoidance lists, and a "knowledge boundary definition" for content moderation.

[0059] Furthermore, the system decomposes the initial explanatory content into semantic segments. This step aims to break down a complete explanatory text (e.g., "The Jadeite Cabbage is a masterpiece of Qing Dynasty jade art, made of half-white and half-green pyroxene, and the grasshoppers and locusts on it symbolize fertility and good fortune") into independent core information units with complete semantics. An effective implementation is to use deep learning-based natural language processing models, such as BERT or ERNIE, to perform semantic boundary detection, thereby accurately segmenting long or complex sentences into multiple semantically coherent but relatively independent segments. After processing, the above example content can be decomposed into two semantic segments: segment one, "The Jadeite Cabbage is a masterpiece of Qing Dynasty jade art, made of half-white and half-green pyroxene," focuses on the object itself; segment two, "The grasshoppers and locusts on it symbolize fertility and good fortune," focuses on its symbolic meaning.

[0060] Furthermore, for each independent semantic fragment, the system enters a creative candidate style prompt template generation stage. Based on the parsed tone style parameter set and terminology usage preferences, the system calls a pre-set template library or utilizes a small generative model to dynamically generate multiple sets of candidate expressions that conform to the character's setting for the semantic fragment. For example, for the fragment one mentioned above, combined with rigorous archaeologist style instructions, the system might generate the following candidate templates: Candidate A (emphasizing dating): Please note that this [object] is dated to the Qing Dynasty, its material characteristics are…; Candidate B (emphasizing academic perspective): From an archaeological perspective, the material of this [object] is characterized as…, its age can be traced back to the Qing Dynasty; Candidate C (neutral professional statement): Regarding this [object], its era belongs to the Qing Dynasty, its material is…. Here, [object] is replaced according to terminology usage preferences.

[0061] Furthermore, the system needs to intelligently filter the generated multiple sets of candidate style suggestion templates to determine the optimal solution. This step can be achieved through a scoring model that comprehensively considers multiple dimensions to score each candidate template, such as: information fidelity (whether it is consistent with the core information of the original semantic segment), style matching degree (the degree of conformity with various indicators of the tone and style parameter set), and text fluency. The scoring model will output a comprehensive score, and the system will select the template with the highest score as the optimal style suggestion template for that semantic segment. For example, after scoring, the system may consider candidate B to perform best in terms of professionalism, and therefore determine it as the optimal one.

[0062] Furthermore, after determining the optimal style cue template for all semantic fragments, the system constructs the final generation instructions. It fills the content of each semantic fragment into the reserved slot of its corresponding optimal style cue template, and then recombines these filled, stylized sentences. This forms one or more complete, structured final generation instructions with explicit style directives. For example, the instructions might be combined as: [System Instruction] Please integrate and refine the following content in the tone of a rigorous archaeologist: From an archaeological perspective, this artifact is named the Jadeite Cabbage, its material is identified as half-white and half-green gabbro, and its age can be traced back to the Qing Dynasty. Furthermore, regarding its symbolic meaning, the grasshoppers and locusts on the artifact symbolize the auspicious meaning of many children and abundant blessings.

[0063] Furthermore, after inputting the final generation instructions into the stylized generation model and obtaining the initial target explanation content, the system performs a crucial content compliance check. This step specifically utilizes the parsed knowledge boundary definitions to review the generated content. The system uses techniques such as keyword matching and topic model classification to check whether the initial target explanation content exceeds the set knowledge domain (e.g., whether it mentions Song Dynasty porcelain) or touches on prohibited topics (e.g., whether it contains valuation terms like "priceless" or unofficial historical content such as "allegedly referring to Empress Dowager Cixi"). Only content that fully passes the compliance check is ultimately confirmed as professional and safe target explanation content that conforms to the character's settings and is output to the user; for content that fails the check, the system can trigger an alarm, make corrections, or choose not to output it, thereby ensuring the professionalism of the virtual character's speech and the consistency of the brand image.

[0064] S208. Load the three-dimensional virtual image of the target virtual character and associate and bind the three-dimensional virtual image with the target explanation content.

[0065] The specific implementation is typically completed within a real-time scene built using a 3D rendering engine, such as Unity or Unreal Engine. First, the system, based on the currently selected target virtual character (e.g., a Qing Dynasty princess), sends a resource loading request to a cloud-based resource server. This request carries the character's unique identifier. The server then returns a resource package containing all the character's 3D assets, including high-precision 3D model files, high-resolution PBR textures, and a preset animation state machine. Upon receiving the resource package, the client engine dynamically instantiates the 3D virtual character in the current scene, completing the model loading and rendering. Next, the system enters the core association and binding operation, a multi-threaded parallel processing process: on one hand, the system sends the target explanatory text to a speech synthesis engine that highly matches the virtual character's voice characteristics, generating a high-quality audio stream; on the other hand, an intelligent lip-sync module simultaneously receives the text and the generated audio, analyzes the phoneme sequence in the audio, calculates the corresponding facial muscle lip movements in real time, and converts this into a series of continuous animation keyframes that drive the 3D model's facial bones or hybrid deformation. Simultaneously, a body language generation module performs semantic and sentiment analysis on the narration text, extracting emotional tendencies such as surprise or admiration, as well as indicative words, such as phrases used to guide the audience's gaze. It also matches and calls corresponding posture, gesture, and micro-expression animation clips from the animation library, such as raising a hand to point or nodding in approval. Finally, the engine's built-in animation controller precisely aligns and synchronizes the audio stream generated by the speech synthesis engine, the facial animation data stream generated in lip-sync, and the body animation sequence scheduled by the body language generation module on a unified timeline. This achieves the final effect where the virtual character speaks personalized lines that match their identity while simultaneously making matching, vivid, and natural lip movements, expressions, and actions, thus achieving a deep binding between content and image.

[0066] Optionally, loading the 3D virtual image of the target virtual character and associating the 3D virtual image with the target explanation content includes: converting the target explanation content into a speech audio stream, and extracting prosodic features and phoneme sequences from the speech audio stream; retrieving lip-sync animation data corresponding to each phoneme in the phoneme sequence from a preset lip-sync primitive library based on the phoneme sequence, and combining the lip-sync animation data to generate lip-sync animation keyframes; and calculating the body micro-movement sequence through a preset posture generation network based on the prosodic features and the posture habit data in the character settings. The system generates a sequence of head poses and gestures; it performs syntactic analysis and semantic entity recognition on the target content to obtain the timestamps and semantic categories of semantic units, and retrieves corresponding gesture animation clips from a preset gesture library based on the timestamps and semantic categories of the semantic units, and then combines the gesture animation clips to generate a gesture animation sequence; it performs time alignment on the speech audio stream, the lip-sync animation keyframes, the head pose sequence, the body micro-movement sequence, and the gesture animation sequence, and merges them into a multimodal driven data stream; it then associates and binds the multimodal driven data stream with the three-dimensional virtual image.

[0067] Specifically, the system needs to transform the target explanation content into a speech audio stream containing rich driving features. Specifically, the system inputs the final text content into a pre-trained text-to-speech engine that matches the voice of the target virtual character. A preferred implementation is to use a neural network-based end-to-end TTS model, such as VITS (an end-to-end speech synthesis model), which can directly generate high-quality raw audio waveforms from text. While generating the audio stream, the system also performs feature extraction tasks in parallel. On one hand, a phoneme analysis module parses the input text or output audio into a sequence of timestamped phonemes; for example, "hello" is parsed as [{n, 0.0s}, {i, 0.1s}, {h, 0.3s}, {ao, 0.4s}]. On the other hand, a prosodic analysis module extracts prosodic features, including pitch curves, energy envelopes, and durations, from the generated speech audio stream. These features reflect the intonation and rhythmic variations of the speech, providing a basis for subsequent non-verbal behavior generation.

[0068] Furthermore, the system generates precise lip-sync animation based on the phoneme sequence extracted in the previous step. The core of this step involves searching and combining elements in a pre-defined lip-sync primitive library. This library pre-stores several basic lip-sync poses, each corresponding to one or more phonemes. In a 3D model, these poses can be represented as a set of mixed deformation weights or a set of facial bone rotation angles. Once the system obtains the phoneme sequence, it iterates through each phoneme in the sequence and retrieves its corresponding lip-sync pose data from the library. Subsequently, based on the timestamp and duration of each phoneme, the system performs linear or nonlinear interpolation on the retrieved discrete lip-sync pose data to generate a series of continuously changing lip-sync animation keyframes. Alternatively, this step can employ an end-to-end deep learning model. This model directly takes the Mel-spectrum of the speech audio stream as input and uses a convolutional neural network or recurrent neural network to directly regress and output the lip-sync mixed deformation weights corresponding to each frame. This approach typically generates more natural and coordinated mouth movements.

[0069] Furthermore, the system utilizes extracted prosodic features and combines them with character settings to generate natural body micro-movements and head postures. Natural human communication is filled with subconscious, synchronized body swaying and head movements in sync with the rhythm of speech. To simulate this effect, the system inputs the extracted prosodic features (especially pitch and energy curves) and posture habit data defined in the character setting file (e.g., "a confident character has a larger head movement amplitude," "an introverted character has a lower body swaying frequency") into a pre-defined posture generation network. This network can be a Generative Adversarial Network (GAN) or a Long Short-Term Memory (LSTM) network, trained to learn the complex mapping relationship between prosodic features and body dynamics. After computation, the network outputs two sets of continuous time-series data: one set is a sequence of body micro-movements, used to drive slight swaying of the virtual character's torso and limbs; the other set is a sequence of head postures, used to drive nodding, shaking, and tilting movements of the head. In this way, the overall posture of the virtual character is no longer rigid, but rather exhibits subtle, lifelike dynamics that change with the intonation of speech.

[0070] Furthermore, to enable the virtual character to make meaningful gestures, the system performs in-depth analysis of the target content to generate gesture animations. This process first uses natural language processing (NLP) technology to perform syntactic analysis and semantic entity recognition on the text to understand the sentence structure and key information. For example, it identifies indicative words (such as "here," "that"), enumerative words (such as "first point," "secondly"), or emotional words (such as "great!"), and determines the precise timestamps corresponding to these semantic units on the timeline. Next, based on the identified semantic categories and timestamps, the system retrieves corresponding gesture animation clips from a pre-defined, categorized gesture library. This library stores a large number of animation clips such as pointing, shrugging, clapping, and counting. After retrieving a suitable clip, the system scales or edits the animation clip as needed according to the duration of the semantic unit, and smoothly splices multiple clips together to generate a complete and highly semantically relevant sequence of gesture animations.

[0071] Furthermore, the system needs to integrate and synchronize all the separate data streams generated in the preceding steps to form a unified multimodal driving data stream, which is then ultimately applied to the 3D virtual avatar. The core of this step is time alignment. Using the audio stream as the main timeline, the system precisely aligns the generated lip-sync animation keyframes, head pose sequences, body micro-motion sequences, and gesture animation sequences to this main timeline according to their respective timestamps. After alignment, all data is merged into a unified, structured data stream, such as a JSON array, where each element represents a frame and contains all the parameters that frame needs to drive, such as audio samples, facial blending deformation weights, rotation and displacement of each bone node, etc. During the rendering phase, a driving controller reads this multimodal driving data stream frame by frame and applies the parameters contained therein to the corresponding parts of the 3D virtual avatar in real time and synchronously. In a preferred implementation, the drive controller passes the data stream to the animation blueprint or animation controller, utilizing its built-in state machine and hybrid tree functionality to achieve more complex logic control and smoother transitions between animation sequences, thereby driving the virtual character to complete the final, highly realistic, and expressive narration performance.

[0072] In one embodiment of this application, a user visits a museum wearing AR (Augmented Reality) glasses. When the user looks at a bronze artifact, the system acquires external event data indicating "the user is viewing bronze artifact XX," and combines this with the event of "China Cultural Heritage Day" to match a virtual "archaeologist" avatar. The 3D image of this virtual avatar appears in the user's AR field of view, as if standing next to a display case. Subsequently, the virtual avatar, in a professional tone and using gestures to point to the actual bronze artifact, explains its historical background and craftsmanship characteristics. This method seamlessly integrates virtual explanations with the real environment, greatly enhancing the immersion and realism of the mixed reality experience.

[0073] This embodiment also discloses an electronic device, as shown in the reference. Figure 3 The electronic device may include: at least one processor 301, at least one communication bus 302, a user interface 303, a network interface 304, and at least one memory 305. The communication bus 302 is used to enable communication between these components. The user interface 303 may include a display screen or a camera; optionally, the user interface 303 may also include a standard wired interface or a wireless interface. The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0074] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory 305 may include a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. Figure 3 As shown, the memory 305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program based on an AI large model and a mixed reality explanation method.

[0075] exist Figure 3In the electronic device shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the user input data; while the processor 301 can be used to call an application stored in the memory 305 that is an explanation method based on AI large model and mixed reality. When executed by one or more processors 301, the electronic device executes one or more methods as described in the above embodiments.

[0076] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the disclosure in this specification. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A teaching method based on AI large-scale models and mixed reality, characterized in that, Applied to a server, the method includes: Acquire external event data, and extract event types and event content from the external event data; Based on the event type, a target virtual character is matched from a preset virtual character setting library, and the target virtual character has character settings associated with the event type; The external event data is fused with data from a pre-defined domain knowledge base to obtain a training dataset; The target explanation model is obtained by training a preset knowledge reasoning model based on the training dataset. The system receives user query requests through user terminal devices and parses the query requests to determine the query intent. The query intent and the event content are associated and input into the target explanation model to obtain the initial explanation content; The initial explanation content is input into a preset stylized generation model along with the character settings to generate target explanation content that conforms to the language style of the target virtual character; Load the 3D virtual image of the target virtual character and associate and bind the 3D virtual image with the target explanatory content.

2. The method according to claim 1, characterized in that, The process of matching a target virtual character from a preset virtual character setting library based on the event type includes: Based on the event type, construct an event feature vector; For each virtual character in the virtual character setting library, construct multiple character feature vectors; The similarity between the event feature vector and all the character feature vectors is calculated to obtain the matching score between each virtual character and the event type. The virtual character with the highest matching score is selected as the target virtual character.

3. The method according to claim 1, characterized in that, The step of fusing the external event data with data from a pre-defined domain knowledge base to obtain a training dataset includes: Extract event entities and event relationships from the external event data to construct an event knowledge subgraph; Based on the event entity, retrieve related knowledge entities and related knowledge paths from the domain knowledge base; The event knowledge subgraph, the associated knowledge entities, and the associated knowledge paths are merged to generate an enhanced knowledge graph; The event content is converted into natural language text and fused with the enhanced knowledge graph to obtain the training dataset.

4. The method according to claim 3, characterized in that, The step of training a preset knowledge reasoning model based on the training dataset to obtain the target explanation model includes: Based on the content differences between the event content and the enhanced knowledge graph, the knowledge missing parts and knowledge conflict parts in the event content are determined. Based on the missing knowledge, supplementary knowledge paths are generated by reasoning in the enhanced knowledge graph, and the supplementary knowledge paths are integrated into the training dataset. Based on the knowledge conflict, the connection weights of the corresponding nodes in the enhanced knowledge graph are adjusted to generate a revised knowledge structure. During the training of the knowledge reasoning model, the training dataset and the modified knowledge structure are used alternately, and iterative optimization is performed by setting a preset hybrid training strategy until the preset convergence condition is met, thereby obtaining the target explanation model.

5. The method according to claim 3, characterized in that, The method further includes: Receive the interactive data stream uploaded by the user terminal device, the interactive data stream including user question text and user heart rate data; The user's question text is compared with the enhanced knowledge graph to identify knowledge points outside the graph, and a graph update candidate set is generated based on the knowledge points outside the graph. Based on the user's heart rate data at the time corresponding to the user's question text, a cognitive load score is calculated for each candidate knowledge point in the map update candidate set; The candidate set for knowledge graph updates is sorted based on the cognitive load score. Candidate knowledge points with cognitive load scores below a preset update threshold are added to the domain knowledge base, and an update operation for the enhanced knowledge graph is triggered.

6. The method according to claim 1, characterized in that, The step of combining the initial explanation content with the character settings input into a preset stylized generation model to generate target explanation content that conforms to the language style of the target virtual character includes: Style instructions are extracted from the character settings, including a set of tone style parameters, terminology usage preferences, and knowledge boundary definitions. The initial explanation is broken down into multiple semantic fragments; For each semantic segment, multiple sets of candidate style prompt templates are generated using the tone style parameter set and the term usage preference; The multiple sets of candidate style suggestion templates are filtered to determine the optimal style suggestion template for each semantic segment; The final generation instruction is constructed based on the optimal style prompt template and the corresponding semantic fragment, and the final generation instruction is input into the stylized generation model to generate preliminary target explanation content; Based on the defined knowledge boundaries, the content compliance of the initial target explanation is verified to obtain the target explanation content.

7. The method according to claim 1, characterized in that, The process of loading the 3D virtual image of the target virtual character and associating and binding the 3D virtual image with the target explanatory content includes: The target explanation content is converted into a speech audio stream, and prosodic features and phoneme sequences are extracted from the speech audio stream; Based on the phoneme sequence, lip-sync animation data corresponding to each phoneme in the phoneme sequence is retrieved from a preset lip-sync primitive library, and the lip-sync animation data is combined to generate lip-sync animation keyframes. Based on the rhythmic features and the posture habit data in the character settings, a body micro-movement sequence and a head posture sequence are calculated through a preset posture generation network. Syntactic analysis and semantic entity recognition are performed on the target explanation content to obtain the timestamp and semantic category of the semantic unit. Based on the timestamp and semantic category of the semantic unit, the corresponding gesture animation clips are retrieved from the preset gesture library and then combined to generate a gesture animation sequence. The speech audio stream, the lip-sync animation keyframes, the head pose sequence, the body micro-motion sequence, and the gesture animation sequence are time-aligned and merged into a multimodal driven data stream; The multimodal driving data stream is associated and bound with the three-dimensional virtual image.

8. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. The user interface and the network interface are both used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-7.

Images