Teaching practical training simulation system and method based on cooperation of large and small models
The teaching and training simulation system, which combines large and small models, solves the problem of insufficient language interaction in virtual classroom teaching systems, realizes a highly realistic virtual training environment, supports multimodal interaction and natural virtual character behavior generation, and enhances the practical ability of teacher trainees.
Patent Information
- Application Number
- CN202511337064.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-18
- Publication Date
- 2025-11-21
AI Technical Summary
Existing virtual classroom teaching systems cannot achieve natural language interaction, lack student behavior generation, have poor immersion, and cannot simulate real classroom teaching scenarios. In particular, teacher trainees lack practical venues and teaching skills honed in front of real students.
The teaching and training simulation system, which adopts a collaboration between large and small models, collects multimodal information in real time through a data acquisition module. It combines the large and small models to perform semantic extraction and intent understanding, and generates the voice, actions and expressions of virtual characters. It supports multimodal interaction, including scenarios such as virtual classrooms, home visit simulations and after-school tutoring.
It has achieved a highly realistic virtual training environment, expanded the perception capabilities of intelligent agents, reduced question-and-answer latency, provided immersive teaching and practice, supported training of various teaching abilities, and improved the naturalness and reaction speed of virtual characters in interaction.
Smart Images
Figure CN120998088A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a teaching and training simulation system and method based on collaboration between large and small models. Background Technology
[0002] In response to the lack of practical training opportunities for teacher trainees and their absence of opportunities to hone their teaching skills in front of real students, a series of virtual classroom teaching simulation platforms have been proposed both domestically and internationally. These platforms simulate real classroom teaching scenarios and construct teaching scenarios using virtual students, thus providing practical training opportunities for teacher trainees.
[0003] Based on the different methods of generating virtual students, virtual classroom teaching platforms can be broadly categorized into three types: human-controlled, computer program-controlled, and large language model-controlled. Among these, large language model-controlled platforms primarily refer to virtual classroom systems based on large language models, representing the latest attempt at generating teaching scenarios. In GPTeach, demonstrated by Stanford University, each virtual student is backed by a large language model, enabling autonomous question-and-answer sessions between virtual students and teachers. Currently, such attempts are limited to chat-based modes, i.e., text-based dialogue. They cannot intuitively and vividly represent the real classroom teaching scenario, cannot achieve natural language interaction, differ from real classrooms, lack the generation of student behavior, and have a poor sense of immersion. Summary of the Invention
[0004] In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide a teaching and training simulation system and method based on large and small model collaboration.
[0005] To achieve the above-mentioned objectives of this invention, this invention provides a teaching and training simulation system based on large-scale model collaboration, comprising:
[0006] The data acquisition module is used to collect real-time multimodal information during the training process, including scene image signals and audio signals; and to pre-record multimodal scene information, which includes at least virtual character portraits and course resources.
[0007] The data perception module employs a large-scale model collaboration mechanism to perform real-time semantic extraction and asynchronous extraction of general semantics from the real-time multimodal information and multimodal scene information.
[0008] The intent understanding module performs intent understanding and analysis based on the real-time semantic information extracted by the data perception module, including intelligent agent response judgment and user command judgment.
[0009] The generation module integrates real-time semantic information and general semantic information extracted by the data perception module, as well as intent information analyzed by the intent understanding module, to generate the voice, actions, and expressions of the virtual character.
[0010] The rendering module receives the voice, action sequence, and facial expression parameters output by the generation module and performs real-time rendering.
[0011] This invention is primarily aimed at teacher trainees and newly hired teachers. By constructing a highly realistic virtual training environment that supports multimodal interaction, it can cover and simulate various virtual scenarios, such as virtual classrooms, simulated home visits, and after-school tutoring. This breaks through the time, space, and resource limitations of traditional training models, expands the perception capabilities of intelligent agents, and reduces question-and-answer latency.
[0012] The data perception module employs a combination of large and small models, supporting both real-time speech and action recognition as well as deep semantic understanding of course resources (such as PPTs and video content). This allows virtual characters to perceive speech content and user posture in real time, as well as understand a wider range of visual or audio information (provided in course resources), broadening the scope of teaching methods for user training. The intent understanding module judges the timing of conversations, avoiding interruptions during real-time interaction, such as the agent interrupting when the user's semantic expression is incomplete and they enter a temporary thinking state.
[0013] Optionally, the scene information may also include character background and teaching scene type.
[0014] Optionally, the data perception module analyzes the real-time multimodal information collected by the data acquisition module to obtain real-time semantic information; it preprocesses or samples the multimodal scene information entered by the data acquisition module across time periods, and obtains general semantic information through general multimodal analysis.
[0015] The inference speed of general multimodal analysis is relatively slow. Therefore, in this alternative, general multimodal analysis does not participate in the real-time data processing of the overall framework. Instead, it preprocesses the scene information or samples it across time periods to avoid problems such as system lag and slow response of virtual characters caused by general multimodal analysis.
[0016] Optionally, the data sensing module includes:
[0017] The real-time multimodal analysis submodule, based on a lightweight speech recognition model and a real-time target detection model, transcribes speech signals in real time and quickly recognizes user posture and pointing gestures, outputting structured text and action labels.
[0018] The general multimodal analysis submodule, relative to the extraction of real-time semantic information, uses a multimodal large model based on an asynchronous processing mechanism to deeply understand the scene information and embeds the extracted general semantic information into the memory of the generation module as the context information of the generation module.
[0019] Optionally, the intent understanding module includes:
[0020] The intelligent agent conversation response judgment module includes a large language model, which constructs conversation response judgment context information based on natural dialogue data, performs parameter fine-tuning training on the large language model, and uses the large language model to judge the timing of conversation. The context information includes dialogue history, user silence time, and intelligent agent silence time.
[0021] The scene instruction recognition module includes a scene instruction recognition model. The parameters of the scene instruction recognition model are fine-tuned and trained based on the teaching language. The scene instruction recognition model is used to recognize the teaching instructions in the training and realize the judgment of user instructions.
[0022] This alternative approach uses a large model for fine-tuning to judge the agent's responses and user commands, and has higher generalization ability compared to traditional text classification methods.
[0023] Optionally, the generation module includes:
[0024] The memory module uses a vector database to store virtual character profiles, scene states, historical interaction contexts, and general semantic information extracted by the data perception module.
[0025] The end-to-end speech generation module generates speech directly from semantic input; when the intent understanding module recognizes that the user's intent is to answer, the end-to-end speech generation module generates an answer based on the context information from the memory module and the question.
[0026] The behavior generation module generates the actions and expressions of each virtual character based on the speech generated by the memory module and the end-to-end speech generation module.
[0027] The introduction of an end-to-end speech generation module in this optional solution accelerates the reasoning speed of the virtual character in content responses, significantly reducing user interaction latency. The introduction of a behavior generation module enables the virtual character not only to express actions within a scene, but also to exhibit more natural facial expressions and movements during speech.
[0028] The behavior generation module includes:
[0029] Scene behavior generation submodule: Based on reinforcement learning, it controls the spontaneous behavior of virtual characters in non-speaking states by relying on virtual character profiles and scene information;
[0030] The speech-action matching model submodule matches lip movements, facial expressions, and auxiliary gestures in real time based on the content and prosody of the speech generated by the end-to-end speech generation module.
[0031] This alternative solution generates scene behavior, avoiding the lack of scene behavior expression ability of traditional digital humans in non-speech states such as classrooms and home visits; through voice-action matching, the virtual character presents coordinated changes in lip shape, facial expression and gestures during the speaking process, avoiding the "rigid" problem of traditional digital human voice playback.
[0032] This invention also proposes a teaching and training simulation method based on large and small model collaboration, comprising the following steps:
[0033] Before the start of the practical training, multimodal scene information should be entered, including at least virtual character portraits and course resources; large models should be used to preprocess or process the scene information across time periods to extract general semantic information.
[0034] During the training process, real-time multimodal information, including scene image signals and audio signals, is collected in real time; a small model is used to process the real-time multimodal information in real time and extract real-time semantic information.
[0035] Analyze the user's intent information based on the real-time semantic information;
[0036] By combining the real-time semantic information, general semantic information, and intent information, the voice, actions, and expressions of the virtual character are generated and rendered and output in real time.
[0037] Furthermore, when analyzing the user's intent information, the timing of the agent's response is analyzed based on the real-time semantic information to determine whether the agent needs to respond.
[0038] The user's instructions are judged based on the real-time semantic information, and the instructions in the real-time semantic information are analyzed.
[0039] Furthermore, when it is determined that the intelligent agent needs to respond, the system generates a response speech based on real-time semantic information, general semantic information, and the analyzed instructions, and generates virtual character actions and expressions that match the speech.
[0040] When it is determined that the agent does not need to respond, facial expressions and scene actions are generated based on the agent's profile and the scene state.
[0041] The beneficial effects of this invention are:
[0042] This invention employs highly realistic digital human technology and integrates cutting-edge artificial intelligence methods such as multimodal semantic analysis, general semantic analysis, large language models, and refined behavior modeling to achieve intelligent driving of virtual characters at the levels of perception, cognition, and behavior. It provides users with an immersive, two-way interactive teaching and training environment, supporting practical training and interactive activities, such as teacher-student interaction, classroom management, emotional communication, and home-school communication, among other teaching skills training.
[0043] Specifically,
[0044] This invention embeds a large language model into a natural interaction agent framework, enabling the agent to generate real-time voice responses based on contextual information while possessing scene awareness and intent understanding. Furthermore, the system employs a separate method for determining the timing of responses, preventing interruptions during real-time interaction.
[0045] This invention employs a combination of small and large models for data perception. The small model participates in real-time interaction with the intelligent agent, extracting user speech recognition, user position, and posture based on audio and video signals, which have high real-time requirements. Meanwhile, the multimodal large model uses analysis to understand general scene information (such as PPTs, teaching resources, etc.), embedding the extracted information into the memory module. This information participates in the generation of intelligent agent feedback through context, improving the speed of intelligent agent response and making the virtual training process smoother.
[0046] This invention is not limited to generating scene actions for students or parents; it also matches audio with actions and facial expressions during the virtual character's speech, making the virtual character have more natural actions and expressions during dialogue.
[0047] This invention reduces the question-and-answer delay to less than 1 second through end-to-end speech generation and lightweight intent recognition, making it particularly suitable for scenarios with high real-time requirements, such as home visits and after-school tutoring.
[0048] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0049] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:
[0050] Figure 1 This is a schematic diagram of the principle framework of the present invention;
[0051] Figure 2 This is a schematic diagram of the data sensing module principle;
[0052] Figure 3 This is a schematic diagram illustrating the principle of the intent-to-understand module;
[0053] Figure 4 This is a schematic diagram of the generation module principle. Detailed Implementation
[0054] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0055] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.
[0056] Example 1
[0057] like Figure 1 As shown, this invention provides a teaching and training simulation system based on large and small model collaboration. This system is primarily aimed at teacher trainees and novice teachers. By constructing a highly realistic virtual training environment that supports multimodal interaction, it covers various typical teaching scenarios such as virtual classrooms, simulated home visits, and after-school tutoring, breaking through the time, space, and resource limitations of traditional training models. Specifically, the system includes: a data acquisition module, a data perception module, an intent understanding module, a generative model, and a rendering module.
[0058] The data acquisition module collects real-time multimodal information during the training process and pre-records multimodal scene information. This real-time multimodal information includes scene image signals and audio signals. Audio signals come from the user's microphone input, and image signals come from an environmental camera, capturing the user's posture and facial expressions. Scene information includes at least virtual character profiles and course resources, and may also include structured non-real-time data such as character backgrounds and teaching scenarios. All virtual characters are intelligent agents, and their profiles are modeled using multi-dimensional methods, including cognitive characteristics, personality traits (extended from the Big Five personality traits), classroom participation, and communication styles. Course resources refer to PPTs, video resources, audio resources, etc., and teaching scenarios can include virtual classrooms, simulated home visits, and after-school tutoring.
[0059] The data perception module employs a collaborative approach between large and small models to understand and analyze multimodal data. Specifically, it includes real-time multimodal analysis and general multimodal analysis. Real-time multimodal analysis is performed in real-time, while general multimodal analysis is performed asynchronously, extracting real-time semantic information and general semantic information.
[0060] Real-time multimodal analysis refers to the analysis of real-time multimodal information through methods such as speech recognition and object detection. For example, it involves transcribing user speech, detecting user posture and scene-specific objects, with the aim of analyzing specific real-time semantic information.
[0061] like Figure 2 As shown, this embodiment uses a real-time multimodal analysis submodule to perform real-time multimodal analysis. The real-time multimodal analysis submodule is a small model that uses a lightweight speech recognition model and a real-time object detection model (such as YOLO) to transcribe speech signals in real time, and quickly recognize user posture and pointing gestures, outputting structured text and action labels.
[0062] General multimodal analysis refers to the analysis of general semantic information in multimodal scenarios using a general multimodal model. General multimodal analysis is not limited to the analysis of specific targets, but rather targets a broader range of scenario content understanding, such as training PPTs, training audio and video comprehension, etc.
[0063] This embodiment employs a general multimodal analysis submodule for general multimodal analysis. Specifically, a large multimodal model (such as GPT-4V or Qwen VL) is used to perform deep understanding of non-real-time elements in the application scenario, including PPT content extraction, semantic summarization of teaching videos, and image understanding of student assignments. Typically, due to the high inference latency of large multimodal models, the inference speed is slow. This submodule uses an asynchronous processing mechanism to preprocess or sample scene information across time periods (i.e., frame extraction), and embeds the extracted general semantic information into a memory module as contextual knowledge for the generation module. During preprocessing, scene information can be processed by frame extraction or frame-by-frame. If real-time processing of scene information is required, cross-time period sampling is performed, and the time interval must be greater than the time interval used in preprocessing.
[0064] The intent understanding module further analyzes and understands user intent based on the real-time semantic information extracted by the data perception module, including agent response judgment and user command judgment. Therefore, as... Figure 3 As shown, in this embodiment, the intent understanding module includes an agent response judgment module and a scene command recognition module.
[0065] The agent's response judgment module determines the timing of a conversation based on contextual information, including dialogue history, user silence time, and agent silence time, all constructed from natural dialogue data. A large language model is built, using the response judgment contextual information as input and whether or not a response occurs as output for parameter fine-tuning training. This large language model is trained to participate in user intent understanding through fine-tuning. The natural dialogues here are taken from real-world audio conversations, and the "response" label is manually assigned. The presence or absence of speaker switching during each dialogue interval is manually determined before the large language model is trained to make these judgments.
[0066] The scene instruction recognition module is used to identify teaching instructions in practical training, enabling the judgment of user commands. It recognizes teaching instructions such as "start discussion," "ask a question," and "please sit down." Traditional scene instruction recognition often requires a large amount of classroom instruction data for training and has poor recognition accuracy for long-tail data. Therefore, this paper constructs a scene instruction recognition model and a scene instruction dataset. The scene instruction recognition model is trained using the scene instruction dataset, and a large-scale model fine-tuning approach is adopted, where the model is trained by inputting teaching language and outputting teaching instructions.
[0067] In this embodiment, the intent understanding module participates in the real-time semantic information processing process. Therefore, both the agent response judgment module and the scene instruction recognition module can use a large text model with low parameters such as Qwen-1.5B as the base model for fine-tuning to achieve response judgment and instruction recognition.
[0068] The generation module is the core of the system, designed to integrate real-time semantic information extracted by the data perception module and general semantic information, as well as intent information analyzed by the intent understanding module, to generate the virtual character's voice, actions, and expressions. Its structure is as follows: Figure 4 As shown, it includes a memory module, an end-to-end speech generation module, and a behavior generation module.
[0069] The memory module employs a vector database to store, retrieve, and forget virtual character profiles, scene states, historical interaction contexts, and general semantic information extracted by the data perception module. Scene states refer to the time, category, and related data of a scene occurrence.
[0070] The end-to-end speech generation module employs a large-scale speech model based on GLM-4-Voice, directly generating speech from semantic input to avoid the latency issues of traditional multi-stage processing. When the user's intent is to answer, the memory module integrates contextual information with the question, and the end-to-end speech generation module generates the answer. The question here is a type of instruction, identified by the scene instruction recognition module within the intent understanding module.
[0071] During character listening, the behavior generation module generates the virtual character's facial expressions and scene actions based on the virtual character profile and scene state from the memory module. In the question-and-answer phase, the end-to-end speech generation module generates speech that matches the virtual character's actions and facial expressions. This mainly includes two sub-modules: a scene behavior generation sub-module and a speech-action matching model sub-module. The scene behavior generation sub-module, based on reinforcement learning (such as PPO), relies on the virtual character profile and scene information to control the virtual character's spontaneous behavior in non-speaking states, such as nodding, daydreaming, raising a hand, etc. The speech-action matching model sub-module uses an audio-video alignment pre-trained model to match lip movements, facial expressions, and auxiliary gestures (such as emphasis gestures and questioning expressions) in real time according to the content and rhythm of the generated speech, ensuring multimodal consistency during the speaking process.
[0072] The rendering module is designed to perform real-time audio playback and render actions and speech based on the speech, action sequences, and facial expression parameters generated by the generation module. This embodiment uses the UE5 high-definition rendering pipeline to construct multi-scene environments (classroom, home, office) and achieves high-fidelity virtual characters through digital human rendering technology (such as the MetaHuman plugin).
[0073] Example 2
[0074] This invention also provides a teaching and training simulation method based on large-scale model collaboration. This method can be implemented using the teaching and training simulation system based on large-scale model collaboration described in Example 1, and mainly includes the following steps:
[0075] Before the start of the practical training, multimodal scene information should be entered, including at least virtual character portraits and course resources; large models should be used to preprocess or process the scene information across time periods to extract general semantic information.
[0076] During the training process, real-time multimodal information, including scene image signals and audio signals, is collected in real time; a small model is used to process the real-time multimodal information in real time and extract real-time semantic information.
[0077] The user's intent is analyzed based on the real-time semantic information. Specifically, the timing of the agent's response is analyzed based on the real-time semantic information to determine whether the agent needs to respond.
[0078] The user's instructions are judged based on the real-time semantic information, and the instructions in the real-time semantic information are analyzed.
[0079] By combining the real-time semantic information, general semantic information, and intent information, the speech, actions, and expressions of the virtual character are generated and rendered in real time. Specifically, when it is determined that the agent needs to respond, a response speech is generated based on the real-time semantic information, general semantic information, and the analyzed instructions, and virtual character actions and expressions matching the speech are generated; when it is determined that the agent does not need to respond, expressions and scene actions are generated based on the character profile and scene state.
[0080] If there are any omissions or omissions in this embodiment, please refer to Embodiment 1 and the prior art; further details will not be provided here.
[0081] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0082] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Claims
1. A teaching and training simulation system based on large and small model collaboration, characterized in that, include: The data acquisition module is used to collect real-time multimodal information during the training process, including scene image signals and audio signals; and to pre-record multimodal scene information, which includes at least virtual character portraits and course resources. The data perception module employs a large-scale model collaboration mechanism to perform real-time semantic extraction and asynchronous extraction of general semantics from the real-time multimodal information and multimodal scene information. The intent understanding module performs intent understanding and analysis based on the real-time semantic information extracted by the data perception module, including intelligent agent response judgment and user command judgment. The generation module integrates real-time semantic information and general semantic information extracted by the data perception module, as well as intent information analyzed by the intent understanding module, to generate the voice, actions, and expressions of the virtual character. The rendering module receives the voice, action sequence, and facial expression parameters output by the generation module and performs real-time rendering.
2. The teaching and training simulation system based on large and small model collaboration according to claim 1, characterized in that, The scene information also includes character background and teaching scene type.
3. The teaching and training simulation system based on large and small model collaboration according to claim 1, characterized in that, The data perception module analyzes the real-time multimodal information collected by the data acquisition module to obtain real-time semantic information; it preprocesses or samples the multimodal scene information entered by the data acquisition module across time periods, and obtains general semantic information through general multimodal analysis.
4. The teaching and training simulation system based on large and small model collaboration according to claim 1 or 3, characterized in that, The data sensing module includes: The real-time multimodal analysis submodule, based on a lightweight speech recognition model and a real-time target detection model, transcribes speech signals in real time and quickly recognizes user posture and pointing gestures, outputting structured text and action labels. The general multimodal analysis submodule, relative to the extraction of real-time semantic information, uses a multimodal large model based on an asynchronous processing mechanism to deeply understand the scene information and embeds the extracted general semantic information into the memory of the generation module as the context information of the generation module.
5. The teaching and training simulation system based on large and small model collaboration according to claim 1, characterized in that, The intent understanding module includes: The intelligent agent conversation response judgment module includes a large language model, which constructs conversation response judgment context information based on natural dialogue data, performs parameter fine-tuning training on the large language model, and uses the large language model to judge the timing of conversation. The context information includes dialogue history, user silence time, and intelligent agent silence time. The scene instruction recognition module includes a scene instruction recognition model. The parameters of the scene instruction recognition model are fine-tuned and trained based on the teaching language. The scene instruction recognition model is used to recognize the teaching instructions in the training and realize the judgment of user instructions.
6. The teaching and training simulation system based on large and small model collaboration according to claim 1, characterized in that, The generation module includes: The memory module stores virtual character portraits, scene states, historical interaction contexts, and general semantic information extracted by the data perception module. The end-to-end speech generation module generates speech directly from semantic input; when the intent understanding module recognizes that the user's intent is to answer, the end-to-end speech generation module generates an answer based on the context information from the memory module and the question. The behavior generation module generates facial expressions and scene actions based on the character profile and scene state from the memory module when the character is listening. During the question-and-answer phase, it uses the speech generated by the end-to-end speech generation module to generate virtual character actions and facial expressions that match the speech.
7. The teaching and training simulation system based on large and small model collaboration according to claim 1, characterized in that, The behavior generation module includes: Scene behavior generation submodule: Based on reinforcement learning, it controls the spontaneous behavior of virtual characters in non-speaking states by relying on virtual character profiles and scene information; The speech-action matching model submodule matches lip movements, facial expressions, and auxiliary gestures in real time based on the content and prosody of the speech generated by the end-to-end speech generation module.
8. A teaching and training simulation method based on large and small model collaboration, characterized in that, Includes the following steps: Before the start of the practical training, multimodal scene information should be entered, including at least virtual character portraits and course resources; Large models are used to preprocess or process scene information across time periods to extract general semantic information; During the training process, real-time multimodal information, including scene image signals and audio signals, is collected. A small model is used to process real-time multimodal information and extract real-time semantic information. Analyze the user's intent information based on the real-time semantic information; By combining the real-time semantic information, general semantic information, and intent information, the voice, actions, and expressions of the virtual character are generated and rendered and output in real time.
9. The teaching and training simulation method based on large and small model collaboration according to claim 8, characterized in that, When analyzing the user's intent information, the timing of the agent's response is analyzed based on the real-time semantic information to determine whether the agent needs to respond. The user's instructions are judged based on the real-time semantic information, and the instructions in the real-time semantic information are analyzed.
10. The teaching and training simulation method based on large and small model collaboration according to claim 9, characterized in that, When it is determined that the intelligent agent needs to respond, the system generates a response speech based on real-time semantic information, general semantic information and the analyzed instructions, and generates virtual character actions and expressions that match the speech. When it is determined that the agent does not need to respond, facial expressions and scene actions are generated based on the agent's profile and the scene state.