Intelligent simulation system and method based on multi-modal interaction and dynamic teaching strategy

Abstract

The invention discloses an intelligent simulation system and method based on multi-modal interaction and a dynamic teaching strategy, and the system comprises a multi-modal interaction emotion perception module, a dynamic teaching strategy generation module, a situational multi-post training module, an automatic guidance and evaluation module, and an integration and optimization module. Specifically, the multi-modal interaction emotion perception module comprises a visual information acquisition module and a voice information acquisition module; and the visual information acquisition module is used for capturing facial expression, eye movement and gesture information of the student through a camera. The invention belongs to the technical field of simulation training and intelligent teaching, and achieves the technical effects that the limitation of the prior art is broken through by organically combining the technologies of multi-modal emotion perception, dynamic teaching strategy generation, situational multi-post training, multi-dimensional intelligent evaluation and the like, and compared with the prior art, the system has the advantages that the system is simple in structure and convenient to use. The method has obvious technical advantages in the aspects of real-time emotion perception, self-adaptive teaching strategies, cross-role interaction, multi-dimensional evaluation and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of simulation training and intelligent teaching, and in particular to an intelligent simulation system and method based on multi-modal interaction and dynamic teaching strategy. Background Art

[0002] Existing simulation training platforms mainly rely on pre-set teaching content and fixed interaction methods, which usually have the following shortcomings:

[0003] 1. Single interaction method: Most of them only support traditional input methods such as mouse and keyboard, lack the perception of students' emotions, attention and other states, and cannot achieve natural and real-time multimodal interaction.

[0004] 2. Fixed teaching strategies: Teaching content and progress are usually pre-set and cannot be dynamically adjusted according to students’ real-time feedback and learning progress, making it difficult to meet individualized learning needs.

[0005] 3. Low level of intelligence: Lack of intelligent retrieval and generation capabilities for large-scale knowledge bases, unable to provide in-depth and accurate teaching guidance and problem solving. Summary of the invention

[0006] In view of the above-mentioned deficiencies, the purpose of the present invention is to provide an intelligent simulation system and method based on multimodal interaction and dynamic teaching strategies. By introducing emotional computing, multimodal data fusion and dynamic teaching strategy generation technology, the virtual teacher can realize real-time perception of the student status and adaptive adjustment of the teaching strategy, thereby improving the interactivity, intelligence and personalization of the training.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] According to a first aspect of the present invention, an intelligent simulation system based on multimodal interaction and dynamic teaching strategies, the system includes a multimodal interaction emotion perception module, a dynamic teaching strategy generation module, a situational multi-position training module, an automated guidance and evaluation module, and an integration and optimization module;

[0009] The multimodal interactive emotion perception module is used to capture the facial information and voice information of the trainees;

[0010] Dynamic teaching strategy generation module, used to provide suitable choices for students and adjust teaching content according to the student information obtained by the multimodal interactive emotion perception module;

[0011] The situational multi-position training module is used to build a learning simulation scenario for students based on the teaching content provided by the dynamic teaching strategy generation module, to help students understand and promote their thinking;

[0012] The automated guidance and evaluation module is used to record the training process of the situational multi-position training module, evaluate the trainees' operations and provide improvement suggestions.

[0013] Furthermore, the multimodal interactive emotion perception module includes a visual information acquisition module and a voice information acquisition module; the visual information acquisition module is used to capture the facial expressions and eye movements of the students through a camera; the voice information acquisition module is used to obtain the students' voice input, including voice content, intonation and speaking speed.

[0014] Furthermore, the dynamic teaching strategy generation module includes a personalized model establishment module, a real-time strategy adjustment module and a teaching content generation module; the personalized model establishment module is used to establish a personalized learning model according to the trainee's position, historical learning data and current emotional state; the real-time strategy adjustment module is used to dynamically adjust the teaching content, difficulty and interaction method according to the personalized model and the real-time identified emotional state; the teaching content generation module is used to select teaching content suitable for the current student status from the RAG knowledge base according to the reinforcement learning algorithm.

[0015] Furthermore, the situational multi-position training module includes a multi-role simulation environment module, a cross-role interaction module and a task-driven learning module; the multi-role simulation environment module is used to construct a simulation scenario including multiple roles of operators, maintenance engineers and dispatchers; the cross-role interaction module is used to allow trainees to interact between different roles and understand the collaborative relationship between positions; the task-driven learning module is used to set up complex task scenarios to promote trainees' active thinking and practice.

[0016] Furthermore, the automated guidance and evaluation module includes a process data collection module, an intelligent evaluation module, and a feedback and improvement suggestion module; the process data collection module is used to record the trainees' operating behaviors, decision-making processes, and interactions in real time; the intelligent evaluation module is used to conduct a multi-dimensional evaluation of the trainees' performance based on the process data using data mining and machine learning techniques; the feedback and improvement suggestion module is used to generate personalized evaluation reports and provide targeted improvement suggestions.

[0017] Furthermore, it also includes an integration and optimization module, including an architecture design module and a data processing module; the integration and optimization module is used to facilitate module docking and ensure the stable operation of system data; the architecture design module is used for modular and standardized interface design to facilitate integration with existing simulation platforms; the data processing module is used for distributed computing and memory data management to ensure the stable operation of the system under large-scale data and high concurrency conditions.

[0018] Further, step 1: obtaining the learner’s visual and voice information through a multimodal interactive emotion perception module to identify the emotional state and attention level in real time;

[0019] Step 2: Based on the students’ personalized model and real-time emotional state, the dynamic teaching strategy generation module is used to dynamically adjust the teaching content and strategy;

[0020] Step 3: In the situational multi-position training module, trainees complete the training tasks through task-driven learning and cross-role interaction;

[0021] Step 4: The automated guidance and multi-dimensional intelligent assessment module records trainees’ operational behaviors in real time, uses an intelligent assessment model to conduct assessments, and provides feedback and improvement suggestions.

[0022] The present invention has the following advantages: by organically combining technologies such as multimodal emotion perception, dynamic teaching strategy generation, situational multi-position training and multi-dimensional intelligent evaluation, the limitations of the existing technology are broken through. Compared with the existing technology, the present invention has obvious technical advantages in real-time emotion perception, adaptive teaching strategy, cross-role interaction and multi-dimensional evaluation, can significantly improve the training effect and learning experience, and has broad application prospects and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A block diagram of the overall system principle provided by the present invention.

[0024] Figure 2 This is an internal diagram of the multimodal interactive emotion perception module provided by the present invention.

[0025] Figure 3 This is an internal diagram of the dynamic teaching strategy generation module provided by the present invention.

[0026] Figure 4 This is an internal diagram of the situational multi-position training module provided by the present invention.

[0027] Figure 5 Internal diagram of the automated guidance and assessment module provided by the present invention.

[0028] Figure 6 Internal diagram of the integration and optimization module provided by the present invention. DETAILED DESCRIPTION

[0029] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Example 1

[0031] like Figures 1 to 6 As shown, the intelligent simulation system based on multimodal interaction and dynamic teaching strategy in the embodiment of the first aspect of the present invention comprises a multimodal interaction emotion perception module, a dynamic teaching strategy generation module, a situational multi-position training module, an automated guidance and evaluation module, and an integration and optimization module;

[0032] The multimodal interactive emotion perception module is used to capture the facial information and voice information of the trainees;

[0033] Dynamic teaching strategy generation module, used to provide suitable choices for students and adjust teaching content according to the student information obtained by the multimodal interactive emotion perception module;

[0034] The situational multi-position training module is used to build a learning simulation scenario for students based on the teaching content provided by the dynamic teaching strategy generation module, to help students understand and promote their thinking;

[0035] Automated guidance and evaluation module, which is used to record the training process of the situational multi-position training module, evaluate the trainees' operations and give improvement suggestions;

[0036] It should be noted that:

[0037] Visual information acquisition uses high-definition cameras combined with OpenCV and deep learning models (such as CNN and RNN) to capture and analyze students' facial expressions and eye movement characteristics in real time.

[0038] Use posture estimation technology to identify students' gestures and body movements.

[0039] Voice information acquisition:

[0040] Combined with advanced speech recognition algorithms (such as DeepSpeech) and natural language processing technology, the speech content of students can be accurately identified, with a speech recognition accuracy rate of over 98%. By analyzing speech features such as intonation, speaking speed, and pauses, the emotional state of students can be inferred. Emotional state recognition and fusion: Using multimodal deep learning models, visual and speech information are integrated, with an emotion recognition accuracy rate of over 90%, achieving real-time monitoring of students' concentration, confusion, etc.

[0041] The dynamic teaching strategy generation module includes a personalized model building module, a real-time strategy adjustment module and a teaching content generation module; the dynamic teaching strategy generation module is used to provide suitable choices for students and adjust teaching content according to the emotional state of students;

[0042] The personalized model building module is used to build a personalized learning model based on the trainee’s position, historical learning data, and current emotional state;

[0043] A real-time strategy adjustment module, which is used to dynamically adjust the teaching content, difficulty and interaction mode according to the personalized model and the real-time recognized emotional state;

[0044] The teaching content generation module is used to select teaching content suitable for the current student status from the RAG knowledge base according to the reinforcement learning algorithm;

[0045] It should be noted that:

[0046] Adaptive adjustment of dynamic teaching strategies: Collect students’ historical learning data, including learning progress, answer accuracy, interest preferences, etc., and use collaborative filtering and cluster analysis to establish a personalized learning model for students.

[0047] Real-time strategy adjustment: Dynamically adjust the difficulty and presentation of teaching content based on the learner's current emotional state and personalization model. If the virtual teacher detects that the learner is confused, he will slow down the explanation and provide more examples and explanations.

[0048] Adaptive teaching content generation: Using reinforcement learning algorithms, we select the content that best matches the students’ current needs from the RAG knowledge base to ensure the relevance, practicality and challenge of the teaching content;

[0049] The situational multi-position training module includes a multi-role simulation environment module, a cross-role interaction module and a task-driven learning module; the situational multi-position training module is used to construct a learning simulation scenario for trainees, to help trainees understand and promote their thinking;

[0050] Multi-role simulation environment module, used to build simulation scenarios including operators, maintenance engineers, and dispatchers;

[0051] The cross-role interaction module is used to allow students to interact between different roles and understand the collaborative relationship between different positions;

[0052] Task-driven learning module, which is used to set up complex task scenarios to promote students' active thinking and practice;

[0053] Construction of multi-role simulation environment: Use engines such as Unity3D or Unreal Engine to build a realistic power plant simulation environment, including roles such as operators, maintenance engineers, and dispatchers.

[0054] Task-driven learning: Setting up complex task scenarios, such as emergency troubleshooting, requires collaboration among multiple roles. Trainees can experience the responsibilities and workflows of different positions, understand the collaborative relationship between roles, and interact across roles: The system supports real-time communication and collaboration among trainees, improving team awareness and communication skills;

[0055] The automated guidance and evaluation module includes a process data acquisition module, an intelligent evaluation module, and a feedback and improvement suggestion module; the automated guidance and evaluation module is used to record and evaluate the trainees' operations and give improvement suggestions;

[0056] Process data collection module, used to record trainees’ operation behaviors, decision-making processes and interactions in real time;

[0057] Intelligent assessment module, which is used to conduct multi-dimensional assessment of trainees’ performance based on process data using data mining and machine learning techniques;

[0058] Feedback and improvement suggestion module, used to generate personalized evaluation reports and provide targeted improvement suggestions;

[0059] The integration and optimization module includes an architecture design module and a data processing module; the integration and optimization module is used to facilitate module docking and ensure the stable operation of system data;

[0060] Architecture design module for modular and standardized interface design to facilitate integration with existing simulation platforms

[0061] Data processing module, used for distributed computing and memory data management, to ensure the stable operation of the system under large-scale data and high concurrency conditions;

[0062] Process data collection: record detailed data such as each operation step of the trainees, decision time, number of errors, etc. Collect information such as the trainees' movement trajectory and visual focus in the simulation environment.

[0063] Intelligent evaluation model: Use data mining technology to analyze students' operation patterns and behavioral characteristics. Use machine learning algorithms to evaluate students' comprehensive abilities, including professional knowledge, operation skills, and adaptability.

[0064] Feedback and improvement suggestions: Automatically generate evaluation reports containing multi-dimensional information, with data accuracy exceeding 95%, and provide targeted learning suggestions, such as strengthening certain knowledge points, improving emergency response speed, etc.

[0065] Example 2

[0066] like Figures 1 to 3 As shown, the intelligent simulation system based on multimodal interaction and dynamic teaching strategy includes all the contents of Example 1. In addition, the multimodal interaction emotion perception module includes a visual information acquisition module and a voice information acquisition module; the visual information acquisition module is used to capture the facial expressions and eye movements of the trainees through a camera; the voice information acquisition module is used to obtain the voice input of the trainees, including voice content, intonation and speaking speed;

[0067] The dynamic teaching strategy generation module includes a personalized model building module, a real-time strategy adjustment module and a teaching content generation module; the personalized model building module is used to build a personalized learning model according to the trainee's position, historical learning data and current emotional state; the real-time strategy adjustment module is used to dynamically adjust the teaching content, difficulty and interaction mode according to the personalized model and the real-time recognized emotional state; the teaching content generation module is used to select teaching content suitable for the current state of the trainee from the RAG knowledge base according to the reinforcement learning algorithm;

[0068] The scenario-based multi-position training module includes a multi-role simulation environment module, a cross-role interaction module, and a task-driven learning module; the multi-role simulation environment module is used to build a simulation scenario including multiple roles of operators, maintenance engineers, and dispatchers; the cross-role interaction module is used to allow trainees to interact between different roles and understand the collaborative relationship between various positions; the task-driven learning module is used to set up complex task scenarios to promote trainees' active thinking and practice;

[0069] The automated guidance and evaluation module includes a process data collection module, an intelligent evaluation module, and a feedback and improvement suggestion module; the process data collection module is used to record the trainee's operation behavior, decision-making process and interaction in real time; the intelligent evaluation module is used to conduct a multi-dimensional evaluation of the trainee's performance based on the process data using data mining and machine learning techniques; the feedback and improvement suggestion module is used to generate a personalized evaluation report and provide targeted improvement suggestions;

[0070] It also includes an integration and optimization module, including an architecture design module and a data processing module; the integration and optimization module is used to facilitate module docking and ensure the stable operation of system data; the architecture design module is used for modular and standardized interface design to facilitate integration with existing simulation platforms; the data processing module is used for distributed computing and memory data management to ensure the stable operation of the system under large-scale data and high concurrency conditions;

[0071] Real-time application of multimodal emotion perception: Different from the existing technology that only focuses on a single interaction mode, the present invention applies multimodal emotion perception technology to the simulation training system for the first time, realizing real-time monitoring and response of trainees' emotions and attention.

[0072] Adaptive adjustment of dynamic teaching strategies: It breaks through the limitations of traditional fixed teaching strategies and innovatively introduces a dynamic teaching strategy generation method based on the real-time status of students, thereby improving the effectiveness of teaching.

[0073] Situational multi-position cross-role interaction: Different from the prior art which only targets single-position training, the present invention supports situational training and cross-role interaction for multiple positions, thus enhancing the collaboration and comprehensiveness of the training.

[0074] Example 3

[0075] like Figures 1 to 6 As shown, a method for operating an intelligent simulation system based on multimodal interaction and dynamic teaching strategies, in addition, step 1: obtaining visual and voice information of trainees through a multimodal interactive emotion perception module, and identifying emotional states and attention levels in real time;

[0076] Step 2: Based on the students’ personalized model and real-time emotional state, the dynamic teaching strategy generation module is used to dynamically adjust the teaching content and strategy;

[0077] Step 3: In the situational multi-position training module, trainees complete the training tasks through task-driven learning and cross-role interaction;

[0078] Step 4: The automated guidance and multi-dimensional intelligent assessment module records the trainees’ operation behaviors in real time, uses an intelligent assessment model to conduct assessments, and provides feedback and improvement suggestions;

[0079] Integration and optimization modules ensure the coordinated work of various modules to achieve stable operation and efficient maintenance of the system;

[0080] Multi-dimensional intelligent evaluation model: A multi-dimensional evaluation method based on operational behavior, decision-making process and interaction situation is proposed, which is more comprehensive and objective than the traditional result-oriented evaluation.

[0081] Open system architecture and high compatibility: It adopts modular and standardized interface design, can be seamlessly integrated with different simulation platforms and hardware environments, and has high scalability.

Claims

1. An intelligent simulation system based on multimodal interaction and dynamic teaching strategy, characterized by: The system includes a multimodal interactive emotion perception module, a dynamic teaching strategy generation module, a situational multi-position training module, an automated guidance and evaluation module, and an integration and optimization module; The multimodal interactive emotion perception module is used to capture the facial information and voice information of the trainees; Dynamic teaching strategy generation module, used to provide suitable choices for students and adjust teaching content according to the student information obtained by the multimodal interactive emotion perception module; The situational multi-position training module is used to build a learning simulation scenario for students based on the teaching content provided by the dynamic teaching strategy generation module, to help students understand and promote their thinking; The automated guidance and evaluation module is used to record the training process of the situational multi-position training module, evaluate the trainees' operations and provide improvement suggestions.

2. The intelligent simulation system based on multimodal interaction and dynamic teaching strategy according to claim 1 is characterized in that: The multimodal interactive emotion perception module includes a visual information acquisition module and a voice information acquisition module; the visual information acquisition module is used to capture the facial expressions and eye movements of the students through a camera; the voice information acquisition module is used to obtain the students' voice input, including voice content, intonation and speaking speed.

3. The intelligent simulation system based on multimodal interaction and dynamic teaching strategy according to claim 1 is characterized in that: The dynamic teaching strategy generation module includes a personalized model building module, a real-time strategy adjustment module and a teaching content generation module; the personalized model building module is used to build a personalized learning model according to the trainee's position, historical learning data and current emotional state; The real-time strategy adjustment module is used to dynamically adjust the teaching content, difficulty and interaction method according to the personalized model and real-time recognized emotional state; the teaching content generation module is used to select teaching content suitable for the current student status from the RAG knowledge base based on the reinforcement learning algorithm.

4. The intelligent simulation system based on multimodal interaction and dynamic teaching strategy according to claim 1 is characterized in that: The situational multi-position training module includes a multi-role simulation environment module, a cross-role interaction module and a task-driven learning module; Multi-role simulation environment module, used to build simulation scenarios including operators, maintenance engineers, and dispatchers; The cross-role interaction module is used to allow students to interact between different roles and understand the collaborative relationship between different positions; The task-driven learning module is used to set up complex task scenarios to promote students' active thinking and practice.

5. The intelligent simulation system based on multimodal interaction and dynamic teaching strategy according to claim 1 is characterized in that: The automated guidance and evaluation module includes a process data collection module, an intelligent evaluation module, and a feedback and improvement suggestion module; the process data collection module is used to record the trainee's operation behavior, decision-making process and interaction in real time; the intelligent evaluation module is used to conduct a multi-dimensional evaluation of the trainee's performance based on the process data using data mining and machine learning techniques; The feedback and improvement suggestion module is used to generate personalized evaluation reports and provide targeted improvement suggestions.

6. The intelligent simulation system based on multimodal interaction and dynamic teaching strategy according to claim 1 is characterized in that: It also includes an integration and optimization module, including an architecture design module and a data processing module; the integration and optimization module is used to facilitate module docking and ensure the stable operation of system data; the architecture design module is used for modular and standardized interface design to facilitate integration with existing simulation platforms; the data processing module is used for distributed computing and memory data management to ensure the stable operation of the system under large-scale data and high concurrency conditions.

7. A method for operating an intelligent simulation system based on multimodal interaction and dynamic teaching strategy as claimed in any one of claims 1 to 6, characterized in that: The following steps are involved: Step 1: Obtain the learner’s visual and voice information through the multimodal interactive emotion perception module to identify the emotional state and attention level in real time; Step 2: Based on the students’ personalized model and real-time emotional state, the dynamic teaching strategy generation module is used to dynamically adjust the teaching content and strategy; Step 3: In the situational multi-position training module, trainees complete the training tasks through task-driven learning and cross-role interaction; Step 4: The automated guidance and multi-dimensional intelligent assessment module records trainees’ operational behaviors in real time, uses an intelligent assessment model to conduct assessments, and provides feedback and improvement suggestions.

Images