Multi-mode virtual teacher dynamic teaching method based on generative AI

Abstract

The invention discloses a multi-modal virtual teacher dynamic teaching method based on generative AI, and relates to the technical field of virtual teacher dynamic teaching, and the method comprises the steps: employing a data collection and preprocessing method to carry out the standardization processing of learning behavior data from various sources, and obtaining the preprocessed multi-modal data; carrying out fusion processing on the preprocessed multi-modal data by adopting an optimized Transform architecture or other cross-modal fusion technologies, and extracting key features by utilizing a self-attention mechanism to obtain a fused multi-modal data representation; analyzing the fused multi-modal data representation by adopting an emotion calculation model, identifying the emotion state of the student, and outputting emotion classification probability distribution; based on the understanding of the current knowledge level and emotional state of the student, adopting a reinforcement learning algorithm to plan a personalized learning path, and obtaining recommended learning resources and an activity sequence; and generating a customized learning material by adopting a generative adversarial network GANs based on a personalized learning path demand.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of virtual teacher dynamic teaching, and particularly relates to a multi-modal virtual teacher dynamic teaching method based on generative AI. BACKGROUND

[0002] Virtual teacher dynamic teaching technology is a teaching method that combines advanced technologies such as artificial intelligence, machine learning, and cross-modal data analysis. It collects and analyzes student behavior data from multiple channels to understand the current knowledge mastery and emotional state of students. Based on this understanding, the technology can dynamically adjust the teaching content, form, and progress, providing personalized learning paths and customized learning materials for each student.

[0003] In the field of virtual teacher dynamic teaching, traditional methods are difficult to effectively extract features from multiple sources and integrate different types of features for comprehensive analysis, resulting in an incomplete understanding of learner behavior, which affects teaching effectiveness. Moreover, existing teaching methods lack the ability to dynamically adjust learning paths based on students' specific emotional states and knowledge levels, resulting in learning resources and activity sequences that cannot fully meet the needs of each student, reducing learning efficiency and satisfaction. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a multi-modal virtual teacher dynamic teaching method based on generative AI to solve the problem that existing teaching methods lack the ability to dynamically adjust learning paths based on students' specific emotional states and knowledge levels, resulting in learning resources and activity sequences that cannot fully meet the needs of each student, reducing learning efficiency and satisfaction.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a multi-modal virtual teacher dynamic teaching method based on generative AI, which includes: The data collection and preprocessing method is used to standardize the learning behavior data from multiple sources to obtain preprocessed multi-modal data; The optimized Transformer architecture or other cross-modal fusion technology is used to fuse the preprocessed multi-modal data, and the self-attention mechanism is used to extract key features to obtain the fused multi-modal data representation; The emotional computing model is used to analyze the fused multi-modal data representation, identify the emotional state of the student, and output the emotional classification probability distribution; Based on the understanding of the current knowledge level and emotional state of students, a reinforcement learning algorithm is used to plan a personalized learning path, and the recommended learning resources and activity sequence are obtained; Based on the demand of personalized learning path, a generative adversarial network (GAN) is used to generate customized learning materials, and teaching resources suitable for the current learning state are obtained; During the interaction between students and generated teaching materials, the performance and emotional changes are continuously monitored, and a feedback mechanism is used to adjust the teaching strategy and content according to the new data; The effect of the whole teaching process is evaluated regularly, feedback is collected, and the method is iteratively improved, and the updated teaching model and algorithm parameters are obtained; The plurality of sources includes a camera, a microphone, and a keyboard input, and the learning behavior data includes text, image, and audio information.

[0008] As a preferred scheme of the multi-modal virtual teacher dynamic teaching method based on generative AI, wherein: the data collection and preprocessing method is used to standardize the learning behavior data from multiple sources, and the preprocessed multi-modal data is obtained, and the specific steps are: An adaptive filtering algorithm is used to automatically adjust the filtering parameters according to the environmental noise, and the audio data collected from the microphone is de-noised to eliminate background noise and obtain clear voice signals; Gaussian blur algorithm is used to de-noise the image data obtained from the camera to reduce noise interference in the image and obtain clear visual information; Convolutional Neural Network (CNN) is used to extract features from preprocessed images to capture key visual information and obtain image feature vectors; Mel Frequency Cepstral Coefficient (MFCC) is used to extract key features from preprocessed audio data to capture important acoustic characteristics in speech and obtain audio feature vectors; TF-IDF weighted word vector model is used to represent text data to quantify the importance of text and obtain text feature vectors; A comprehensive feature extraction function is introduced to uniformly process multi-modal data, and the expression is: ; Wherein, represents a multi-modal data set, , , represent image, audio and text data respectively, , , is a weight parameter that balances the importance of each modality, , , Respectively refer to the feature extraction method of each field; All extracted features are standardized by Z-score standardization method, so that data from different sources are comparable, and preprocessed multi-modal data are obtained.

[0009] As a preferred scheme of the multi-modal virtual teacher dynamic teaching method based on generative AI, wherein: the multi-modal data after preprocessing are fused by using the optimized Transformer architecture or other cross-modal fusion technology, and the key features are extracted by using the self-attention mechanism to obtain the fused multi-modal data representation, and the specific steps are: The multi-modal data after standardization are input encoded by using the optimized Transformer architecture, and the text, image and audio information are converted into unified vector representation form to obtain the preliminary encoded multi-modal data; The self-attention mechanism is used to extract features from the preliminary encoded multi-modal data to capture the relevance and importance weight of the data, and the expression is: ; Among them, are query, key and value matrices, is the dimension of the key vector; The multi-head self-attention mechanism is used to refine the feature extraction process, and the results of the above self-attention mechanism are independently linearly transformed and combined multiple times to ensure that complex patterns in the input data can be captured from multiple perspectives; Each multi-head attention module operates independently, and the results are finally spliced and integrated through a linear layer; The data processed by the multi-head self-attention mechanism is nonlinearly transformed by using the feedforward neural network, and nonlinear factors are introduced; The data at each position is processed by the same feedforward network, but the parameters are not shared, so that the position-specific information is retained, and the fused multi-modal data representation is obtained.

[0010] As a preferred scheme of the multi-modal virtual teacher dynamic teaching method based on generative AI, wherein: the fused multi-modal data representation is analyzed by using the emotion computing model to identify the emotional state of the student, and the emotional classification probability distribution is output, and the specific steps are: The image part in the fused multi-modal data representation obtained from the Transformer architecture is feature extracted by using the convolutional neural network CNN to capture the key features in the facial expression and obtain the visual information related to the emotion; The recursive neural network RNN and its variant long short-term memory network LSTM are used to process the audio data part, capture the emotional changes in the speech tone, obtain the emotion-related auditory information, and the expression is: ; Among them, represents the hidden state of the time step , and are the weight matrices of the hidden layer to the hidden layer and the input to the hidden layer respectively, is the input of the time step , is the bias term; The pre-trained language model is used to encode the text data part, understand the emotional tendency in the language content, and extract the emotion-related information; The feature vectors of the above three modalities are integrated by using the full connection layer to form a unified emotional feature representation; In the process of integrating the feature vectors, the importance of each modality is adjusted by the weight parameter learned, and the expression is: ; Among them, is the final emotional feature representation, and are the weight matrix and bias term of the full connection layer respectively, indicates that the feature vectors of the three modalities are spliced; The integrated emotional feature representation is classified by using the Softmax classifier, the emotion category is identified, and the probability distribution of the emotion classification is output, and the expression is: ; Among them, is the probability distribution of the emotion classification, and are the weight matrix and bias term of the classification layer respectively.

[0011] As a preferred scheme of the multi-modal virtual teacher dynamic teaching method based on the generative AI, wherein: based on the understanding of the current knowledge level and emotional state of the student, the reinforcement learning algorithm is used to plan the personalized learning path, and the recommended learning resources and activity sequence are obtained, and the specific steps are: The emotion classification probability distribution obtained from the full connection layer is analyzed by using the sentiment analysis module, the current emotional state of the student is identified, and the emotional label of the student is obtained; The learning history data of the student is analyzed by using a knowledge tracking model, and the current knowledge mastery of the student is evaluated, the knowledge tracking model is constructed based on deep knowledge tracking (DKT), and the expression is: ; Among them, represents the performance of the student on the knowledge point at the time step , given all previous interactions and scores , and are weight matrix and bias term respectively, is a hidden state vector; According to the emotional label and knowledge state of the student, a personalized learning path is formulated, and the optimal learning path is selected by simulating different learning paths and evaluating their expected effects; The recommendation mechanism is used to select specific learning resources and activity order according to the personalized learning path, and the abstract learning path is converted into a specific and operable learning plan.

[0012] As a preferred scheme of the multi-modal virtual teacher dynamic teaching method based on generative AI, wherein: based on the personalized learning path requirement, the generative adversarial network (GAN) is used to generate customized learning materials, and the teaching resources suitable for the current learning state are obtained, and the specific steps are: The learning resources and activity order list obtained from the recommendation mechanism are used as input and converted into a form that can be processed by GANs; The generator in the conditional generative adversarial network (CGAN) framework is used to process the above input to generate preliminary customized learning materials; In the CGAN, the generator not only receives random noise as input, but also receives specific conditions to generate content that meets specific requirements; The discriminator is used to evaluate the learning materials generated by the generator to determine whether the authenticity and quality of the materials meet the standard; The generator and the discriminator are optimized by using the adversarial training strategy, so that the generator generates high-quality learning materials, and the discrimination ability of the discriminator is improved; The post-processing module is used to adjust the format and correct the content of the learning materials optimized by the adversarial training, so that the generated content meets the teaching standards and format requirements.

[0013] As a preferred scheme of the multi-modal virtual teacher dynamic teaching method based on generative AI, the performance and emotional changes of the student during the interaction with the generated teaching materials are continuously monitored, and a feedback mechanism is used to adjust the teaching strategy and content according to the new data. The specific steps are as follows: The real-time data acquisition module is used to collect the behavior data of the student when using the generated teaching materials; The data includes the student's correct answer rate, task completion time, facial expression, and voice tone; The emotional state of the student is identified by analyzing the image and audio parts of the standardized real-time behavior data using an affective computing model, and the emotion classification probability distribution is outputted; The emotional state includes confusion, excitement, or fatigue; The knowledge tracking model is used to evaluate the student's learning performance data and update the student's mastery of knowledge points; The adaptive learning path adjustment algorithm dynamically adjusts the personalized learning path according to the student's emotional label and the latest knowledge state representation.

[0014] As a preferred scheme of the multi-modal virtual teacher dynamic teaching method based on generative AI, the effectiveness of the entire teaching process is periodically evaluated, feedback is collected, and the method is iteratively improved to obtain updated teaching models and algorithm parameters. The specific steps are as follows: The performance data of the student during the entire learning period is summarized using the effectiveness evaluation module; The data includes the student's test scores, task completion, participation, and emotional state changes; The effectiveness of the teaching process is evaluated by quantitatively analyzing each index in the comprehensive performance data set using statistical analysis methods; Subjective feedback information is collected from students and teachers through questionnaires or direct interviews to obtain their opinions on teaching content, methods, and interactive experiences; The key factors affecting the teaching effectiveness are identified by fusing and analyzing the comprehensive performance data and subjective feedback data using machine learning models; The existing teaching models and algorithm parameters are adjusted to improve the teaching effectiveness using Bayesian optimization algorithms based on the above analysis results; The optimized teaching models and algorithm parameters are applied to the actual teaching environment through the deployment update mechanism, so that all students can benefit from the latest improvements.

[0015] In a second aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, implements any step of the method for dynamic teaching of a multi-modal virtual teacher based on generative AI according to the first aspect of the present application.

[0016] In a third aspect, the present application provides a computer-readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements any step of the method for dynamic teaching of a multi-modal virtual teacher based on generative AI according to the first aspect of the present application.

[0017] The present application has the beneficial effects that: by using the adaptive filtering algorithm to remove noise from the audio data collected from the microphone, the clear voice signal is obtained, ensuring that the subsequent emotion calculation and knowledge tracking module can accurately analyze based on high-quality audio data, thereby improving the reliability and effectiveness of the overall method, by using the Gaussian blur algorithm to perform noise reduction processing on the image data obtained from the camera, the clear visual information is extracted, improving the accuracy of the emotion calculation model, which helps to more accurately capture the emotional state of the students, providing more accurate data support for personalized learning path planning, by introducing the comprehensive feature extraction function to uniformly process multi-modal data, and using the Z-score standardization method to standardize all extracted features, the consistency of data from different sources is realized, enhancing the effect of cross-modal fusion technology, so that the virtual teacher can fully understand the learning state of the students, and then provide more personalized teaching suggestions. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 The flowchart of the method for dynamic teaching of a multi-modal virtual teacher based on generative AI in embodiment 1. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0021] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be appreciated that the present application can be practiced in a variety of ways beyond the specific details set forth herein, having regard to the contents of the whole patent document, and that the present application can be practiced in other but essentially similar ways. Thus, the present application should not be limited to the specific embodiments described herein, but should be given the widest coverage under the patent statute.

[0022] Secondly, the "one embodiment" or "embodiment" referred to herein means a specific feature, structure or characteristic that can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an embodiment that is independent or alternative to other embodiments.

[0023] Embodiment 1, reference Figure 1 For the first embodiment of the present application, the embodiment provides a multi-modal virtual teacher dynamic teaching method based on generative AI, including the following steps:

[0024] S1, using data collection and preprocessing method to standardize the learning behavior data from multiple sources, and obtaining the preprocessed multi-modal data; Further, an adaptive filtering algorithm (LMS or RLS) is used to automatically adjust the filtering parameters according to the environmental noise, and the audio data collected from the microphone is removed from the noise to eliminate the background noise, and the clear voice signal is obtained; Gaussian blur algorithm is used to reduce noise interference in the image, and clear visual information is obtained; Convolutional Neural Network (CNN) (ResNet or VGG) is used to extract features from preprocessed images to capture key visual information and obtain image feature vectors; Mel Frequency Cepstral Coefficient (MFCC) is used to extract key features from preprocessed audio data to capture important acoustic characteristics in speech and obtain audio feature vectors; TF-IDF weighted word vector model is used to represent text data to quantify the importance of text and obtain text feature vectors; A comprehensive feature extraction function is introduced to uniformly process multi-modal data , the expression is: ; Among them, represents a multi-modal data set, , , , respectively represent image, audio and text data, , , is a weight parameter that balances the importance of each modality, , , respectively refer to the feature extraction method of each field; Z-score standardization method is adopted to standardize all extracted features, so that data from different sources are comparable, and pre-processed multi-modal data are obtained; It should be noted that by adopting adaptive filtering algorithm and Gaussian blur algorithm to pre-process audio and image data respectively, not only the accuracy of subsequent emotion calculation and knowledge tracking module is improved, but also the data from different sources can be compared in the same dimension, which lays a foundation for cross-modal fusion. In addition, the introduction of comprehensive feature extraction function and Z-score standardization method enables the method to process multiple types of data and ensures the consistency and comparability of these data, thereby improving the robustness and flexibility of the entire method.

[0025] S2, the optimized Transformer architecture or other cross-modal fusion technology is adopted to fuse the pre-processed multi-modal data, and the self-attention mechanism is used to extract key features, so as to obtain the fused multi-modal data representation; Further, the optimized Transformer architecture is adopted to input and encode the standardized multi-modal data, and the text, image and audio information are converted into unified vector representation form, so as to obtain the preliminary encoded multi-modal data; The self-attention mechanism is adopted to extract features from the preliminary encoded multi-modal data to capture the relevance and importance weight of the data, and the expression is: ; Among them, are query, key and value matrices respectively, is the dimension of the key vector; The multi-head self-attention mechanism is adopted to refine the feature extraction process, and by performing multiple independent linear transformations on the results of the above self-attention mechanism and combining them, it is ensured that complex patterns in the input data can be captured from multiple perspectives; Each multi-head attention module runs independently, and finally the results are spliced and integrated through a linear layer; The feedforward neural network is adopted to perform nonlinear transformation on the data processed by the multi-head self-attention mechanism, and nonlinear factors are introduced; The data of each position is processed through the same feedforward network, but the parameters are not shared, so as to retain the position-specific information, and the fused multi-modal data representation is obtained; It should be noted that the optimized Transformer architecture and its self-attention mechanism not only effectively convert text, image and audio information into unified vector representation, but also capture the complex relationship between different modalities, enhancing the depth and breadth of feature extraction. The multi-head self-attention mechanism further refines this process, ensuring that the method can comprehensively understand the key features in the input data from multiple perspectives, providing more rich information support for subsequent sentiment analysis and personalized learning path planning.

[0026] S3, using a sentiment computing model to analyze the fused multi-modal data representation, identifying the emotional state of the student, and outputting the emotional classification probability distribution; Furthermore, a convolutional neural network (CNN) is used to extract features from the image part of the fused multi-modal data representation obtained from the Transformer architecture, capturing key features in facial expressions and obtaining visual information related to emotions. A recurrent neural network (RNN) and its variant, long short-term memory network (LSTM), are used to process the audio data part, capturing emotional changes in speech tone (fundamental frequency, energy change and speech rate), obtaining auditory information related to emotions, and the expression is: ; where, represents the hidden state of time step , and are the weight matrices of the hidden layer to the hidden layer and the input to the hidden layer, is the input of time step , is the bias term; A pre-trained language model (BERT or RoBERTa) is used to encode the text data part, understand the sentiment tendency (positive, negative and neutral) in the language content, and extract information related to emotions; A fully connected layer is used to integrate the feature vectors of the above three modalities to form a unified emotional feature representation; During the integration of the feature vectors, the importance of each modality is adjusted by the learned weight parameters, and the expression is: ; where, is the final emotional feature representation, and are the weight matrix and bias term of the fully connected layer, represents concatenating the feature vectors of the three modalities; The Softmax classifier is used to classify the integrated emotional feature representation, identify the emotional categories (confused, excited, tired, and focused), and output the probability distribution of the emotional classification, which is expressed as: ; in, is the probability distribution of emotion classification, and are the weight matrix and bias term of the classification layer respectively; The emotional state recognition results are compared and verified with standard emotion databases (FERA, RAVDESS and SEMAINE) to ensure that the model has high accuracy and generalization ability; It should be noted that by combining CNN, RNN / LSTM and pre-trained language models, the method can comprehensively capture students' emotional states from three dimensions: vision, hearing and text, forming a comprehensive emotional feature representation. This method not only improves the accuracy of emotion recognition, but also more carefully reflects the emotional changes of students during the learning process, helping virtual teachers to adjust teaching strategies according to students' emotional dynamics, thereby improving the effectiveness of interaction and student participation.

[0027] S4. Based on the understanding of students’ current knowledge level and emotional state, a reinforcement learning algorithm is used to plan personalized learning paths and obtain recommended learning resources and activity sequences; Furthermore, the sentiment analysis module is used to analyze the emotion classification probability distribution obtained from the fully connected layer to identify the student’s current emotional state and obtain the student’s emotion label; The emotional labels include confused, focused, and exhausted; The emotional state is continuously predicted by collecting data (facial expressions, voice intonation and answering speed) and compared with historical emotional trends to determine whether it is in an abnormal state; The knowledge tracking model DKT is used to analyze students’ learning history data and evaluate their current knowledge mastery. The knowledge tracking model is constructed based on deep knowledge tracking DKT and is expressed as: ; in, Indicates that at time step , given all previous interactions and grades , predict students’ knowledge points On performance, and are the weight matrix and bias term respectively, is the hidden state vector, and the knowledge state is represented as a vector, where each dimension corresponds to the mastery probability of a knowledge point; According to the emotional label and knowledge state representation of the student, a personalized learning path is formulated, and the optimal learning path is selected by simulating different learning paths and evaluating their expected effects; The learning path is defined as the arrangement and combination of knowledge points and corresponding teaching resources (videos, exercises, and explanations), and the path effect is quantitatively evaluated by the predicted mastery improvement speed and emotional stability; A recommendation mechanism is used to select specific learning resources and activity sequences according to the personalized learning path, and to convert the abstract learning path into a specific and operable learning plan; It should be noted that the method of formulating a personalized learning path based on emotional labels and knowledge state representations fully considers the student's learning progress and emotional state, ensuring the flexibility and adaptability of teaching strategies. By simulating different learning paths and evaluating their expected effects, the method can select the optimal path to maximize student learning efficiency, not only improving the effectiveness of teaching, but also enhancing the learning experience of students and promoting their continuous progress and development.

[0028] S5, based on the demand of personalized learning path, using generative adversarial network GANs to generate customized learning materials, get the teaching resources suitable for the current learning state; Further, the list of learning resources and activity sequences obtained from the recommendation mechanism is used as input and converted into a form that GANs can handle (keywords, difficulty levels, and knowledge point IDs); The input condition vector includes the student's current knowledge mastery level, emotional state, and preferred style (text, video, and interactive exercises); The generator in the conditional generative adversarial network framework CGANs is used to process the above input to generate preliminary customized learning materials (questions, explanation texts, and animation scripts); In the CGANs, the generator not only receives random noise as input, but also receives specific conditions to generate content that meets specific needs; The discriminator is used to evaluate the learning materials generated by the generator to determine whether the authenticity and quality of the materials meet the standards; An adversarial training strategy is used to optimize the generator and discriminator, so that the generator generates high-quality learning materials while improving the discriminator's ability to distinguish; A post-processing module is used to adjust the format and correct the content of the learning materials optimized by the adversarial training, so that the generated content meets the teaching standards and format requirements; It should be noted that the use of the CGANs framework enables the method to generate high-quality and targeted teaching resources based on specific learning needs. By continuously optimizing the generator and discriminator through adversarial training strategies, the method can generate content that meets teaching standards while ensuring its authenticity and quality. The post-processing module further ensures that the format and content of the generated content meet actual teaching requirements, thereby providing students with an efficient and personalized learning environment.

[0029] S6. Continuously monitor students’ performance and emotional changes during their interaction with the generated teaching materials, and use feedback mechanisms to adjust teaching strategies and content based on new data; Furthermore, a real-time data acquisition module is used to collect students’ behavioral data when using the generated teaching materials; The data includes students’ accuracy in answering questions, time to complete tasks, facial expressions, and voice intonation; The learning performance data refers to quantifiable indicators generated by students during the learning process, including accuracy rate of answers, reaction time, number of repeated wrong answers, and task completion time; The affective computing model is used to analyze the image and audio parts of the standardized real-time behavioral data, identify the student's current emotional state, and output the emotion classification probability distribution; Said emotional states include confusion, excitement, or exhaustion; Use the knowledge tracking model to evaluate students' learning performance data and update their understanding of knowledge points; The knowledge state update mechanism is based on Bayesian reasoning and dynamically adjusts the probability of mastering knowledge points by combining new and old data; Adopting an adaptive learning path adjustment algorithm to dynamically adjust the personalized learning path based on the student's emotional tags and the latest knowledge status; If the emotional state shows "confused" and the knowledge point mastery rate is below the threshold, basic explanations are recommended first; if the emotional state shows "focused" and the mastery rate is high, advanced exercises are recommended; It should be noted that the combination of the real-time data acquisition module and the emotional computing model enables the method to instantly obtain students' behavior and emotional state, ensuring that the teaching strategy can be dynamically adjusted according to the latest data. The continuous monitoring and feedback mechanism not only improves the response speed of the method, but also enhances the adaptability of personalized learning paths, helping students maintain the best state during the learning process, thereby improving learning outcomes.

[0030] S7. Regularly evaluate the effectiveness of the entire teaching process, collect feedback and iteratively improve the method to obtain updated teaching models and algorithm parameters; Furthermore, an effectiveness evaluation module is used to summarize students’ performance data throughout the entire learning cycle; The data includes students' test scores, task completion, participation, and emotional state changes. Statistical analysis methods are used to quantitatively analyze each indicator in the comprehensive performance dataset, evaluating the effectiveness of the teaching process. Subjective feedback information is collected from students and teachers through questionnaires or direct interviews, obtaining their views on teaching content, methods, and interactive experiences. Machine learning models are used to analyze the comprehensive performance data and subjective feedback data, identifying key factors affecting teaching effectiveness. Bayesian optimization algorithms are used to adjust existing teaching models and algorithm parameters based on the above analysis results to improve teaching effectiveness. A deployment update mechanism is used to apply the optimized teaching models and algorithm parameters to the actual teaching environment, so that all students can benefit from the latest improvements. It should be noted that the effectiveness of the teaching process is regularly evaluated, and iterative improvements are made based on statistical analysis results and subjective feedback information. This not only helps identify key factors affecting teaching effectiveness, but also continuously optimizes teaching models and algorithm parameters. The method ensures that the system can continuously improve itself to adapt to changing teaching needs, ultimately providing more high-quality and effective teaching services to all students.

[0031] The embodiment also provides a computer device suitable for the case of the multi-modal virtual teacher dynamic teaching method based on generative AI, which comprises a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the multi-modal virtual teacher dynamic teaching method based on generative AI proposed in the above embodiment.

[0032] The computer device can be a terminal, which comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through Wi-Fi, operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device. It can also be an external keyboard, touchpad or mouse, etc.

[0033] The embodiment also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the method for implementing a multi-modal virtual teacher dynamic teaching method based on a generative AI proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0034] To sum up, by using the adaptive filtering algorithm to remove noise from the audio data collected by the microphone, the clear speech signal is obtained, ensuring that the subsequent emotion calculation and knowledge tracking module can accurately analyze based on high-quality audio data, thereby improving the reliability and effectiveness of the overall system. By using the Gaussian blur algorithm to perform noise reduction processing on the image data obtained from the camera, clear visual information is extracted, improving the accuracy of the emotion calculation model, which helps to more accurately capture the emotional state of the student, providing more accurate data support for personalized learning path planning. By introducing a comprehensive feature extraction function to uniformly process multi-modal data, and using the Z-score standardization method to standardize all extracted features, the consistency of data from different sources is achieved, enhancing the effect of cross-modal fusion technology, so that the virtual teacher can fully understand the learning state of the student, and then provide more personalized teaching suggestions.

[0035] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, which should be covered by the scope of the claims of the present application.

Claims

1. A multimodal virtual teacher dynamic teaching method based on generative AI, characterized by: include: Data collection and preprocessing methods are used to standardize learning behavior data from various sources to obtain preprocessed multimodal data; Use the optimized Transformer architecture or other cross-modal fusion technologies to fuse the pre-processed multimodal data, use the self-attention mechanism to extract key features, and obtain the fused multimodal data representation; The emotional computing model is used to analyze the fused multimodal data representation, identify the students' emotional state, and output the emotion classification probability distribution; Based on the understanding of students' current knowledge level and emotional state, reinforcement learning algorithms are used to plan personalized learning paths and obtain recommended learning resources and activity sequences; Based on the needs of personalized learning paths, we use generative adversarial networks (GANs) to generate customized learning materials and obtain teaching resources suitable for the current learning status. As students interact with the generated teaching materials, their performance and emotional changes are continuously monitored, and feedback mechanisms are used to adjust teaching strategies and content based on new data; Regularly evaluate the effectiveness of the entire teaching process, collect feedback and iteratively improve the method to obtain updated teaching models and algorithm parameters; The multiple sources include cameras, microphones, and keyboard inputs, and the learning behavior data includes text, images, and audio information.

2. The multimodal virtual teacher dynamic teaching method based on generative AI according to claim 1, characterized in that: The data collection and preprocessing method is used to standardize learning behavior data from multiple sources to obtain preprocessed multimodal data. The specific steps are as follows: Adopting adaptive filtering algorithm, the filter parameters are automatically adjusted according to the ambient noise, and the audio data collected from the microphone is de-noised to eliminate background noise and obtain clear speech signals; The Gaussian blur algorithm is used to perform noise reduction on the image data obtained from the camera to reduce noise interference in the image and obtain clear visual information; Convolutional neural network (CNN) is used to extract features from preprocessed images, capture key visual information, and obtain image feature vectors. Mel-frequency cepstral coefficients (MFCCs) are used to extract key features from preprocessed audio data, capture important acoustic characteristics in speech, and obtain audio feature vectors. The TF-IDF weighted word vector model is used to represent text data to quantify the importance of the text and obtain the text feature vector; A comprehensive feature extraction function is introduced to uniformly process multimodal data. The expression is: ； in, represents a multimodal dataset, , , Represent image, audio and text data respectively, , , is a weight parameter that balances the importance of each mode, , , They refer to the feature extraction methods in their respective fields; The Z-score standardization method is used to standardize all the extracted features to make the data from different sources comparable and obtain preprocessed multimodal data.

3. The multimodal virtual teacher dynamic teaching method based on generative AI according to claim 2, characterized in that: The optimized Transformer architecture or other cross-modal fusion technologies are used to fuse the pre-processed multimodal data, and the self-attention mechanism is used to extract key features to obtain the fused multimodal data representation. The specific steps are as follows: The optimized Transformer architecture is used to encode the standardized multimodal data input, converting text, image, and audio information into a unified vector representation to obtain preliminarily encoded multimodal data. The self-attention mechanism is used to extract features from the preliminarily encoded multimodal data to capture the internal relevance and importance weight of the data. The expression is: ； in, are query, key, and value matrices respectively, is the dimension of the key vector; A multi-head self-attention mechanism is used to refine the feature extraction process. By applying multiple independent linear transformations to the results of the self-attention mechanism and combining them, it ensures that complex patterns in the input data can be captured from multiple perspectives. Each multi-head attention module runs independently, and finally the results are concatenated and integrated through a linear layer; A feedforward neural network is used to perform nonlinear transformation on the data processed by the multi-head self-attention mechanism, and nonlinear factors are introduced; The data at each position passes through the same feedforward network, but the parameters are not shared, thereby retaining position-specific information and obtaining a fused multimodal data representation.

4. The multimodal virtual teacher dynamic teaching method based on generative AI according to claim 3, characterized in that: The emotional computing model is used to analyze the fused multimodal data representation, identify the student's emotional state, and output the emotion classification probability distribution. The specific steps are: A convolutional neural network (CNN) is used to extract features from the image portion of the fused multimodal data representation obtained from the Transformer architecture, capturing key features in facial expressions and obtaining visual information related to emotions. The recursive neural network (RNN) and its variant, the long short-term memory (LSTM) network, are used to process the audio data, capture the emotional changes in the voice intonation, and obtain auditory information related to emotions. The expression is: ; in, Represents the time step The hidden state of and are the weight matrices from hidden layer to hidden layer and from input to hidden layer, is the time step Input, is the bias term; Use a pre-trained language model to encode text data, understand the emotional tendencies in the language content, and extract emotion-related information; A fully connected layer is used to integrate the feature vectors of the above three modalities to form a unified emotional feature representation; During the integration of the feature vectors, the importance of each mode is adjusted by the learned weight parameter, which is expressed as: ； in, is the final emotional feature representation, and are the weight matrix and bias term of the fully connected layer, It means concatenating the eigenvectors of the three modes; The Softmax classifier is used to classify the integrated emotional feature representation, identify the emotional category, and output the probability distribution of the emotional classification, which is expressed as: ; in, is the probability distribution of emotion classification, and are the weight matrix and bias term of the classification layer respectively.

5. The multimodal virtual teacher dynamic teaching method based on generative AI according to claim 4, characterized in that: Based on the understanding of students' current knowledge level and emotional state, a reinforcement learning algorithm is used to plan personalized learning paths and obtain recommended learning resources and activity sequences. The specific steps are as follows: The sentiment analysis module is used to analyze the emotion classification probability distribution obtained from the fully connected layer to identify the student's current emotional state and obtain the student's emotional label; The knowledge tracking model is used to analyze students’ learning history data and evaluate their current knowledge mastery. The knowledge tracking model is constructed based on deep knowledge tracking (DKT) and is expressed as: ; in, Indicates that at time step , given all previous interactions and grades , predict students’ knowledge points On performance, and are the weight matrix and bias term respectively, is the hidden state vector; Develop personalized learning paths based on students' emotional labels and knowledge status, and select the optimal learning path by simulating different learning paths and evaluating their expected effects; A recommendation mechanism is used to select specific learning resources and activity sequences based on personalized learning paths, and to transform abstract learning paths into specific and actionable learning plans.

6. The multimodal virtual teacher dynamic teaching method based on generative AI according to claim 5, characterized in that: Based on the needs of personalized learning paths, the generative adversarial network (GANs) is used to generate customized learning materials to obtain teaching resources suitable for the current learning status. The specific steps are as follows: It takes the sequential list of learned resources and activities obtained from the recommendation mechanism as input and transforms it into a form that GANs can process; The generator in the conditional generative adversarial network framework CGANs is used to process the above input to generate preliminary customized learning materials; In the CGANs described above, the generator receives not only random noise as input, but also specific conditions to generate content that meets specific requirements; A discriminator is used to evaluate the learning materials generated by the generator to determine whether the authenticity and quality of the materials meet the standards; Adopting adversarial training strategies to optimize the generator and discriminator, enabling the generator to generate high-quality learning materials while improving the discriminator’s ability to distinguish between different classes. The post-processing module is used to adjust the format and correct the content of the learning materials optimized by adversarial training, so as to make the generated content meet the teaching standards and format requirements.

7. The multimodal virtual teacher dynamic teaching method based on generative AI according to claim 6, characterized in that: During the interaction between students and the generated teaching materials, the performance and emotional changes of students are continuously monitored, and the teaching strategy and content are adjusted according to the new data using a feedback mechanism. The specific steps are as follows: A real-time data acquisition module is used to collect students’ behavioral data when using the generated teaching materials; The data includes students’ accuracy in answering questions, time to complete tasks, facial expressions, and voice intonation; The affective computing model is used to analyze the image and audio parts of the standardized real-time behavioral data, identify the student's current emotional state, and output the emotion classification probability distribution; Said emotional states include confusion, excitement, or exhaustion; Use the knowledge tracking model to evaluate students' learning performance data and update their understanding of knowledge points; An adaptive learning path adjustment algorithm is used to dynamically adjust the personalized learning path based on the students' emotional labels and the latest knowledge status representation.

8. The multimodal virtual teacher dynamic teaching method based on generative AI according to claim 7, characterized in that: The effectiveness of the entire teaching process is regularly evaluated, feedback is collected, and the method is iteratively improved to obtain an updated teaching model and algorithm parameters. The specific steps are as follows: Use the effectiveness evaluation module to summarize students' performance data throughout the learning cycle; The data includes students’ test scores, task completion, engagement, and changes in emotional state; Statistical analysis methods were used to quantitatively analyze the various indicators in the comprehensive performance data set to evaluate the effectiveness of the teaching process; Use questionnaires or direct interviews to collect subjective feedback from students and teachers to obtain their views on teaching content, methods, and interactive experiences; Use machine learning models to integrate and analyze comprehensive performance data and subjective feedback data to identify key factors affecting teaching effectiveness; The Bayesian optimization algorithm is used to adjust the existing teaching model and algorithm parameters according to the above analysis results to improve the teaching effect; A deployment update mechanism is used to apply the optimized teaching models and algorithm parameters to the actual teaching environment, so that all students can benefit from the latest improvements.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the generative AI-based multimodal virtual teacher dynamic teaching method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the generative AI-based multimodal virtual teacher dynamic teaching method according to any one of claims 1 to 7 are implemented.

Images