Teaching intervention method, system and device based on multi-modal intelligent analysis

Abstract

The application provides a teaching intervention method, system and device based on multi-modal intelligent analysis, and relates to the technical field of intelligent education. The method comprises the following steps: collecting multi-modal learning data of a learner, extracting each modal learning feature, and inferring the current learning state of the learner; if the current learning state is abnormal, at least one main factor causing the abnormality is screened out based on the causal relationship between a plurality of preset candidate factors and the abnormal learning state, and a corresponding first set of intervenable variables is determined; the causal relationship is obtained based on learning portrait analysis including learning state history records; an intelligent agent is called, and a teaching plan and teaching intention for the current learning state are generated based on an intelligent agent data set including the learning portrait, the main factors, the first set of intervenable variables and the teaching task of the current round; and a digital person is driven to execute the teaching plan and the teaching intention. The application can comprehensively perceive the state of the learner, and realize collaborative optimization of teaching strategies and teaching interactive behaviors.

Description

Technical Field

[0001] This invention relates to the field of intelligent education technology, and in particular to a teaching intervention method, system and device based on multimodal intelligent analysis. Background Technology

[0002] The development of artificial intelligence technology has driven the evolution of education towards personalization. Personalized education and intelligent tutoring systems can provide customized learning support based on the individual differences of learners, thereby improving teaching effectiveness, and have become an important direction for educational technology research.

[0003] Existing technologies include various personalized education assistance methods. For example, rule-based expert systems generate recommended learning information based on preset knowledge point dependencies; collaborative filtering-based educational resource recommendation systems predict learner needs by analyzing the behavioral patterns of similar learners; and knowledge graph-based tutoring systems provide learning guidance to learners by constructing a subject knowledge system.

[0004] However, the aforementioned existing technologies have significant shortcomings: rule-based learning information is difficult to adapt to the dynamic changes of learners; educational resources recommended by collaborative filtering algorithms may be of insufficient quality due to differences among learners; and learning guidance provided by knowledge graphs cannot perceive the learner's status in real time. Furthermore, these solutions fail to integrate learning status identification, instructional strategy generation, and instructional interaction, and fail to dynamically adjust instructional interventions based on the learner's actual difficulties, making it difficult to achieve efficient personalized educational assistance.

[0005] Therefore, there is an urgent need for an educational intervention method that can comprehensively perceive learners' status and achieve synergistic optimization of teaching strategies and teaching interaction behaviors, so as to improve the accuracy of personalized teaching and the adaptability of teaching interaction. Summary of the Invention

[0006] This invention provides a teaching intervention method, system, and device based on multimodal intelligent analysis to improve the accuracy of personalized teaching and the adaptability of teaching interaction.

[0007] In a first aspect, embodiments of the present invention provide a teaching intervention method based on multimodal intelligent analysis, comprising: Collect learners' multimodal learning data and extract learning features for each modality; infer the learner's current learning state based on the learning features for each modality; When the current learning state becomes abnormal, based on the causal relationship between multiple preset candidate factors and the abnormal learning state, at least one main factor causing the current learning state to become abnormal is selected, and a first set of interventionable variables corresponding to each main factor is determined; the causal relationship is obtained based on learning profile analysis; the learning profile includes the learning state history record. The intelligent agent is invoked to generate a teaching plan and teaching intention for the current learning state based on the intelligent agent dataset; wherein, the intelligent agent dataset includes the learning profile, the main factors, the first set of interventionable variables, and the teaching task for this round; Drive the digital human to execute the teaching plan and the teaching intent.

[0008] In one possible implementation, the causal relationship includes the correlation between each candidate factor and different abnormal learning states; the step of filtering out at least one main factor causing the current learning state to become abnormal based on the causal relationship between multiple preset candidate factors and abnormal learning states, and determining a first set of interventionable variables corresponding to each main factor, includes: The correlation between each candidate factor and the current learning state is determined based on the causal relationship. The at least one candidate factor with the highest correlation to the current learning state is identified as the primary factor; For each of the main factors, the first set of interventionable variables corresponding to the main factor is extracted from a pre-defined library of interventionable variables.

[0009] In one possible implementation, the method for determining the correlation between each candidate factor and different abnormal learning states includes: Based on the learning status history and historical teaching records of the learning profile, the frequency distribution of learning status before intervention and the frequency distribution of learning status after intervention for each interventionable variable are statistically analyzed; wherein, the historical teaching records include the interventionable variables executed in historical intervention events; For each candidate factor, perform the following steps: Obtain the second set of interventionable variables corresponding to the candidate factor; Based on the frequency distribution of learning states before intervention and the frequency distribution of learning states after intervention of each of the interventionable variables, the causal effect values ​​between the interventionable variables in the second set of interventionable variables and different abnormal learning states are calculated, and the correlation between the candidate factor and different abnormal learning states is determined based on the causal effect values.

[0010] In one possible implementation, inferring the learner's current learning state based on the learning features of each modality includes: Evaluate the confidence level of the learned features for each modality; Each modal learning feature is input into the corresponding learning state inference model to obtain the predicted probability of each modal learning feature for different learning states; Based on the confidence level of each modal learning feature, the prediction probabilities of each modal learning feature for different learning states are weighted and fused to obtain the fused prediction probabilities of different learning states. The current learning state is determined based on the predicted probabilities of the different learning states after fusion.

[0011] In one possible implementation, evaluating the confidence level of each modality-learned feature includes: For each modality's learned features, perform the following steps: The quality score of the modality learning feature is determined based on the effective data ratio and signal-to-noise ratio of the modality learning feature; The historical consistency score of the modality learning feature is determined based on the similarity between the modality learning feature and its historical values. Based on the preset confidence fusion weights, the weighted sum of the quality score and the historical consistency score is calculated to obtain the confidence of the modality learning feature.

[0012] In one possible implementation, the intelligent agent includes a diagnostic intelligent agent, a policy intelligent agent, and an expression planning intelligent agent; the teaching plan includes a sequence of teaching actions, and the teaching intention includes teaching performance parameters corresponding to the teaching action sequence; the invocation of the intelligent agent, based on the intelligent agent dataset, generates a teaching plan and teaching intention for the current learning state, including: Based on the aforementioned agent dataset, the diagnostic agent is invoked to execute the diagnostic verification sub-task extracted from the current teaching task, the policy agent is invoked to execute the policy formulation sub-task extracted from the current teaching task, and the expression planning agent is invoked to execute the expression planning sub-task extracted from the current teaching task. The diagnostic verification subtask includes generating verification questions and verifying the key factors; the strategy formulation subtask includes generating the instructional action sequence based on the verified key factors and the first set of interventionable variables; and the expression planning subtask includes generating the instructional performance parameters.

[0013] In one possible implementation, the intelligent agent further includes a content-generating intelligent agent and an evaluation intelligent agent; the invoking intelligent agent, based on the intelligent agent dataset, generates a teaching plan and teaching intent for the current learning state, further including: Based on the aforementioned agent dataset, the content generation agent is invoked to execute a content generation sub-task extracted from the current teaching task, and the evaluation agent is invoked to execute an evaluation design sub-task extracted from the current teaching task. The content generation sub-task includes generating explanation content and practice questions corresponding to the teaching action sequence; the evaluation design sub-task includes determining teaching evaluation points and evaluation criteria for the learner's feedback data.

[0014] In one possible implementation, after the digital human executes the instructional plan and the instructional intent, the method further includes: Collect feedback data from the learners; evaluate the quality of completion of the current teaching task based on the feedback data and the evaluation criteria, and update the historical teaching records.

[0015] Secondly, embodiments of the present invention provide a teaching intervention system based on multimodal intelligent analysis, comprising: The feature extraction module is used to collect learners' multimodal learning data, extract learning features for each modality, and infer the learner's current learning state based on the learning features for each modality. The factor analysis module is used to, when the current learning state becomes abnormal, filter out at least one main factor that causes the current learning state to become abnormal based on the causal relationship between multiple preset candidate factors and the abnormal learning state, and determine the first set of interventionable variables corresponding to each main factor; the causal relationship is obtained based on learning profile analysis; the learning profile includes the learning state history record; The strategy generation module is used to invoke the intelligent agent to generate a teaching plan and teaching intent for the current learning state based on the intelligent agent dataset; wherein, the intelligent agent dataset includes the learning profile, the main factors, the first set of interventionable variables, and the teaching task for this round. An interactive execution module is used to drive the digital human to execute the teaching plan and the teaching intent.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0017] In this embodiment of the invention, multimodal learning data of learners is collected and learning features of each modality are extracted, enabling comprehensive perception and quantitative evaluation of learners' multidimensional learning behaviors, effectively improving the reliability and anti-interference ability of learning state identification; based on the learning features of each modality, the current learning state and the learner's learning profile are inferred, enabling continuous tracking of the trajectory of changes in the learning state and providing time-series data support for personalized teaching decisions; when the current learning state is abnormal, based on the causal relationship between multiple preset candidate factors and changes in the learning state, the main factors and the first set of interventionable variables corresponding to each main factor are determined, realizing causal reasoning and factor localization of learning difficulties, transforming ambiguous learning problems into operable intervention variables, significantly improving the pertinence and effectiveness of teaching assistance; through the intelligent agent, a teaching plan including teaching planning and teaching intentions is generated, ensuring the verifiability, traceability and logical consistency of the teaching decision-making process; driving the digital human to execute the teaching plan and teaching intentions forms a full-process teaching system of perception, decision-making and execution, realizing real-time dynamic adjustment of teaching assistance schemes, and continuously improving the adaptability and effectiveness of personalized teaching.

[0018] The embodiments of the present invention realize intelligentization of the entire process from accurate identification of learning status to dynamic generation of teaching strategies and execution of teaching behaviors, thereby improving the accuracy of diagnosis of difficulties and problems, the pertinence of teaching strategies and the adaptability of teaching interaction in personalized education systems. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the implementation of the teaching intervention method based on multimodal intelligent analysis provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the structure of the teaching intervention system based on multimodal intelligent analysis provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] See Figure 1 This invention provides a teaching intervention method based on multimodal intelligent analysis, detailed below: Step S101: Collect learners' multimodal learning data and extract learning features for each modality; infer the learners' current learning status based on the learning features for each modality.

[0022] Specifically, collecting learners' multimodal learning data includes: at the start of learning, acquiring the current teaching task and constraints, generating a session identifier and a unified time reference for the current learning process; continuously collecting learners' video data, audio data, handwriting data, and input text data, as well as a log of the answering process during the learning process. Teaching constraints may include teaching duration, knowledge scope, difficulty level, or expression style constraints; audio data can be collected by a microphone and encoded into an audio stream at a preset sampling rate (e.g., 16kHz), while recording the start and end times of the audio; video data can be collected by a camera, with the collection area limited to the learner's face and upper body, at a frame rate of 30 frames per second; handwriting data can be collected by a touchscreen or electronic pen device, recording the learner's pen tip coordinates, pressure value, and time information; input text data can be collected by a text editor, recording input content, editing operation type, content accuracy, and corresponding time information; the log of the answering process can be generated by an event monitoring device, recording operation events such as opening, submitting, modifying, and returning questions, along with their time information. Optionally, learners' screen interaction data can also be collected, recording behaviors such as clicking, swiping, and switching within the operating system. All collected data is associated with time information and session identifiers for subsequent learning status analysis and historical teaching record tracing.

[0023] Optionally, the multimodal learning data is also preprocessed, including time alignment of each modality of the learning data.

[0024] Specifically, extracting learning features for each modality can include: estimating the gaze direction vector from video data using facial key points and head posture, and projecting the gaze direction vector onto the coordinate system of the teaching screen plane to obtain the position of the gaze point. When the gaze point is located within the attention area corresponding to the question stem or writing area in multiple consecutive video frames, a video modality feature indicating that the gaze is focused on the question stem area is generated; performing frame-by-frame processing on speech data, calculating the pause duration between speech segments, speech rate changes, and pitch stability, and generating a speech modality feature indicating speech hesitation if the pause duration is significantly higher than the learner's historical average and the pitch is lower; and analyzing continuous writing time, pause duration between strokes, number of pauses, and number of revisions on writing data. If the pause duration between strokes exceeds a preset threshold and the number of revisions is higher than the historical average within a time window containing key problem-solving steps, a writing modality feature indicating increased pauses and revisions in key steps can be generated.

[0025] In one possible implementation, extracting the learning features for each modality may include the following detailed steps: Face detection and key point localization are performed on the acquired video data. Specifically, a face detection model based on convolutional neural networks is used to locate the face region in each frame of video image, and a key point detection network is used to extract the coordinates of 68 key points of the face, including the center points of the eyes, the tip of the nose, and the corners of the mouth. Calculate the gaze direction vector based on the center points of both eyes and the tip of the nose: ,in Let's say we have the coordinates of the tip of the nose. The coordinates of the center points of both eyes; Calculate the angle of change of the gaze direction vector in consecutive video frames: When θ is greater than the preset threshold of 15°, it is determined as a gaze shift event. Video modal features of the gaze being focused on the question stem area are generated, and the number of gaze shifts within a unit time window is counted.

[0026] The acquired speech data undergoes noise reduction and speech endpoint detection processing. Specifically, a noise suppression algorithm based on spectral subtraction is used to reduce the noise in the speech data, and a short-time energy threshold-based endpoint detection method is used to extract valid speech segments. A short-time Fourier transform is performed on each valid speech segment to extract the fundamental frequency. And calculate the fundamental frequency fluctuation coefficient: ,in The standard deviation of the fundamental frequency of speech. It represents the average value of the fundamental frequency of speech, and generates speech modal features based on the pause duration of effective speech segments and the fundamental frequency fluctuation coefficient.

[0027] The collected writing data is analyzed, and the stroke speed is calculated based on the handwriting coordinate sequence: The proportion of stroke pause time within a unit time window is statistically analyzed to generate writing modal features.

[0028] In one possible implementation, the learner's current learning state is inferred based on the learning features of each modality, including: Evaluate the confidence level of the learned features for each modality; Each modality learning feature is input into the corresponding learning state inference model to obtain the prediction probability of each modality learning feature for different learning states; Based on the confidence of each modality learning feature, the prediction probabilities of each modality learning feature for different learning states are weighted and fused to obtain the fused prediction probabilities of different learning states. The current learning state is determined based on the predicted probabilities of the different learning states after fusion.

[0029] Specifically, learning status can include the learner's knowledge mastery status and non-cognitive status. The knowledge mastery status reflects the learner's level of understanding and application of knowledge. The non-cognitive status can include attention status, confusion status, emotional status, or fatigue status, which reflects the learner's level of mental adaptation.

[0030] The learning state inference model can be a machine learning model trained using historical data of any modality learning features and corresponding learning state labels, used to generate predicted probabilities for different learning states based on the learning features of that modality.

[0031] In one possible implementation, inferring the learner's current learning state based on the learning features of each modality can include the following detailed steps: Evaluate the confidence level of the learned features for each modality; The learning features of each modality are input into the corresponding learning state inference model. The learning state inference model can be a multilayer perceptron classification network, and its output is the predicted probability distribution of the learning features of modality i for different learning states. ; The predicted probabilities of different learning states for each modality's learning features are weighted and fused to obtain the fused predicted probabilities of different learning states: ,in Let be the predicted probability distribution of the learning features of modality i for different learning states. Let i be the modal weights. , Let i be the confidence level of mode i; For example, when video modal features indicate that the gaze has not deviated, and writing modal features and speech modal features reflect pauses in thinking rather than interruptions in writing, after fusing the predicted probabilities of different learning states from the learning features of each modality, a probability distribution of learning states with higher probabilities of confusion and uncertainty and lower probabilities of distraction is obtained. The current learning state is determined based on the probability distribution of the fused learning states. When the probability value of the learning state with the highest probability is greater than the preset learning state filtering threshold, the corresponding learning state is output as the current learning state. Specifically, the learning state filtering threshold can be 0.6, so that when the maximum probability value of the learning state is greater than 0.6, the learning state with the highest probability is adopted as the current learning state.

[0032] Furthermore, the learning state with the highest probability in the fused learning state probability distribution can be selected as the current learning state.

[0033] In one possible implementation, the confidence level of each modality-learned feature is evaluated, including: For each modality's learned features, perform the following steps: The quality score of the modality learning feature is determined based on the effective data ratio and signal-to-noise ratio of the modality learning feature; The historical consistency score of the modality learning feature is determined based on the similarity between the modality learning feature and its historical values. Based on the preset confidence fusion weights, the weighted sum of the quality score and the historical consistency score is calculated to obtain the confidence of the modality learning feature.

[0034] For example, when the confidence level of a certain modality learning feature is lower than a preset threshold, the learning feature can be temporarily blocked to avoid low-quality data interfering with the judgment of the learning state.

[0035] Specifically, based on the confidence level of the learned features of evaluation mode i It can include: confidence level of mode i ;in , The confidence level is preset based on experience and the fusion weights are used; for example, =0.7, =0.3.

[0036] in Let i be the mass fraction of mode i. ×Integrity Indicators+ ×Signal-to-noise ratio index+ × Stability metrics; Completeness metrics are the percentage of effective data for the learned features of modality i. For example, for video modalities, the percentage of effective data can be the ratio of the number of effective frames to the total number of frames; Signal-to-noise ratio metrics are the data signal-to-noise ratio of modality i; Stability metrics are the data variance or standard deviation of modality i. , , The preset quality normalization weights can reflect the different levels of importance of different indicators to the quality score. For example, =0.4, =0.3, =0.3.

[0037] in The historical consistency score of mode i is specifically the current feature of mode i. Similarity to the feature mean vector within the historical window.

[0038] In this embodiment of the invention, deep fusion and accurate state inference of multimodal learning data are achieved: First, by constructing a unified time benchmark and collecting multimodal data such as video, voice, writing, text, and operation logs, a comprehensive and three-dimensional perception of the learning process is realized, overcoming the one-sidedness and limitations of traditional single-modal analysis; Second, a confidence assessment mechanism is introduced, comprehensively considering the quality scores and historical consistency of each modality's data, effectively filtering out interference from low-quality or abnormal data on state judgment, and significantly improving the accuracy and robustness of learning state inference; Finally, a weighted fusion strategy is adopted to organically integrate the predicted probabilities of each modality's features, which can not only accurately identify the learner's knowledge mastery status, but also capture subtle changes in non-cognitive states such as attention, confusion, emotion, and fatigue in real time, providing a reliable state basis for personalized teaching intervention.

[0039] Step S102: When the current learning state is abnormal, based on the causal relationship between multiple preset candidate factors and the abnormal learning state, at least one main factor that causes the current learning state to be abnormal is selected, and the first set of interventionable variables corresponding to each main factor is determined; the causal relationship is obtained based on the learning profile analysis; the learning profile includes the learning state history record.

[0040] In one possible implementation, the causal relationship includes the correlation between each candidate factor and different anomalous learning states; based on the causal relationship between multiple pre-defined candidate factors and anomalous learning states, at least one main factor causing the current learning state to become anomalous is selected, and a first set of interventionable variables corresponding to each main factor is determined, including: The correlation between each candidate factor and the current learning state is determined based on causal relationships; Identify at least one candidate factor that has the strongest correlation with the current learning state as the primary factor; For each major factor, a first set of interventionable variables corresponding to the major factor is extracted from a pre-defined pool of interventionable variables.

[0041] In one possible implementation, the method for determining the correlation between each candidate factor and different abnormal learning states includes: Based on the learning profile's historical learning status history and historical teaching records, the frequency distribution of learning status before intervention and the frequency distribution of learning status after intervention for each modifiable variable were statistically analyzed; among them, the historical teaching records include the modifiable variables executed in historical intervention events. For each candidate factor, perform the following steps: Obtain the second set of interventionable variables corresponding to the candidate factor; Based on the frequency distribution of learning states before intervention and the frequency distribution of learning states after intervention for each interventionable variable, the causal effect values ​​between the interventionable variables in the second set of interventionable variables and different abnormal learning states are calculated, and the degree of association between the candidate factor and different abnormal learning states is determined based on the causal effect values.

[0042] For example, when calculating the causal effect values ​​of different modifiable variables, we comprehensively consider the degree of matching between the modifiable variable and the trend of state trajectory changes before and after the occurrence of abnormal learning states, the performance of the modifiable variable in the corresponding intervention effects in similar historical scenarios, and the support and stability of the current multimodal learning features for the modifiable variable. When the analysis shows that the causal effect value of the modifiable variable corresponding to the factor of "missing steps and strategies" is higher than that of "missing concepts," "missing steps and strategies" is identified as the main factor.

[0043] Specifically, the multiple preset candidate factors, the first set of interventionable variables, and the second set of interventionable variables are all data pre-built and stored in the knowledge base by the system. Their specific content is configured based on educational theories, expert experience, and historical data analysis.

[0044] Among them, candidate factors refer to a set of common reasons that may lead to abnormal learning states, which may include conceptual gaps, lack of step-by-step strategies, misinterpretation of questions, transfer failures, fluctuations in attention, and emotional influences.

[0045] Both the first and second sets of interventionable variables are pre-defined sets of multiple interventionable variables corresponding to candidate factors. They represent the complete set of all possible teaching interventions for that candidate factor. For example, for the candidate factor of "distraction," the corresponding second set of interventionable variables might include various intervention strategies such as "inserting interactive Q&A," "switching teaching media formats," and "adjusting the pace of content."

[0046] In one possible implementation, when the current learning state becomes abnormal, it can be: the current learning state is detected to conform to any of the preset abnormal learning states; For example, interventionist variables may include prompt intensity, explanation granularity, example type, exercise difficulty, pacing control, feedback tone, review interval, and follow-up questioning style. Different primary factors correspond to different priorities of interventionist variables. Specifically: when the primary factor is a lack of concept, priority is given to increasing explanation granularity, supplementing basic examples, and reducing transfer difficulty; when the primary factor is a lack of step-by-step strategies, priority is given to increasing step-by-step follow-up questions, prompting key steps, and comparative questions; when the primary factor is fluctuations in attention or emotional influence, priority is given to adjusting pacing control, feedback tone, and expression style, without immediately changing the knowledge content. Furthermore, priority, direction of action, and expected impact descriptions can be assigned to interventionist variables, forming the decision-making basis for the teaching support intervention process.

[0047] In one possible implementation, based on the frequency distribution of learning states before intervention and the frequency distribution of learning states after intervention for each interventionable variable, the causal effect values ​​between the interventionable variables in the second set of interventionable variables and different abnormal learning states are calculated, and the correlation between the candidate factor and different abnormal learning states is determined based on the causal effect values, including the following detailed steps: A causal graph of learning states is constructed based on the learner's historical learning state records. The nodes of the causal graph include the knowledge mastery, attention level, cognitive load, and response performance of each record point in the historical learning state records. Based on the frequency distribution of learning states in different nodes of the causal graph, and the correspondence between each node and historical intervention records, the causal effect values ​​of each modifiable variable on different abnormal learning states are calculated: , where X is an operable variable and Y is an abnormal learning state; The frequency distribution of Y before intervention, The frequency distribution of Y after X intervention; The correlation between the candidate factor and each abnormal learning state is determined by the average of the causal effect values ​​between all the modifiable variables in the second set of modifiable variables corresponding to the candidate factor and each abnormal learning state.

[0048] In this embodiment of the invention, intelligent decision-making from abnormal learning states to precise intervention strategies is realized: First, by constructing a causal relationship model based on historical data of learning profiles, the correlation between candidate factors and abnormal learning states is quantitatively analyzed, realizing the transformation from traditional experience-based judgment to data-driven causal reasoning, significantly improving the objectivity and accuracy of identifying factors leading to abnormalities; Second, a multi-dimensional comprehensive calculation method for causal effect values ​​is adopted to ensure the scientificity and reliability of the selection of main factors; Finally, through a structured mapping between a pre-set candidate factor library and an interventionist variable library, combined with real-time causal analysis results, the main factors are quickly located and the corresponding first set of interventionist variables is extracted, realizing automation from problem diagnosis to intervention strategy generation, providing a precise decision-making basis for personalized teaching intervention.

[0049] In one possible implementation, updating the learning profile can include writing the current learning state into the learning profile, or comparing and analyzing it with historical learning state records to form a learning state trajectory, including knowledge mastery, error type distribution, intervention history, intervention effects, and state change trends. The learning state trajectory also records the sequence of changes in the learner's learning state over time, used to identify trends in learning state changes, trigger intervention behaviors, and evaluate intervention effects. For example, by analyzing a learner's historical answers to similar questions and their learning state trajectory, it can be found that the learner has a high accuracy rate on questions related to basic function concepts, but fluctuates significantly in the problem-solving steps involving "extracting key conditions from the analytical expression and transforming them into a function graph." This analysis result is recorded in the learning state trajectory of the learning profile.

[0050] In one possible implementation, the process of updating the learning profile includes the following detailed steps: Learning status includes knowledge mastery status and non-cognitive status. Knowledge mastery status can include knowledge mastery degree, problem-solving strategy stability, and knowledge transfer ability. Knowledge mastery degree can be updated using a knowledge tracking model, and the update method is as follows: ,in As the learner's current level of knowledge mastery, The current response result is given, and α is the preset learning rate. A learning state trajectory is constructed based on a continuous sequence of learning states to describe the trend of learning state changes over time.

[0051] Specifically, the learning profile is obtained through continuous updates to the learning status; the mastery of knowledge points is statistically estimated through a knowledge tracking model based on the answer results, error type distribution, and status improvement; the error type distribution is calculated by aggregating error labels from the input text data; historical teaching records include executed teaching actions and corresponding time points; the intervention effect is obtained by analyzing changes in learning status and feedback data before and after the intervention; the trend of status change is obtained through time series analysis of the status trajectory; the learning profile is updated with each time window, forming a status trajectory that reflects the evolution of the learning status.

[0052] In this embodiment of the invention, by comparing and analyzing the current learning state with historical records, a multi-dimensional learning state trajectory is constructed, including knowledge mastery, error type distribution, intervention history, intervention effect, and state change trend. This achieves a panoramic tracking and quantitative description of the learner's cognitive development process, overcoming the limitation of traditional learning record updates that only record discrete state points. Secondly, the knowledge mastery is dynamically updated, providing a scientific basis for pre-intervention. Finally, the learning state trajectory not only records the learner's performance fluctuations on specific knowledge points, but also quantitatively evaluates the intervention effect by comparing the state before and after intervention, forming a closed-loop feedback mechanism of "state monitoring - trajectory analysis - effect evaluation," which significantly enhances the practical value of learning profiles in personalized teaching decisions.

[0053] Step S103: Invoke the intelligent agent to generate a teaching plan and teaching intention for the current learning state based on the intelligent agent dataset; wherein, the intelligent agent dataset includes a learning profile, main factors, a first set of interventionable variables, and the teaching task for this round.

[0054] In one possible implementation, the intelligent agent includes a diagnostic intelligent agent, a policy intelligent agent, and an expression planning intelligent agent; the instructional plan includes a sequence of instructional actions, and the instructional intent includes instructional performance parameters corresponding to the sequence of instructional actions; the intelligent agent is invoked to generate an instructional plan and instructional intent for the current learning state based on the intelligent agent dataset, including: Based on the agent dataset, the diagnostic agent is invoked to execute the diagnostic verification sub-task extracted from the current teaching task, the policy agent is invoked to execute the policy formulation sub-task extracted from the current teaching task, and the expression planning agent is invoked to execute the expression planning sub-task extracted from the current teaching task. The diagnostic validation subtask includes generating confirmatory questions and validating key factors; the strategy formulation subtask includes generating a sequence of instructional actions based on the validated key factors and the first set of interventionable variables; and the expressive planning subtask includes generating instructional performance parameters.

[0055] In one possible implementation, the intelligent agent also includes a content-generating intelligent agent and an evaluation intelligent agent; invoking the intelligent agent to generate a teaching plan and teaching intentions for the current learning state based on the intelligent agent dataset also includes: Based on the agent dataset, the content generation agent is invoked to execute the content generation sub-task extracted from the current teaching task, and the evaluation agent is invoked to execute the evaluation design sub-task extracted from the current teaching task. The content generation sub-task includes generating the explanation content and practice questions corresponding to the teaching action sequence; the evaluation design sub-task includes determining the teaching evaluation points and the evaluation criteria for the learner's feedback data.

[0056] For example, the agent dataset also includes the current learning state, learning state trajectory, candidate factors, teaching constraints, and quality assessment results corresponding to the latest feedback data. Each agent reads input based on this agent dataset and writes the output results back to the agent dataset in a structured form to ensure information consistency among multiple agents.

[0057] In one possible implementation, the diagnostic agent and the policy agent first generate results in parallel, and then verify the validation questions generated by the diagnostic agent and the instructional action sequences generated by the policy agent. If the instructional action sequences given by the policy agent satisfy the validation questions generated by the diagnostic agent, the content generation and expression planning stage begins; otherwise, an inconsistency is triggered, and a factor verification task is initiated. The factor verification task can be a round of follow-up questions, a quiz, a key-step judgment question, or a partial transfer task, used to further differentiate factors at a lower cost. The subsequent instructional action sequences are selected or modified based on the results of the factor verification task. After completing the factor verification task, the instructional plan and instructional intent are output. The instructional plan may include the instructional action sequence, the execution order of each instructional action, the corresponding instructional objectives, assessment points, and backtracking rules; the instructional intent may include the instructional behavior type, expression strategy, target focus, and intensity level corresponding to each instructional action.

[0058] In one possible implementation, multiple agents are invoked to generate teaching plans and teaching intentions based on the agent dataset, including the following detailed steps: Based on the first set of interventionable variables Construct a teaching task request object TaskRequest. TaskRequest includes the target variable identifier, the current learning state St, the learning state trajectory Ht, and the teaching constraints C. The teaching constraints may include time constraints and cognitive load thresholds.

[0059] Constructing an expert intelligent agent set The system comprises three agent sets: Adiag (diagnostic agent) generates confirmatory questions to validate key factors; Astrategy (strategy agent) generates instructional action sequences; Generation (content generation agent) generates explanation content and / or practice questions corresponding to the instructional action sequences; and Aeval (evaluation agent) determines evaluation points and criteria for feedback data. The expert agent set also includes an expression planning agent that generates instructional performance parameters corresponding to the instructional action sequences. Each agent receives input data and outputs structured results through a unified state representation interface.

[0060] The strategy agent Astrategy performs a strategy template matching and parameterization generation process for each operable variable vi in ​​the first set of operable variables V, generating a structured teaching strategy Strategyi. Specifically, this may include: A teaching strategy template library, TemplateLib, is constructed. Each teaching strategy template is defined as Template=(TriggerCondition, ActionSequence, ParamSchema), where TriggerCondition represents the applicable conditions, ActionSequence represents the preset teaching behavior sequence in the template, and ParamSchema represents the parameter structure definition. Based on the modifiable variable vi and the main factors, a matching teaching strategy template Templatei is selected from the template library; Based on ParamSchema, the execution parameters are populated to generate a specific teaching strategy: Strategyyi=(ActionSequence, ParamSet), where ParamSet includes the following parameters: content parameters, including knowledge point identifiers and explanation depth levels; performance parameters, including speaking speed, pause intervals, and emphasis weights; interaction parameters, including question frequency, prompt intensity, and feedback delay; and difficulty parameters, including question difficulty coefficients and step breakdown granularity. Multiple teaching strategies (Strategy) are sorted by influence_weight to generate a candidate teaching strategy set (StrategySet). The process of generating teaching strategies must meet the following constraints: each teaching strategy is generated based on a one-to-one mapping of the modifiable variables; each teaching strategy contains two components, ActionSequence and ParamSet, neither of which can be omitted; the ranking of teaching strategies must be calculated based on the causal effect value influence_weight.

[0061] In one possible implementation, the candidate instructional strategy set (StrategySet) is transformed into an executable instructional plan (Plan). Specifically, this includes: Construct a task graph structure TaskGraph, where each node represents a teaching action Actionj, and the edges represent the execution order and dependencies of the teaching actions; Define a teaching action node Actionj=(action_type, content_id, duration, control_param), where action_type represents the type of teaching action, content_id represents the corresponding knowledge point or question identifier, duration represents the execution duration, and control_param represents the performance control parameters; Based on ActionSequence, each teaching strategy in the candidate teaching strategy set is expanded into a task graph substructure and then merged. Conflict detection and constraint verification are performed on the task graph, including time constraint verification, cognitive load threshold verification, and content redundancy verification; the final teaching plan Plan=(TaskGraph, Schedule, ControlParamSet) is determined and integrated to obtain the fused teaching action sequence actions[], which is used as the teaching plan output.

[0062] For example, a teaching plan may include actions[] (a sequence of teaching actions, such as explanation / questioning / example / practice / summary); it may also include goals (the teaching task for this round, such as mastering a knowledge point, correcting a mistake, etc.), checkpoints[] (teaching assessment points, including assessment time, assessment content, assessment criteria), constraints (teaching constraints, such as total duration, difficulty limit, compliant language, prohibited strategies, etc.), and fallback rules (failure fallback rules, such as switching the teaching plan if two assessments fail, etc.).

[0063] For example, teaching intent can include pedagogy_intent (teaching behavior performance parameters, such as follow-up questioning / correction / inspiration / summarization, etc.), expression_intent (expression strategy performance parameters, such as slowing down / emphasizing / gentle correction / encouragement, etc.), intensity (intent intensity performance parameters, such as weak / medium / strong or 0–1), and target_focus (guiding focus performance parameters, such as keywords in the question stem / key points in the steps / formulas, etc.).

[0064] In this embodiment of the invention, by breaking down teaching tasks into sub-tasks such as diagnostic verification, strategy formulation, content generation, evaluation design, and expression planning, and scheduling multiple expert agents to work collaboratively based on a unified agent dataset, teaching decisions are transformed from fuzzy recommendations to interpretable and traceable structured generation. This effectively solves the problems of opaque strategies and lack of reasoning verification caused by single-model decision-making. At the same time, by constructing a teaching plan that includes teaching action sequences, evaluation points, constraints, and backtracking rules, as well as teaching intentions that include teaching behavior types, expression strategies, target focus, and intensity levels, and by introducing factor verification tasks, the teaching strategy generation process has causal basis, verifiability, and dynamic adaptability, thereby improving the decision-making credibility and teaching execution effectiveness of the personalized education system.

[0065] Step S104: Drive the digital human to execute the teaching plan and teaching intention.

[0066] In one possible implementation, driving the digital human to execute instructional plans and intentions involves mapping the instructional plans and intentions to a multi-channel execution parameter instruction set for the digital human, and then driving the digital human to perform corresponding instructional behaviors. Execution parameters may include the digital human's speech prosody parameters, facial expression parameters, gaze parameters, gesture / posture parameters, and lip-sync parameters. The execution parameters of the digital human are dynamically adjusted according to the instructional intention Intent. For example, in a scenario where the main factor is "missing step-by-step strategies," when a "verifying follow-up question" appears in the instructional action sequence, the Intent is configured for "heuristic follow-up questioning," and the digital human adopts a slower speech rate, questioning tone, forward-leaning posture, and guiding gestures; when a "demonstration of key steps" appears in the instructional action sequence, the Intent is configured for "emphasis," and the digital human adopts a slower speech rate, stronger tone, gestures pointing to key content, and focused facial expressions.

[0067] In one possible implementation, after driving the digital human to execute instructional plans and intentions, the method also includes: Collect learner feedback data; evaluate the quality of completion of the current round of teaching tasks based on the feedback data and assessment criteria, and update historical teaching records.

[0068] For example, learner feedback data is collected, the quality of the current teaching task is evaluated based on the feedback data, and historical teaching records are updated. This includes triggering follow-up questions, quizzes, or transfer questions at the assessment points set in the teaching plan, while continuously collecting multimodal learning data from learners to evaluate the quality of the current teaching task in real time. When the multimodal learning data indicates that learners mainly experience confusion, decreased attention, or emotional fluctuations but have not yet shown a clear knowledge gap, the next teaching task is prioritized for expression-level intervention (adjusting speaking speed, pauses, facial expression intensity, error correction intensity, etc.); when the multimodal learning data indicates that learners have not yet mastered the relevant knowledge or steps, the next teaching task is prioritized for content-level intervention (adjusting examples, explanation granularity, exercise difficulty, etc.), and a corresponding teaching plan is regenerated. The real-time feedback multimodal learning data is compared and analyzed with the multimodal learning data before the implementation of the current teaching plan and intentions. Specifically, paired t-tests can be used to calculate the significance of the differences before and after the data to analyze the quality of the current teaching task. The quality indicators of the current teaching task can include one of the following: changes in accuracy, changes in reaction time, changes in repeated error rate, and changes in the probability of confused learning state. When an increase in the accuracy of variation questions, a decrease in pauses in writing key steps, and a decrease in the probability of confused learning state are detected, the quality of the current teaching task is judged to be good. The evidence units, factor judgments, teaching strategies, digital human execution parameters, and quality assessment results in the current teaching process are structured and stored, and historical teaching records are updated for reuse and strategy optimization in subsequent teaching interventions.

[0069] In this embodiment of the invention, by accurately mapping teaching intentions to multi-channel execution parameter instructions of digital humans (including speech prosody, facial expressions, gaze, gestures / postures, and lip-sync parameters), a seamless transformation from abstract intentions to concrete expressions is achieved in teaching intervention. This makes the teaching behavior of digital humans interpretable, consistent, and context-adaptable, effectively solving the problems of rigid expression and distorted intention transmission in traditional digital human teaching. At the same time, a real-time quality assessment mechanism based on feedback data is constructed. The teaching effect can be scientifically quantified through statistical methods such as paired t-tests. Based on the feedback data, expression-level intervention and content-level intervention are intelligently distinguished, forming a closed-loop teaching system of "execution-feedback-evaluation-optimization," which improves the teaching accuracy and continuous optimization capability of the intelligent education system.

[0070] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0071] The following are system embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0072] Figure 2 A schematic diagram of the teaching intervention system based on multimodal intelligent analysis provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: like Figure 2 As shown, the teaching intervention system 2 based on multimodal intelligent analysis includes: Feature extraction module 21 is used to collect learners' multimodal learning data, extract learning features of each modality, and infer the learner's current learning state based on the learning features of each modality. The factor analysis module 22 is used to, when the current learning state is abnormal, filter out at least one main factor that causes the current learning state to be abnormal based on the causal relationship between multiple preset candidate factors and the abnormal learning state, and determine the first set of interventionable variables corresponding to each main factor; the causal relationship is obtained based on the learning profile analysis; the learning profile includes the learning state history record. The strategy generation module 23 is used to invoke the agent to generate a teaching plan and teaching intention based on the agent dataset for the current learning state; wherein, the agent dataset includes the learning profile, key factors, the first set of interventionable variables, and the teaching task for this round. Interactive execution module 24 is used to drive the digital human to execute teaching plans and teaching intentions.

[0073] In one possible implementation, the factor analysis module 22 is used to determine the degree of correlation between each candidate factor and the current learning state based on causal relationships; Identify at least one candidate factor that has the strongest correlation with the current learning state as the primary factor; For each major factor, a first set of interventionable variables corresponding to the major factor is extracted from a pre-defined pool of interventionable variables.

[0074] In one possible implementation, the factor analysis module 22 is also used to statistically analyze the frequency distribution of learning states before intervention and the frequency distribution of learning states after intervention for each interventionable variable based on the learning state history record and historical teaching record of the learning profile; wherein, the historical teaching record includes the interventionable variables executed in the historical intervention event; For each candidate factor, perform the following steps: Obtain the second set of interventionable variables corresponding to the candidate factor; Based on the frequency distribution of learning states before intervention and the frequency distribution of learning states after intervention for each interventionable variable, the causal effect values ​​between the interventionable variables in the second set of interventionable variables and different abnormal learning states are calculated, and the degree of association between the candidate factor and different abnormal learning states is determined based on the causal effect values.

[0075] In one possible implementation, the feature extraction module 21 is used to evaluate the confidence level of the learned features for each modality; Each modality learning feature is input into the corresponding learning state inference model to obtain the prediction probability of each modality learning feature for different learning states; Based on the confidence of each modality learning feature, the prediction probabilities of each modality learning feature for different learning states are weighted and fused to obtain the fused prediction probabilities of different learning states. The current learning state is determined based on the predicted probabilities of the different learning states after fusion.

[0076] In one possible implementation, the feature extraction module 21 is further configured to learn features for each modality and perform the following steps: The quality score of the modality learning feature is determined based on the effective data ratio and signal-to-noise ratio of the modality learning feature; The historical consistency score of the modality learning feature is determined based on the similarity between the modality learning feature and its historical values. Based on the preset confidence fusion weights, the weighted sum of the quality score and the historical consistency score is calculated to obtain the confidence of the modality learning feature.

[0077] In one possible implementation, the policy generation module 23 is used to, based on the agent dataset, call the diagnostic agent to execute the diagnostic verification sub-task extracted from the current teaching task, call the policy agent to execute the policy formulation sub-task extracted from the current teaching task, and call the expression planning agent to execute the expression planning sub-task extracted from the current teaching task. The diagnostic validation subtask includes generating confirmatory questions and validating key factors; the strategy formulation subtask includes generating a sequence of instructional actions based on the validated key factors and the first set of interventionable variables; and the expressive planning subtask includes generating instructional performance parameters.

[0078] In one possible implementation, the policy generation module 23 is further configured to, based on the agent dataset, invoke the content generation agent to execute the content generation sub-task extracted from the current teaching task, and invoke the evaluation agent to execute the evaluation design sub-task extracted from the current teaching task; wherein, the content generation sub-task includes generating the explanation content and practice questions corresponding to the teaching action sequence; the evaluation design sub-task includes determining the teaching evaluation points and the evaluation criteria for the learner's feedback data.

[0079] In this embodiment of the invention, a complete closed-loop system from multimodal data collection to personalized teaching intervention is constructed through the organic synergy of four modules: feature extraction, factor analysis, strategy generation, and interactive execution. This system achieves intelligent processing of the entire process, including data perception, state inference, factor localization, strategy generation, precise execution, and effect evaluation. It effectively solves the technical problems of poor data correlation, ambiguous decision-making, and extensive intervention in traditional educational support solutions. Through innovative mechanisms such as confidence-weighted multimodal state fusion, factor screening based on causal effect values, structured teaching planning generation through multi-agent collaboration, and multi-channel parameterized execution by digital humans, the system transforms teaching intervention from experience-driven to data-driven, from single-point recommendation to system decision-making, and from static content to dynamic adaptation. This significantly improves the accuracy, interpretability, and teaching effectiveness of the intelligent education system.

[0080] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 3 As shown, the electronic device 3 of this embodiment includes a processor 30 and a memory 31. The memory 31 stores a computer program 32. When the processor 30 executes the computer program 32, it implements the steps in the various method embodiments described above. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the various device embodiments described above.

[0081] For example, computer program 32 may be divided into one or more modules / units, which are stored in memory 31 and executed by processor 30 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 32 in electronic device 3.

[0082] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 3 may also include input / output devices, network access devices, buses, etc.

[0083] The processor 30 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0084] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store the computer program 32 and other programs and data required by the electronic device 3. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0085] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0086] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0087] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0088] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0089] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A teaching intervention method based on multimodal intelligent analysis, characterized in that, include: Collect learners' multimodal learning data and extract learning features for each modality; The learner's current learning state is inferred based on the learning features of each modality; When the current learning state becomes abnormal, based on the causal relationship between multiple preset candidate factors and the abnormal learning state, at least one main factor causing the current learning state to become abnormal is selected, and a first set of interventionable variables corresponding to each main factor is determined; the causal relationship is obtained based on learning profile analysis; the learning profile includes the learning state history record. The intelligent agent is invoked to generate a teaching plan and teaching intention for the current learning state based on the intelligent agent dataset; wherein, the intelligent agent dataset includes the learning profile, the main factors, the first set of interventionable variables, and the teaching task for this round; Drive the digital human to execute the teaching plan and the teaching intent.

2. The teaching intervention method based on multimodal intelligent analysis according to claim 1, characterized in that, The causal relationship includes the correlation between each candidate factor and different abnormal learning states; based on the causal relationship between multiple preset candidate factors and abnormal learning states, at least one main factor causing the current learning state to become abnormal is selected, and a first set of interventionable variables corresponding to each main factor is determined, including: The correlation between each candidate factor and the current learning state is determined based on the causal relationship. The at least one candidate factor with the highest correlation to the current learning state is identified as the primary factor; For each of the main factors, the first set of interventionable variables corresponding to the main factor is extracted from a pre-defined library of interventionable variables.

3. The teaching intervention method based on multimodal intelligent analysis according to claim 2, characterized in that, The methods for determining the correlation between each candidate factor and different abnormal learning states include: Based on the learning status history and historical teaching records of the learning profile, the frequency distribution of learning status before intervention and the frequency distribution of learning status after intervention for each interventionable variable are statistically analyzed; wherein, the historical teaching records include the interventionable variables executed in historical intervention events; For each candidate factor, perform the following steps: Obtain the second set of interventionable variables corresponding to the candidate factor; Based on the frequency distribution of learning states before intervention and the frequency distribution of learning states after intervention of each of the interventionable variables, the causal effect values ​​between the interventionable variables in the second set of interventionable variables and different abnormal learning states are calculated, and the correlation between the candidate factor and different abnormal learning states is determined based on the causal effect values.

4. The teaching intervention method based on multimodal intelligent analysis according to claim 1, characterized in that, The step of inferring the learner's current learning state based on the learning features of each modality includes: Evaluate the confidence level of the learned features for each modality; Each modal learning feature is input into the corresponding learning state inference model to obtain the predicted probability of each modal learning feature for different learning states; Based on the confidence level of each modal learning feature, the prediction probabilities of each modal learning feature for different learning states are weighted and fused to obtain the fused prediction probabilities of different learning states. The current learning state is determined based on the predicted probabilities of the different learning states after fusion.

5. The teaching intervention method based on multimodal intelligent analysis according to claim 4, characterized in that, The evaluation of the confidence level of each modality-learned feature includes: For each modality's learned features, perform the following steps: The quality score of the modality learning feature is determined based on the effective data ratio and signal-to-noise ratio of the modality learning feature; The historical consistency score of the modality learning feature is determined based on the similarity between the modality learning feature and its historical values. Based on the preset confidence fusion weights, the weighted sum of the quality score and the historical consistency score is calculated to obtain the confidence of the modality learning feature.

6. The teaching intervention method based on multimodal intelligent analysis according to claim 1, characterized in that, The intelligent agent includes a diagnostic intelligent agent, a strategy intelligent agent, and an expression planning intelligent agent; the teaching plan includes a sequence of teaching actions, and the teaching intention includes teaching performance parameters corresponding to the sequence of teaching actions. The invoking agent generates a teaching plan and teaching intent for the current learning state based on the agent dataset, including: Based on the aforementioned agent dataset, the diagnostic agent is invoked to execute the diagnostic verification sub-task extracted from the current teaching task, the policy agent is invoked to execute the policy formulation sub-task extracted from the current teaching task, and the expression planning agent is invoked to execute the expression planning sub-task extracted from the current teaching task. The diagnostic verification subtask includes generating verification questions and verifying the key factors; the strategy formulation subtask includes generating the instructional action sequence based on the verified key factors and the first set of interventionable variables; and the expression planning subtask includes generating the instructional performance parameters.

7. The teaching intervention method based on multimodal intelligent analysis according to claim 6, characterized in that, The intelligent agent also includes a content generation intelligent agent and an evaluation intelligent agent; The invocation of the intelligent agent, based on the intelligent agent dataset, to generate a teaching plan and teaching intent for the current learning state, further includes: Based on the aforementioned agent dataset, the content generation agent is invoked to execute a content generation sub-task extracted from the current teaching task, and the evaluation agent is invoked to execute an evaluation design sub-task extracted from the current teaching task. The content generation sub-task includes generating explanation content and practice questions corresponding to the teaching action sequence; the evaluation design sub-task includes determining teaching evaluation points and evaluation criteria for the learner's feedback data.

8. The teaching intervention method based on multimodal intelligent analysis according to claim 7, characterized in that, After the digital human is driven to execute the teaching plan and the teaching intent, the method further includes: Collect feedback data from the learners; evaluate the quality of completion of the current teaching task based on the feedback data and the evaluation criteria, and update the historical teaching records.

9. A teaching intervention system based on multimodal intelligent analysis, characterized in that, include: The feature extraction module is used to collect learners' multimodal learning data and extract learning features for each modality. The learner's current learning state is inferred based on the learning features of each modality; The factor analysis module is used to, when the current learning state becomes abnormal, filter out at least one main factor that causes the current learning state to become abnormal based on the causal relationship between multiple preset candidate factors and the abnormal learning state, and determine the first set of interventionable variables corresponding to each main factor; the causal relationship is obtained based on learning profile analysis; the learning profile includes the learning state history record; The strategy generation module is used to invoke the intelligent agent to generate a teaching plan and teaching intent for the current learning state based on the intelligent agent dataset; wherein, the intelligent agent dataset includes the learning profile, the main factors, the first set of interventionable variables, and the teaching task for this round. An interactive execution module is used to drive the digital human to execute the teaching plan and the teaching intent.

10. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 8.

Images