Adaptive teaching adjustment method, system and device based on sentiment analysis

Abstract

This application provides an adaptive teaching adjustment method, system, and device based on sentiment analysis, relating to the field of intelligent education. It solves the technical problem that existing technologies cannot effectively integrate multimodal sentiment data for precise teaching adjustments. The method includes: collecting multimodal sentiment data from students during the teaching process, inputting it into a pre-constructed dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector; constructing a composite state vector based on the sentiment state representation vector and the students' cognitive level assessment results; and inputting the composite state vector into a pre-trained reinforcement learning strategy network to output teaching adjustment instructions and dynamically adjust the teaching content. This application effectively solves the problem of difficult multimodal data fusion, achieves comprehensive perception and response to students' emotional and cognitive states, and improves the accuracy and intelligence of teaching adjustments.

Description

Technical Field

[0001] This application relates to the field of intelligent education, and in particular to adaptive teaching adjustment methods, systems and devices based on sentiment analysis. Background Technology

[0002] Currently, in the field of intelligent education, adaptive teaching systems typically focus on adjusting the content and difficulty of instruction based on students' cognitive levels (such as accuracy in answering questions and mastery of knowledge points). However, the learning process is a complex psychological activity, and students' emotional states (such as confusion, anxiety, focus, and excitement) have a crucial impact on learning efficiency and knowledge internalization. Most existing adaptive teaching solutions neglect students' emotional changes during the teaching process or only use a single modality (such as facial expressions) for simple emotion recognition, lacking the ability to comprehensively process multimodal emotional data (visual, audio, and text interaction). Furthermore, multimodal data often exhibit temporal asynchrony and semantic inconsistencies, and direct fusion can lead to inaccurate emotional state judgments. Simultaneously, existing teaching strategy adjustment mechanisms are mostly simple rule-based mappings, lacking the ability to predict and optimize the long-term effects of teaching strategies, making it difficult to achieve truly personalized, emotion-aware intelligent teaching adjustments.

[0003] Therefore, how to integrate multimodal emotional data and make precise and efficient adjustments to teaching based on it has become a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides an adaptive teaching adjustment method, system, and device based on sentiment analysis, which solves the technical problem that existing technologies cannot effectively integrate multimodal sentiment data and make precise teaching adjustments.

[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, it provides adaptive teaching adjustment methods based on sentiment analysis, including: Collect multimodal emotional data of students during the teaching process. Multimodal emotional data includes at least visual data, audio data, and text interaction data. Multimodal sentiment data is input into a pre-built dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector. The dynamic cross-modal graph-temporal alignment network is used to handle the temporal asynchrony and semantic inconsistency between multimodal sentiment data. Based on the emotional state representation vector and the students' cognitive level assessment results, a composite state vector is constructed. The composite state vector is input into a pre-trained reinforcement learning policy network, which then outputs instructional adjustment commands for the current learning content. Based on the instructions for adjusting the teaching, the teaching content will be dynamically adjusted, including the presentation method, difficulty, and interaction mode of the teaching content.

[0006] Based on the above technical solutions, the adaptive teaching adjustment method based on sentiment analysis provided in this application suffers from the common drawback of traditional teaching systems that "emphasize cognition but neglect emotion." These systems struggle to perceive students' complex emotional states in real time and cannot deeply integrate emotional and cognitive information to guide precise intervention, resulting in delayed teaching adjustments, crude strategies, and negatively impacting learning efficiency and experience. Therefore, there is an urgent need for an adaptive teaching mechanism that can simultaneously model emotion and cognition and make intelligent decisions accordingly. This application effectively solves these problems by constructing a closed loop of "perception-integration-decision-execution." Its core benefits are as follows: First, by utilizing a dynamic cross-modal graph-temporal alignment network, it not only captures the cross-modal correlations within multimodal emotional data (such as facial expressions, speech, and posture) but also models their evolutionary patterns over time, generating a high-dimensional, robust fusion emotional state representation. Second, it organically integrates this emotional representation with independently assessed cognitive levels into a composite state vector, comprehensively depicting the student's "mind-body integration" learning state. Finally, with the help of a pre-trained reinforcement learning strategy network, it learns optimal teaching strategies in a simulated environment, enabling it to output personalized and forward-looking teaching adjustment instructions (such as switching difficulty, inserting incentives, and adjusting pace) in real time based on the composite state. This technical solution achieves a leap from single-dimensional feedback to multi-dimensional state-driven approaches, enabling teaching interventions to not only "understand knowledge acquisition" but also "observe emotional changes," significantly improving the accuracy, humanization, and adaptability of teaching.

[0007] In conjunction with the first aspect above, in one possible implementation, the method for generating the fused emotional state representation vector includes: Extract contextual semantic vectors from the current teaching content; Based on contextual semantic vectors, a cross-modal semantic relation matrix is ​​generated through a learnable relation prior generator. The cross-modal semantic relation matrix is ​​used to characterize the idealized semantic association strength that should exist between feature elements of different modalities in the current teaching context. An initial node set is constructed based on multimodal sentiment data, and a cross-modal semantic relationship matrix and a cross-modal similarity matrix calculated based on the data are weighted and fused to jointly determine the edge weights between nodes, thereby constructing a context-aware dynamic heterogeneous graph. By using a graph neural network to perform message passing on a context-aware dynamic heterogeneous graph, cross-modal context information is aggregated to obtain initially aligned node embeddings. The initially aligned nodes are embedded and reassembled into a new temporal sequence according to the timestamp, and then input into a gated recurrent unit for temporal modeling, outputting a fused emotional state representation vector.

[0008] In conjunction with the first aspect above, in one possible implementation, the step of weighted fusion of the cross-modal semantic relationship matrix and the cross-modal similarity matrix calculated based on data to jointly determine the edge weights between nodes includes: For any two nodes from mode i and mode j respectively, their edge weights Calculated using the following formula: ; in, For contextual semantic vectors, , Let i and j be the feature vectors of nodes i and j, respectively. This is the normalized similarity calculated based on cosine similarity. For the normalized cross-modal semantic relation strength, σ( () is the Sigmoid function. These are the learnable parameters of the gated network. For the bias term, [ ; ; ] indicates vector concatenation.

[0009] In conjunction with the first aspect above, in one possible implementation, the method for constructing the composite state vector includes: Obtain the sentiment state representation vector generated from multimodal sentiment data analysis; Obtain the cognitive level assessment vector output by student answer behavior, interaction logs, or knowledge tracing model, and simultaneously obtain the uncertainty measure of the cognitive level assessment; Based on the uncertainty measure, the cognitive level assessment vector is calibrated with confidence to obtain the calibrated cognitive vector; wherein, the change in the uncertainty measure is proportional to the degree to which the amplitude of each element of the calibrated cognitive vector is compressed; The emotional state representation vector and the calibrated cognitive vector are fused in a higher order. The higher order interaction fusion is achieved through a learnable third-order coupling parameter tensor, which is used to capture the bilinear interaction relationship between the emotional dimension and the cognitive dimension. The results of high-order interaction fusion are superimposed with their respective independent linear transformation components and bias terms, and then a composite state vector is generated by the hyperbolic tangent activation function.

[0010] In conjunction with the first aspect above, in one possible implementation, the reinforcement learning policy network includes: The reinforcement learning policy network is a hierarchical reinforcement learning framework, including a high-level policy module and a low-level policy module; The high-level strategy module determines the short-term sub-goals that need to be achieved at present, based on the preset long-term teaching objectives. The underlying strategy module combines short-term sub-goals and integrates emotional state representations to generate specific adaptive teaching adjustment instructions; In this process, the underlying strategy module further performs counterfactual reasoning when generating adaptive teaching adjustment instructions in order to evaluate and optimize the expected effects of the generated instructions.

[0011] In conjunction with the first aspect above, in one possible implementation, the execution of counterfactual reasoning includes: Obtain the student's fused emotional state representation vector at time t. and candidate instruction vectors ; Using a pre-trained world model, for each candidate instruction, predict the student's emotional state vector at the next time step t+1 after executing the instruction. ;in, For the mapping function of the world model; Based on the goals of the current teaching task, students' knowledge mastery level, and individual emotional baseline, an ideal emotional state vector is dynamically constructed. Ideal emotional state represents the emotional state that students should achieve in an ideal teaching context; Calculate the counterfactual loss for each candidate instruction; the counterfactual loss is used to predict the sentiment state. With ideal emotional state The square of the Euclidean distance between them: ; The candidate instruction that minimizes counterfactual loss is selected as the final adaptive instruction adjustment instruction and executed.

[0012] In conjunction with the first aspect above, in one possible implementation, the pre-trained world model includes: The world model is trained offline using historical interaction data, which includes time series of students' multimodal emotional states under different teaching instructions. The training objective is to minimize the reconstruction error between the predicted emotional state and the actual observed emotional state.

[0013] In conjunction with the first aspect above, in one possible implementation, the multimodal sentiment data is further preprocessed before being input into the pre-constructed dynamic cross-modal graph-temporal alignment network: The system detects whether there are missing modalities in the multimodal sentiment data; if not, no action is taken; if yes, the adversarial missing modality reconstruction module is activated, which includes a generator and a discriminator. The generator receives data on the current non-missing modalities and temporal context information extracted from the historical sentiment state memory bank to generate potential representations of the missing modalities. The discriminator performs dual discrimination on latent representations: on the one hand, it judges their consistency with the distribution of real modal data, and on the other hand, it judges their emotional reasonableness with the current teaching task difficulty assessment results; The original missing data is replaced with latent representations, and the repaired multimodal data is input into a dynamic crossmodal graph-temporal alignment network.

[0014] Secondly, this application provides an adaptive teaching adjustment system based on sentiment analysis, comprising: a data acquisition module, an adjustment module, and an execution module; wherein, the data acquisition module is used to acquire multimodal sentiment data of students during the teaching process; the adjustment module is used to input the multimodal sentiment data into a pre-constructed dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector; based on the sentiment state representation vector and the student's cognitive level assessment results, a composite state vector is constructed; the composite state vector is input into a pre-trained reinforcement learning strategy network, which outputs teaching adjustment instructions for the current learning content; the execution module is used to dynamically adjust the teaching content according to the teaching adjustment instructions.

[0015] Thirdly, an adaptive teaching adjustment device based on sentiment analysis is provided, comprising: a communication unit and a processing unit; wherein, the communication unit is used to collect multimodal sentiment data of students during the teaching process; the processing unit is used to input the multimodal sentiment data into a pre-constructed dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector; based on the sentiment state representation vector and the student's cognitive level assessment results, a composite state vector is constructed; the composite state vector is input into a pre-trained reinforcement learning strategy network, which outputs teaching adjustment instructions for the current learning content; and the teaching content is dynamically adjusted according to the teaching adjustment instructions.

[0016] The adaptive teaching adjustment method, system, and device based on sentiment analysis provided in this application effectively solve the core problems of weak emotional state perception, delayed teaching intervention, and lack of personalized dynamic adaptation capabilities in traditional intelligent education systems. Existing systems mostly rely on cognitive indicators such as answer accuracy for feedback, neglecting the crucial impact of emotional factors such as emotion, motivation, and focus on learning outcomes. They struggle to proactively intervene before students experience negative states such as anxiety, fatigue, or distraction, leading to decreased learning efficiency and even resistance. This application constructs an integrated closed-loop control system of "perception-fusion-prediction-optimization" by deeply integrating multimodal emotion recognition, cognitive assessment, and reinforcement learning decision-making mechanisms. Its beneficial effects are reflected in: first, accurately extracting students' real-time emotional states using a dynamic cross-modal graph-temporal alignment network and generating high-dimensional composite state vectors based on cognitive levels to comprehensively characterize learners' internal states; second, introducing world models and counterfactual reasoning mechanisms to proactively predict the impact of different teaching instructions on future emotions and select the optimal strategy guided by ideal emotional goals; and third, supporting adversarial reconstruction of missing modalities, ensuring the system's robustness in real and complex environments. Therefore, this application can not only respond to the current situation in a timely manner, but also proactively guide students into a positive and efficient learning emotional zone, significantly improving the personalization, intelligence and humanization of teaching, and providing a reliable technical path for realizing a new generation of adaptive education that is "teaching according to emotions and guiding learning according to abilities".

[0017] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0018] Figure 1 A flowchart illustrating an adaptive teaching adjustment method based on sentiment analysis provided in this application embodiment; Figure 2 A flowchart illustrating a method for obtaining a fused emotional state representation vector, provided in an embodiment of this application; Figure 3 A flowchart illustrating a method for obtaining a composite state vector provided in an embodiment of this application; Figure 4 A flowchart illustrating a counterfactual reasoning method for executing a teaching adjustment instruction, provided as an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a teaching adjustment device A provided in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of a teaching adjustment device B provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] The adaptive teaching adjustment method based on sentiment analysis provided in this application embodiment can be applied to an adaptive teaching adjustment system based on sentiment analysis, which includes: a collection module, an adjustment module, and an execution module. The data acquisition module is used to collect multimodal emotional data of students during the teaching process; The adjustment module is used to input multimodal sentiment data into a pre-built dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector; based on the sentiment state representation vector and the student's cognitive level assessment results, a composite state vector is constructed; the composite state vector is input into a pre-trained reinforcement learning policy network, which outputs teaching adjustment instructions for the current learning content. The execution module is used to dynamically adjust the teaching content based on teaching adjustment instructions.

[0021] To address the technical problem that existing technologies cannot effectively integrate multimodal emotional data for precise teaching adjustments, this application provides an adaptive teaching adjustment method based on sentiment analysis. This method includes: collecting multimodal emotional data from students during the teaching process; inputting the multimodal emotional data into a pre-constructed dynamic cross-modal graph-temporal alignment network to generate a fused emotional state representation vector; constructing a composite state vector based on the emotional state representation vector and the student's cognitive level assessment results; inputting the composite state vector into a pre-trained reinforcement learning policy network, which outputs teaching adjustment instructions for the current learning content; and dynamically adjusting the teaching content according to the teaching adjustment instructions. Based on this, this application effectively solves the problems of temporal asynchrony and semantic inconsistency between multimodal data by collecting multimodal emotional data and using a dynamic cross-modal graph-temporal alignment network to generate fused emotional state representations, significantly improving the accuracy and robustness of emotion recognition; further, by combining cognitive level to construct a composite state vector and having the reinforcement learning policy network output teaching adjustment instructions, it achieves personalized teaching intervention driven by both emotion and cognition. This method can dynamically optimize the presentation, difficulty, and interaction mode of teaching content based on students' real-time status, thereby improving learning focus, alleviating negative emotions, enhancing learning motivation, and ultimately improving teaching effectiveness and learning experience.

[0022] Example 1, such as Figure 1 As shown in the embodiments of this application, the adaptive teaching adjustment method based on sentiment analysis includes: S101. Collect multimodal emotional data of students during the teaching process. Multimodal emotional data includes at least visual data, audio data, and text interaction data.

[0023] Specifically, the collection of multimodal emotional data is the foundation of the entire adaptive teaching adjustment process. Visual data can be collected through cameras installed on teaching terminals or in classrooms, capturing image information such as students' facial expressions, head postures, and eye movements. Audio data can be collected through microphone arrays, capturing sound signals such as students' speech tone, speaking speed, and pauses. Text interaction data can be obtained by recording students' answers, forum posts, chat logs, and other text information on the teaching platform. It should be understood that multimodal emotional data is not limited to the above three types; it can also include physiological signal data (such as heart rate and skin conductance response), etc. Any data that can reflect students' emotional state is within the scope of protection of this application. By collecting multimodal data, students' emotional changes can be comprehensively captured from multiple dimensions, avoiding the misjudgment or information loss problems that may exist with single-modal data.

[0024] S102. Input the multimodal sentiment data into a pre-constructed dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector. The dynamic cross-modal graph-temporal alignment network is used to handle the temporal asynchrony and semantic inconsistency between the multimodal sentiment data.

[0025] Specifically, the dynamic cross-modal graph-temporal alignment network is one of the core components of this embodiment. Because data from different modalities such as visual, audio, and text are often misaligned on the timeline (e.g., a student frowns first, then sighs a few seconds later), and the emotional semantics expressed by each modality may conflict (e.g., a facial expression shows confusion, but the tone of voice is calm), direct splicing or simple fusion can lead to inaccurate emotional state judgments. This network models the complex relationships between different modalities by constructing a dynamic heterogeneous graph and uses a temporal alignment mechanism to solve the asynchronicity problem. For example, the network can use features from different modalities as nodes in the graph, and learn the edge weights between nodes to capture the semantic consistency relationships across modalities, thereby generating a fused emotional state representation vector that comprehensively reflects the student's current emotional state. This vector is a high-dimensional numerical representation that can be efficiently processed by subsequent modules.

[0026] S103. Based on the emotional state representation vector and the students' cognitive level assessment results, construct a composite state vector.

[0027] Specifically, students' cognitive level assessment results can be obtained by analyzing their historical answering behavior, interaction logs, or using knowledge tracing models to quantitatively assess their mastery of knowledge points. The purpose of constructing a composite state vector is to deeply integrate students' emotional and cognitive states, thereby forming a more comprehensive description of their learning state. For example, the emotional state representation vector and the cognitive level assessment vector can be concatenated, weighted, or fused through attention mechanisms. The composite state vector not only contains cognitive information about whether students "understand" or "don't understand," but also emotional information about whether they "want to learn" and "are anxious," providing richer evidence for subsequent teaching decisions.

[0028] S104. Input the composite state vector into the pre-trained reinforcement learning policy network, and the reinforcement learning policy network outputs teaching adjustment instructions for the current learning content.

[0029] Specifically, a reinforcement learning strategy network can be viewed as an intelligent decision-making brain. Through continuous interaction with the environment (i.e., the teaching system), it learns which actions (i.e., teaching adjustment instructions) under different states will yield the greatest long-term cumulative rewards (such as improved student learning outcomes and emotional state). This network is pre-trained with a large amount of historical teaching interaction data, enabling it to output the optimal teaching strategy based on the current workload vector. Teaching adjustment instructions can be discrete action labels (such as "reduce difficulty," "provide hints," "switch teaching style") or continuous parameter values ​​(such as "adjust explanation speed to 0.8 times," "set question difficulty coefficient to 0.6"). Through reinforcement learning, the system can move beyond simple rule matching, achieving more proactive and personalized teaching adjustments.

[0030] S105. Based on the teaching adjustment instructions, dynamically adjust the teaching content, including the presentation method, difficulty, and interaction mode of the teaching content.

[0031] Specifically, the presentation of teaching content can include various forms such as text, images, videos, animations, and virtual reality (VR). For example, when the system detects that a student is in a "confused" state, the plain text explanation can be adjusted to a more intuitive animated demonstration. The difficulty of the teaching content can be dynamically adjusted based on the student's cognitive level assessment results. Interaction modes can include active questioning, guided inquiry, and gamified interaction. For example, when the system detects that a student is "distracted," it can trigger an interactive mini-game to re-attract the student's attention. Through the closed-loop processing of the above steps, this embodiment realizes a fully adaptive teaching process from data collection, sentiment analysis, state fusion to strategy decision-making and content adjustment, which can effectively improve students' learning experience and learning outcomes.

[0032] Example 2, as Figure 2 As shown in the example, this embodiment, based on Example 1, provides a detailed explanation of the specific process for generating the fused emotional state representation vector. Specifically, this process is achieved by constructing a context-aware dynamic heterogeneous graph and fusing multimodal information using a gating mechanism.

[0033] S201. Extract the contextual semantic vector from the current teaching content.

[0034] Specifically, the teaching system can semantically encode the text of the currently being taught knowledge points, question descriptions, or teaching scripts. For example, it can use the BERT model to extract high-dimensional semantic features, obtaining a contextual semantic vector c. This vector c contains thematic information about the current teaching scenario, such as "properties of quadratic function graphs" or "analysis of English relative clauses," providing a benchmark for subsequently judging the semantic relationships between different modalities of data.

[0035] S202. Based on the context semantic vector, a cross-modal semantic relation matrix is ​​generated through a learnable relation prior generator. The cross-modal semantic relation matrix is ​​used to characterize the idealized semantic association strength that should exist between feature elements of different modalities in the current teaching context.

[0036] Specifically, the relation prior generator can be a multilayer perceptron (MLP) or an attention network, whose input is a context semantic vector c, and whose output is a matrix R of a predetermined dimension. The elements in this matrix R... This represents the theoretically expected correlation strength between features of modality i (such as the visual feature of a student frowning) and features of modality j (such as the audio feature of a student sighing) under the current teaching theme. For example, frowning and sighing may be strongly correlated when explaining difficult math problems; while smiling and rising intonation may be strongly correlated when practicing light English conversation. By introducing relational priors, the system can utilize prior knowledge from the teaching domain to guide the fusion of multimodal data, avoiding erroneous associations caused by data noise or individual differences.

[0037] S203. Construct an initial node set based on multimodal sentiment data, and use the cross-modal semantic relationship matrix and the cross-modal similarity matrix calculated based on the data to perform weighted fusion to jointly determine the edge weights between nodes, thereby constructing a context-aware dynamic heterogeneous graph.

[0038] Specifically, the initial node set consists of visual feature nodes, audio feature nodes, and text feature nodes. To construct the graph structure, the edge weights between nodes need to be determined. This embodiment does not simply calculate feature similarity, but rather employs a dynamic weighted fusion mechanism.

[0039] For any two nodes from mode i and mode j respectively, their edge weights Calculated using the following formula: ; in, For contextual semantic vectors, , Let i and j be the feature vectors of nodes i and j, respectively. This is the normalized similarity calculated based on cosine similarity. For the normalized cross-modal semantic relation strength, σ( () is the Sigmoid function. These are the learnable parameters of the gated network. For the bias term, [ ; ; ] indicates vector concatenation.

[0040] In this formula, This represents surface similarity driven by data. For example, the visual feature vector of "frowning" and the audio feature vector of "sighing" may be close in distance and have a high similarity in Euclidean space. This represents semantic relevance driven by knowledge. In other words, within the current teaching context, should these two features logically coexist? (Gated networks) It plays a dynamic adjustment role, with its output value ranging from 0 to 1. The system adjusts its output based on the current context c and node characteristics. , Automatic decision-making relies more on data-driven similarity. Or do we place more trust in knowledge-driven semantic relationships? .

[0041] For example, in some cases, a student's visual characteristics may not match their current emotional state due to habitual actions (such as resting their chin on their hand). In such cases, simply relying on similarity is insufficient. This may introduce noise. However, through gating mechanisms, the system can reduce noise. Weighting, increasing The system assigns weights based on prior knowledge of the teaching context to correct erroneous associations. Conversely, if student performance is highly typical and clear, the system can improve... The weighted fusion method effectively overcomes the limitations of relying solely on data similarity or prior knowledge, significantly improving the robustness and accuracy of graph construction.

[0042] S204. Message passing is performed on the context-aware dynamic heterogeneous graph through a graph neural network, and cross-modal context information is aggregated to obtain preliminary aligned node embeddings.

[0043] Specifically, graph neural networks (such as GAT or GCN) use multiple iterations to allow each node to aggregate information from its neighboring nodes. For example, a visual node aggregates information from its associated audio and text nodes, thus incorporating contextual information from other modalities into the node's features. This process achieves cross-modal semantic alignment, resolving the inconsistency problem in the semantic space between data from different modalities.

[0044] S205. Embed the initially aligned nodes and reassemble them into a new temporal sequence according to the timestamp, and input them into the gated recurrent unit for temporal modeling, and output the fused emotional state representation vector.

[0045] Specifically, the node embeddings processed by the graph neural network already contain cross-modal contextual information. The system rearranges these embedding vectors according to their original acquisition timestamps, forming a new temporal sequence. The gated recurrent unit (GRU) processes this sequence to capture the dynamic patterns of emotional states changing over time, such as the evolution from "confusion" to "anxiety" and then to "understanding." The final output fused emotional state representation vector not only integrates multimodal information but also includes temporal evolution features, comprehensively and accurately reflecting the student's current emotional state.

[0046] Based on the above technical solutions, the proposed solution aims to address two core challenges in multimodal emotion data fusion: temporal asynchrony (different sampling rates and response times for different modalities) and semantic inconsistency (conflicting representations of the same emotion across different modalities). Direct splicing or simple weighted fusion can lead to distortions in emotion state representation due to noise, missing data, or intermodal misalignment, thus affecting the accuracy of teaching decisions. To address this, this solution constructs a context-aware dynamic heterogeneous graph-temporal alignment network. Its beneficial effects are reflected in the following aspects: First, by introducing a learnable relation prior generator, the contextual semantics of the current teaching content are utilized to provide knowledge-driven idealized constraints for the association between cross-modal features, effectively guiding the direction of semantic alignment. Second, this prior relation matrix is ​​gated and weighted to fuse with a data-driven similarity matrix, constructing a dynamic graph structure that retains the real-time characteristics of the data while incorporating prior knowledge from the teaching domain, significantly improving the robustness and accuracy of graph construction. Finally, through a two-stage processing of "graph neural network aggregation + gated recurrent unit temporal modeling," cross-modal semantic alignment is first completed in the graph space, and then the dynamic evolution of emotions is captured in the temporal dimension, thereby generating a high-fidelity, context-sensitive fused emotional state representation vector. This vector provides a solid and reliable state foundation for subsequent precise teaching interventions, fundamentally improving the intelligence level and personalized service capabilities of the adaptive teaching system.

[0047] Example 3, as Figure 3 As shown, this embodiment, based on Embodiment 1, provides a detailed explanation of the specific process for constructing a composite state vector. Specifically, this process involves introducing an uncertainty metric for confidence calibration and utilizing a high-order interaction fusion mechanism to deeply integrate emotional and cognitive states.

[0048] S301. Obtain the emotional state representation vector generated by multimodal emotional data analysis. This vector is the fused emotional state representation vector output in S102 of Example 1, which contains the student's multidimensional emotional characteristics at the current moment.

[0049] S302. Obtain the cognitive level assessment vector output by the student's answer behavior, interaction log, or knowledge tracing model, and simultaneously obtain the uncertainty measure of the cognitive level assessment.

[0050] Specifically, cognitive level assessment vectors can be obtained by analyzing students' historical answer accuracy rates, knowledge point mastery maps, or using deep knowledge tracing models to quantitatively assess students' cognitive states. However, due to random factors such as guessing and carelessness in student answers, or sparse interaction log data, cognitive level assessment results often have a certain degree of uncertainty. This embodiment introduces an uncertainty metric to quantify the reliability of this assessment. The uncertainty metric can be obtained by calculating the variance, entropy, or posterior distribution based on a Bayesian neural network. For example, if the Monte Carlo Dropout method is used, multiple forward propagations can be performed during the inference phase, and the variance of the output results can be calculated as an uncertainty metric. The larger the variance, the less certain the model is about the cognitive assessment result, and the higher the uncertainty.

[0051] S303. Based on the uncertainty measure, the confidence level assessment vector is calibrated to obtain the calibrated cognitive vector; wherein, the change in the uncertainty measure is proportional to the degree to which the amplitude of each element of the calibrated cognitive vector is compressed.

[0052] Specifically, the purpose of confidence calibration is to reduce the impact of unreliable cognitive assessments on the final decision. The calibration logic can employ an adaptive scaling mechanism: when the uncertainty metric is high, indicating unreliable cognitive assessment results, the system compresses the magnitudes of each element in the cognitive level assessment vector towards the zero vector, giving them a smaller weight in subsequent fusion; conversely, when the uncertainty metric is low, indicating reliable assessment results, the original magnitudes are retained or only slightly compressed. For example, the calibration formula can be designed as follows: ;in, This is a confidence-calibrated cognitive level assessment vector. Let U be the original cognitive vector, and U be the uncertainty measure. The hyperparameters are adjustable. This design effectively prevents teaching strategy errors caused by erroneous cognitive assessments (such as misjudging "correct guess" as "mastery of knowledge"), thus improving the robustness of the system.

[0053] S304. The emotional state representation vector and the calibrated cognitive vector are fused in a higher order. The higher order interaction fusion is achieved through a learnable third-order coupling parameter tensor, which is used to capture the bilinear interaction relationship between the emotional dimension and the cognitive dimension.

[0054] Specifically, traditional vector fusion methods typically employ simple concatenation or weighted summation, assuming that emotion and cognition are independent and ignoring the complex nonlinear coupling between them. For example, when a student is in an "anxious" state, their "high cognitive level" may not function properly; conversely, when a student is in an "excited" state, their "low cognitive level" may be masked. To capture this interaction effect, this embodiment introduces a third-order coupling parameter tensor T for higher-order fusion. In this way, the system can explicitly model "how specific emotional states modulate the impact of cognitive states on teaching decisions," thereby generating more discriminative composite features.

[0055] S305. After superimposing the results of the high-order interaction fusion with their respective independent linear transformation components and bias terms, a composite state vector is generated by the hyperbolic tangent activation function.

[0056] Specifically, to preserve the independent information of emotion and cognition, the system performs linear transformations on the emotional state representation vector and the calibrated cognitive vector respectively, and then superimposes the results onto the higher-order interaction fusion result. The superimposed vector is then nonlinearly mapped using a hyperbolic tangent activation function, compressing the numerical range to the (-1,1) interval, thus obtaining the final composite state vector. This vector not only integrates both emotional and cognitive information but also contains deep interaction features between the two, enabling it to more comprehensively and accurately reflect the student's true learning state at the current moment, providing high-quality input for the subsequent decision-making of the reinforcement learning strategy network.

[0057] Based on the above technical solutions, this application proposes a solution aimed at addressing the core problems of existing adaptive teaching systems, such as the simple splicing of emotional and cognitive states, lack of consideration for the reliability of cognitive assessment, and lack of deep interactive modeling. Traditional methods treat emotion and cognition as independent dimensions and linearly fuse them, ignoring their complex coupling relationship (e.g., anxiety can inhibit high cognitive performance). Furthermore, directly using uncalibrated cognitive assessment results makes them susceptible to noise interference such as guesswork or data sparsity, leading to distortion of the composite state and consequently, erroneous teaching decisions. To address this, this solution achieves precise fusion through a dual innovative mechanism: First, it introduces a confidence calibration mechanism based on uncertainty measurement to dynamically compress the amplitude of unreliable cognitive assessments, effectively suppressing the negative impact of low-confidence cognitive information on decision-making and improving system robustness. Second, it designs a high-order interactive fusion module based on a learnable third-order tensor to explicitly model the bilinear interaction relationship between the emotional and cognitive dimensions, capturing complex situations such as "how specific emotions regulate cognitive utility" (e.g., in an "excited" state, moderate cognitive performance may be amplified into efficient learning). The resulting composite state vector not only integrates both emotional and cognitive information but also contains deep interactive features, providing high-fidelity, context-aware state input for reinforcement learning strategy networks. This supports the generation of more precise and humanized teaching adjustment instructions, significantly improving the scientific nature and effectiveness of personalized teaching.

[0058] Example 4, based on Example 1, provides a detailed description of the specific structure and decision optimization mechanism of the reinforcement learning policy network. Specifically, this reinforcement learning policy network is a hierarchical reinforcement learning framework, including a high-level policy module and a low-level policy module.

[0059] In adaptive learning scenarios, learning objectives are often long-term and hierarchical. For example, a semester's learning objective might be "mastering quadratic functions," a long-term goal that needs to be broken down into short-term objectives across multiple class periods and knowledge points. Using only a single-layer strategy network makes it difficult to simultaneously address both long-term goal achievement and immediate short-term responses. Therefore, this embodiment introduces a hierarchical reinforcement learning framework. The high-level strategy module is responsible for macro-level planning, determining the short-term sub-goals to be achieved at any given moment based on pre-set long-term learning objectives (such as syllabus requirements and student ability models) and the current overall learning progress of students. For example, the high-level strategy module might break down "mastering quadratic functions" into short-term sub-goals such as "understanding the properties of quadratic function graphs" and "mastering the vertex coordinate formula," setting deadlines for each sub-goal. The low-level strategy module is responsible for micro-level execution, generating specific adaptive learning adjustment instructions based on the short-term sub-goals issued by the high-level module and real-time generated fusion emotional state representations, such as "showing a basic exercise problem" or "playing an animated explanation." This hierarchical architecture allows the system to both track long-term learning objectives and flexibly respond to students' real-time emotional fluctuations and cognitive changes.

[0060] Furthermore, when generating adaptive teaching adjustment instructions, the underlying strategy module performs counterfactual reasoning to evaluate and optimize the expected effects of the generated instructions.

[0061] Specifically, the core idea of ​​counterfactual reasoning is "deducing the future and making the best decision." Because the teaching process is irreversible, executing incorrect instructional adjustments (such as forcibly increasing the difficulty when students are anxious) may lead to a loss of interest or even dropping out. Therefore, before actually executing the instructions, the system needs to deduce the consequences of different instructions in a "virtual environment." For example... Figure 4 As shown, the specific process of counterfactual reasoning includes the following steps: S401. Obtain the student's fused emotional state representation vector at the current time t. and candidate instruction vectors .

[0062] Among them, candidate teaching instruction vectors It consists of multiple alternative actions generated by the underlying strategy module based on the current state, such as "reduce difficulty", "provide hints", "switch explanation style", etc., and each action is encoded as a vector.

[0063] S402. Using a pre-trained world model, predict the student's emotional state vector at the next time step t+1 after executing each candidate teaching instruction. ;in, This is the mapping function for the world model. The world model is essentially a simulator that learns the dynamics of the learning environment. For example, it can be based on the current student's state of "anxiety." and the "reduce difficulty" instruction It can be predicted that the student's state may change to "calm" or "focused" in the next moment.

[0064] S403. Based on the objectives of the current teaching task, students' knowledge mastery level, and individual emotional baseline, dynamically construct an ideal emotional state vector. Ideal emotional state represents the emotional state that students should achieve in an ideal teaching context.

[0065] For example, when dealing with challenging concepts, the ideal state might be "moderate tension but focused attention"; for review material, the ideal state might be "relaxed and confident." This ideal state is not fixed but dynamically adjusted according to the teaching context.

[0066] S404. Calculate the counterfactual loss for each candidate instruction. The counterfactual loss is the predicted sentiment state. With ideal emotional state The square of the Euclidean distance between them: The smaller the loss value, the closer the student's state is to the ideal state after executing the instruction.

[0067] S405. By calculating the counterfactual loss of all candidate instructions, select the candidate instruction that minimizes the counterfactual loss as the final adaptive teaching adjustment instruction and execute it.

[0068] To more clearly illustrate the advantages of counterfactual reasoning, let's assume the student is currently in a state of "confusion," with candidate instructions including "continue explaining" and "provide hints." Without counterfactual reasoning, the policy network might directly choose "continue explaining" based on historical experience. However, the world model predicts that "continue explaining" in the "confusion" state will lead to a state of "anxiety" in the next moment, deviating significantly from the ideal state of "understanding," resulting in high counterfactual loss. "Provide hints," on the other hand, predicts a state of "thinking" in the next moment, which is closer to the ideal state, resulting in low counterfactual loss. Therefore, the system ultimately chooses "provide hints," preventing the student from falling into anxiety.

[0069] In this embodiment, the world model is trained offline using historical interaction data. This historical interaction data includes time series of students' multimodal emotional states under different teaching instructions. For example, the dataset records tens of thousands of transition samples from "student state - teaching action - next student state". The training objective is to minimize the reconstruction error between the predicted emotional state and the actual observed emotional state. In this way, the world model can learn the complex causal relationships in the teaching environment, providing a reliable predictive basis for counterfactual reasoning. It should be understood that the world model can be implemented using deep learning architectures such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or Transformers, as long as it can fit the state transition function.

[0070] Based on the aforementioned technical solutions, this application proposes a solution aimed at addressing the critical issue that traditional adaptive teaching systems rely solely on immediate feedback or static rules for instructional adjustments, lacking forward-looking modeling of the long-term emotional impact of instructional interventions. Existing methods often only respond passively after students have already experienced negative emotions (such as frustration or distraction), making it difficult to achieve proactive guidance and emotional optimization, resulting in low teaching efficiency and even learning resistance. To address this, this solution introduces a counterfactual reasoning mechanism based on a world model. Its beneficial effects are as follows: First, by simulating the causal impact of different instructional instructions on future emotional states through a pre-trained world model, the system possesses "predictability," enabling it to assess the emotional consequences of each option before action. Second, by dynamically constructing an ideal emotional state vector by combining teaching objectives, cognitive levels, and individual emotional baselines, it ensures that emotional guidance aligns with educational goals while respecting individual differences. Finally, by using the minimization of the counterfactual loss between predicted and ideal emotions as the optimization criterion, the optimal instruction is selected from candidate instructions, achieving a paradigm shift from "reactive regulation" to "proactive guidance." This mechanism can not only effectively avoid teaching strategies that may trigger negative emotions, but also proactively cultivate a positive, focused, and motivated learning emotional state. In this way, while improving the efficiency of knowledge transfer, it can significantly enhance the learning experience and psychological resilience, and promote the evolution of human-computer collaborative education towards a higher level of intelligent emotional interaction.

[0071] Example 5, building upon Example 1, details the preprocessing of multimodal sentiment data. In real-world teaching scenarios, data acquisition devices may experience temporary data loss in a particular modality due to network fluctuations, hardware malfunctions, or environmental obstructions. For example, a student might temporarily block the camera, resulting in lost visual data, or a microphone connection interruption might lead to lost audio data. Directly inputting this missing data into the dynamic cross-modal graph-temporal alignment network would cause a mismatch in network input dimensions, severely impacting the accuracy of sentiment analysis. Therefore, this example introduces an adversarial missing modality reconstruction module to improve the system's robustness.

[0072] Specifically, the preprocessing of multimodal sentiment data includes the following steps: S501. Detect the presence of missing modalities in the multimodal sentiment data. After receiving the data packet uploaded by the acquisition module, the system will first check the integrity of the data packet.

[0073] For example, check if the visual data frame is empty or a completely black image, check if the audio data is a silent segment or an empty signal, and check if the text interaction log contains records. If the detection results show that all modal data are complete, no processing is required, and the raw data is directly input into the subsequent network.

[0074] S502. If the detection results show the presence of a missing modality, the adversarial missing modality reconstruction module is activated. The adversarial missing modality reconstruction module includes a generator and a discriminator. This module is essentially a deep learning model based on a generative adversarial network (GAN) architecture, specifically designed to infer and generate a reasonable representation of the missing modality based on information from the remaining modalities in the case of missing modalities.

[0075] S503, The generator receives data on the current non-missing modalities and temporal context information extracted from the historical emotional state memory bank, and generates potential representations of the missing modalities.

[0076] Specifically, suppose visual data is missing at the current moment, but audio and text interaction data are normal. The generator (e.g., a multi-layer Transformer encoder) takes audio and text feature vectors as input, while retrieving the student's emotional state sequence from the past few minutes from a historical emotional state memory bank as temporal context. The generator learns the association between the not-missing and missing modalities through a cross-modal attention mechanism. For example, the generator might learn that "rapid tone and incorrect text answer" usually corresponds to "facial anxiety." Based on this, the generator outputs a high-dimensional latent representation vector that simulates the missing visual features in the semantic space.

[0077] The discriminator performs a dual discrimination on latent representations: on the one hand, it judges their consistency with the distribution of real modal data; on the other hand, it judges their emotional plausibility with the difficulty assessment results of the current teaching task. This dual discrimination mechanism is the core innovation of this embodiment, aiming to ensure that the generated representations are not only statistically realistic but also logically plausible in the teaching context.

[0078] The first layer of discrimination focuses on distribution consistency. The discriminator (such as a convolutional neural network) calculates the distribution distance (e.g., Wasserstein distance) between the generated latent representations and the real visual feature dataset. The goal is to force the features generated by the generator to be as close as possible to the numerical distribution of the real-world visual features, avoiding the generation of anomalous data that deviates from the norm.

[0079] The second layer of discrimination focuses on the rationality of emotional responses. The discriminator considers the difficulty assessment results of the current teaching task (e.g., the difficulty coefficient of the current question is 0.8, which is considered high difficulty) to determine whether the generated emotional representation is reasonable. For example, under a high-difficulty question, if the generator produces a "relaxed and happy" emotional representation, the discriminator will determine it as unreasonable and impose a penalty; conversely, if it generates a "focused or confused" emotional representation, it will be determined as reasonable. This mechanism ensures that the reconstructed data conforms to the teaching logic and avoids generating semantically incorrect emotional features due to blindly pursuing similar distributions.

[0080] S504. Replace the original missing data with latent representations and input the repaired multimodal data into the dynamic cross-modal graph-temporal alignment network.

[0081] Based on the above technical solution, the system can still construct complete and semantically consistent multimodal inputs even when a certain modality is missing, ensuring the normal operation of the subsequent sentiment analysis network. It should be understood that the adversarial missing modality reconstruction module in this embodiment is not only applicable to the mutual reconstruction between visual, audio, and text modalities, but can also be extended to reconstruction scenarios of more modalities (such as physiological signals). This module can function as long as cross-modal semantic relationships exist. This preprocessing mechanism significantly improves the robustness and usability of the adaptive teaching system in complex and unstable teaching environments, avoiding system crashes or misjudgments caused by occasional data loss.

[0082] The foregoing mainly describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as an adaptive teaching adjustment device based on sentiment analysis, includes at least one of the hardware structures and software modules corresponding to the execution of each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] This application embodiment can divide the adaptive teaching adjustment device based on sentiment analysis into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0084] When using integrated units, Figure 5 A possible structural schematic diagram of the adaptive teaching adjustment device based on emotion analysis (hereinafter referred to as teaching adjustment device A) involved in the above embodiments is shown. The teaching adjustment device A includes a communication unit 601 and a processing unit 602, and may also include a storage unit 603. Figure 5 The schematic diagram shown can be used to illustrate the structure of the adaptive teaching adjustment device based on emotion analysis involved in the above embodiments.

[0085] when Figure 5 The schematic diagram shown illustrates the structure of the emotion-based adaptive teaching adjustment device involved in the above embodiments. The communication unit 601 is used for the emotion-based adaptive teaching adjustment device to communicate with other devices, the processing unit 602 is used to control and manage the actions of the emotion-based adaptive teaching adjustment device, and the storage unit 603 is used to store the program code and data of the emotion-based adaptive teaching adjustment device.

[0086] For example, communication unit 601 is used to collect multimodal emotional data of students during the teaching process; Processing unit 602 is used to input multimodal sentiment data into a pre-built dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector.

[0087] In one possible implementation, the processing unit 602 also constructs a composite state vector based on the emotional state representation vector and the student's cognitive level assessment results.

[0088] In one possible implementation, the processing unit 602 is also used to construct a composite state vector based on the emotional state representation vector and the student's cognitive level assessment results.

[0089] The processing unit 602 can be a processor or a controller, and the communication unit 601 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 603 can be a memory. When the teaching adjustment device A is a chip, the processing unit 602 can be a processor or a controller, and the communication unit 601 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 603 can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., read-only memory (ROM), random access memory (RAM, etc.).

[0090] The communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the teaching adjustment device A can be considered as the communication unit 601 of the teaching adjustment device A, and the processor with processing functions can be considered as the processing unit 602 of the teaching adjustment device A. Optionally, the device in the communication unit 601 used to implement the receiving function can be considered as the communication unit. The communication unit is used to execute the receiving steps in the embodiments of this application, and the communication unit can be a receiver, a receiver circuit, etc. The device in the communication unit 601 used to implement the transmitting function can be considered as the transmitting unit. The transmitting unit is used to execute the transmitting steps in the embodiments of this application, and the transmitting unit can be a transmitter, a transmitter, a transmitting circuit, etc.

[0091] Figure 5 If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0092] Figure 5 The units in the process can also be called modules; for example, a processing unit can be called a processing module.

[0093] This application also provides a hardware structure diagram of an adaptive teaching adjustment device based on sentiment analysis (denoted as teaching adjustment device B), see [link to diagram]. Figure 6The teaching adjustment device B includes a processor 701, and optionally, a memory 702 connected to the processor 701.

[0094] In the first possible implementation, see Figure 6 The teaching adjustment device B also includes a transceiver 703. The processor 701, memory 702, and transceiver 703 are connected via a bus. The transceiver 703 is used to communicate with other devices or communication networks. Optionally, the transceiver 703 may include a transmitter and a receiver. The device in the transceiver 703 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 703 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.

[0095] Based on the first possible implementation method Figure 6 The schematic diagram shown can be used to illustrate the structure of the adaptive teaching adjustment device based on emotion analysis involved in the above embodiments.

[0096] in, Figure 6 The diagram also illustrates the system chip in an adaptive teaching adjustment device based on sentiment analysis. In this case, the actions performed by the aforementioned adaptive teaching adjustment device based on sentiment analysis can be implemented by this system chip; the specific actions performed are described above and will not be repeated here.

[0097] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

Claims

1. An adaptive teaching adjustment method based on sentiment analysis, characterized in that, include: Collect multimodal emotional data of students during the teaching process. Multimodal emotional data includes at least visual data, audio data, and text interaction data. Multimodal sentiment data is input into a pre-built dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector; the dynamic cross-modal graph-temporal alignment network is used to handle the temporal asynchrony and semantic inconsistency between multimodal sentiment data. The method for generating the fused emotional state representation vector includes: Extract contextual semantic vectors from the current teaching content; Based on contextual semantic vectors, a cross-modal semantic relation matrix is ​​generated through a learnable relation prior generator. The cross-modal semantic relation matrix is ​​used to characterize the idealized semantic association strength that should exist between feature elements of different modalities in the current teaching context. An initial node set is constructed based on multimodal sentiment data, and a cross-modal semantic relationship matrix and a cross-modal similarity matrix calculated based on the data are weighted and fused to jointly determine the edge weights between nodes, thereby constructing a context-aware dynamic heterogeneous graph. The method for determining the edge weights between nodes includes: For any two nodes from mode i and mode j respectively, their edge weights Calculated using the following formula: ； in, For contextual semantic vectors, , Let i and j be the feature vectors of nodes i and j, respectively. This is the normalized similarity calculated based on cosine similarity. For the normalized cross-modal semantic relation strength, σ( () is the Sigmoid function. These are the learnable parameters of the gated network. For the bias term, [ ; ; ] indicates vector concatenation; By using a graph neural network to perform message passing on a context-aware dynamic heterogeneous graph, cross-modal context information is aggregated to obtain initially aligned node embeddings. The initially aligned nodes are embedded and reassembled into a new temporal sequence according to the timestamp, and then input into a gated recurrent unit for temporal modeling, outputting a fused emotional state representation vector; Based on the emotional state representation vector and the students' cognitive level assessment results, a composite state vector is constructed, including: Obtain the sentiment state representation vector generated from multimodal sentiment data analysis; Obtain the cognitive level assessment vector output by student answering behavior, interaction logs, or knowledge tracing model, and simultaneously obtain the uncertainty measure of the cognitive level assessment; Based on the uncertainty measure, the cognitive level assessment vector is calibrated with confidence to obtain the calibrated cognitive vector; wherein, the change in the uncertainty measure is proportional to the degree to which the amplitude of each element of the calibrated cognitive vector is compressed; The emotional state representation vector and the calibrated cognitive vector are fused in a higher order. The higher order interaction fusion is achieved through a learnable third-order coupling parameter tensor, which is used to capture the bilinear interaction relationship between the emotional dimension and the cognitive dimension. The results of high-order interaction fusion are superimposed with their respective independent linear transformation components and bias terms, and then a composite state vector is generated by the hyperbolic tangent activation function. The composite state vector is input into a pre-trained reinforcement learning policy network, which then outputs instructional adjustment commands for the current learning content. Based on the instructions for adjusting the teaching, the teaching content will be dynamically adjusted, including the presentation method, difficulty, and interaction mode of the teaching content.

2. The adaptive teaching adjustment method based on sentiment analysis according to claim 1, characterized in that, The reinforcement learning policy network includes: The reinforcement learning policy network is a hierarchical reinforcement learning framework, including a high-level policy module and a low-level policy module; The high-level strategy module determines the short-term sub-goals that need to be achieved at present, based on the preset long-term teaching objectives. The underlying strategy module combines short-term sub-goals and integrates emotional state representations to generate specific adaptive teaching adjustment instructions; In this process, the underlying strategy module further performs counterfactual reasoning when generating adaptive teaching adjustment instructions in order to evaluate and optimize the expected effects of the generated instructions.

3. The adaptive teaching adjustment method based on sentiment analysis according to claim 2, characterized in that, The execution of counterfactual reasoning includes: Obtain the student's fused emotional state representation vector at time t. and candidate instruction vectors ; Using a pre-trained world model, for each candidate instruction, predict the student's emotional state vector at the next time step t+1 after executing the instruction. ;in, For the mapping function of the world model; Based on the goals of the current teaching task, students' knowledge mastery level, and individual emotional baseline, an ideal emotional state vector is dynamically constructed. Ideal emotional state represents the emotional state that students should achieve in an ideal teaching context; Calculate the counterfactual loss for each candidate instruction; the counterfactual loss is used to predict the sentiment state. With ideal emotional state The square of the Euclidean distance between them: ; The candidate instruction that minimizes counterfactual loss is selected as the final adaptive instruction adjustment instruction and executed.

4. The adaptive teaching adjustment method based on sentiment analysis according to claim 3, characterized in that, The pre-trained world model includes: The world model is trained offline using historical interaction data, which includes time series of students' multimodal emotional states under different teaching instructions. The training objective is to minimize the reconstruction error between the predicted emotional state and the actual observed emotional state.

5. The adaptive teaching adjustment method based on sentiment analysis according to claim 1, characterized in that, The multimodal sentiment data is preprocessed before being input into the pre-constructed dynamic cross-modal graph-temporal alignment network. Detecting whether missing modalities exist in multimodal sentiment data; No, no action will be taken. If yes, then the adversarial missing mode reconstruction module is activated. The adversarial missing mode reconstruction module includes a generator and a discriminator. The generator receives data on the current non-missing modalities and temporal context information extracted from the historical sentiment state memory bank to generate potential representations of the missing modalities. The discriminator performs dual discrimination on latent representations: on the one hand, it judges their consistency with the distribution of real modal data, and on the other hand, it judges their emotional reasonableness with the current teaching task difficulty assessment results; The original missing data is replaced with latent representations, and the repaired multimodal data is input into a dynamic crossmodal graph-temporal alignment network.

6. An adaptive teaching adjustment system based on sentiment analysis, operating based on the adaptive teaching adjustment method based on sentiment analysis as described in any one of claims 1-5, characterized in that, It includes a data acquisition module, an adjustment module, and an execution module; The acquisition module is used to collect students' multimodal emotional data during the teaching process; The adjustment module is used to input multimodal sentiment data into a pre-constructed dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector. The composite state vector is constructed based on the emotional state representation vector and the student's cognitive level assessment results. The composite state vector is input into a pre-trained reinforcement learning policy network, which then outputs instructional adjustment commands for the current learning content. The execution module is used to dynamically adjust the teaching content according to the teaching adjustment instructions.

7. An adaptive teaching adjustment device based on sentiment analysis, operating based on the adaptive teaching adjustment method based on sentiment analysis as described in any one of claims 1-5, characterized in that, Includes a communication unit and a processing unit; The communication unit is used to collect multimodal emotional data of students during the teaching process; The processing unit is used to input multimodal sentiment data into a pre-constructed dynamic cross-modal graph-temporal alignment network to generate a fused sentiment state representation vector. Based on the emotional state representation vector and the students' cognitive level assessment results, a composite state vector is constructed. The composite state vector is input into a pre-trained reinforcement learning policy network, which then outputs instructional adjustment commands for the current learning content. The teaching content will be dynamically adjusted according to the instructions for adjusting the teaching.

Images