Method and System for Analyzing the Emotional Evolution of Learners Based on Causal Graph Neural Network

Abstract

The beneficial effects of the method and system for analyzing the emotional evolution of learners based on causal graph neural networks provided by the embodiments of the present application are as follows: The present application designs a multi-level emotional association semantic description framework based on "learner-activity-emotion" to obtain multi-modal time-series data of learners; a cross-modal time-series fusion method based on Z-Transformer is used to fuse the multi-modal time-series data; according to the fused time-series data of learners, an emotional structure causal model for fine-grained emotional evolution of learners is constructed; according to the emotional structure causal model, an intervention variational graph auto-encoder is used to establish an emotional heterogeneous causal graph; according to the emotional heterogeneous causal graph, a causal graph neural network is constructed to analyze the emotional evolution of learners. The present application can dynamically track the real emotional state and its trend of learners, deeply analyze the spatio-temporal causal relationship of learners' emotional changes, realize the deep revelation of the emotional changes and their traceability laws of learners in the classroom, and help with precise teaching in the classroom.

Description

Technical Field

[0001] This application relates to the field of graph neural networks, and in particular to a method and system for analyzing the emotional evolution of learners based on causal graph neural networks. Background Art

[0002] As the main body of the classroom, the emotions of learners are the inner experiences and corresponding external manifestations generated during learning and cognitive activities. They are the key to improving the quality and efficiency of education and teaching, and also an important basis for improving the quality of education. Artificial intelligence technology is an important driving engine for the innovation of high-quality classroom teaching, and plays a significant role in achieving the "emotional attitude and values" among the three-dimensional goals of classroom teaching and cultivating all-round quality talents. However, the emotions of classroom learners present characteristics such as the difficulty in perceiving emotions under the complexity of the interpersonal interaction network, the dynamic variability of emotions under the high degree of human-machine collaboration, and the multi-source correlation of emotions under the group intelligence of classroom activities. All these lead to the difficulty in obtaining and understanding the emotions throughout the classroom process, seriously hindering the improvement of the efficiency and effectiveness of classroom teaching and learning, as well as the all-round development of learners. Therefore, taking the classroom as a support environment, under the support of the new generation of artificial intelligence, a method for timely perceiving the emotional state and evolution trend of learners throughout the classroom process, understanding the dynamic complex spatio-temporal causal relationship of multi-dimensional elements of emotions, and realizing the in-depth revelation of the emotional changes of classroom learners and their traceability laws is helpful to promote the improvement of the quality and efficiency of the classroom.

[0003] Learner emotion analysis is to simulate human intelligence, with the ability to update and optimize data, and can identify, understand, express, and adapt to the emotions of learners in combination with specific situations, and continuously self-upgrade to approach the best decision-making of people. The existing related technologies for emotion analysis mainly include: based on the facial expression data of students, using AdaBoost with Haar-like features to mark the faces of students, constructing a convolutional neural network model to identify the learning emotions of students as positive emotions and negative emotions and feedback them to teachers; or, by extracting signals such as eye movements and heart rates that are most relevant to the emotional state of learners, fusing deep and shallow features, and proposing machine learning methods such as long short-term memory networks to identify four emotions: interested, confused, bored, and happy. The main disadvantages of the related technologies mainly include: (1) Collecting multi-modal homogeneous information such as faces, eye movements, and physiological levels for emotion analysis, without considering various types of heterogeneous information related to emotions such as the differences among learners (gender, hobbies, cognitive levels, etc.) and multi-dimensions of emotions (polarity, intensity, etc.) throughout the classroom teaching process, and unable to comprehensively perceive the fine-grained emotional state of classroom learners; (2) Only classifying the static emotional state of learners, ignoring the temporal emotional evolution in complex teaching and learning scenarios, and unable to achieve in-depth learner emotion analysis and attribution; (3) Collecting data on the dynamic changes of facial expressions to identify only the emotions of learners at different times, without analyzing the temporal dynamic emotional evolution trend; (4) Not analyzing the causal relationship that triggers the emotional changes of learners, unable to make accurate traceability and give targeted interventions, and even producing negative effects.

[0004] Therefore, the above technical problems existing in the related art need to be solved urgently. Summary of the Invention

[0005] This application aims to solve one of the technical problems in the related art. To this end, an embodiment of this application provides a learner emotion evolution analysis method and system based on a causal graph neural network with high accuracy and robustness. Starting from the perspective of emotions and their changes under the complexity of the interpersonal interaction network presented by learner emotions, using multi-modal data associated with learner emotions, learning the complex spatio-temporal causal relationships of learners' dynamic emotions, analyzing the trend of learner emotion evolution, and assisting in precise teaching in the classroom.

[0006] According to one aspect of the embodiments of this application, a learner emotion evolution analysis method based on a causal graph neural network is provided. The method includes:

[0007] Design a multi-level emotion-associated semantic description framework based on "learner-activity-emotion" to obtain multi-modal time-series data of learners;

[0008] Based on the cross-modal time-series fusion method of Z-Transformer, fuse the multi-modal time-series data;

[0009] According to the fused learner time-series data, construct an emotion structure causal model for fine-grained emotion evolution of learners;

[0010] According to the emotion structure causal model, use an intervention variational graph autoencoder to establish an emotion heterogeneous causal graph;

[0011] According to the emotion heterogeneous causal graph, construct a causal graph neural network to analyze the emotion evolution of learners.

[0012] In one of the embodiments, the design of a multi-level emotion-associated semantic description framework based on "learner-activity-emotion" to obtain multi-modal time-series data of learners includes:

[0013] Construct a learner description layer: Learners and teachers are the main body and the dominant in classroom learning. This layer involves interaction relationship elements such as "learner-teacher-learning content", etc. Among them, on the learner side, individual factors such as learners' classroom emotions, ages, genders, and homework completion degrees are included. Formalize the above factors and construct a learner instance set S = {s1,..., s n}; on the teacher side. Related factors such as their ages, genders, and professional titles will also affect learners' emotions. Construct a teacher factor instance set D = {d1,..., d n}; in addition, learning content factors such as the relevant disciplines and difficulties of the course, as a bridge connecting learners and teachers, also affect learners' emotions. Construct a course factor instance set C = {c1,..., cn};

[0014] Construct the activity layer description: Based on the existing factors at the learner level, in activities such as classroom teaching and learning activities, learner participation activities include classroom listening, course reading, peer communication, classroom discussion, answering questions, etc., and teacher participation activities include teacher's course reading, classroom questioning, etc. Collect and quantify the data of classroom activity factors to construct a set of classroom activity instances A = {a1,…,a n};

[0015] Construct the emotion layer description: Based on the learner level and the activity layer, perceive the time-ordered classroom teaching and learning process. The learner's emotional state is affected by the data in the above two-layer sets. Define the learner's emotional state as Define the learner's emotional evolution trend as The emotional type state of learner i is represented as a triple where h t represents the learner's emotional state at time t, h p is the emotional polarity of the learner, and h q is the emotional intensity of the learner. The interaction throughout the classroom teaching and learning activities stimulates the emotional changes in the learner's cognitive process. The evolution trends of these emotions are characterized from two aspects. Then the learner's emotions have four evolution trends: negative rising, negative falling, positive rising, and positive falling. Therefore, define the learner's emotional evolution as a triple where represents the emotional evolution trend of learner i at time t, y p is the emotional evolution polarity of the learner, including two states: positive 1 and negative -1, and y q is the change in the learner's emotional intensity, including two changes: rising 1 and falling -1.

[0016] In one embodiment, the cross-modal time-series fusion method based on Z-Transformer fuses the multi-modal time-series data, including:

[0017] According to the semantically processed set of learner emotion correlation instances and their corresponding multi-modal time-series data such as images, audio, and text, construct a cross-modal attention fusion model. Input the multi-modal data into the cross-modal attention fusion model, and align the time series of the fused data and input it into the recurrent network function stacked with cross-modal multi-head cross-self-attention;

[0018] Among them, the input set of the modal time-series data fusion model for images, audio, and text is Z ∈ M, and the cross-modal time-series data fusion process is:

[0019]

[0020]

[0021] In the formula, Z is the input set defined by the multi-modal data fusion model, M is the set of data modal types, etc. are the sets of sentiment association instances under different modalities, Transformer is the self-attention mechanism, θ is the parameter, LN is the normalization process, Z' is the data of the fused temporal sentiment association instance set, and η t represents the diffusion factor at time t, σ is the activation function, and U z∈M is the traversal symbol, and [] is the feature splicing.

[0022] In one of the embodiments, constructing an emotion structure causal model for fine-grained emotion evolution of learners based on the fused learner temporal data includes:

[0023] Aiming at the dynamic complex spatio-temporal causal characteristics of multi-dimensional elements of learners' emotions, through the fused learner temporal data set, the emotion structure causal model SSCM is defined as a quadruple <U, V, f, P(U)>, including exogenous variable U, endogenous variable set V, a set of functional equations f, and a probability function P(U) defined on the domain of U;

[0024] In order to construct an accurate emotion structure causal model and remove the confounding effects of confounding factors, it is necessary to perform counterfactual reasoning on the causal effects of learners' emotions. By performing counterfactual reasoning do(x) on the link relationship of emotion instances, the causal relationship estimation result P of this emotion state is obtained (θ) , when the observation value set V i exists and the emotion changes, it is 1, and when there is no emotion change, it is 0, and the emotion relationship effect estimation inference function is calculated to eliminate confounding variables;

[0025] Finally, based on the counterfactual calculation results, the mathematical formula for constructing the emotion structure causal model through emotion causal constraints is:

[0026]

[0027] In the formula, g is the constructed directed acyclic graph, and G s is the graph structure data implicitly included in SSCM, i is the emotion state node, and f i is the set of inference functions, Z i is a set of causal variable instances, U i is the set of noise variables, pa is the parent node of the emotion node i, and V i is the observation value set, v is the parameter range, and u is a single noise variable, is the set of relational meta-path, and σ is the sigmoid activation function, The path corresponding to the i emotional state node, j is the meta-path weight parameter corresponding to the i emotion, and Gumbel is a random variable.

[0028] In one embodiment, according to the emotional structure causal model, an intervention variational graph autoencoder is used to establish an emotional heterogeneous causal graph, including:

[0029] According to the above emotional structure causal model, the interconnection of multiple variables constitutes the heterogeneity of the information related to the emotions of classroom learners. In this heterogeneous emotional causality, multiple types of learners and their relationships coexist, and rich structural and semantic information is contained in different emotional relationship meta-paths. Using the intervention variational graph autoencoder, the graph structure data implicitly contained in the emotional causal model SSCM of the classroom learning process is reconstructed;

[0030] The adjacency matrix of the reconstructed temporal SSCM graph is input into the input of the encoder. Through the intervention variational graph autoencoder, the structural neighbors and attributes of the graph network nodes are calculated to learn the representations of the graph network nodes, and the emotional heterogeneous causal graph structure data set is output through the GNN decoding layer for different node data in the temporal state.

[0031] In one embodiment, according to the emotional heterogeneous causal graph, a causal graph neural network is constructed to analyze the emotional evolution of learners, including:

[0032] Embed the nodes of the learner's emotional heterogeneous causal graph;

[0033] Calculate the emotional causal attention of learners under the support of long-term and short-term multi-scale time series, and capture the multi-scale evolution characteristics and causal relationship characteristics of learners' emotions respectively;

[0034] Analyze the emotional state and its change trend of learners through graph convolution update operation.

[0035] In one embodiment, embedding the nodes of the learner's emotional heterogeneous causal graph includes:

[0036] Embed the nodes of the learner's emotional heterogeneous causal graph, construct causal node attention to learn the weights based on meta-path neighbors; make the final node embedding according to the weights obtained by repeating the preset number of times for all meta-paths of the heterogeneous causal graph, so that the nodes with causal relationships in the graph are still similar in the embeddings of different feature spaces.

[0037] In one embodiment, the calculation of the emotional causal attention of learners under the support of long-term and short-term multi-scale time series, which captures the multi-scale evolution characteristics and causal relationship characteristics of learners' emotions respectively, includes:

[0038] After embedding the heterogeneous graph at a specific moment, the weighted average of all learners' short-term feature vectors of emotions is calculated, and the weighted average sum of multi-scale feature vectors is calculated to obtain the long-term feature vector of learners' emotions;

[0039] According to the above-extracted short-term and long-term features, the multi-scale sampling results are incorporated into the feature-level attention calculation by using normalized weighted geometric mean approximation, the causal relationship between the short-term features and the long-term features is calculated, and a causal feature vector matrix is obtained;

[0040] In one embodiment, the graph convolution update operation analyzes the emotional state of the learner and its changing trend, including:

[0041] According to the above-learned causal feature vector matrix, the state of the learner and its changing trend at a certain moment are output through the graph convolution update operation, and the operation is as follows:

[0042]

[0043]

[0044] In the formula, i is the emotional state node, is the emotional state at time t, is the emotional evolution state of the i-th emotional node at time t, is the learned causal feature vector matrix, W l is the linear transformation weight, b l is the bias term, and σ and ReLu are activation functions, j is the meta-path weight parameter corresponding to the i-th emotion, v is the parameter range, is the mapping function, is the weight information of the heterogeneous neighbor nodes.

[0045] According to one aspect of the embodiments of the present application, a learner emotional evolution analysis system based on a causal graph neural network is provided, and the system includes:

[0046] An acquisition module, configured to design an emotional association semantic description framework based on a multi-level "learner-activity-emotion" to obtain multi-modal time-series data of the learner;

[0047] A fusion module, configured to fuse the multi-modal time-series data based on the cross-modal time-series fusion method of Z-Transformer;

[0048] A first construction module, configured to construct an emotional structure causal model for the fine-grained emotional evolution of the learner according to the fused time-series data of the learner;

[0049] A second construction module, according to the emotional structure causal model, uses an intervention variational graph autoencoder to establish an emotional heterogeneous causal graph;

[0050] An analysis module, configured to construct a causal graph neural network to analyze the emotional evolution of the learner according to the emotional heterogeneous causal graph.

[0051] The beneficial effects of the learner emotion evolution analysis method and system based on causal graph neural network provided by the embodiment of the present application are as follows: the present application designs a multi-level emotion association semantic description framework based on "learner-activity-emotion" to obtain learner multimodal time series data; based on the cross-modal time series fusion method of Z-Transformer, the multimodal time series data is fused; based on the fused learner time series data, an emotion structure causal model for learner fine-grained emotion evolution is constructed; based on the emotion structure causal model, an emotion heterogeneous causal graph is established using the intervention variational graph autoencoder; based on the emotion heterogeneous causal graph, a causal graph neural network is constructed to analyze learner emotion evolution. The present application can dynamically track learners' real emotional states and trends, deeply analyze the spatiotemporal causal relationships of learners' emotion changes, realize the in-depth revelation of learners' emotion changes and their traceability laws in the classroom, and help to accurately teach in the classroom.

[0052] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A flowchart of a method for analyzing learner emotion evolution based on a causal graph neural network provided in an embodiment of the present application;

[0055] Figure 2 A learner emotion category and evolution representation diagram provided in the embodiment of the present application;

[0056] Figure 3 Establishing an emotional heterogeneous causal graph for the intervention variational graph autoencoder provided in the embodiment of the present application;

[0057] Figure 4 The causal graph neural network provided in the embodiment of the present application analyzes the learner's emotional evolution;

[0058] Figure 5 The long-short-time multi-scale time series feature calculation provided in the embodiment of the present application;

[0059] Figure 6 Causal attention calculation provided for embodiments of the present application. DETAILED DESCRIPTION

[0060] To enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0061] The terms "first", "second", "third", "fourth", etc. in the description, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0062] Referring to "embodiments" herein means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of this application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0063] As the main body of the classroom, the emotions of learners are the inner experiences and corresponding external manifestations generated during learning and cognitive activities. They are the key to improving the quality and efficiency of education and teaching, and also an important basis for enhancing the quality of education. Artificial intelligence technology is an important driving engine for the innovation of high-quality classroom teaching, and plays a significant role in achieving the "emotional attitude and values", one of the three-dimensional goals of classroom teaching, and cultivating all-round quality talents. However, the emotions of classroom learners present characteristics such as the difficulty in perceiving emotions under the complexity of the interpersonal interaction network, the dynamic variability of emotions under the highly collaborative process between humans and machines, and the multi-source correlation of emotions under the group intelligence of classroom activities. All these lead to the difficulty in obtaining and understanding the emotions throughout the classroom process, seriously hindering the improvement of the efficiency and effectiveness of classroom teaching and learning and the all-round development of learners. Therefore, taking the classroom as the support environment, a method for timely perceiving the emotional state and evolution trend of learners throughout the classroom process is proposed under the support of the new generation of artificial intelligence, understanding the dynamic complex spatio-temporal causal relationship of multi-dimensional emotional elements, and realizing the in-depth revelation of the emotional changes of classroom learners and their traceability laws, which helps to promote the improvement of the quality and efficiency of the classroom.

[0064] Learner emotion analysis is the ability to simulate human intelligence, update and optimize data, and recognize, understand, express, and adapt to learners' emotions in specific situations, continuously self-upgrading to approach the best human decision-making. Existing emotion analysis technologies mainly include: Based on students' facial expression data, using AdaBoost with Haar-like features to mark students' faces, constructing a convolutional neural network model to identify students' learning emotions as positive and negative emotions and feedback to teachers; or, by extracting signals such as eye movements and heart rates most relevant to learners' emotional states, fusing deep and shallow features, and proposing machine learning methods such as long short-term memory networks to identify four emotions: interested, confused, bored, and happy. The disadvantages of related technologies mainly include: (1) Collecting multi-modal homogeneous information such as faces, eye movements, and physiological levels for emotion analysis, without considering various types of heterogeneous information related to emotions such as learners' differences (gender, hobbies, cognitive levels, etc.) and multi-dimensional emotions (polarity, intensity, etc.) involved in the whole process of classroom teaching, unable to comprehensively perceive the fine-grained emotional states of classroom learners; (2) Only classifying learners' static emotional states, ignoring the temporal emotional evolution in complex teaching and learning scenarios, unable to achieve in-depth learner emotion analysis and attribution; (3) Collecting data on the dynamic changes of learners' facial expressions for emotion recognition only at different times, without analyzing the trend of temporal dynamic emotion evolution; (4) Not analyzing the causal relationships that trigger learners' emotional changes, unable to make accurate traceability and give targeted interventions, and even producing negative effects.

[0065] To solve the above problems, this application proposes a learner emotion evolution analysis method and system based on causal graph neural networks.

[0066] Figure 1 The flowchart of the learner emotion evolution analysis method based on causal graph neural networks provided by the embodiments of this application is as Figure 1 shown. This application provides a learner emotion evolution analysis method based on causal graph neural networks, including:

[0067] S101. Design of an emotion-related semantic description framework based on the "learner-activity-emotion" multi-level.

[0068] S102. Cross-modal temporal data fusion based on Z-Transformer.

[0069] S103. Construction of an emotion structure causal model for learners' fine-grained emotion evolution.

[0070] S104. Establishment of an emotion heterogeneous causal graph supported by an intervention variational graph autoencoder.

[0071] S105. Learner emotion evolution analysis based on causal graph neural networks.

[0072] Next, the main definitions of this application are introduced:

[0073] (1) Learner emotion categories and evolution trends

[0074] Learner emotion is an important implicit feature of classroom activities. In the complex classroom environment of teaching and learning, it has a series of attributes, including emotion type, emotion polarity, emotion intensity, and emotion evolution trend, etc. To accurately mine learners' emotions, as Figure 2 shown, this application constructs a learner emotion category and its evolution representation diagram based on an academic emotion scale. Considering that diverse emotion classifications will convey different emotions with different polarities, the emotion classification of learners is further refined. According to the emotions experienced in the cognitive interaction process, this application divides the fine-grained emotions of learners into two dimensions and 20 types, namely positive and negative emotion polarities p and high and low emotion intensities q. The value range of each dimension is from -1 to 1, and each emotion has a two-dimensional emotion state quantization value. For example, the emotion state value (p, q) of "relaxed" is (0.5, -0.2). The emotion type state of learner i is represented as a triple where h t represents the learner's emotion state at time t, h p is the learner's emotion polarity, and h q is the learner's emotion intensity. The interaction throughout the whole process of classroom teaching and learning activities stimulates the emotional changes in the learner's cognitive process. The evolution trends of these emotions are characterized from two aspects. Then, there are four evolution trends for learner emotions: negative rising, negative falling, positive rising, and positive falling. Therefore, the learner emotion evolution is defined as a triple where represents the emotion evolution trend of learner i at time t, y p is the emotion evolution polarity of the learner, including two states: positive 1 and negative -1, and y q is the change in the learner's emotion intensity, including two changes: rising 1 and falling -1.

[0075] (2) Heterogeneous information related to classroom learner emotions

[0076] The emotion-related elements of classroom learners are multi-source and complex, and various types of learners have diverse interactions through teaching activities. In this process, learners experience different emotion intensities, polarities and fine-grained emotional states. It can be seen that the interconnection of multiple elements in the classroom constitutes the heterogeneity of information related to the emotions of classroom learners. Multiple types of learners and their relationships coexist in this heterogeneous emotional network, and different emotional relationship meta-paths contain rich structural and semantic information, which is convenient for capturing complex high-order semantic relationships in heterogeneous emotional association information, thereby providing a new and accurate and explainable way to discover implicit emotional causal patterns. Therefore, this application analyzes the heterogeneous information set composed of the interaction of a large number of emotion-related objects of different types in the classroom, and adopts multi-level emotional causal semantic description analysis such as learners-teaching activities-emotions. It is assumed that the emotion-related heterogeneous information of classroom learners is composed of learner i's differentiated related attributes, including learner factor set S, teacher factor set D, course factor set C and teaching activity set A and their interaction relationships. The above objects and their relationships constitute the classroom learner emotion heterogeneous network set Each object v belongs to the object set V, E is the relationship link set, T represents the time series, is the mapping function, each linked path is called a meta-path e∈E, which constitutes a set of sentiment relations A specific emotional causal relationship type weight value is defined in the node, and a node can establish multiple relationship links at different times.

[0077] The design of the emotion-related semantic description framework based on the multi-level "learner-activity-emotion" in step S101 specifically includes:

[0078] In the complex teaching and learning classroom process, relevant tools and questionnaires are used to collect multimodal time-series data related to classroom activities and learners. Due to the large number of emotional factors related to classroom learners, different emotional association meta-paths contain rich structural and semantic information. In order to capture the complex high-order semantic relationships in heterogeneous emotional association information, this application starts from the perspective of the causal relationship between the emotional influence of time-series activities in the classroom learning process, and uses the perceptible semantic mechanism of classroom learning activity process data to analyze the hierarchical related factors in the current classroom data. The emotional factors of classroom learners are designed based on the three-level semantic description framework of "learner-activity-emotion". The three-level framework can be formally described as:

[0079] person(student,teacher),attributes.collect{S,D,C}……(1)

[0080] action(S,D,activities,t),egread:{SA,DA}......(2)

[0081]

[0082] Among them, Equation (1) describes the learner level, where "person" represents the learner-related entity, and "attributes.collect" represents attribute collection. Equation (2) analyzes the activity level, where "action" represents the activity, e.g., "read" represents obtaining the activity data related to the learner's teacher. Equation (3) is for the emotional level. To implement this description framework, it is necessary to establish a set of instances of relevant influencing factors based on the data of the emotional correlation factors of learners in the teaching and learning process.

[0083] In the learner level, the learner and the teacher respectively constitute the main body and the leading role in classroom learning, which is closely related to the learner's emotions in the classroom. Semantic analysis and quantification of the individual emotions of learners are carried out. The classroom emotions of learners are closely related to individual factors such as the learner's age, gender, and homework completion... Collect the data of learner factors, quantify them, and construct the set of learner factors S = {s1,..., s n}. Factors related to the teacher, such as age, gender, and professional title, will also have an impact on the learner's emotions. Collect the time-series data of teacher factors, quantify them, and construct the time-series set of teacher factors D = {d1,..., d n}. In addition, factors related to the course, such as subject and difficulty..., as a bridge connecting learners and teachers, also affect the learner's emotions. Collect the relevant data, quantify them, and construct the time-series set of course factors C = {c1,..., c n}.

[0084] In the activity level, the self-efficacy theory provides a basis for the learner's participation in classroom learning behaviors. Based on the existing factors at the learner level, in classroom teaching and learning activities and other behaviors, there are learner participation activities such as classroom listening, course reading, peer communication, classroom discussion, answering questions..., and teacher participation activities such as teacher's course reading and classroom questioning. Collect the time-series data of activities, quantify them, and construct the time-series set of classroom activities A = {a1,..., a n}.

[0085] On the basis of the learner level and the activity level, the emotional level perceives the time-series classroom teaching and learning process. The emotional state of the learner is affected by the data in the above two-level sets. Define the learner's emotional state as Define the evolution trend of the learner's emotion as The emotional type state of learner i is represented as a triple Among them, h t represents the emotional state of the learner at time t, h p is the emotional polarity of the learner, h qis the emotional intensity of the learner. The interaction throughout the whole process of classroom teaching and learning activities stimulates the emotional changes in the cognitive process of the learner. The evolutionary trends of these emotions are characterized from two aspects. Then, there are four evolutionary trends of the learner's emotion: negative increase, negative decrease, positive increase, and positive decrease. Therefore, the learner's emotional evolution is defined as a triple where represents the emotional evolution trend of learner i at time t, and y p is the polarity of the learner's emotional evolution, including two states: positive 1 and negative -1. y q is the change in the learner's emotional intensity, including two changes: increase 1 and decrease -1.

[0086] The cross-modal temporal data fusion based on Z-Transformer in step S102 specifically includes:

[0087] Construct a cross-modal attention fusion model, and input the multi-modal data into the cross-modal attention fusion model; Align the time series of the fused data and input it into the recurrent network function of the cross-modal multi-head cross self-attention stack.

[0088] First of all, usually traditional emotion computing models often analyze and predict emotional states from the perception and input of single-modal data. However, the elements related to the emotions of classroom learners are multi-source and complex, with a large amount of learnable semantic information, and include multi-modal temporal data such as images, audio, and text. Based on the collected data set of semantic learner emotion association instances, it is necessary to clean, process, fuse, and unify these multi-modal temporal data with a large number of semantic information variables.

[0089] This application constructs a cross-modal temporal fusion method based on Z-Transformer, uses the self-attention mechanism to connect multi-modal data, calculates the correlation calculation function, and defines the input set of the model by fusing modal data such as images, audio, and text as Z ∈ M, where etc. are sets of emotion association instances in different modalities. Z represents the modality containing the learner emotion association set data in the T period, and M is the data modality type including images, audio, and text. Cross-modal cross-attention fusion enhancement is adopted, and the correlation degree between any two modal data at time t can be calculated through the residual connection and joint optimization normalization of the random gated neural network. The cross-modal attention fusion module is designed as:

[0090]

[0091] In the formula, Transformer is the self-attention mechanism, θ is the parameter, LN is the normalization process, and the time series aligned data in different periods is input into the recurrent network function of the cross-modal multi-head cross self-attention stack:

[0092]

[0093] Among them, Z′ is the data of the set of fused sentiment association instances, and η t represents the diffusion factor at time t, σ is the activation function, [] is feature concatenation, U is the traversal symbol, and the cross-modal time-series data synchronization fusion of the classroom learner sentiment association instance set is completed by means of the above method.

[0094] The construction of the sentiment structure causal model for the fine-grained sentiment evolution of learners in step S103 specifically includes:

[0095] For the dynamic complex spatio-temporal causal characteristics of the multi-dimensional elements of learners' sentiment, through the fused learner time-series data set, this application constructs the classroom learner sentiment association instance set into a sentiment structure causal model (SentimentStructural Causal Model, SSCM), which contains the graph structure data relationship determined by the set of structural equations implicitly controlled, and carries an ordered quadruple <U, V, f, P(U)> of classroom sentiment instance variables and their causal relationships. U is an exogenous variable, V is a set of endogenous variables {V 1, V 2, …, V n}, f is a set of functional equations {f 1, f 2, …, f n}, through f, the relationship weights of endogenous variables can be deduced from exogenous variables for the fused modal data, and P(U) is a probability function defined on the domain of U, and the SSCM can be constructed by counterfactual reasoning of the instance set through the fused learner time-series data set.

[0096] In the set of learner sentiment association instances, there is a set of observed endogenous variable sets V related to the vertices of the directed acyclic graph. All learner sentiment-related variables can be divided into dependent variables and causal variables. The causal variables are the classroom learner sentiment states, and the dependent variables are the factors causing the changes in learners' sentiment, which are variables that can be intervened. The noise variables refer to the factors irrelevant to the learners' sentiment relationship, and the noise variables can be removed through randomized experiments. Each sentiment state i is composed of a set of causal variable instances Z i and a set of noise variables U i , and each observed value is obtained according to the following formula:

[0097] V i = f i {Z i , U i}, (i = 1, …, n)……(6)

[0098] A randomized experiment is conducted on a set of causal variable instances to obtain a probability distribution. The randomized experiment adopts three methods: Bernoulli randomization, complete randomization, and stratified randomization. Causal inference is used to calculate the causal effect from observational data. Causal analysis constructs a directed acyclic graph. To determine the independence of some potential outcomes in the directed acyclic graph with respect to other variable relationships, the d-separation algorithm is needed. A directed graph of all variables in the probability expression is constructed. The parent nodes of each node are connected pairwise, and the directed edges are replaced with undirected edges. If there is any given variable in the independence problem, the variable and all its connections are deleted from the directed acyclic graph. If there is no path between the variables to be judged in this graph, they are independent. The emotional causal relationship is calculated by combining the instance set relationship with counterfactual intervention reasoning:

[0099]

[0100]

[0101] Among them, P (θ) represents the estimated result of the learner's emotional causal relationship, θ represents the emotional data reasoning parameter, do(x) is the counterfactual intervention parameter, g is the constructed directed acyclic graph, is the effect estimation reasoning function, pa is the parent node of this node, u is the noise variable, is the set of relational meta-paths, and σ is the sigmoid activation function. By constructing a directed acyclic graph for the instance set and performing counterfactual reasoning do(x) on the link relationship, the estimated result P (θ) of the causal relationship of this emotional state is obtained. When the observed value set V i has an emotional change, it is 1; when there is no emotional change, it is 0. The effect estimation reasoning function of this emotional relationship is calculated to eliminate the confounding variables. Finally, based on the counterfactual calculation results, an emotional structural causal model is constructed through emotional causal constraints:

[0102]

[0103] The SSCM quadruple is obtained Among them, G S is the graph structure data implicitly included in the SSCM, is the path corresponding to the node in the i emotional state, j is the meta-path weight parameter corresponding to the i emotion, and f i is the set of reasoning functions, and Gumbel is a random variable.

[0104] In step S104, the establishment of the sentiment heterogeneous causal graph supported by the intervention variational graph autoencoder includes: using the intervention variational graph autoencoder (IVGAE) to reconstruct the graph structure data implicitly contained in the sentiment causal model of the classroom learning process (SSCM); calculating the structural neighbors and attributes of the graph network nodes to learn the representation of the graph network nodes, and constructing the sentiment heterogeneous graph structure data through the output of the GNN decoding layer.

[0105] According to the above sentiment structure causal model, the interconnection of multiple variables constitutes the heterogeneity of the information related to the emotions of classroom learners. In this heterogeneous sentiment causality, multiple types of learners and their relationships coexist, and rich structural and semantic information is contained in different sentiment relationship meta-paths. SSCM leads to a graph structure data G contained in a set of structural equations that implicitly control the causal relationships between the variables to be modeled. S In recent years, emerging deep learning methods represented by the Graph Neural Network (GNN) have the ability to efficiently process complex graph structure data. However, the changes in topology and node attributes in the temporal graph structure data contained in the temporal SSCM cannot be well captured and learned by the graph neural network. It is necessary to reconstruct the graph data to learn its effective features, while ensuring that the properties of the graph data can also be corresponding in the vector space.

[0106] For this reason, this application uses the intervention variational graph autoencoder (Interventional Variational Graph Auto-Encoders, IVGAE) to reconstruct the graph structure data implicitly contained in the sentiment causal model of the classroom learning process (SSCM). Its essence is to find suitable Embedding vectors for the nodes in the graph, and use the encoder and decoder to reconstruct the graph structure sample data. For the graph structure data G contained in the learner sentiment causal relationship SSCM constructed from the classroom learning process data S Encoding and reconstruction are as Figure 3 shown.

[0107] Through IVGAE, the adjacency matrix A of the temporal SSCM graph G can be input into the input of the encoder. The encoder is composed of a graph neural network. Through the encoder layer, the low-dimensional vector representation μ and variance ω of the nodes are learned, and the probability distribution function f is calculated θ :

[0108] GNN(G S , A G ) = D -1 / 2 A G D -1 / 2 RELU(D -1 / 2 A G D -1 / 2 G S W)……(10)

[0109] μ = GMN (G S , A G )……(11)

[0110] logω = GNN ω (G S , A G )……(12)

[0111]

[0112] For the feature mapping space, perform the mapping:

[0113] F = ∑ i f θ (U ij , V ij )……(14)

[0114] where GNN refers to the graph neural network as the encoder, G S is the graph structure implicit in the sentiment structure causal model, A G is the adjacency matrix of graph G S , D -1 / 2 is the degree matrix, RELU is the activation function, and F is the mapped feature space.

[0115] The decoding process randomly samples from each latent state distribution as the input to the decoder layer, calculates the link prediction through the probability distribution function, and pairwise calculates the probability of the existence of an edge between two points to reconstruct the graph, thereby generating new sentiment heterogeneous graph structure data G:

[0116] g θ = f θ σ(F i U ij )……(15)

[0117]

[0118] g θ is the matrix calculation decoding function, σ is the sigmoid activation function, f ij is the node meta-path probability calculation function, which calculates the structure U i of the sentiment network nodes and the attributes of neighbors U j to intervene in the sentiment node representation of the mapped feature space. P(G|U) represents the graph structure data constructed under the support of sentiment node intervention. Finally, the different node data in the time series state are output through the GNN decoding layer as the sentiment heterogeneous causal graph structure data set G.

[0119] Optionally, in step S105, the analysis of the learner's emotional evolution based on the causal graph neural network includes: embedding the heterogeneous causal graph nodes of the learner's emotions; calculating the emotional causal attention of the learner under the support of long-short-term multi-scale time series, respectively capturing the multi-scale evolution characteristics and causal relationship characteristics of the learner's emotions; and analyzing the learner's emotional state and its change trend through graph convolution update operations.

[0120] It should be noted that the embedding of the heterogeneous causal graph nodes of the learner's emotions includes: embedding the heterogeneous causal graph nodes of the learner's emotions, constructing the weights of the causal node attention learning based on the meta-path neighbors; making the final node embedding according to the weights obtained by repeating the preset number of times for all meta-paths of the heterogeneous causal graph, so that the nodes with causal relationships in the graph are still similar in the embeddings of different feature spaces;

[0121] The calculation of the emotional causal attention of the learner under the support of long-short-term multi-scale time series, respectively capturing the multi-scale evolution characteristics and causal relationship characteristics of the learner's emotions, includes: after embedding the heterogeneous graph at a specific moment, calculating the weighted average of all short-term feature vectors of the learner's emotions, and summing the weighted average of the multi-scale feature vectors to obtain the long-term feature vector of the learner's emotions; according to the above-extracted long and short-term features, using the normalized weighted geometric mean approximation to integrate the multi-scale sampling results into the feature layer attention calculation, calculating the causal relationship between the short-term feature and the long-term feature, and obtaining the causal feature vector matrix.

[0122] This application uses the obtained heterogeneous causal graph data set of the classroom learner's emotional causal relationship Combined with the graph neural network technology, a causal graph neural network is constructed to analyze the learner's emotional evolution, and the learner's emotional state and evolution trend are obtained, such as Figure 4 As shown, specifically, the analysis of the learner's emotional evolution based on the causal graph neural network includes:

[0123] (1) Embedding the heterogeneous causal graph nodes of the learner's emotions

[0124] First, it is necessary to perform node embedding on the heterogeneous causal graph at different moments, aiming to map each node in the graph to a low-dimensional vector representation. The traditional local linear embedding and Laplacian eigenmap embedding methods are mainly based on the calculation of matrix eigenvectors. Since the above two methods are too complex and not suitable for large-scale causal graph data containing complex semantic information, this application uses heterogeneous causal node embedding, so that the nodes with causal relationships in the graph are still similar in the embeddings of the feature space. The learner's emotional time-series heterogeneous graph data set is represented as Composed of the object set V and the link set E, and also associated with the node link relationship mapping function The path of each causal link node pair (i, j) linked during the classroom activity and learner link process is called a meta-path The initial node feature is s i and the node feature after feature engineering is s i ’. The neighbors of node i based on the meta-path are Since the nodes of the heterogeneous causal graph have different feature spaces, we define a specific type of transformation matrix to transform the features of different types of nodes and project them onto the same feature space. This application uses causal node-level attention to learn the weights of the neighbors based on the meta-path, and catt node represents the deep neural network of causal node attention. The node similarity function that aggregates to obtain the causal relationship semantic node embedding is

[0125]

[0126]

[0127]

[0128] After repeating the learning for all meta-paths of the heterogeneous causal graph to obtain the causal relationship semantic node similarity function, the heterogeneous causal graphs at different times are used for the final node embedding:

[0129]

[0130] Obtain a set of feature vector matrices

[0131] (2) Calculate the emotional causal attention of the learner under the support of long-short-term multi-scale time series, and capture the multi-scale evolution features and causal relationship features of the learner's emotion respectively

[0132] The emotional evolution of the learner during the classroom process is a dynamic process, and its core lies in how to effectively capture the short-term and trial cycle time series laws in the emotional sequence and consider the interdependence between multiple activities. Since there will be interactions between multiple types of activities in the classroom, such as listening, collaboration, discussion, etc., the emotional changes of the learner will be affected by the time series in different periods, showing different data representations. For example, predicting the emotion and its changes during the collaborative interaction process of the learner in the classroom environment can provide useful information from the directly adjacent time period (such as the previous collaborative interaction period), the same activity period of the corresponding previous class, and the same time period a week ago, while the data of the teaching period in this class provides much less information. In order to obtain sufficient time series information and reduce the influence of irrelevant historical information, this application proposes a long-short-term multi-scale time series mechanism and a causal attention mechanism to capture the multi-scale evolution features and causal relationship features of the learner's emotion respectively.

[0133] Given that the constructed heterogeneous graph time-series data features contain a large number of learnable representations of emotional evolution, therefore, after the heterogeneous graph is embedded at a specific moment in this application, the weighted average calculation is performed on all short-term feature vectors, and the multi-scale feature vector X t ∈R n is summed by weighted average to obtain the long-term feature vector X s , as shown in Figure 5 .

[0134] X s =[(X t0 +X t1 )+…+(X t(n-1) +X tn )] / 2n……(21)

[0135] After obtaining the long-term features, in the process of emotional evolution analysis at each specific moment, the short-term features and long-term features are fused to construct the calculation of emotional causal attention. The traditional attention mechanism can be summarized as the linear transformation matrix operation of Query-Key-Value. V (Value) is the vector representing the input features, and Q (Query) and K (Key) are the feature vectors for calculating the causal weights. Due to the heterogeneity of nodes, different types of nodes have different feature spaces. Through the causal effect mediator, the short-term and long-term features of the input set are transmitted to the target set. In this application, the causal attention is integrated into the causal graph neural network task, and the normalized weighted geometric average approximation is used to incorporate the multi-scale sampling results into the feature layer attention calculation, and the causal relationship between the short-term features and long-term features is calculated. In the causal attention function supported by short-term and long-term multi-scales, the similarity between the current Q and all K matrices is calculated, and the similarity values are passed through the Softmax layer to obtain a set of weights. The causal attention Value value is obtained by multiplying this set of weights with the corresponding V matrix, and finally the vector is sent to the stacked calculation, as shown in Figure 6 .

[0136] Among them, first, the short-term feature vector X i and the long-term feature vector X s at this moment are input, that is, (X i , X s ) is the input of the causal attention. The QK is multiplied by the V matrix through matrix operations, and the output vector X' at this moment is linked to the input stacked calculation to obtain the calculated feature vector matrix Y i , and then sent to the feed-forward residual network operation to output the causal feature vector matrix

[0137]

[0138]

[0139]

[0140] Among them, the QK matrix operation X' connects the output and stacking, and Softmax is the normalized exponential function. is the weight of the matrix linear transformation, and Embed() represents the feed-forward residual network operation. is the weight of the matrix linear transformation of the meta-path node matrix, [.] represents the concatenation operation, and finally the time-series causal feature vector matrix is output through the feed-forward residual network operation.

[0141] (3) The graph convolution update operation analyzes the emotional state of the learner and its changing trend.

[0142] Based on the causal feature vector matrix This application constructs an emotional graph convolutional neural network GCN update operation to analyze the emotional evolution state, where the emotional state at time t is The emotional evolution state at time t defines the evolution output under the i-th emotional node. W l is the weight of the linear transformation, and b l is the bias term, while σ and ReLu are activation functions. is the mapping function. is the weight information of heterogeneous neighbor nodes. The graph convolution operation updates the learning emotional evolution operation as:

[0143]

[0144]

[0145] Suppose that the heterogeneous data set related to the learner's emotion in a certain classroom teaching process is processed by the above method and constructed into a causal graph data set. Analyze and calculate the learner's emotional state at time t. And the learner's emotional evolution trend The process is as follows:

[0146] (1) Obtain the causal graph data set related to calculating the learner's emotional evolution state.

[0147] (2) The graph embeds the learner's emotional heterogeneous causal nodes to obtain the feature vector matrix.

[0148] (3) The local feature vectors are weighted and fused with the global feature vectors through node attention and sent to the causal attention module for calculation to obtain the causal feature vector matrix.

[0149] (4) The causal feature vector matrix is sent into the graph convolution operation to update the learner's emotional state at time t.

[0150] (5) Calculate the emotional evolution trend of the learner at time t

[0151] Most of the existing methods stay at the static homogeneous graph model. However, in classroom teaching, it is necessary to process the characteristics of time series and causal relationships. Therefore, this application proposes an analysis method for the emotional evolution of classroom learners based on causal graph neural networks, dynamically tracking the real emotional state and its trend of learners, deeply analyzing the spatio-temporal causal relationship of learners' emotional changes, realizing the deep revelation of the emotional changes and their traceability laws of learners in the classroom, and helping to implement precise teaching in the classroom.

[0152] In addition, this application also provides a learner emotional evolution analysis system based on causal graph neural networks, and the system includes:

[0153] An acquisition module, used to design a multi-level emotional association semantic description framework based on "learner-activity-emotion" to obtain multi-modal time series data of learners;

[0154] A fusion module, used to fuse the multi-modal time series data based on the cross-modal time series fusion method of Z-Transformer;

[0155] A first construction module, used to construct an emotional structure causal model for the fine-grained emotional evolution of learners according to the fused learner time series data;

[0156] A second construction module, according to the emotional structure causal model, uses an intervention variational graph autoencoder to establish an emotional heterogeneous causal graph;

[0157] An analysis module, used to construct a causal graph neural network to analyze the emotional evolution of learners according to the emotional heterogeneous causal graph.

[0158] In some alternative embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of this application are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated, where the order of various operations is changed and the sub-operations described as part of a larger operation are executed independently.

[0159] In addition, although the present application has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It should also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present application. Rather, considering the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Therefore, those skilled in the art can implement the present application as set forth in the claims without undue experimentation. It should also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0160] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with the instruction execution system, apparatus, or device.

[0161] In the foregoing description of the present specification, the descriptions referring to the terms "one embodiment / example", "another embodiment / example", or "certain embodiments / examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0162] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0163] Above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for analyzing the emotional evolution of learners based on a causal graph neural network, characterized in that The method includes: Designing an emotional association semantic description framework based on the multi - level learner - activity - emotion to obtain multi - modal time - series data of learners; Fusing the multi - modal time - series data by means of cross - modal time - series fusion based on Z - Transformer; Constructing an emotional structure causal model for fine - grained emotional evolution of learners according to the fused multi - modal time - series data; Establishing an emotional heterogeneous causal graph using an intervention variational graph auto - encoder according to the emotional structure causal model; Constructing a causal graph neural network to analyze the emotional evolution of learners according to the emotional heterogeneous causal graph, including: Embedding the nodes of the learner's emotional heterogeneous causal graph; Calculating the emotional causal attention of learners under long - and short - term multi - scale time - series support to capture the multi - scale evolution features and causal relationship features of learners' emotions respectively; Analyzing the emotional state and its change trend of learners through graph convolution update operation.

2. The method for analyzing the emotional evolution of learners based on a causal graph neural network according to claim 1, characterized in that, The design of the emotional association semantic description framework based on the multi - level learner - activity - emotion to obtain multi - modal time - series data of learners is specifically: Construct the description of the learner layer: Learners and teachers are the main body and the dominant factor in classroom learning. This layer involves elements of interaction relationships, including learner-teacher-learning content. Among them, on the learner side, there are individual factors, and the individual factors include learners' classroom emotions, ages, genders, and homework completion rates. Formalize the above factors and construct a set of learner instances ; On the teacher side, their relevant ages, genders, and professional titles will also affect learners' emotions. Construct a set of teacher factor instances ; In addition, the learning content factors include the courses' related disciplines and difficulties. As a bridge connecting learners and teachers, it also affects learners' emotions. Construct a set of course factor instances C= ; Constructing the description of classroom activities: Based on the existing factors at the learner level, classroom teaching activities include learner participation activities and teacher participation activities; among them, the learner participation activities include listening in class, reading courses, peer communication, class discussion, and answering questions; the teacher participation activities include teacher course reading and classroom questioning, collecting data on classroom activity factors for quantification, and constructing a set A of classroom activity examples ; Construct a description of the emotional layer: Based on the learner level and activity level, perceive the teaching and learning process in the sequential classroom. The emotional state of the learner is affected by the combined data of the above two levels. The learner at the emotional type state of the moment is represented as a triple , where represents the emotional state of the learner at the moment, is the emotional polarity of the learner, is the emotional intensity of the learner; The interaction throughout the whole process of classroom teaching and learning activities stimulates the emotional changes in the cognitive process of the learner. The evolution trends of these emotions are characterized from two aspects. Then the learner's emotions have four evolution trends: negative increase, negative decrease, positive increase, and positive decrease. Therefore, the learner at the emotional evolution trend type of the moment is defined as a triple , where represents the emotional evolution trend of the learner at the moment, is the emotional evolution polarity of the learner, including two states: positive 1 and negative -1, is the change in the emotional intensity of the learner, including two changes: increase 1 and decrease -1.

3. The method for analyzing the emotional evolution of learners based on a causal graph neural network according to claim 1, characterized in that The cross - modal time - series fusion method based on Z - Transformer for fusing the multi - modal time - series data includes: According to the semanticized set of learner emotional association instances and their corresponding multi - modal time - series data, including images, audio, and text, constructing a cross - modal attention fusion model, inputting the multi - modal time - series data into the cross - modal attention fusion model, and aligning the fused multi - modal time - series data to input into a recurrent network function with cross - modal multi - head cross - self - attention stacked; Among them, the input set of the cross-modal attention fusion model is , Z ∈ M, and the cross-modal temporal data fusion process is as follows: ； ； where M is the set of data modality types, is the set of emotional association instances of learners, is the set of emotional association instances of teachers, is the set of emotional association instances of course factors, is the set of emotional association instances of classroom activities; Transformer is the self-attention mechanism, is a parameter, is the normalization process, is the data of the fused temporal emotional association instance set, represents the diffusion factor at time is the activation function, is the traversal symbol, is the feature concatenation.

4. The method for analyzing the emotional evolution of learners based on a causal graph neural network according to claim 1, wherein The construction of an emotional structure causal model for fine - grained emotional evolution of learners according to the fused multi - modal time - series data includes: Regarding the dynamic, complex spatio-temporal causal characteristics of the multi-dimensional emotional elements of learners, through the fused multi-modal time-series data set, the emotional structural causal model is defined as a quadruple consisting of an exogenous variable U, a set of endogenous variables V, a set of functional equations f, and a probability function P(U) defined on the domain of U; In order to construct an accurate emotional structure causal model, it is necessary to perform counterfactual reasoning on the causal effect of learners' emotions to eliminate the confounding effect brought by confounding factors; Through counterfactual reasoning on the link relationship of emotional instances , obtain the estimation result of the causal relationship of this emotional state , when the observation value set is 1 when there is an emotional change and 0 when there is no emotional change, calculate the estimation inference function of the causal relationship of this emotional state , eliminate confounding variables; Finally, based on the counterfactual calculation results, the mathematical formula for constructing the emotional structure causal model through emotional causal constraints is: ； In the formula, is the constructed directed acyclic graph, is the graph structure data implicitly included in SSCM, is the emotional state node, is the set of inference functions, is a set of causal variable instances, is the set of noise variables, pa is the parent node of the emotional node of, is the set of observed values, is the parameter range, u is a single noise variable, φ is the set of relational meta-paths, σ is the sigmoid activation function, is the path corresponding to the emotional state node, is the weight parameter of the meta-path corresponding to the emotion, and Gumbel is a random variable.

5. The method for analyzing the emotional evolution of learners based on a causal graph neural network according to claim 1, wherein The establishment of an emotional heterogeneous causal graph using an intervention variational graph auto - encoder according to the emotional structure causal model includes: According to the above - mentioned emotional structure causal model, the interconnection of multiple variables constitutes the heterogeneity of information related to the emotions of classroom learners. In the emotional structure causal model, multiple types of learners and their relationships coexist, and rich structural and semantic information is contained in different emotional relationship meta - paths; using an intervention variational graph auto - encoder to reconstruct the graph - structure data implicitly contained in the emotional structure causal model SSCM of the classroom learning process; Inputting the adjacency matrix of the reconstructed time - series SSCM graph into the input of the encoder, and through the intervention variational graph auto - encoder, calculating the structural neighbors and attributes of the graph network nodes to learn the representation of the graph network nodes, and outputting a set of emotional heterogeneous causal graph structure data through the GNN decoding layer for different node data in the time - series state.

6. The method for analyzing the emotional evolution of learners based on a causal graph neural network according to claim 1, characterized in that, The construction of a causal graph neural network to analyze the emotional evolution of learners according to the emotional heterogeneous causal graph includes: Embedding the nodes of the learner's emotional heterogeneous causal graph; Calculate the emotional causal attention of learners under the support of multi-scale time series for length and shortness, and capture the multi-scale evolution characteristics and causal relationship characteristics of learners' emotions respectively; The graph convolution update operation analyzes the emotional state and its change trend of learners.

7. The method for analyzing the emotional evolution of learners based on a causal graph neural network according to claim 6, characterized in that, The nodes of the embedded learner emotional heterogeneous causal graph include: Embed the nodes of the learner emotional heterogeneous causal graph, construct the weights of the causal node attention learning based on the meta-path neighbors; make the final node embedding according to the weights obtained by repeating the learning of all meta-path pairs of the heterogeneous causal graph for a preset number of times, so that the nodes with causal relationships in the graph are still similar in the embeddings of different feature spaces.

8. The method for analyzing the emotional evolution of learners based on a causal graph neural network according to claim 6, wherein The calculation of the emotional causal attention of learners under the support of multi-scale time series for length and shortness, which captures the multi-scale evolution characteristics and causal relationship characteristics of learners' emotions respectively, includes: After embedding the heterogeneous graph at a specific moment, perform weighted average calculation on all short-term feature vectors of learners' emotions, and perform weighted average summation on the multi-scale feature vectors to obtain the long-term feature vector of learners' emotions; According to the above-extracted short-term feature vector and long-term feature vector, use the normalized weighted geometric mean approximation to integrate the multi-scale sampling results into the feature layer attention calculation, calculate the causal relationship between the short-term feature vector and the long-term feature vector, and obtain the causal feature vector matrix.

9. The method for analyzing the emotional evolution of learners based on a causal graph neural network according to claim 6, wherein The graph convolution update operation analyzes the emotional state and its change trend of learners, including: According to the above-learned causal feature vector matrix, output the state and its change trend of learners at a certain moment through the graph convolution update operation, and its operation is: ； ； In the formula, is the emotional type state of the learner at time is the emotional evolution trend type of the learner at time is the learned causal feature vector matrix, is the linear transformation weight, is the bias term, while and are activation functions, is the weight parameter of the meta-path corresponding to the emotion, is the parameter range, is the weight information of heterogeneous neighbor nodes.

10. A learner's emotional evolution analysis system based on a causal graph neural network, characterized in that, The system includes: An acquisition module, used to design an emotional association semantic description framework based on the multi-level of learner-activity-emotion to obtain multi-modal time series data of learners; A fusion module, used to fuse the multi-modal time series data based on the cross-modal time series fusion method of Z-Transformer; A first construction module, used to construct an emotional structure causal model for the fine-grained emotional evolution of learners according to the fused multi-modal time series data; A second construction module, according to the emotional structure causal model, uses the intervention variational graph auto-encoder to establish an emotional heterogeneous causal graph; An analysis module, used to construct a causal graph neural network to analyze the emotional evolution of learners according to the emotional heterogeneous causal graph, including: Embed the nodes of the learner emotional heterogeneous causal graph; Calculate the emotional causal attention of learners under the support of multi-scale time series for length and shortness, and capture the multi-scale evolution characteristics and causal relationship characteristics of learners' emotions respectively; The graph convolution update operation analyzes the emotional state and its change trend of learners.

Images