A cross-modal emotion causal tracing method

Abstract

The present application discloses a cross-modal emotion causal tracking method, which relates to the field of emotion recognition. The method uses a front-door adjustment method based on encoding semantic learning to train the emotion extraction model, and uses an adaptive gradient adjustment method to adjust the gradient of the multimodal encoder in the emotion extraction model; combines knowledge distillation and noise gating mechanisms to train a sparse hybrid expert model to obtain a multimodal hybrid expert model; trains a teaching scenario causal emotion tracking model to obtain a trained teaching scenario causal emotion tracking model; collects multimodal emotion time series data of different teaching scenarios in actual classrooms according to the time sequence of teaching activities, and passes the multimodal emotion time series data through a trained emotion extraction model, a multimodal hybrid expert model, and a trained teaching scenario causal emotion tracking model in turn to obtain a causal mapping tracking of teaching activities and student emotions. The present application can accurately track the causal relationship between teaching activities and student multimodal emotions.

Description

Technical Field

[0001] The present application relates to the field of emotion recognition, and in particular to a cross-modal emotion causal tracing method. Background Art

[0002] Emotion is a key driving force in the learning process of students. It can promote students' intrinsic motivation and enhance their enthusiasm and persistence in learning. Emotion not only affects students' concentration and memory retention, but is also directly related to learning motivation and engagement. Students' emotional state is affected by many factors, including teaching content, teaching methods, and personal psychological state. Therefore, accurately identifying and tracking students' emotional changes is a key task for teachers, which can help teachers adjust teaching methods and strategies in time to meet the emotional experiences of different students. With the rapid development of artificial intelligence technology, especially in the field of education, multimodal learning has gradually become a research hotspot. This deep learning method improves the interactivity and personalization of teaching by integrating data from different modalities, such as text, images, and sounds.

[0003] Existing multimodal sentiment analysis models focus on model fusion and assume that the information of each modality and each label information is evenly distributed, so the main work is focused on designing architectures such as convolutional neural networks, tensor fusion networks, and Transformer models to promote modal fusion. However, in real data sets, this approach faces many challenges. The main problem is that the model often overfits the majority class and under-recognizes the minority class, resulting in a decrease in overall performance. In addition, commonly used evaluation indicators may not accurately reflect the performance of the model for the minority class, resulting in bias in the evaluation indicators. The model training process may be unstable due to data imbalance, and feature learning may also become unbalanced due to unequal information between modalities. In teaching scenarios, the imbalance of teaching data is a common problem, that is, the data of some categories is far more than that of other categories, which may cause the learning model to produce bias in prediction, affect the generalization ability of the model, and make it difficult for existing technologies to accurately capture and analyze these complex emotional expressions, especially in the presence of noise interference. The information between different modalities is unequal and competes with each other. At the same time, the complexity of emotion recognition and the noise problem of teaching data make accurate recognition and tracking of emotions a challenge. Existing models are prone to overfitting or insufficient generalization when faced with noisy data, and lack robustness. Summary of the invention

[0004] The purpose of this application is to provide a cross-modal emotion causal tracking method, which can accurately track the causal relationship between teaching activities and students' multimodal emotions.

[0005] To achieve the above objectives, this application provides the following solutions.

[0006] The present application provides a cross-modal emotion causal tracing method, comprising: constructing an emotion extraction model; the emotion extraction model comprises a multimodal encoder and a plurality of causal representation models; a modal encoder is connected to a causal representation model, the causal representation model is used to obtain a feature vector representing the causal relationship between a teaching scene and a student's emotion in each modality according to the modal encoding information output by the modal encoder; obtaining multimodal information samples generated by students in different teaching scenes; using the multimodal information samples, adopting a front-door adjustment method based on encoding semantic learning to train the emotion extraction model, and using an adaptive gradient adjustment method to adjust the gradient of the multimodal encoder in the emotion extraction model to obtain a trained emotion extraction model; inputting the multimodal information samples into the trained emotion extraction model, and outputting a feature vector representing the causal relationship between the teaching scene and the student's emotion in each modality; A noise gating mechanism is established; the feature vector of each modality representing the causal relationship between the teaching scene and the student's emotions is used, combined with knowledge distillation and the noise gating mechanism, to train a sparse mixed expert model to obtain a multimodal mixed expert model; the feature vector of each modality representing the causal relationship between the teaching scene and the student's emotions is input into the multimodal mixed expert model, and the student's emotion representation vector in each teaching scene is output; the teaching scene causal emotion tracking model is trained using the student's emotion representation vector in a variety of teaching scenes to obtain a trained teaching scene causal emotion tracking model; multimodal emotion time series data of different teaching scenes in actual classrooms are collected in chronological order of teaching activities, and the multimodal emotion time series data is sequentially passed through a trained emotion extraction model, a multimodal mixed expert model and a trained teaching scene causal emotion tracking model to obtain a causal mapping tracking of teaching activities and student emotions.

[0007] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0008] The present application provides a cross-modal emotion causal tracking method. The causal representation model in the emotion extraction model can deeply analyze the emotion causal relationship between different modal data through causal reasoning technology. Compared with traditional sentiment analysis, it not only focuses on the recognition of emotions, but also pays more attention to the causes and consequences of emotions, thereby achieving a deeper level of emotion understanding. The front-door adjustment method based on encoding semantic learning and the adaptive gradient adjustment method are used to estimate the degree of bias, and adaptive bias removal and competition relief are performed to overcome the problems of teaching data imbalance and modal competition. Knowledge distillation improves the performance of the sparse mixed expert model, a small model, through knowledge transfer. At the same time, the noise gating mechanism is introduced, and the impact of noise data on model performance is reduced through robust training methods, thereby improving the generalization and stability of the model. In addition, the sparse mixed expert model improves the model's ability to fuse data of different modalities and improves the model's recognition accuracy of emotions, thereby achieving accurate tracking of the causal relationship between teaching activities and students' multimodal emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 Schematic diagram of a flow chart of a cross-modal emotion causal tracing method in one embodiment of the present application.

[0011] Figure 2 Schematic diagram of the principle of a cross-modal emotion causal tracing method in one embodiment of the present application.

[0012] Figure 3 This is a brief flowchart of the training process in a cross-modal emotion causal tracing method in one embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] In order to make the above-mentioned objects, features and advantages of the present application more obvious and understandable, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0015] In an exemplary embodiment, Figure 1 As shown, the present application provides a cross-modal emotion causal tracing method, including the following steps 101 to 109.

[0016] Step 101: construct an emotion extraction model; the emotion extraction model includes a multimodal encoder and multiple causal representation models; a modal encoder is connected to a causal representation model, and the causal representation model is used to obtain a feature vector representing the causal relationship between the teaching scene and the student's emotions in each modality according to the modal encoding information output by the modal encoder.

[0017] Step 102: Obtain multimodal information samples generated by students in different teaching scenarios.

[0018] Step 103: Utilize the multimodal information samples and adopt the front-door adjustment method based on encoding semantic learning to train the emotion extraction model, and use the adaptive gradient adjustment method to adjust the gradient of the multimodal encoder in the emotion extraction model to obtain a trained emotion extraction model.

[0019] Step 104: Input the multimodal information samples into the trained emotion extraction model, and output a feature vector for each modality representing the causal relationship between the teaching scene and the student's emotions.

[0020] Step 105: Establish a noise gating mechanism.

[0021] Step 106: Using the feature vector of each modality to represent the causal relationship between the teaching scenario and the student's emotions, combined with knowledge distillation and noise gating mechanism, train the sparse mixed expert model to obtain a multimodal mixed expert model.

[0022] Step 107: Input the feature vector of each modality representing the causal relationship between the teaching scenario and the student's emotions into the multimodal hybrid expert model, and output the student's emotion representation vector in each teaching scenario.

[0023] Step 108: Use the emotion representation vectors of students in various teaching scenarios to train the teaching scenario causal emotion tracking model to obtain a trained teaching scenario causal emotion tracking model.

[0024] Step 109: Collect multimodal emotion time series data of different teaching scenes in the actual classroom according to the chronological order of teaching activities, and pass the multimodal emotion time series data in turn through the trained emotion extraction model, the multimodal hybrid expert model and the trained teaching scene causal emotion tracking model to obtain the causal mapping tracking of teaching activities and students' emotions.

[0025] Implement steps 101 to 109 above, use a sparse mixed expert model to improve the model's ability to fuse data of different modalities, use large model knowledge to enhance the accuracy of emotion recognition, and use gradient adjustment to alleviate modal competition problems. Introduce a noise gating mechanism to reduce the impact of noise data on model performance through a robust training method, thereby improving the generalization and stability of the model. In addition, use causal reasoning technology to analyze and track the emotional causal relationship between data of different modalities, further explore the causal mapping between different teaching activities and student emotions, and provide deeper insights into teaching. This application aims to provide a more accurate, robust and adaptable multimodal emotion causal tracking and training method to support more effective teaching decisions and personalized learning experiences.

[0026] In another exemplary embodiment of the present application, the causal representation model in the above step 101 is a two-branch neural network; the two-branch neural network includes a first branch, a second branch and a semantic fusion module. The first branch is a self-attention network, and the second branch is a cross-attention network. The self-attention network is used to perform internal sampling from the modal coding information samples output by the modal encoder, and output the internal sampling feature vector. The cross-attention network is used to interact with the global dictionary, perform cross-sample modal sampling, and output a cross-sample feature vector; the global dictionary includes all modal coding information samples output by the multimodal encoder. The semantic fusion module is used to fuse the internal sampling feature vector and the cross-sample feature vector through a vector splicing operation to obtain a feature vector in each modality that represents the causal relationship between the teaching scene and the student's emotions.

[0027] In another exemplary embodiment of the present application, the data generated in the actual teaching process has the characteristic of category imbalance, which is specifically manifested in that in the sentiment analysis task, the samples of positive emotions may be far more than the samples of negative emotions. Category imbalance will cause pseudo-correlation between modality and label, causing the model to make predictions based on incorrect correlations. The present application uses a front-door adjustment method based on encoding semantic learning to obtain a causal representation of unbalanced teaching data. Therefore, the front-door adjustment method based on encoding semantic learning in the above step 103 can be replaced by the following steps 201 to 203.

[0028] Step 201: Introduce a mediating variable that characterizes the cross-sample modal sampling process to cut off the spurious causal path.

[0029] Step 202: Based on the mediating variable, according to the formula , obtain the feature vector representing the causal relationship between teaching scenes and students’ emotions in each mode; among them, represents the feature vector, represents multimodal sentiment labels, Indicates the front door adjustment object mode, Indicates a mode other than the front door adjustment object mode. Indicates another mode besides the front door adjustment object mode. represents the probability mapping function, represents the causal representation model, represents the expected calculation function, represents the in-sample sampling process of the front door adjustment object mode, represents the mediating variable, express Operator.

[0030] Step 203: Based on the encoding semantics, the loss function of the global dictionary in the cross-sample modality sampling process is defined as: ;in, represents the loss function of the global dictionary, Indicates that the gradient stops operating, represents the pre-trained model of the front-gate adjusted object modality encoder, express Sampling results from the global dictionary based on the results of the most recent dictionary query, represents the learning rate hyperparameter, Represents the Euclidean norm operation.

[0031] When the multimodality includes text modality, audio modality and visual modality, taking the text modality as an example, the front-door adjustment method based on encoding semantic learning is: To apply front-door adjustment in multimodal learning analysis, a mediating variable needs to be introduced To transmit the true causal effect and cut off the pseudo-correlation path. Taking the front-door adjustment of the text modality as an example, the causal estimation method can be described as the following formula.

[0032]

[0033] in, The object representing the front door adjustment is the text modal, Indicates the audio mode, Indicates the visual mode, represents the probability calculation function, represents multimodal sentiment labels, represents the expected calculation function, represents the in-sample sampling process for the text modality, As an intermediary variable, it represents the cross-sample text modality sampling process. The global dictionary contains all multimodal information samples, providing a contextual verification environment for the current in-sample knowledge to evaluate whether the in-sample knowledge of different fragments can still maintain emotional semantics in different contexts. In order to achieve end-to-end learning, the global dictionary cannot rely on simple clustering. This application constructs a new dictionary learning method based on encoding semantics.

[0034] The global dictionary learning based on encoding semantics is introduced as follows: The purpose of dictionary learning is to maintain the semantics of all training sets as much as possible in end-to-end learning, and provide reliable context guarantee for front-door adjustment. The global dictionary is randomly initialized, and the loss function of the global dictionary is used. Optimize the learning of the global dictionary. By ensuring that the distance between the query results in the dictionary and the text latent variables is minimized, that is, the encoding is minimized, a global dictionary that conforms to the text information in all multimodal information samples is constructed.

[0035] This application can alleviate the pseudo-correlation problem between modalities and labels in the teaching process by using the above-mentioned front-door adjustment method based on encoded semantic learning for the three modalities of text modality, visual modality, and audio modality, and lays a foundation at the single-modal level for emotional causal tracking in teaching scenarios.

[0036] In another exemplary embodiment of the present application, when multi-modality is used for unified network training, different modalities will not only promote each other and supplement information, but will also compete with each other, ultimately leading to the problem of underfitting of several modalities. The present application constructs an adaptive gradient adjustment based on the gradient norms of different modality encoders to unify the modality preemption problem in multi-modal training and form an adaptive gradient adjustment method for inter-modal balance. Then the use of the adaptive gradient adjustment method in the above step 103 to adjust the gradient of the multi-modal encoder in the emotion extraction model can be replaced by the following steps 301 to 302.

[0037] Step 301: According to the formula , determine the modal preemption factor of each modality; where, Representing modality The modal preemption factor of Representing modality The L2 norm of the modal encoder gradient, Representing modality The L2 norm of the modal encoder gradient, Representing modality The L2 norm of the modal encoder gradient.

[0038] Step 302: If the modal preemption factor is greater than 1, then according to the formula Adjust the gradient of the modal encoder; where, Indicates the mode before adjustment The corresponding modal encoder gradient, Indicates the adjusted mode The corresponding modal encoder gradient, represents the hyperbolic tangent function, Represents the coefficient that controls the strength of the gradient adjustment.

[0039] Below is the modal For text mode, modal For audio mode, mode Taking visual modality as an example, the adaptive gradient adjustment method for inter-modality balance is described in detail. The adaptive gradient adjustment method for inter-modality balance includes modality preemption state evaluation and adaptive gradient adjustment.

[0040] Modal preemption state evaluation: Gradient norm plays an important role in the optimization process. This application estimates the modal preemption state by evaluating the gradient of three types of encoders. In represents the L2 norm of the text modality encoder gradient, represents the L2 norm of the audio encoder gradient, represents the L2 norm of the visual encoder gradient. If , that is, the gradient of the text modality encoder is greater than the average of the norms of the audio modality encoder and the visual modality encoder. This means that the optimization of the text modality may lead to under-optimization of the other two modalities. In this case, the gradient of the text encoder needs to be adjusted, otherwise it does not need to be adjusted.

[0041] The preemption status evaluation method of other modalities is consistent with that of the text modality, that is, if the L2 norm of the current modality encoder is greater than the average of the L2 norms of the other two modality encoders, the current modality preempts other modalities and gradient adjustment is required.

[0042] Adaptive gradient adjustment: The cause of multimodal competition is the difference in characteristics and performance of different modalities, which leads to the model optimization being mainly dominated by the dominant modality. The text modality has richer semantics. In the unified training paradigm, the optimization process is likely to be dominated by text, which directly leads to insufficient optimization of the audio and visual modalities. In order to solve the above modality competition problem, the gradients of different encoders need to be adjusted. In the formula middle, Characterizes the degree of modal preemption. The greater the The corresponding mode The encoder gradient of will be scaled smaller, that is, It will be relatively smaller, thus solving the problem that the gradient only flows to the dominant mode during unified training.

[0043] In another exemplary embodiment of the present application, the sparse mixed expert model in the above step 106 includes: Layer-wise multi-channel Transformer; each layer of multi-channel Transformer includes: a multi-head self-attention layer and a multi-channel expert, and the multi-head self-attention layer is connected to the multi-channel expert respectively.

[0044] The formula for multi-way experts is: ;in, Indicates that each layer selects The output of road experts, represents the probability mapping function, represents the input of the sparse mixture of experts model, represents the latent vector output by the self-attention network in the sparse mixture of experts model, According to Select The sorting gating function activated by the road expert, Indicates Output from road experts. Represents the learnable matrix, the mapping relationship between the output of the multi-head self-attention layer and the multi-way expert assignment.

[0045] In another exemplary embodiment of the present application, the knowledge of the existing multimodal large model is applied to the sparse mixed expert model through the knowledge distillation technology, which can significantly improve the efficiency and practicality of the sparse mixed expert model. The present application pre-trains a sparse mixed expert model through a two-stage distillation method for the multimodal large model. Through this process, different expert models can inherit the specific deep features and knowledge of the multimodal large model, so that they can still make classification decisions quickly and accurately in a resource-constrained environment. In addition, knowledge distillation also helps to improve the robustness of the model, and maintain a high level of accuracy even in the face of insufficient data or noisy data. The knowledge distillation in the above step 106 includes: single-modal expert distillation and multi-modal expert distillation.

[0046] Single-modal expert distillation means: each layer of multi-way experts in the sparse mixed expert model is divided into multiple groups, each group corresponds to a modality; when a single-modal dataset is used to train a sparse mixed expert model, only the expert group corresponding to the modality of the single-modal dataset is distilled and trained.

[0047] The loss function of unimodal expert distillation is: ;in, represents the loss function of unimodal expert distillation, represents the guidance intensity coefficient of the control teacher network, represents the cross entropy loss function, represents the output of the sparse mixture expert model during single-modal expert distillation, represents the true unimodal label, represents the KL divergence function, represents the output of the teacher network, Indicates student network The output features of the layer, Indicates student network The output features of the layer, Indicates the L2 loss strength of the control layer inter-message.

[0048] Multimodal expert distillation means that when a sparse mixed expert model is trained using a multimodal dataset, all experts in the sparse mixed expert model are distilled and trained. The loss function of multimodal expert distillation is: ; where ℒ3 represents the loss function of multimodal expert distillation, It indicates that the sparse mixed expert model output of all experts is aggregated during multimodal expert distillation. represents the true multimodal label, Represents the cosine similarity calculation function, Represents the aggregation mode The sparse mixture expert model output after the multi-way expert, Representing modality Other modes, Represents the aggregation mode The sparse mixture expert model output after the multi-way expert, Representing modality The input data, Representing modality The input data.

[0049] The following uses multimodality including text modality, audio modality and visual modality as an example to introduce knowledge distillation in more detail.

[0050] First stage distillation - single-modal expert distillation: Multimodal datasets are much smaller in number than single-modal datasets. In order to enable the model to learn more robust knowledge from a larger dataset, in the first stage, this application uses single-modal datasets (including text datasets, audio datasets, and visual datasets) to train different experts, so that different experts have specific processing capabilities for different modalities. This application initializes the multi-channel experts in each layer. Multi-channel experts are evenly divided into three modal unique experts, namely, text modality, audio modality, and visual modality, respectively. When training with a single-modal dataset (such as a text dataset), only the current modality (such as text modality) corresponds to experts will perform distillation training, and the corresponding Experts do not participate in the training to ensure that information of a specific modality (such as text modality) is temporarily retained only in a specific The above distillation operation is performed for text modality, audio modality and visual modality to ensure that each expert contains specific modality information. Given a large model in a single modality domain (including text, audio, and vision), the loss function is as follows.

[0051]

[0052] Second stage distillation - multimodal expert distillation: This application uses multimodal data including image-text pairs, audio-text pairs, image-audio pairs, and trimodal combinations for training when all experts are activated, that is, all experts unfreeze and fine-tune, and the teacher network is a large model of mixed-modal perception and analysis. Considering the possible modality loss problem in teaching scenarios, this application takes into account both single-modal performance and multi-modal performance, and designs the following loss function.

[0053]

[0054] The teacher network mentioned above is an existing multimodal large model, and the student network is a sparse mixed expert model. This application introduces the cross-modal feature space consistency loss-cosine similarity in multimodal knowledge distillation, which can adapt to the modality missing problem in different teaching scenarios and better meet the needs of actual teaching scenarios.

[0055] In another exemplary embodiment of the present application, the present application uses noise gating to avoid overfitting problems during training, remain flexible in the face of changing data, and dynamically adjust the use of experts according to the training stage. Gaussian noise is added to the gating to control the flexible processing mechanism when different experts work. The noise gating mechanism established in the above step 105 is as follows.

[0056]

[0057] in, represents the first The weight of road experts, represents the input of the sparse mixture of experts model, represents the latent vector output by the self-attention network in the sparse mixture of experts model, represents a learnable vector, represents Gaussian noise, represents the activation function, represents a learnable noise map.

[0058] In another exemplary embodiment of the present application, a load balancing strategy is designed to ensure that the computing resources of each expert are reasonably allocated, to avoid overloading of some experts while other experts are idle, to more efficiently utilize computing resources, and to improve overall computing efficiency. The present application uses an importance loss function to balance the balance of tokens accepted by different experts. When training the sparse mixed expert model in the above step 106, an importance loss function is introduced as a constraint term in the original loss function. The introduced importance loss function is as follows.

[0059]

[0060] in, represents the importance loss function, represents a multimodal dataset, represents the input of the sparse mixture of experts model, Represents the number of paths per expert layer in the sparse mixture of experts model. Represents an adjustable scaling factor that controls the weight of the importance loss in the total loss. Indicates that the gating network is Road experts target The generated gating value reflects the relevance or weight of the expert in processing the sample. The gating network is integrated in the hybrid expert model and is a weight allocator in the hybrid expert model. It is used to allocate the first Road experts target The noise gating mechanism is integrated into the gating network, so the gating value corresponds to . is the average of the experts’ importance, The average value among different experts is obtained to measure the benchmark of expert activation frequency. Indicates all In the Road Expert's Gating Value The cumulative sum of Total importance of road expert activated.

[0061] Through the above importance loss function, the data in a batch is maintained to enter different experts as evenly as possible, so that each expert branch can be load balanced and fully optimized.

[0062] In another exemplary embodiment of the present application, the actual classroom data contains a large amount of noise, in order to filter out causal information directly related to emotion discrimination. The cross-modal attention mechanism improves the model's ability to integrate information by learning the correlation between different modalities, but it cannot identify noise information and cannot track emotional causal clues. The cross-modal counterfactual attention mechanism proposed in this application not only focuses on the relationship between the current modalities, but also considers the impact on other modalities and the overall task completion if the information in a certain modality changes. This mechanism helps the model to understand the complex relationship between different modalities more deeply by simulating different possibilities, and provides a fusion level of assistance for tracking causal emotional clues. Therefore, when the above step 106 trains the sparse mixed expert model, the cross-modal attention mechanism and the cross-modal counterfactual attention mechanism are used to fuse the feature vectors, and the expression of the cross-modal counterfactual attention mechanism is as follows.

[0063]

[0064] in, represents the counterfactual fusion result, express Operator, represents the attention matrix sampled from a uniform distribution, Indicates attention Forced to be set , Represents the input of cross-modal attention vector, Indicates that The value of is set to the modal The value of represents a multimodal encoder with cross-attention, represents the first attention matrix sampled from a uniform distribution, represents the first sampled from a uniform distribution An attention matrix.

[0065] The core of the cross-modal counterfactual attention mechanism is to mine unstable noise for causal mining in the multimodal hybrid expert model. The cross-modal counterfactual attention mechanism provides a negative example, and its role is to capture emotion-irrelevant context, such as noise, visual background and other information. By jointly optimizing it with the cross-modal attention mechanism, the original multimodal encoder will be able to effectively filter out irrelevant context information. When training the sparse hybrid expert model, the joint optimization loss function of the cross-modal attention mechanism and the cross-modal counterfactual attention mechanism is as follows.

[0066]

[0067] in, represents the joint optimization loss function, represents the mean absolute error loss function, represents a multimodal encoder, represents the assembly of a cross-modal counterfactual attention encoder, represents the true multimodal label, Representing modality Other modes, and As and The core of this application is to train robust Capturing causal emotional clues, It mainly captures irrelevant noise information.

[0068] Both the cross-modal attention mechanism and the cross-modal counterfactual attention mechanism are executed in the multi-head self-attention layer in the sparse hybrid expert model. In another exemplary embodiment of the present application, in observational studies, usually only one potential result can be observed, that is, the result of the student's actual teaching activities. To determine the counterfactual result (that is, what would happen if the individual received different interventions), it is necessary to infer the possible results under different interventions. This involves dealing with the so-called "selection bias" or "confounding factors", which may affect the choice of intervention and the generation of results, and it is difficult to measure the causal relationship between different teaching activities and student emotions. In order to solve the above-mentioned counterfactual reasoning problem, emotional data in actual classrooms are collected and arranged in chronological order of teaching activities to track the dynamic process of emotional changes. And by embedding teaching activity slices, the causal mapping between teaching activities and student emotions in the time series dimension is explored. The loss function used in the above step 108 to train the teaching scene causal emotion tracking model is as follows.

[0069]

[0070] in, represents the loss function used to train the causal emotion tracking model for teaching scenarios, represents the feature mapping function, ' represents a bidirectional long short-term memory time series prediction network, represents the output of the sparse mixture expert model. Student emotion feature vectors, Indicates that for The teaching activities and measures for each student, Indicates The emotion representation vector label of each student, represents the hyperparameter that controls the strength of the balance penalty, A measure of the difference in distributions, and represents the distribution among different teaching activity groups, represents a hyperparameter that controls the strength of the proximity penalty, Indicates The nearest neighbor students of each student who receives different teaching activities, The emotional representation vector labels of students receiving different teaching activities, Indicates the number of students. Indicates that for The teaching activities and measures for each student, Indicates that for There are no teaching activities for students. and They respectively represent whether there are teaching activities for students.

[0071] From the above loss function, we can see that the teaching scenario causal emotion tracking model includes the feature mapping function And bidirectional LSTM time series prediction network When training the causal emotion tracking model for teaching scenarios, the optimized feature mapping function and the bidirectional LSTM time series prediction network are trained simultaneously.

[0072] The first term of the above loss function is It measures the accuracy of the model's predictions of the observed outcomes. 'It can record the temporal characteristics of several previous teaching activities. Indicates the penalty for the balance of distribution, which is used to punish different teaching activities. Between represents the distribution ( and ) imbalance. It is a measure of distribution difference. This application uses the spectral norm of the second moment of the distribution to measure the distribution difference. The third item is to encourage the model. When making counterfactual predictions, the predicted value should be close to the most recent observation of the corresponding teaching activity measures.

[0073] The above steps 108 and 109 constitute a method for tracking causal emotions in a teaching scenario in a temporal manner.

[0074] With the assistance of the causal emotion tracking model in teaching scenarios, this application can also measure the emotional differences brought about by the same student in different teaching activities.

[0075] Figure 2 This is the overall schematic diagram of a cross-modal emotion causal tracking method for this application. Figure 2 As shown, the sample text modal information is “The sharing of the previous group was really wonderful, we…” Figure 3 This is a brief flowchart of the training process in a cross-modal emotion causal tracking method for this application. Figure 3 As shown, the brief flow of the training process in a cross-modal emotion causal tracking method of the present application is: causal representation for unbalanced teaching data, adaptive gradient adjustment method for inter-modal balance, sparse mixed experts for multi-modal large model distillation, robust training of noise-gated load balancing, cross-modal counterfactual-guided emotion causal discovery and teaching scene temporal causal emotion tracking method. Among them, the causal representation for unbalanced teaching data corresponds to step 101 and step 103, the adaptive gradient adjustment method for inter-modal balance and sparse mixed experts for multi-modal large model distillation correspond to steps 102 and 103, the robust training of noise-gated load balancing corresponds to steps 105 and 106, the cross-modal counterfactual-guided emotion causal discovery corresponds to step 106, and the teaching scene temporal causal emotion tracking method corresponds to steps 108 and 109.

[0076] This application proposes an innovative method in the field of multimodal sentiment analysis, and its comprehensiveness is reflected in the following three points.

[0077] 1. Cross-modal emotion causal tracking technology: This application uses causal reasoning technology to deeply analyze the emotional causal relationship between different modal data. This technology can reveal the deep connection between teaching activities and students' emotions, and provide more accurate feedback and improvement directions for teaching. Compared with traditional sentiment analysis, this method not only focuses on the recognition of emotions, but also pays more attention to the causes and consequences of emotions, thereby achieving a deeper level of emotional understanding.

[0078] 2. Fusion of adaptive gradient adjustment and sparse hybrid expert model: To solve the problem of unbalanced data set,

[0079] Front-gate adjustment and gradient adjustment are used to estimate the degree of bias and perform adaptive bias removal and competition mitigation. At the same time, the sparse mixed expert model is used to improve the model's ability to integrate multimodal data. This structure not only optimizes the feature learning process, but also enhances the model's integration and expression of different modal features, significantly improving the generalization and robustness of the model.

[0080] 3. Combination of large model distillation and noise gating mechanism: In order to improve the accuracy of emotion recognition and the robustness of the model, this application adopts large model distillation technology to improve the performance of small models through knowledge transfer. At the same time, a noise gating mechanism is introduced to reduce the impact of noisy data on model performance through robust training methods. This combination not only improves the model's recognition accuracy for emotions, but also enhances the model's stability and reliability when facing complex and noisy data.

[0081] These three points together constitute the technical core of this application, which not only solves the application problem of multimodal sentiment analysis on unbalanced data sets, but also provides strong technical support for teaching decisions and personalized learning experience, demonstrating the comprehensiveness and innovation of this application in the field of multimodal sentiment analysis.

[0082] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A cross-modal emotion causal tracing method, characterized in that: include: Constructing an emotion extraction model; the emotion extraction model includes a multimodal encoder and multiple causal representation models; a modal encoder is connected to a causal representation model, and the causal representation model is used to obtain a feature vector representing the causal relationship between the teaching scene and the student's emotions in each modality according to the modal encoding information output by the modal encoder; Obtain multimodal information samples generated by students in different teaching scenarios; Using multimodal information samples, the emotion extraction model is trained by adopting a front-door adjustment method based on encoding semantic learning, and the gradient of the multimodal encoder in the emotion extraction model is adjusted by using an adaptive gradient adjustment method to obtain a trained emotion extraction model; Input the multimodal information samples into the trained emotion extraction model, and output a feature vector for each modality representing the causal relationship between the teaching scene and the student's emotions; Establish a noise gating mechanism; Using the feature vector of each modality to represent the causal relationship between the teaching scene and the student's emotions, combined with knowledge distillation and noise gating mechanism, the sparse mixed expert model is trained to obtain a multi-modal mixed expert model; Inputting the feature vector of each modality representing the causal relationship between the teaching scenario and the student's emotions into the multimodal hybrid expert model, and outputting the student's emotion representation vector in each teaching scenario; Using the emotion representation vectors of students in various teaching scenarios, the causal emotion tracking model of teaching scenarios is trained to obtain a trained causal emotion tracking model of teaching scenarios; Multimodal emotion time series data of different teaching scenes in actual classrooms are collected according to the chronological order of teaching activities, and the multimodal emotion time series data are successively passed through a trained emotion extraction model, a multimodal hybrid expert model and a trained teaching scene causal emotion tracking model to obtain the causal mapping tracking of teaching activities and students' emotions.

2. The cross-modal emotion causal tracing method according to claim 1, characterized in that: The causal representation model is a dual-branch neural network; the dual-branch neural network includes a first branch, a second branch and a semantic fusion module; the first branch is a self-attention network, and the second branch is a cross-attention network; The self-attention network is used to perform in-sampling from the modality encoding information sample output by the modality encoder, and output an in-sampling feature vector; The cross-attention network is used to interact with the global dictionary, perform cross-sample modality sampling, and output a cross-sample feature vector; the global dictionary includes all modality encoding information samples output by the multimodal encoder; The semantic fusion module is used to fuse the internal sampling feature vector and the cross-sample feature vector through a vector splicing operation to obtain a feature vector in each modality that represents the causal relationship between the teaching scene and the student's emotions.

3. The cross-modal emotion causal tracing method according to claim 2, characterized in that: The front door adjustment method based on encoding semantic learning specifically includes: Introduce a mediating variable that characterizes the sampling process across sample modalities to cut off the spurious causal path; Based on the mediating variables, according to the formula , obtain the feature vector representing the causal relationship between teaching scenes and students’ emotions in each mode; among them, represents the feature vector, represents multimodal sentiment labels, Indicates the front door adjustment object mode, Indicates a mode other than the front door adjustment object mode. Indicates another mode besides the front door adjustment object mode. represents the probability mapping function, represents the causal representation model, represents the expected calculation function, represents the in-sample sampling process of the front door adjustment object mode, represents the mediating variable, express Operator; Based on the encoding semantics, the loss function of the global dictionary in the cross-sample modality sampling process is defined as: ;in, represents the loss function of the global dictionary, Indicates that the gradient stops operating, represents the pre-trained model of the front-gate adjusted object modality encoder, express Sampling results from the global dictionary based on the results of the most recent dictionary query, represents the learning rate hyperparameter, Represents the Euclidean norm operation.

4. The cross-modal emotion causal tracing method according to claim 1, characterized in that: The gradient of the multimodal encoder in the emotion extraction model is adjusted using an adaptive gradient adjustment method, specifically including: According to the formula , determine the modal preemption factor of each modality; where, Representing modality The modal preemption factor of Representing modality The L2 norm of the modal encoder gradient, Representing modality The L2 norm of the modal encoder gradient, Representing modality The L2 norm of the modal encoder gradient; If the modal preemption factor is greater than 1, then according to the formula Adjust the gradient of the modal encoder; where, Indicates the mode before adjustment The corresponding modal encoder gradient, Indicates the adjusted mode The corresponding modal encoder gradient, represents the hyperbolic tangent function, Represents the coefficient that controls the strength of the gradient adjustment.

5. The cross-modal emotion causal tracing method according to claim 1, characterized in that: The sparse mixed expert model comprises: Layer-Multiple Transformer; Each layer of multi-channel Transformer includes: a multi-head self-attention layer and a multi-channel expert, and the multi-head self-attention layer is connected to the multi-channel expert respectively; The formula for multi-way experts is: ;in, Indicates that each layer selects The output of road experts, represents the probability mapping function, represents the learnable matrix, represents the input of the sparse mixture of experts model, represents the latent vector output by the self-attention network in the sparse mixture of experts model, Indicates based on Select The sorting gating function activated by the road expert, Indicates Output from road experts.

6. The cross-modal emotion causal tracing method according to claim 1, characterized in that: The noise gating mechanism is: ； in, represents the first The weight of road experts, represents the input of the sparse mixture of experts model, represents the latent vector output by the self-attention network in the sparse mixture of experts model, represents a learnable vector, represents Gaussian noise, represents the activation function, represents a learnable noise map.

7. The cross-modal emotion causal tracing method according to claim 1, characterized in that: The knowledge distillation includes: single-modal expert distillation and multi-modal expert distillation; The single-modal expert distillation refers to: dividing each layer of multi-channel experts in the sparse mixed expert model into multiple groups, each group corresponding to one modality; when using a single-modal dataset to train the sparse mixed expert model, only the expert group corresponding to the modality of the single-modal dataset is subjected to distillation training; The loss function of the unimodal expert distillation is: ;in, represents the loss function of unimodal expert distillation, represents the guidance intensity coefficient of the control teacher network, represents the cross entropy loss function, represents the output of the sparse mixture expert model during single-modal expert distillation, represents the true unimodal label, represents the KL divergence function, represents the output of the teacher network, Indicates student network The output features of the layer, Indicates student network The output features of the layer, Indicates the L2 loss strength of the control layer inter-message; The multimodal expert distillation refers to: when a sparse mixed expert model is trained using a multimodal dataset, distillation training is performed on all experts in the sparse mixed expert model; The loss function of the multimodal expert distillation is: ;in, represents the loss function of multimodal expert distillation, It indicates that the sparse mixed expert model output of all experts is aggregated during multimodal expert distillation. represents the true multimodal label, Represents the cosine similarity calculation function, Represents the aggregation mode The sparse mixture expert model output after the multi-way expert, Representing modality Other modes, Represents the aggregation mode The sparse mixture expert model output after the multi-way expert, Representing modality The input data, Representing modality The input data.

8. The cross-modal emotion causal tracing method according to claim 1, characterized in that: When training the sparse mixture of experts model, an importance loss function is introduced to balance the expert load; The importance loss function is: ； in, represents the importance loss function, represents a multimodal dataset, represents an adjustable scaling factor, represents the input of the sparse mixture of experts model, Indicates that the gating network is Road experts target The generated gate value, is the average of the experts’ importance, Represents the number of paths per expert layer in the sparse mixture of experts model.

9. The cross-modal emotion causal tracing method according to claim 1, characterized in that: When training the sparse mixture of experts model, the cross-modal attention mechanism and the cross-modal counterfactual attention mechanism are used to fuse the feature vectors; The expression of the cross-modal counterfactual attention mechanism is: ； in, represents the counterfactual fusion result, express Operator, represents the attention matrix sampled from a uniform distribution, Indicates attention Forced to be set , Represents the input of cross-modal attention vector, Indicates that Set the value of the modal The value of represents a multimodal encoder with cross-attention, represents the first attention matrix sampled from a uniform distribution, represents the first sampled from a uniform distribution An attention matrix; When training the sparse mixture of experts model, the joint optimization loss function of the cross-modal attention mechanism and the cross-modal counterfactual attention mechanism is: ； in, represents the joint optimization loss function, represents the mean absolute error loss function, represents a multimodal encoder, represents the assembly of a cross-modal counterfactual attention encoder, represents the true multimodal label, Representing modality Other modes, and As and Input.

10. The cross-modal emotion causal tracing method according to claim 1, characterized in that: The loss function used to train the causal emotion tracking model for teaching scenarios is: ； in, represents the loss function used to train the causal emotion tracking model for teaching scenarios, represents the feature mapping function, ' represents a bidirectional long short-term memory time series prediction network, represents the output of the sparse mixture expert model. Student emotion feature vectors, Indicates that for The teaching activities of each student Indicates The emotion representation vector label of each student, represents the hyperparameter that controls the strength of the balance penalty, A measure of the difference in distributions, and represents the distribution among different teaching activity groups, represents a hyperparameter that controls the strength of the proximity penalty, Indicates The nearest neighbor students of each student who receives different teaching activities, The emotional representation vector labels of students receiving different teaching activities, Indicates the number of students.

Images