Video learning emotion recognition method based on emotion infection tracking and multi-modal fusion

Abstract

The application discloses a video learning emotion recognition method based on emotion infection tracking and multi-modal fusion, which comprises the following steps: 1) data collection; 2) data preprocessing; 3) feature structure enhancement; 4) emotion infection driven feature enhancement and emotion recognition; 5) testing and evaluation. This method uses eye movement physiological signals to model the bias modulation of individual differences and emotional feedback of learners, and uses it as a guide to track the teaching video inducing factors and infect the feature weighting, and then realizes the deep enhancement fusion of physiological response and video features through the one-way emotion infection tracking module and the two-way cross-modal attention mechanism, finally improves the precision and robustness of emotion recognition.

Description

Technical Field

[0001] This invention relates to computer intelligent recognition technology, and in particular to multimodal emotion recognition technology based on emotion contagion modeling in video learning scenarios. Specifically, it is a video learning emotion recognition method based on emotion contagion tracking and multimodal fusion. Background Technology

[0002] Multimodal Emotion Recognition (MER) effectively captures complex human psychological states by collaboratively analyzing multi-source information, including visual, semantic, and physiological signals, demonstrating significant application value in fields such as intelligent interaction, intelligent education, and psychological assistance. With the popularization of online education, the demand for emotion recognition in video learning scenarios is becoming increasingly urgent. It not only monitors learners' psychological fluctuations in real time but also provides crucial support for building adaptive learning environments with emotional awareness. However, the lack of real-time interactive feedback in video learning environments makes accurately understanding learners' emotional evolution during video viewing a highly challenging technical problem.

[0003] While physiological signals, such as eye movements, have unique advantages in emotion recognition due to their objectivity and immediacy, these signals are essentially physiological feedback triggered by emotions. Without in-depth modeling of external eliciting factors, it is difficult to reveal the intrinsic logic of emotion generation. Researchers have begun to attempt cross-modal fusion of video semantics and physiological signals, but existing methods mostly focus on feature alignment and weighting, neglecting the deep causal relationship and emotion transmission path between the teaching stimulus and the respondent. This results in insufficient inference ability of the model to understand the deep emotion-driving mechanism.

[0004] Currently, multimodal emotion recognition technology in video learning scenarios still faces the following challenges: (1) Lack of modeling of emotional influence mechanism: Existing mainstream fusion methods mainly focus on the simple alignment or interaction of modal features, usually treating video features and physiological features as information sources of equal status. For example, black-box models based on Transformer or graph convolution, although improving feature fitting ability, rely solely on statistical correlation for discrimination, lacking explicit modeling of the emotional contagion process of "how teaching cues induce physiological responses". This results in a lack of interpretability and robustness of recognition results when facing data noise or complex contexts due to the lack of mechanism guidance. (2) Low efficiency of cross-modal collaborative enhancement: Existing feature enhancement schemes often do not fully consider the unidirectional influence features between modalities. In video learning, the learner's infection by the video content is a unidirectional driving process, while existing bidirectional interactive models are prone to introducing irrelevant physiological noise to interfere with video representation. Due to the lack of accurate tracking of such unidirectional infection factors, the model is difficult to establish a robust logical mapping between semantic features and physiological features, which limits the modeling accuracy under complex emotions; Individual sensitivity differences are ignored: Different learners have significantly different susceptibility to the same emotional cues in videos due to differences in psychological traits, concentration levels, etc. This sensitivity bias at the sensory and psychological levels directly affects the intensity and form of physiological responses. However, existing methods generally use globally consistent fusion weights, ignoring the differences in infection bias between individuals. This makes it difficult for the model to adapt to the personalized emotional expressions of different learners, affecting the generalization performance of the system in cross-individual applications.

[0005] Therefore, there is an urgent need for a multimodal emotion recognition method that can explicitly track emotional contagion factors, integrate individual response biases, and achieve contagion-driven cross-modal feature enhancement to improve the recognition accuracy, robustness, and psychological interpretability of the model in video learning scenarios, and promote the deep application of affective computing in intelligent education decision support. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a video-based emotion recognition method based on emotion contagion tracking and multimodal fusion. This method uses eye-tracking physiological signals to bias-modulate and model learners' individual differences and emotional feedback. This model guides the tracking of inducing factors in instructional videos and the weighting of contagion features. Furthermore, a unidirectional emotion contagion tracking module and a bidirectional cross-modal attention mechanism achieve deep enhancement and fusion of physiological responses and video features, ultimately improving the accuracy and robustness of emotion recognition.

[0007] The technical solution to achieve the objective of this invention is: A video-based emotion recognition method based on emotion contagion tracking and multimodal fusion includes the following steps: 1) Data Acquisition: Learners' eye movements and the physical characteristics of the instructional videos are collected during viewing. The mPLUG-Owl large model is then used to generate semantic descriptions of the video clips, including: 1-1) Eye movement signal acquisition: The learner's eye movement data during the learning process was collected using a flat-panel eye tracker, including fixation count, saccade count, fixation speed, saccade speed, fixation time, left eye pupil diameter, right eye pupil diameter, and average pupil diameter. 1-2) Physical feature acquisition of teaching videos: The dynamic change features of the video are characterized by calculating the pixel change rate between adjacent frames, and the mean, extreme values, variance and total pixel change statistics are extracted within the time window; 1-3) Video semantic extraction: The rich semantic information of the video is captured by generating video text descriptions. This text description includes detailed text information describing the scenes, objects, actions and plots in the video, as well as the emotions and emotional backgrounds that express the video content. The semantic description of the video segment is generated using the mPLUG-Owl large model. 2) Data Preprocessing: Preprocessing is performed on the acquired eye-tracking data, video physical features, and video semantic information, including: 2-1) Eye movement signal processing: Linear interpolation is performed to fill in missing eye movement values ​​and baseline correction is applied, specifically as follows: The Pearson correlation coefficients between these features and emotional states were calculated. The p-value was used to determine the significance of the correlation coefficient, i.e., the association between each feature and emotional state. When the p-value was less than 0.05, the correlation coefficient was considered significant, indicating a significant linear relationship between the two groups of samples. When the p-value was less than 0.01, the correlation coefficient was considered highly significant. Finally, 24 eye movement features that were significantly correlated with emotional state (i.e., p-values ​​less than 0.05) were selected: fixation count, saccade count, fixation speed, fixation time, left pupil diameter, right pupil diameter, maximum mean pupil diameter, fixation speed, saccade speed, left pupil diameter, right pupil diameter, minimum mean pupil diameter, fixation speed, saccade speed, fixation time, left pupil diameter, right pupil diameter, mean mean pupil diameter, left pupil diameter, right pupil diameter, and mean pupil diameter, standard deviation and variance of the mean pupil diameter. The formula for the Pearson correlation coefficient is defined as follows: , in , It is the number of data points. and These are the first two variables. The value of each data point and These are the means of the two variables, respectively, based on the Pearson correlation coefficient. To calculate the p-value, first calculate the statistic. When the sample size is Then: , Then based on the degrees of freedom Find the p-value corresponding to the t-statistic in the t-distribution table; Linear interpolation is used to fill in missing values ​​in eye-tracking data. The formula for linear interpolation is defined as follows: , in The time points of the frames adjacent to the missing value. for The corresponding eye movement data values ​​are: x represents the time point corresponding to the missing eye movement data, and y represents the missing eye movement data. Then, baseline correction is performed on these filled eye movement data, and the eye movement features are normalized to obtain 19-dimensional features. 2-2) Video Physical Feature Processing: Specifically, the videos watched by learners were analyzed. The learning videos were divided into 20 frames per second, and the hue, saturation, and brightness of each frame were calculated. The mean, maximum, minimum, variance, and standard deviation of the hue, saturation, and brightness of multiple images within each time window were calculated. The click-through rate of the video on the video website, the time-absolute time corresponding to the time of the learner's emotional state in the entire experimental process, and the time-relative time of the learner's emotional state relative to the current video were extracted, totaling 18 features as knowledge features. Then, the Pearson correlation coefficient and significance coefficient between the 18 knowledge features and the emotional state were calculated. Based on the significance coefficient, 17 knowledge features were retained. The calculation formulas for the hue, saturation, and brightness feature values ​​are shown in the following formulas: , , Where Br is brightness, S is saturation, H is hue, and R, G, and B refer to the RGB values ​​of the image. The visual features are then normalized to obtain 17-dimensional features. 2-3) Video semantic information processing: The generated video semantic description text is subjected to content inspection and cleaning, including typo correction, deletion of extra spaces, and verification of text length to ensure that the text length does not exceed the maximum input token limit of the subsequent feature extraction model BERT-Base, thereby reducing the large model illusion phenomenon and ensuring that the semantic description accurately reflects the content of the video segment. Preliminary features are extracted from the input BERT-Base, and the PCA dimensionality reduction algorithm is used to reduce the dimensionality of the features and further remove redundant information, finally obtaining a semantic feature vector with 25 dimensions; 3) Feature Structuring Enhancement: The preprocessed eye-tracking physiological signals, video physical features, and video semantic features are structurally enhanced, including: 3-1) Multimodal unified embedding projection: Utilizing independent linear layers to embed the preprocessed heterogeneous modal features in These represent eye-tracking, visual, and semantic projections onto a shared embedding space in a unified dimension. Preliminary indications were received. : ; 3-2) Structure-aware denoising: With relation matrix Multiply and obtain structural enhancement features by applying the GELU activation function and layer normalization. : , Noise suppression operators are generated using a structure-driven masking mechanism. The denoised representation is obtained by element-wise multiplication with the enhanced features. : , Using MLP combined with the Sigmoid activation function, based on the original features Generate attention weights : , The weights are applied to the denoised features to obtain the weighted enhanced representation. : , To preserve the integrity of the original features and prevent gradient vanishing, the original representation is... With modulated representation Residual aggregation is performed, and finally, layer normalization and Dropout processing are applied to output the final structured enhanced features. : , Through the above structured enhancement process, each heterogeneous modality is transformed into an enhancement vector with high semantic space consistency. This provides data support for accurately capturing the temporal infection correlation between video stimuli and physiological responses in the future; 4) Emotion-Driven Feature Enhancement and Emotion Recognition: This stage is the core of this technical solution, aiming to simulate the unidirectional driving mechanism of emotion transmission from teaching videos to learners' physiological states. It specifically includes the following detailed steps: 4-1) Tracing and Inference of Emotional Contagion Factors: Constructing a Video Visual-Semantic Composite Modality Using Low-Rank Outer Product With eye-tracking modality Similarity tensors between : , Project the similarity tensor back into a higher-dimensional space. By using Softmax normalization, we obtain the attention distribution that reflects the intensity of the teaching content's impact. ; 4-2) Introduce a bias modulator to correct for differences in sensitivity among learners to the same instructional stimuli: Based on eye movement features After processing through two fully connected layers, the individual bias vector is obtained. With regulation operator : , , The bias is injected into the infection attention tensor to generate the corrected personalized infection weights for each student. : , Introducing KL divergence loss To ensure consistency between the student distribution and the target distribution, entropy regularization loss is employed. Determinism of constrained predictions: , , according to By integrating teaching modalities, we can ultimately achieve emotional engagement. : ; 4-3) Infection-driven cross-modal collaborative enhancement: with For the query, perform cross-modal attention aggregation on video features, combined with Enhanced features are obtained : , by Adjust eye-tracking information to provide feedback to the query and generate enhanced visual features. With semantic features : ; 4-4) Adaptive Modality Fusion and Classification: The Modality Importance Estimator (MIE) is used to calculate the activation mean of each enhanced feature. with standard deviation : , Through the generated fusion weights Obtain the final multimodal sentiment features The Softmax classifier is used to output the probability of the sentiment category. : , , , Using classification loss With emotional contagion loss The weighted sum is used as the overall optimization objective: , , ; 5) Testing and Evaluation: Conduct comprehensive testing and evaluation of the multimodal emotion recognition system, perform cross-validation on the model, and use accuracy and F1 score to measure the model's performance.

[0008] Building upon current technology, this technical solution aims to more effectively and accurately simulate learners' cognitive and emotional states during video learning. Based on the theory of emotional contagion, it uses an eye tracker to collect eye-movement physiological signals in real time and extracts statistical features while learners watch instructional videos. Furthermore, this solution employs a multimodal large model mPLUG-Owl to extract visual and semantic descriptive information from instructional video clips and encodes it into video semantic feature vectors using BERT-Base. The core innovation of this technical solution lies in constructing a causal interaction tensor between the video stimulus modality and the eye-movement physiological modality using low-rank outer product technology to track potential emotional contagion factors. Simultaneously, a bias modulator is introduced to generate an individual sensitivity bias vector using eye-movement signals, which is used to personalize the contagion weights. Then, through a bidirectional cross-modal attention mechanism, a deep enhancement and fusion of physiological responses and video features is achieved, ultimately enabling accurate identification of four emotional states: interest, confusion, boredom, and happiness.

[0009] This technical solution has the following advantages: 1. Personalized Emotional Contagion Modeling: To address the issue that existing methods ignore the differences in learners' sensitivity to teaching stimuli, this technical solution uses a bias modulator to quantify individual sensory biases into calculable weights using eye-tracking physiological signals. This bias modulation mechanism based on physiological feedback can accurately simulate the personalized responses of different learners in the process of emotional contagion, effectively correcting the biases caused by uniform modeling, and making the model's portrayal of emotional states closer to the real psychological processing. 2. Precise and efficient tracking of infectious factors: To address the problem that traditional multimodal fusion models have huge computational overhead and are difficult to capture deep correlations, this technical solution adopts low-rank outer product technology to achieve deep interaction between video stimuli and physiological responses with extremely low computational load. By explicitly inferring potential emotional infectious factors, it avoids noise interference introduced by redundant features and achieves a high balance between performance and efficiency. 3. Infection-driven feature alignment and enhancement: To address the limitation of sentiment reasoning caused by the semantic gap between multimodal data, this technical solution adopts a bidirectional cross-modal attention mechanism guided by infection factors to model the unidirectional driving path of teaching cues to physiological responses. By introducing a hybrid loss function, including KL divergence loss and marginal loss, multi-objective joint optimization is carried out to ensure the consistency and stability of features in the semantic space, and significantly improve the robustness and interpretability of the model in the context of incomplete data or noisy environments. 4. Applicable to video learning monitoring scenarios: This technical solution is particularly suitable for online video learning environments. It can effectively simulate the dynamic process of emotions being transmitted from teaching videos to learners. By tracking physiological feedback in real time, it helps teachers and intelligent education systems to deeply understand the logic of learners' psychological changes, providing reliable technical support and psychological explanation for building personalized teaching interventions with emotional perception capabilities.

[0010] This method uses eye-tracking physiological signals to bias-modulate and model learners’ individual differences and emotional feedback. Based on this, it tracks the inducing factors of teaching videos and weights the infection features. Then, through a one-way emotion infection tracking module and a two-way cross-modal attention mechanism, it achieves a deep enhancement fusion of physiological response and video features, ultimately improving the accuracy and robustness of emotion recognition. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the overall process of the method in the embodiment; Figure 2 This is a schematic diagram of the multimodal data processing flow in the embodiment. Detailed Implementation

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0013] Example: Reference Figure 1 , Figure 2 A multimodal emotion recognition method based on probabilistic modeling and knowledge enhancement includes the following steps: 1) Data Collection: This stage aims to simultaneously collect learners' physiological response signals during the viewing of instructional videos, as well as emotional cues used to represent the instructional content, as detailed below: 1-1) Acquisition of Eye-Tracking Physiological Feedback Signals: In this example, a non-contact tablet eye tracker—the Tobii TX300—was used. The eye tracker was fixed below the display terminal to ensure high-fidelity data acquisition without interfering with the learner's natural learning state. Sampling and Calibration: The data sampling rate was set to 60Hz, i.e., 60 samples per second. Before formal data acquisition, the learner needed to complete device calibration by watching the calibration points on the screen. Feature Dimensions: Physiological responses of the learner during the 1-3 minute viewing of the instructional video were collected and statistically analyzed, as shown in Table 1. Table 1. List of 24 eye movement features , Finally, 24 eye movement features that were significantly related to emotional state were selected and obtained, covering fixation count, saccade count, fixation speed, fixation time, and the maximum, minimum, average, standard deviation and variance of the left and right pupil diameters. These data are used as physiological results after emotional infection and are used for subsequent source tracing of infectious agents. 1-2) Acquisition of Video Physical Stimulus Features: The dynamic visual stimuli of the image are described by calculating the pixel change rate between adjacent frames of the teaching video. Within a selected 1-second sliding window, the color attributes and change intensity features of the video image are extracted. Specifically, 17 key physical features are extracted as shown in Table 2, including the average, maximum, minimum, variance, and standard deviation of hue; the average, maximum, variance, and standard deviation of saturation; the average, maximum, minimum, variance, and standard deviation of brightness; the video's click-through rate on the video website; and the absolute and relative times of the video. Table 2 List of features of 17 video frames ; 1-3) Video semantic extraction: The multimodal large model mPLUG-Owl is used to parse the content of teaching video clips. The generated semantic description includes detailed text information, covering the scene layout, key objects, character actions, plot development, and the emotional background and atmosphere of the video content. The generated descriptive text is then input into the pre-trained BERT-Base model for deep semantic analysis. 2) Data Preprocessing: Preprocessing is performed on the acquired eye-tracking data, PPG data, and video semantic information, including: 2-1) Eye-tracking data processing: Missing values ​​in the eye-tracking data are filled in using a linear interpolation method. The formula for the linear interpolation method is defined as follows: , in The time points of the frames adjacent to the missing value. for The corresponding eye movement data values, where x is the time point corresponding to the missing eye movement data and y is the missing eye movement data, are then baseline corrected for these filled eye movement data. 2-2) Standardization of video physical features: Content verification is performed on the extracted physical indicators such as hue, saturation, and brightness. Since the physical parameters of different videos vary greatly, the features of 17 video frames need to be normalized and mapped to the 0-1 range to reduce the impact of numerical magnitude on model training. 2-3) Text cleaning and deep encoding of video semantic information: The video description text generated by mPLUG-Owl is corrected, redundant spaces and special characters are removed, and the text length is checked to ensure that it meets the input constraints of the subsequent pre-trained model in order to reduce the illusion phenomenon in the large model generation process. The pre-trained BERT-Base model is used to perform deep encoding on the cleaned semantic description and extract high-dimensional semantic feature vectors containing video plot, scene and emotional background. 3) Feature Structuring Enhancement: The preprocessed eye-tracking physiological signals, video physical features, and video semantic features are structurally enhanced to eliminate distribution differences between modalities and suppress redundant noise, as detailed below: 3-1) Multimodal unified embedding projection: Utilizing independent linear layers to embed the preprocessed heterogeneous modal features in (representing eye movement, vision, and semantics respectively) projected onto a shared embedding space of a unified dimension. Preliminary indications were received. : ; 3-2) Structure-aware denoising: With relation matrix Multiply and obtain structural enhancement features by applying the GELU activation function and layer normalization. : , Noise suppression operators are generated using a structure-driven masking mechanism. The denoised representation is obtained by element-wise multiplication with the enhanced features. : , Using MLP combined with the Sigmoid activation function, based on the original features Generate attention weights : , The weights are applied to the denoised features to obtain the weighted enhanced representation. : , To preserve the integrity of the original features and prevent gradient vanishing, the original representation is... With modulated representation Residual aggregation is performed, and finally, layer normalization and Dropout processing are applied to output the final structured enhanced features. : , Through the above structured enhancement process, each heterogeneous modality is transformed into an enhancement vector with high semantic space consistency. This provides data support for accurately capturing the temporal infection correlation between video stimuli and physiological responses in the future; 4) Emotionally contagious feature enhancement and emotion recognition: including: 4-1) Tracing and Inference of Emotional Contagion Factors: Constructing a Video Visual-Semantic Composite Modality Using Low-Rank Outer Product With eye-tracking modality Similarity tensors between : , Project the similarity tensor back into a higher-dimensional space. By using Softmax normalization, we obtain the attention distribution that reflects the intensity of the teaching content's impact. ; 4-2) Introduce a bias modulator to correct for differences in sensitivity among learners to the same instructional stimuli: Based on eye movement features After processing through two fully connected layers, the individual bias vector is obtained. With regulation operator : , , The bias is injected into the infection attention tensor to generate the corrected personalized infection weights for each student. : , Introducing KL divergence loss Constrain the consistency between the student distribution and the target distribution; employ entropy regularization loss. Determinism of constrained predictions: , , according to By integrating teaching modalities, we can ultimately achieve emotional engagement. : ; 4-3) Infection-driven cross-modal collaborative enhancement: with For the query, perform cross-modal attention aggregation on video features, combined with Enhanced features are obtained : , by Adjust eye-tracking information to provide feedback to the query and generate enhanced visual features. With semantic features : ; 4-4) Adaptive Modality Fusion and Classification: The modality importance estimator (MIE) is used to calculate the activation mean of each enhanced feature. with standard deviation : , Through the generated fusion weights Obtain the final multimodal sentiment features And use a Softmax classifier to output the probability of the sentiment category. : , , , Using classification loss With emotional contagion loss The weighted sum is used as the overall optimization objective: , , ; 5) Testing and Evaluation: A comprehensive test and evaluation of the multimodal emotion recognition system was conducted. Five-fold cross-validation was performed on the model, and accuracy, F1 score, and recall were used to measure the model's performance. The test results are shown in Table 3 below. Table 3 Comparison of model performance under different aggregation methods , Experimental results demonstrate the superior performance and robust effectiveness of this method. On the VLMED dataset, the model achieved an accuracy of 97.32%, an F1 score of 96.94%, and a recall of 97.01%. Compared to traditional machine learning and deep learning models such as SVM, DNN, and Transformer, all metrics show significant improvements of 7% to 18%, proving the superior ability of this method in handling complex video learning emotion recognition tasks. Ablation experiments further confirm the value of the core modules: removing the feature structuring enhancement module or the emotion contagion tracking fusion module reduced the model accuracy by 1.44% and 6.2%, respectively. This strongly demonstrates the key driving role of the emotion contagion mechanism in capturing the deep correlation between teaching stimuli and physiological responses, as well as the necessity of structured feature enhancement in suppressing modal noise and improving recognition accuracy.

Claims

1. A video-based emotion recognition method based on emotion contagion tracking and multimodal fusion, characterized in that, Includes the following steps: 1) Data Acquisition: Learners' eye movements and the physical characteristics of the instructional videos are collected during viewing. The mPLUG-Owl large model is then used to generate semantic descriptions of the video clips, including: 1-1) Eye movement signal acquisition: The learner's eye movement data during the learning process was collected using a flat-panel eye tracker, including fixation count, saccade count, fixation speed, saccade speed, fixation time, left eye pupil diameter, right eye pupil diameter, and average pupil diameter. 1-2) Physical feature acquisition of teaching videos: The dynamic change features of the video are characterized by calculating the pixel change rate between adjacent frames, and the mean, extreme values, variance and total pixel change statistics are extracted within the time window; 1-3) Video semantic extraction: The rich semantic information of the video is captured by generating video text descriptions. This text description includes detailed text information describing the scenes, objects, actions and plots in the video, as well as the emotions and emotional backgrounds that express the video content. The semantic description of the video segment is generated using the mPLUG-Owl large model. 2) Data Preprocessing: Preprocessing is performed on the acquired eye-tracking data, video physical features, and video semantic information, including: 2-1) Eye movement signal processing: Linear interpolation is performed to fill in missing eye movement values ​​and baseline correction is applied, specifically as follows: The Pearson correlation coefficients between these features and emotional states were calculated. The p-value was used to determine the significance of the correlation coefficient, i.e., the association between each feature and emotional state. When the p-value was less than 0.05, the correlation coefficient was considered significant, indicating a significant linear relationship between the two groups of samples. When the p-value was less than 0.01, the correlation coefficient was considered highly significant. Finally, 24 eye movement features that were significantly correlated with emotional state (i.e., p-values ​​less than 0.05) were selected: fixation count, saccade count, fixation speed, fixation time, left pupil diameter, right pupil diameter, maximum mean pupil diameter, fixation speed, saccade speed, left pupil diameter, right pupil diameter, minimum mean pupil diameter, fixation speed, saccade speed, fixation time, left pupil diameter, right pupil diameter, mean mean pupil diameter, left pupil diameter, right pupil diameter, and mean pupil diameter, standard deviation and variance of the mean pupil diameter. The formula for the Pearson correlation coefficient is defined as follows: ， in , It is the number of data points. and These are the first two variables. The value of each data point and These are the means of the two variables, respectively, based on the Pearson correlation coefficient. To calculate the p-value, first calculate the statistic. When the sample size is Then: ， Then based on the degrees of freedom Find the p-value corresponding to the t-statistic in the t-distribution table; Linear interpolation is used to fill in missing values ​​in eye-tracking data. The formula for linear interpolation is defined as follows: ， in The time points of the frames adjacent to the missing value. for The corresponding eye movement data values ​​are: x represents the time point corresponding to the missing eye movement data, and y represents the missing eye movement data. Then, baseline correction is performed on these filled eye movement data, and the eye movement features are normalized to obtain 19-dimensional features. 2-2) Selection of Video Physical Features: The videos watched by learners were analyzed. The learning videos were divided into 20 frames per second. The hue, saturation, and brightness of each frame were calculated. The mean, maximum, minimum, variance, and standard deviation of hue, saturation, and brightness of multiple images within each time window were calculated. The click-through rate of the video on the video website, the time-absolute time corresponding to the time of the learner's emotional state in the entire experimental process, and the time-relative time of the learner's emotional state relative to the current video were extracted, totaling 18 features as knowledge features. Then, the Pearson correlation coefficient and significance coefficient between the 18 knowledge features and the emotional state were calculated. Based on the significance coefficient, 17 knowledge features were retained. The calculation formulas for the hue, saturation, and brightness feature values ​​are shown in the following formulas: ， ， Where Br is brightness, S is saturation, H is hue, and R, G, and B refer to the RGB values ​​of the image. The visual features are then normalized to obtain 17-dimensional features. 2-3) Video semantic information processing: The generated video semantic description text is checked and cleaned, including typo correction, deletion of extra spaces, and the text length is checked to ensure that the text length does not exceed the maximum input token limit of the subsequent feature extraction model BERT-Base. Preliminary features are extracted from the input BERT-Base, and the PCA dimensionality reduction algorithm is used to finally obtain a semantic feature vector with 25 dimensions. 3) Feature Structuring Enhancement: The preprocessed eye-tracking physiological signals, video physical features, and video semantic features are structurally enhanced, including: 3-1) Multimodal unified embedding projection: Utilizing independent linear layers to embed the preprocessed heterogeneous modal features in These represent eye-tracking, visual, and semantic projections onto a shared embedding space in a unified dimension. Preliminary indications were received. : ； Will With relation matrix Multiply and obtain structural enhancement features by applying the GELU activation function and layer normalization. : ， 3-2) Structure-aware denoising: Noise suppression operators are generated through a structure-driven masking mechanism. The denoised representation is obtained by element-wise multiplication with the enhanced features. : ， Using MLP combined with the Sigmoid activation function, based on the original features Generate attention weights : ， The weights are applied to the denoised features to obtain the weighted enhanced representation. : ， Original representation With modulated representation Residual aggregation is performed, and finally, layer normalization and Dropout processing are applied to output the final structured enhanced features. : ， Through the above structured enhancement process, each heterogeneous modality is transformed into an enhancement vector with high semantic space consistency. ; 4) Emotionally contagious feature enhancement and emotion recognition: including: 4-1) Tracing and Inference of Emotional Contagion Factors: Constructing a Video Visual-Semantic Composite Modality Using Low-Rank Outer Product With eye-tracking modality Similarity tensors between : ， Project the similarity tensor back into a higher-dimensional space. By using Softmax normalization, we obtain the attention distribution that reflects the intensity of the teaching content's impact. ; 4-2) Introduce a bias modulator to correct for differences in sensitivity among learners to the same instructional stimuli: Based on eye movement features After processing through two fully connected layers, the individual bias vector is obtained. With regulation operator : ， ， The bias is injected into the infection attention tensor to generate the corrected personalized infection weights for each student. : ， Introducing KL divergence loss To ensure consistency between the student distribution and the target distribution, entropy regularization loss is employed. Determinism of constrained predictions: ， ， according to By integrating teaching modalities, we can ultimately achieve emotional engagement. : ； 4-3) Infection-driven cross-modal collaborative enhancement: with For the query, perform cross-modal attention aggregation on video features, combined with Enhanced features are obtained : ， by Adjust eye-tracking information to provide feedback to the query and generate enhanced visual features. With semantic features : ； 4-4) Adaptive Modality Fusion and Classification: The Modality Importance Estimator (MIE) is used to calculate the activation mean of each enhanced feature. with standard deviation : ， Through the generated fusion weights Obtain the final multimodal sentiment features The Softmax classifier is used to output the probability of the sentiment category. : ， ， ， Using classification loss With emotional contagion loss The weighted sum is used as the overall optimization objective: ， ， ； 5) Testing and Evaluation: Conduct comprehensive testing and evaluation of the multimodal emotion recognition system, perform cross-validation on the model, and use accuracy and F1 score to measure the model's performance.

Images