Emotion analysis method based on multi-mode electroencephalogram eye movement fusion

Abstract

The invention discloses an emotion analysis method and system based on multi-mode electroencephalogram eye movement fusion. An emotional induction stimulation sequence is alternately or synchronously displayed through a stimulation presentation module according to preset time sequences such as 5-second vision and 3-second auditory sense, a 64 conductive electrode cap (10-20 system layout, the sampling rate is larger than or equal to 1000 Hz) is triggered to collect electroencephalogram data of a prefrontal lobe, a temporal lobe and the like, and meanwhile eye movement data are obtained through an infrared pupil tracking technology (the sampling frequency is larger than or equal to 120 Hz). 0.5-70 Hz band-pass filtering and ICA artifact removal processing are carried out on the electroencephalogram data, smooth interpolation and Kalman filtering optimization are carried out on the eye movement data, and then electroencephalogram alpha / beta / gamma wave power spectrums, eye movement pupil change rates and other characteristics are extracted respectively. A Transform model based on an attention mechanism is adopted, deep fusion of electroencephalogram eye movement features is realized through feature embedding and multi-head self-attention calculation, and emotion recognition accuracy greater than or equal to 85% is achieved through confusion matrix optimization and five-fold cross validation in combination with an SVM or DNN classifier.

Description

Technical Field

[0001] This invention provides a sentiment analysis method, and particularly relates to a sentiment analysis method based on multimodal EEG-eye-tracking fusion. Background Technology

[0002] In the field of emotion analysis technology, emotion analysis devices aim to identify an individual's emotional state by collecting and processing human physiological signals or behavioral data, and are widely used in fields such as human-computer interaction and mental health monitoring. The basic structure of existing emotion analysis devices typically includes single or a few modal acquisition modules, such as relying solely on facial expression cameras and microphones to collect data, or using simple EEG acquisition equipment (sparse electrode layout, low sampling rate); the data processing modules mostly use traditional machine learning algorithms (such as Naive Bayes and decision trees) to perform simple classification of single-modal data; although some devices involve multimodal fusion, they only process data through simple methods such as feature splicing, lacking in-depth correlation analysis of EEG, eye-tracking, and other data.

[0003] However, existing devices have significant shortcomings. In terms of data acquisition, single-modal approaches are easily affected by environmental factors (such as facial expressions being affected by lighting, and speech being affected by noise), resulting in poor acquisition accuracy and stability, making it difficult to comprehensively reflect emotional states. EEG acquisition devices suffer from unreasonable electrode layouts and low sampling rates, failing to capture complete emotion-related EEG signals. In the data processing stage, traditional algorithms have limited classification capabilities, making it difficult to handle complex emotional scenarios. Simple multimodal fusion methods cannot uncover the dynamic synergistic relationship between EEG and eye-tracking data, leading to generally low emotion recognition accuracy (typically below 75%), and lacking individual difference calibration mechanisms, they cannot adapt to the emotional expression characteristics of different users, making it difficult to meet the needs of scenarios such as intelligent education and precise mental health analysis. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a multimodal situational awareness analysis system and its analysis method, which solves the problems of poor accuracy, shallow multimodal fusion, and weak individual adaptation in traditional sentiment analysis.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a sentiment analysis method based on multimodal EEG-eye-tracking fusion, comprising:

[0006] 1.1 Stimulus presentation steps: Displaying an emotionally evoking stimulus sequence, the stimulus sequence includes visual stimuli (such as emotional pictures, video clips) and auditory stimuli (such as emotional voice), and each stimulus is presented alternately or synchronously according to a preset time sequence (such as 5 seconds of visual stimulus → 3 seconds of auditory stimulus).

[0007] 1.2 Multimodal data acquisition steps: EEG and eye movement data were acquired simultaneously. EEG data acquisition used a 64-electrode cap with an international 10-20 system layout, focusing on covering emotion-related brain regions such as the prefrontal cortex (Fp1, Fp2) and temporal lobe (T3, T4), with a sampling rate ≥1000Hz. Eye movement data acquisition used infrared pupil tracking technology to record parameters such as fixation point position, pupil diameter changes, and saccade speed, with a sampling frequency ≥120Hz.

[0008] 1.3 Data preprocessing steps: bandpass filtering (0.5-70Hz) and independent component analysis (ICA) were performed on the EEG data to remove EEG artifacts, and smoothing interpolation was performed on the eye movement data.

[0009] 1.4 Feature extraction steps: extract power spectrum features of alpha waves (8-13Hz), beta waves (13-30Hz), and gamma waves (30-70Hz) and event-related potential (ERP) components (such as P300 and N170) from the preprocessed EEG data; extract features such as pupil diameter change rate, fixation duration, and saccade frequency from the eye movement data.

[0010] 1.5 Multimodal fusion analysis steps: The attention-based multimodal Transformer model is used to deeply fuse EEG time-frequency features with eye movement temporal features, and feature complementarity enhancement is achieved through self-attention weight allocation;

[0011] 1.6 The emotion classification and recognition steps utilize fused features to train a support vector machine (SVM) or deep neural network (DNN) classifier to recognize basic emotional states such as joy, sadness, anger, fear, and neutrality, with a classification accuracy of ≥85%.

[0012] Preferably, in the stimulus presentation step, visual stimuli use a standardized emotional picture library (such as the International Emotional Picture System IAPS), auditory stimuli use a standardized emotional speech library (such as the Berlin Emotional Speech Database), and stimulus intensity is calibrated using a subjective rating scale (such as the Self-Assessment Manikin, SAM).

[0013] Preferably, in the data preprocessing step, the EEG data preprocessing further includes adaptive noise cancellation and wavelet transform denoising, and the eye movement data preprocessing uses the Kalman filter algorithm to smooth the trajectory, so that the eye movement positioning error is ≤0.5° of visual angle.

[0014] Preferably, in the feature extraction step, the time-frequency features extracted from the EEG data include wavelet packet decomposition coefficients and differential entropy features, and the features extracted from the eye-tracking data include fixation point distribution entropy and pupil dynamic response curve features.

[0015] Preferably, in the multimodal fusion analysis step, the multimodal Transformer model includes:

[0016] 5.1 Feature embedding layer, which maps EEG feature vectors and eye-tracking feature vectors to a unified dimensional space;

[0017] 5.2 Multi-head self-attention layer: Calculate the association weights between EEG and eye movement features using the following formula:

[0018] Attention(Q,K,V)=softmax(QK^T / √d_k)V

[0019] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively, and d_k is the dimension of the key vector;

[0020] 5.3 Residual connectivity and layer normalization enhance model training stability;

[0021] 5.4 Location encoding layer, adding temporal information to preserve the temporal dependence of features.

[0022] Preferably, in the emotion classification and recognition step, a confusion matrix is ​​used to optimize the classifier parameters, five-fold cross-validation is used to evaluate the model's generalization ability, and finally, the probability distribution of the emotion state and the confidence score are output.

[0023] Preferred: It also includes a real-time monitoring and feedback step for emotional state, continuously updating the emotion recognition results through a sliding time window (such as a 5-second window with a step size of 1 second), and triggering a feedback mechanism (such as pushing emotion adjustment suggestions) according to a preset threshold.

[0024] Preferred: It also includes an individual difference calibration step, which establishes a user-specific emotion feature template through the initial baseline acquisition stage, and dynamically adjusts the classification threshold in subsequent analysis to improve the accuracy of individual emotion recognition by ≥15%.

[0025] Preferably, the method can be applied to scenarios such as learning emotion monitoring in intelligent education systems, screening for emotional disorders in mental health assessments, and optimizing emotional immersion experiences in virtual reality interactions.

[0026] A sentiment analysis system based on multimodal EEG-eye-tracking fusion includes:

[0027] 10.1 A stimulus presentation module, used to perform the stimulus presentation steps as described in claim 1;

[0028] 10.2 A multimodal data acquisition module, including an EEG acquisition device (such as a 64-channel EEG amplifier) ​​and an eye-tracking device (such as an infrared eye tracker), is used to perform the data acquisition steps described in claim 1;

[0029] 10.3 A data processing server, configured with a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of claims 1-9.

[0030] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0031] This invention, based on the principle of multimodal information complementarity, achieves precise emotion analysis by constructing a data fusion system for electroencephalography (EEG) and eye movement (EMG). First, visual and auditory stimulus sequences are used to induce emotional responses in subjects. Simultaneously, 64 electrode caps (in a 10-20 system layout) are triggered to collect electrical activity signals from emotion-related brain regions such as the prefrontal and temporal lobes at a sampling rate of ≥1000Hz. Infrared pupil tracking technology is used to capture eye movement parameters such as fixation point and pupillary changes at a frequency of ≥120Hz, obtaining multidimensional physiological response data. The EEG data is denoised using algorithms such as bandpass filtering and independent component analysis. Kalman filtering is then used to smooth the eye movement trajectory, eliminating interference factors. After this process, features such as EEG α / β / γ wave power spectra, event-related potentials, pupillary change rate, and fixation duration are extracted. A Transformer model based on attention mechanisms maps two types of features to a unified space through feature embedding. Multi-head self-attention mechanisms are used to dynamically calculate the correlation weights between EEG and EMG features, uncovering the synergistic patterns of neural electrical activity and visual behavior in emotional expression, achieving deep fusion. Finally, by using a support vector machine or deep neural network classifier, combined with confusion matrix optimization and cross-validation, the fused features are mapped to emotion categories such as joy and sadness, and the probability distribution and confidence are output. At the same time, individual baseline calibration and sliding time window are used for real-time monitoring to adapt to individual differences and achieve dynamic tracking of emotional state, and to analyze emotional features from the dual dimensions of physiological signal essence and behavioral performance.

[0032] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0033] Figure 1 This is a time-series flowchart of a multimodal emotion analysis method based on multimodal EEG-eye-tracking fusion according to the present invention.

[0034] Figure 2 This is a system hardware architecture connection diagram of an emotion analysis method based on multimodal EEG-eye-tracking fusion according to the present invention;

[0035] Figure 3 This is a flowchart illustrating the multimodal data processing of an emotion analysis method based on multimodal EEG-eye-tracking fusion according to the present invention.

[0036] Figure 4 This is a flowchart illustrating the individual difference calibration process for an emotion analysis method based on multimodal EEG-eye-tracking fusion according to the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] It should be noted that the terms "vertical," "horizontal," "up," "down," "left," "right," and similar expressions used in this article are for illustrative purposes only and do not represent the only possible implementation.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0040] like Figure 1 and Figure 2 As shown, an emotion analysis method based on multimodal EEG-eye-tracking fusion first presents a sequence of evoked stimuli, including visual (emotional images, videos) and auditory (emotional speech) stimuli, alternating or simultaneously in a preset time sequence such as 5 seconds of visual stimuli → 3 seconds of auditory stimuli. Following a multimodal data acquisition step, EEG data is acquired using a 64-electrode cap (10-20 system layout, covering the prefrontal and temporal lobes, sampling rate ≥1000Hz), and eye-tracking data is acquired using infrared pupil tracking technology (sampling frequency ≥120Hz). In the data preprocessing step, the EEG data undergoes 0.5-70Hz bandpass filtering, ICA artifact removal, and adaptive noise cancellation. The process involves: smoothing and interpolating eye-tracking data, followed by Kalman filtering (positioning error ≤ 0.5° visual angle); extracting features such as α / β / γ wave power spectra from EEG and pupil change rate from eye-tracking; employing a Transformer model based on attention mechanisms for deep fusion of EEG and eye-tracking features through feature embedding, multi-head self-attention calculation, and residual connections; and training an SVM or DNN classifier using the fused features, combined with confusion matrix optimization and five-fold cross-validation to achieve an emotion recognition accuracy of ≥ 85%. Further improvements can be made through real-time monitoring and individual difference calibration.

[0041] In this implementation, the stimulus presentation module is connected to a 4K display screen via an HDMI interface, and the audio output port is connected to a high-fidelity speaker. It outputs stimulus content from a standardized emotional image library (such as IAPS) and the Berlin emotional speech database in a preset sequence of 5 seconds visual → 3 seconds auditory. The 64-channel EEG acquisition electrode caps are precisely positioned in the prefrontal lobe (Fp1, Fp2), temporal lobe (T3, T4), and other areas according to a 10-20 system. They are connected to a 16-bit ADC precision EEG amplifier via shielded cables. The amplifier transmits EEG data with a sampling rate of ≥1000Hz to the data processing host via a USB 3.2 interface. The infrared pupil-tracking eye tracker is installed 15cm below the display screen, maintaining a working distance of 45cm from the subject's eyes. It acquires eye movement data at a frequency of ≥120Hz and transmits it back in real time via gigabit Ethernet. The preprocessing module built into the data processing host sequentially performs 0.5-70Hz bandpass filtering, ICA artifact removal, and adaptive noise cancellation on the EEG data. After cubic spline interpolation and smoothing, the eye movement data is then processed by a Kalman filter algorithm to control the positioning error to ≤0.5° of visual angle. The feature extraction module analyzes the α / β / γ wave power spectrum and P300 component from the EEG signal and extracts parameters such as pupil dilation rate and fixation entropy from the eye movement signal. After normalization, the two types of features are input into a Transformer model based on the attention mechanism. The EEG and eye movement features are mapped to a 256-dimensional space through the feature embedding layer. The cross-modal feature weights are dynamically calculated using a multi-head self-attention mechanism. Deep fusion is achieved by combining residual connections and layer normalization. Finally, the emotional state recognition is completed by an SVM or DNN classifier. Through precise physical layout, high-speed data transmission links, and algorithmic collaboration, the components innovatively break through the limitations of traditional single-modal emotion analysis. The precise layout of EEG electrodes and high-frequency sampling ensure the complete capture of neural electrical activity, while the high-precision tracking of the eye tracker enables dynamic monitoring of visual behavior. The introduction of the Transformer model solves the problem of heterogeneous fusion of multimodal features, improving the accuracy of emotion recognition to ≥85%. At the same time, the individual difference calibration module establishes personalized feature templates through baseline acquisition, effectively overcoming the problem of differences in emotional expression among different users. This provides a high-precision and universal emotion analysis solution for fields such as intelligent education and mental health.

[0042] like Figure 3 and Figure 4As shown, an emotion analysis system and method based on multimodal EEG-eye-tracking fusion is presented. The system includes a stimulus presentation module, a multimodal data acquisition module (including a 64-channel EEG amplifier and an infrared eye tracker), and a data processing server. The data processing server performs steps from stimulus presentation and multimodal data acquisition to data preprocessing, feature extraction, multimodal fusion analysis based on the Transformer model, and SVM / DNN emotion classification and recognition. The stimulus sources use a standardized emotion database and are calibrated with subjective scales. The data processing integrates multiple algorithms to improve accuracy. This system and method can be applied to scenarios such as learning emotion monitoring in intelligent education, emotional disorder screening in mental health assessment, and optimization of emotional immersion experience in virtual reality interaction. It achieves real-time emotion monitoring and feedback through a sliding time window and optimizes recognition accuracy based on individual baseline calibration, providing an efficient solution for multi-domain emotion analysis.

[0043] In this implementation scheme, during the use of this device, conductive gel for EEG electrodes (such as Ten20 conductive gel) is applied to the contact area between the 64 conductive electrode cap and the scalp to reduce electrode-skin impedance and ensure high-quality acquisition of EEG signals. An ergonomic headrest and chin support are used to stabilize the subject's head, reducing artifacts caused by head movement. The support is made of medical-grade ABS plastic, combining strength and comfort. Eye tracker calibration requires a standard nine-point calibration board, coupled with existing automatic calibration algorithms, to complete device calibration and ensure the accuracy of eye-tracking data acquisition. For data transmission, an industrial-grade PoE switch establishes a stable network connection between the eye tracker and the data processing host. The EEG amplifier and host use shielded USB 3.2 Gen2 cables for data transmission to avoid signal attenuation and electromagnetic interference. Data storage utilizes an enterprise-grade solid-state drive array (such as Samsung 870 QVO), employing RAID 10 technology to ensure data security and read / write speeds. Before presenting emotional stimuli, standardized images from the International Emotional Image System (IAPS) and audio files from the Berlin Emotional Speech Database (EMO-DB) were used, and the stimulus sequences were arranged using existing audio and video editing software such as Adobe Premiere Pro. During the experiment, existing ambient light sensors (such as BH1750) were used to monitor the laboratory light intensity in real time to ensure environmental consistency. Simultaneously, a soundproof room (with polyester fiber sound-absorbing panels on the interior walls) and an electromagnetic shielding mesh (copper woven mesh) were used to reduce external noise and electromagnetic interference. In the data processing stage, Transformer model training was accelerated using the NVIDIA CUDA parallel computing platform. Python data analysis libraries (such as NumPy, SciPy, and PyTorch) were deployed using the Anaconda environment management tool, and interactive development and debugging of algorithm code were achieved using Jupyter Notebook. Finally, existing data visualization tools such as Tableau were used to visually present the sentiment analysis results in the form of dynamic heatmaps and trend curves, facilitating users' quick understanding of changes in emotional state.

[0044] Specifically, in practical implementation, the stimulus presentation module can adopt a dual-screen display system. The main screen is used to present visual stimuli, and the secondary screen serves as the control terminal. Stimulus sequences and parameter settings are arranged using existing graphical user interface (GUI) software (such as a dedicated control program written in Qt). During eye-tracking acquisition, the infrared pupil tracker is fixed on a tripod, and a level is used to ensure the device is horizontal. The lens spacing is adjusted according to the subject's interpupillary distance, and the device's built-in calibration program (such as the nine-point calibration method for EyeLink series eye trackers) is used for accuracy calibration. Before wearing the EEG acquisition electrode caps, the subject's scalp must be cleaned with medical alcohol wipes to remove oil. The tightness of the electrode caps is adjusted individually using Velcro, ensuring that the contact impedance of each electrode is below 5kΩ. The accompanying software for EEG acquisition systems such as BioSemiActiveTwo is used for electrode impedance detection and data acquisition start / stop control.

[0045] In the data transmission stage, a high-speed data transmission network is built using fiber optic network switches. The EEG amplifier and eye tracker transmit data to a data processing server equipped with a 10 Gigabit Ethernet card via fiber optic interfaces. Existing zero-copy technologies (such as DPDK) are used to achieve rapid data reception and processing. In the data preprocessing stage, bandpass filtering of the EEG signals is performed using filtering functions in the MATLAB SignalProcessingToolbox, and the ICA algorithm in the EEGLAB toolbox is used to remove artifacts. Smoothing interpolation of the eye-tracking data is implemented using the SciPy library in Python. During feature extraction, the MNE-Python library is used to extract power spectrum features and event-related potential components of the EEG signals, and the OpenCV library is used for trajectory analysis and feature extraction of the eye-tracking data.

[0046] Multimodal fusion analysis employs a Transformer model built on the PyTorch framework. During training, the Adam optimizer and learning rate scheduler (such as CosineAnnealingLR) are used to update model parameters. Existing distributed training techniques (such as Horovod) are utilized for parallel training on multiple GPUs (such as NVIDIA RTX 3090) to shorten training time. For sentiment classification and recognition, a support vector machine classifier is built using the Scikit-learn library, with hyperparameter tuning performed through grid search and cross-validation; alternatively, a deep neural network is built using TensorFlow, employing EarlyStopping to prevent overfitting. During system operation, a monitoring platform using Prometheus and Grafana is established to monitor the real-time running status and resource usage of each module, ensuring stable system operation. The final analysis results can be output via a RESTful API interface, facilitating integration into other application systems and enabling multi-scenario applications of sentiment analysis results.

[0047] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the claims.

Claims

1. A sentiment analysis method based on multimodal EEG-eye movement fusion, characterized in that, include: 1.1 Stimulus presentation steps: Displaying an emotionally evoking stimulus sequence, the stimulus sequence includes visual stimuli (such as emotional pictures, video clips) and auditory stimuli (such as emotional voice), and each stimulus is presented alternately or synchronously according to a preset time sequence (such as 5 seconds of visual stimulus → 3 seconds of auditory stimulus). 1.2 Multimodal data acquisition steps: EEG and eye movement data were acquired simultaneously. EEG data acquisition used a 64-electrode cap with an international 10-20 system layout, focusing on covering emotion-related brain regions such as the prefrontal cortex (Fp1, Fp2) and temporal lobe (T3, T4), with a sampling rate ≥1000Hz. Eye movement data acquisition used infrared pupil tracking technology to record parameters such as fixation point position, pupil diameter changes, and saccade speed, with a sampling frequency ≥120Hz. 1.3 Data preprocessing steps: bandpass filtering (0.5-70Hz) and independent component analysis (ICA) were performed on the EEG data to remove EEG artifacts, and smoothing interpolation was performed on the eye movement data. 1.4 Feature extraction steps: extract power spectrum features of alpha waves (8-13Hz), beta waves (13-30Hz), and gamma waves (30-70Hz) and event-related potential (ERP) components (such as P300 and N170) from the preprocessed EEG data; extract features such as pupil diameter change rate, fixation duration, and saccade frequency from the eye movement data. 1.5 Multimodal fusion analysis steps: The attention-based multimodal Transformer model is used to deeply fuse EEG time-frequency features with eye movement temporal features, and feature complementarity enhancement is achieved through self-attention weight allocation; 1.6 The emotion classification and recognition steps utilize fused features to train a support vector machine (SVM) or deep neural network (DNN) classifier to recognize basic emotional states such as joy, sadness, anger, fear, and neutrality, with a classification accuracy of ≥85%.

2. The emotion analysis method based on multimodal EEG-eye movement fusion according to claim 1, characterized in that: In the stimulus presentation step, visual stimuli use a standardized emotional picture library (such as the International Emotional Picture System IAPS), auditory stimuli use a standardized emotional speech library (such as the Berlin Emotional Speech Database), and stimulus intensity is calibrated using a subjective rating scale (such as the Self-Assessment Manikin SAM).

3. The emotion analysis method based on multimodal EEG-eye movement fusion according to claim 1, characterized in that: In the data preprocessing steps, EEG data preprocessing also includes adaptive noise cancellation and wavelet transform denoising, and eye movement data preprocessing uses the Kalman filter algorithm to smooth the trajectory, so that the eye movement positioning error is ≤0.5° of visual angle.

4. The emotion analysis method based on multimodal EEG-eye movement fusion according to claim 1, characterized in that: In the feature extraction step, the time-frequency features extracted from EEG data include wavelet packet decomposition coefficients and differential entropy features, while the features extracted from eye-tracking data include fixation point distribution entropy and pupil dynamic response curve features.

5. The emotion analysis method based on multimodal EEG-eye movement fusion according to claim 1, characterized in that: In the multimodal fusion analysis step, the multimodal Transformer model includes: 5.1 Feature embedding layer, which maps EEG feature vectors and eye-tracking feature vectors to a unified dimensional space; 5.2 Multi-head self-attention layer: Calculate the association weights between EEG and eye movement features using the following formula: Attention(Q,K,V)=softmax(QK^T / √d_k)V Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively, and d_k is the dimension of the key vector; 5.3 Residual connectivity and layer normalization enhance model training stability; 5.4 Location encoding layer, adding temporal information to preserve the temporal dependence of features.

6. The emotion analysis method based on multimodal EEG-eye movement fusion according to claim 1, characterized in that: In the emotion classification and recognition step, a confusion matrix is ​​used to optimize the classifier parameters, and five-fold cross-validation is used to evaluate the model's generalization ability. Finally, the probability distribution of the emotion state and the confidence score are output.

7. The emotion analysis method based on multimodal EEG-eye movement fusion according to claim 1, characterized in that: It also includes a real-time monitoring and feedback step for emotional state, continuously updating the emotion recognition results through a sliding time window (such as a 5-second window with a step size of 1 second), and triggering a feedback mechanism (such as pushing emotion adjustment suggestions) based on a preset threshold.

8. The emotion analysis method based on multimodal EEG-eye movement fusion according to claim 1, characterized in that: It also includes an individual difference calibration step, which establishes user-specific emotion feature templates through the initial baseline acquisition stage and dynamically adjusts the classification threshold in subsequent analysis to improve the accuracy of individual emotion recognition by ≥15%.

9. The emotion analysis method based on multimodal EEG-eye movement fusion according to claim 1, characterized in that: The method can be applied to scenarios such as learning emotion monitoring in intelligent education systems, screening for emotional disorders in mental health assessments, and optimizing emotional immersion experiences in virtual reality interactions.

10. An emotion analysis system based on multimodal EEG-eye-tracking fusion, characterized in that, include: 10.1 A stimulus presentation module, used to perform the stimulus presentation steps as described in claim 1; 10.2 A multimodal data acquisition module, including an EEG acquisition device (such as a 64-channel EEG amplifier) ​​and an eye-tracking device (such as an infrared eye tracker), is used to perform the data acquisition steps described in claim 1; 10.3 A data processing server, configured with a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of claims 1-9.

Images