Multi-modal emotion recognition system based on facial expression and eye movement tracking

Abstract

The invention discloses a multi-modal emotion recognition system based on facial expression and eye movement tracking. The multi-modal emotion recognition system comprises a data acquisition module, a data cleaning and quality control module, a visual analysis module, a multi-dimensional feature extraction module and a result feedback module. The data acquisition module is used for acquiring multi-modal original data; the data cleaning and quality control module is used for executing structured processing on the input multi-modal original data; the visual analysis module is used for presenting visual behavior characteristics of the subject during stimulation; the multi-dimensional feature extraction module is used for extracting multi-modal features describing gazing behaviors, physiological reactions and image contents from the eye movement data, the stimulation pictures and the face videos; the result feedback module is used for converting the analysis result of the multi-modal features into visual display content, and generating final emotion classification output according to the eye movement signals, the robustness and generalization ability of emotion classification are greatly improved, and the scene adaptability is high.

Description

Technical Field

[0001] This invention relates to the field of intelligent human-computer interaction technology, specifically to a multimodal emotion recognition system based on facial expressions and eye tracking. Background Technology

[0002] Emotion recognition is a key technology in human-computer interaction, mental health monitoring, and affective computing. Its goal is to identify an individual's emotional state in different situations through computational means. Traditionally, emotion recognition most often relies on facial expression recognition and voice emotion analysis. Meanwhile, eye-tracking technology uses indicators such as gaze direction, gaze pattern, and pupil changes to reflect an individual's attentional distribution and emotional response to external stimuli, offering advantages such as high temporal resolution, strong objectivity, and less susceptibility to external interference. Eye-tracking systems typically estimate eye position accurately through the positional relationship between the pupil center and the corneal reflective point (PCCR technology). The system generates corneal reflective points using an infrared light source and captures their displacement changes in real time to calculate gaze direction and acquire multiple types of data, including gaze trajectory, saccade patterns, and pupil size changes.

[0003] However, current eye-tracking-based emotion analysis still has significant shortcomings. On the one hand, existing research largely relies on existing general-purpose eye-tracking toolkits, which mainly focus on eye-tracking data collection or basic indicator calculations, without being optimized for emotion recognition tasks. On the other hand, existing emotion recognition research generally suffers from limitations such as single stimuli, limited sample size, and a lack of unified standardized toolchains, making it difficult to compare and reproduce results from different studies. Furthermore, existing toolchains lack adaptability to emotion recognition tasks, exhibiting deficiencies in multimodal fusion capabilities, complex operation and high usage barriers, data quality issues, and a lack of unified processes, posing challenges specific to the elderly population. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal emotion recognition system based on facial expressions and eye tracking to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal emotion recognition system based on facial expressions and eye tracking, comprising: a data acquisition module, a data cleaning and quality control module, a visualization analysis module, a multidimensional feature extraction module, and a result feedback module.

[0006] Preferably, the data acquisition module is used to collect multimodal raw data of the subjects in the emotional task. The specific operation process includes: real-time recording of the subjects' binocular fixation point coordinates, left and right pupil diameters, and eye movement trajectory changes during the emotional task; outputting raw sequences including fixation point coordinates, pupil values, and eye movement validity markers using an eye tracker; and synchronizing the timestamps of the raw records to align the eye movement trajectory points, pupil points, and facial frame sequences on the same time reference.

[0007] Preferably, the data cleaning and quality control module is used to perform structured processing on the input multimodal raw data. The specific operation process includes: marking abnormal samples in the eye movement raw recordings whose data frames and pupil diameters exceed a set threshold as missing, and excluding long-term missing segments; using linear interpolation and forward padding to recover short-term interruptions for gaze point coordinates, and uniformly marking coordinate points exceeding the screen resolution as missing; performing Savitzky-Golay smoothing on the eye movement velocity sequence; and performing face region detection, cropping, illumination normalization, and key point localization initialization on the facial video during the preprocessing stage.

[0008] Preferably, the visualization analysis module is used to generate a gaze distribution map, a gaze heatmap, and a pupil change trend map based on the cleaned eye movement and pupil data, presenting the visual behavioral characteristics of the subject during the stimulation period. The specific operation process includes: generating gaze-related visualization results based on the preprocessed eye movement data, including visualization graphics showing the distribution of gaze points, the proportion of gaze points, and changes in viewing patterns; and simultaneously generating a trend map of pupil diameter changes over time and the gaze proportion corresponding to the emotional stimulus category, visually presenting the subject's visual attention and physiological response during the task.

[0009] Preferably, the multi-dimensional feature extraction module is used to extract multimodal features describing gaze behavior, physiological response, and image content from eye-tracking data, stimulus images, and facial videos; the specific operation process includes: extracting multimodal features including eye-tracking behavior features, facial expression features, and image content features, and constructing a feature matrix for emotion recognition using the three types of features.

[0010] Preferably, the result feedback module is used to transform the analysis results of multimodal features into visual display content and generate the final emotion classification output based on eye movement signals. The specific operation process includes: forming a multimodal feature matrix from eye movement features, image visual features, and facial emotion features; inputting the matrix into a multimodal fusion model for emotion classification after quantile standardization and PCA dimensionality reduction; the system outputs the probabilities of three types of emotions: positive, neutral, and negative; and generating a visualization result including gaze trajectory, pupil change trend, and emotion proportion distribution.

[0011] Compared with the prior art, the beneficial effects achieved by the present invention are: (1) This invention constructs a standardized eye-tracking-face-image three-modal analysis process, which automates the entire process from data preprocessing and feature extraction to emotion classification, enabling different environments and different users to obtain consistent and reproducible recognition results based on a unified process, significantly reducing the threshold of existing toolkits that rely on programming experience and professional equipment.

[0012] (2) In terms of dynamic quantification of emotion-related eye movement features, this invention uses high-precision eye tracking data to automatically calculate indicators such as the number of fixations, the number of saccades, pupil changes, fixation entropy, trajectory smoothness and pupil reaction time. It can sensitively capture changes in emotional arousal, differences in visual search methods and attention distribution strategies. It not only reveals the temporal pattern of emotion regulating visual behavior, but also makes up for the shortcomings of traditional facial or voice emotion recognition in obtaining internal physiological reactions.

[0013] (3) In terms of the expression and visualization of multimodal information, this invention proposes a multidimensional visualization scheme such as gaze heatmap, three-dimensional eye movement trajectory, pupil time series and emotion viewing ratio map, which makes the emotion processing process more interpretable and easier for non-technical personnel to understand and use, thus breaking through the problem of "black box" in existing emotion recognition models.

[0014] (4) This invention integrates eye movement features, image visual features and facial expression emotion features to form a joint representation, and achieves effective fusion through cross-modal attention mechanism. It can capture subtle differences in emotions that are difficult to identify in a single modality, greatly improve the robustness and generalization ability of emotion classification, and is particularly suitable for complex situations such as weakened facial expressions and changes in visual attention in the elderly.

[0015] In summary, this invention not only improves the accuracy, stability, and interpretability of emotion recognition, but also has strong scene adaptability, and can be widely applied to various scenarios such as elderly care, mental health screening, human-computer interaction design, and multimedia emotion analysis. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the system architecture provided in an embodiment of the present invention; Figure 2 This is a flowchart of the experimental paradigm provided in the embodiments of the present invention; Figure 3 This is the system initial interface provided in the embodiments of the present invention; Figure 4 This is a system pupil line graph provided in an embodiment of the present invention; Figure 5 This is a system heat map provided in an embodiment of the present invention; Figure 6 This is a statistical chart of the proportion of eye movement emotions provided in an embodiment of the present invention; Figure 7 This is an eye-tracking feature extraction map provided in an embodiment of the present invention; Figure 8 This is a three-dimensional eye-tracking trajectory diagram provided in an embodiment of the present invention; Figure 9 This is a graph showing the emotion prediction results provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Combination Figures 1-9 As shown, the present invention provides a technical solution: a multimodal emotion recognition system based on facial expression and eye tracking, comprising: a data acquisition module, a data cleaning and quality control module, a visualization analysis module, a multidimensional feature extraction module, and a result feedback module.

[0019] In this embodiment, the data acquisition module is used to collect multimodal raw data from the subjects during the emotional task. The data acquisition process is strictly controlled to ensure the synchronization and stability of eye-tracking data, facial expression videos, and emotional stimulus images to the greatest extent possible. The high-precision eye tracker Pupil.IO outputs the raw sequence at a sampling rate of 200Hz.

[0020] For example, the specific operation process of the data acquisition module includes: The data collection process consisted of two consecutive phases. First, participants completed a psychological scale assessment, which included 15 items closely related to emotional state, covering the Geriatric Depression Scale (GDS) and the Generalized Anxiety Scale (GAD). The scale items were read aloud by the experimenter, and the participants responded immediately. The raw scores were then uniformly coded and normalized to construct a baseline indicator of subsequent emotional state.

[0021] Next, participants entered an emotion-evoking task, viewing a six-panel emotional image database from the Cornell Emotion6 Image Database. Each trial presented a 2×3 image matrix consisting of six images, categorized by emotional valence as positive, neutral, and negative. Each six-panel contained two positive images, two neutral images, and two negative images. Each stimulus was presented for 10 seconds, and a total of 16 six-panel images were displayed throughout the experiment. To avoid positional and order bias, the order of the images in the six-panel was randomized across trials and participants. During the presentation, participants were instructed to view the screen content naturally, without performing any judgment or key presses, to closely approximate a natural emotional response.

[0022] In both of the aforementioned stages, the system synchronously records the raw data output by the eye tracker, including timestamps, validity markers, coordinates of left and right eye fixations, pupil diameters, and fields related to visual angles. Simultaneously, facial video sequences are recorded for subsequent extraction of changes in key facial areas such as eyebrows, eyes, and mouth. The presentation order, category labels, and presentation duration of emotion images are also synchronously written into the experimental file during this process, providing temporal reference information for subsequent data processing and feature alignment.

[0023] In this embodiment, the data cleaning and quality control module performs structured processing on the input multimodal raw data to ensure the stability and continuity of subsequent analysis. Data preprocessing is a core component of the emotion recognition toolkit, aiming to systematically clean and correct the raw eye-tracking records, thereby improving the stability and repeatability of subsequent analysis and model building. Because eye trackers are easily affected by factors such as blinking, head movements, lighting changes, and calibration biases during data acquisition, invalid values, missing values, or sudden jumps often appear in the raw records. Therefore, this embodiment constructs a complete data preprocessing workflow.

[0024] For example, the specific operation process of the data cleaning and quality control module includes: First, duplicate rows and data with a validity flag of 0 are removed from the original records to eliminate redundant information and ensure the uniqueness of each record. Then, for short-term missing segments generated during the acquisition process, the gaze point coordinates are... Linear interpolation and forward padding were used to estimate the time series data and restore its continuous structure. To ensure the physical validity of the gaze coordinates, all records exceeding the experimental screen resolution of 1920×1080 pixels were marked as missing. Furthermore, based on the general physiological range of adult pupil diameter, pupil values ​​less than 1.5 mm or greater than 9.0 mm were considered abnormal and uniformly processed as missing to further filter out extreme noise.

[0025] After completing missing point repair and anomaly removal, key fields (such as gaze coordinates and pupil diameter) were uniformly converted to floating-point types to eliminate calculation errors caused by string or mixed data types. At the same time, the data index was reconstructed in chronological order, making the entire record monotonically increasing, which is beneficial for accurate alignment with stimulus presentation time and facial video timeline.

[0026] To obtain more stable velocity and direction-related features, a Savitzky-Golay filter is introduced at the end of the preprocessing stage to smooth the velocity sequence. This method performs local fitting with a low-order polynomial within a sliding window, which can reduce high-frequency noise while preserving the overall trend of the velocity sequence. In this embodiment, the window length is set to 15 and the order of the fitting polynomial is set to 3 to achieve a balance between noise suppression and detail preservation. It should be noted that the filtering may have a slight impact on individual instantaneous velocity peaks, but it has little impact on the identification of the start and end points of fixation and saccade events, and will not change the final analysis results of the overall shape of the eye movement trajectory and related feature indicators.

[0027] In this embodiment, the visualization analysis module is used to present the visual behavioral characteristics of the subjects during stimulation. After data cleaning, this module generates gaze-related visualization results based on the preprocessed eye-tracking data, including visualizations showing the distribution of gaze points, the proportion of gaze points, and changes in viewing patterns; it also generates trend graphs of pupil diameter changes over time and gaze proportions corresponding to emotional stimulus categories, thus providing an intuitive presentation of the subjects' visual attention and physiological responses during the task.

[0028] In this embodiment, the multi-dimensional feature extraction module is used to extract multimodal features describing gaze behavior, physiological response and image content from eye-tracking data, stimulus images and facial videos for subsequent emotion recognition modeling.

[0029] For example, the specific operation process of the multidimensional feature extraction module includes: Eye movement characteristics analysis: Fixation frequency is an important indicator for characterizing an individual's visual persistence behavior in a specific task, often used to reflect the subject's attentional focus on key information and attentional allocation patterns. A valid fixation event is defined as a fixation point maintaining a small spatial dispersion range and persisting for a duration exceeding a preset minimum fixation time. Different emotional stimuli trigger different fixation patterns; for example, negative emotions are typically accompanied by a higher frequency of repetitive gazing, while positive emotions tend to produce more stable visual persistence in the pleasure zone. Therefore, the number of fixations and their distribution across different zones are significant in distinguishing different emotional states.

[0030] In actual calculations, a series of candidate fixation segments are first divided based on the eye-tracking time series. The requirement is that within a given time window, the spatial dispersion of all sampling points is less than a fixation spatial threshold, and the duration of the segment is not less than the shortest fixation duration. Let the... The start and end times of each candidate segment are and If the segment length reduce Not less than the set threshold If a segment is observed, it is considered a valid fixation. Therefore, the total number of fixations by the subject in one trial is... Represented as: Among them, Represents the total number of candidate segments, symbol This indicates an indicator function that takes the value of one when the condition is met and zero otherwise. This definition allows for a more robust statistical analysis of gaze events while ensuring time and space constraints.

[0031] Analysis of fixation frequency: Salivation frequency reflects the frequency of rapid eye movements between different fixation points and is an important indicator for characterizing visual search strategies and information acquisition methods. Unlike the relatively stable characteristics of fixation, saccades typically involve significant spatial displacement of the eye position within a short period. Without differentiation, minute eye movements or sampling noise may be misjudged as saccades. Therefore, it is necessary to set clear thresholds for the displacement amplitude of eye movement trajectories to make the statistically obtained saccade events more reliable and interpretable.

[0032] In specific calculations, let the first... The start and end positions of the gaze between adjacent fixation points are vectors. and When the distance between two line-of-sight vectors is not less than the saccade amplitude threshold. At that time, the event is determined to be a valid saccadic. Therefore, the total number of saccadics in a single trial can be calculated. Represented as: in The value is fifty pixels, symbol This also refers to indicator functions.

[0033] Pupil Difference Standard Deviation Analysis: The pupil difference standard deviation is used to measure the degree of variation in the diameter of an individual's pupils in both eyes. This indicator mainly reflects the degree of variation in an individual's physiological response during a task by calculating the difference between the pupil diameters of the left and right eyes and their average values. A higher pupil standard deviation indicates a higher level of variation in pupil size. A larger standard deviation typically indicates a greater change in pupil diameter, which may be related to physiological states such as emotional excitement or concentration. A lower standard deviation, on the other hand, indicates a smaller change in pupil diameter, reflecting a more stable emotional state or a lower physiological response. The calculation formula is: in, These are the values ​​of the left and right pupil diameters for each data point. It is the average diameter of the left and right pupils. It represents the number of data points.

[0034] Fixation entropy analysis: Fixation entropy is an eye movement index derived from information theory, used to quantitatively characterize an individual's information search patterns in a task. It can reflect the complexity and diversity of the visual information acquisition process from an overall perspective, and more comprehensively reveal the attention distribution characteristics of the subjects in the spatial range. This invention uses two types of fixation entropy indices, namely saccade fixation entropy and static fixation entropy, to characterize the fixation distribution characteristics at different time scales and their differences between emotional states.

[0035] Saccade fixation entropy measures the frequency of random eye saccades between regions of interest, reflecting an individual's efficiency in information retrieval; a higher value indicates a more complex search path and stronger exploratory nature. The calculation method is as follows: in, Indicates that within all observed time periods, the first... The probability of occurrence of each region of interest Indicates two areas of interest and The transition probability between areas, i.e., eye movement from the area of ​​interest Jump to the area of ​​interest frequency, This indicates the total number of regions of interest.

[0036] Static fixation entropy measures the uniformity of fixation distribution across different regions of interest, reflecting the dispersion of an individual's fixation patterns; a higher value indicates a more uniform fixation distribution. The calculation method is as follows: Eye movement smoothness analysis: The fixation smoothness index measures the smoothness of eye movements during continuous motion and is closely related to an individual's emotional state. When experiencing tension or anxiety, eye movements tend to exhibit more reversals and abrupt changes, with increased turning angles between consecutive fixations, resulting in decreased overall smoothness. Conversely, when emotions are relatively relaxed or stable, eye movements typically proceed in a more coherent direction, with less variation in gaze movement, leading to a relatively higher smoothness index. Therefore, eye movement smoothness can provide a supplementary representation of emotional state from the perspective of motion geometry.

[0037] In this invention, during calculation, the eye movement trajectory is first represented as a series of fixation points ordered by time, and then vectorized. This means that for each group of three consecutive points... Construct two adjacent displacement vectors , The degree of transition is measured by the cosine of the angle between these two displacements. : When the line of sight travels in almost the same direction, the two displacements are nearly collinear, and the cosine of the angle between them is close to one. When the line of sight suddenly turns back or sharply, the angle becomes larger, even approaching 180 degrees, and the cosine of the angle tends to be negative. In actual implementation, each... To avoid exceeding the limit due to numerical errors, the eye-tracking smoothness index is defined as the average of the cosines of the angles between all valid turning points, confined to a range of -1 to 1. in, The more the index is close to one, the more consistent the direction of adjacent gaze movements are, and the smoother the overall trajectory is. This is common in situations where attention is focused or emotions are relatively stable. The lower the index, or even close to a negative value, the more frequently the gaze jumps between different directions, and the trajectory is full of sharp turns. This is more likely to occur in a state of high alertness, anxiety, or large emotional fluctuations. Therefore, the fluency index provides an intuitive and compact kinematic measure for identifying different emotional states.

[0038] Pupil Reaction Time Calculation: Pupil reaction time is an important physiological indicator for measuring an individual's reaction speed to external stimuli (such as emotional triggers, visual stimuli, etc.). In a state of emotional excitement or anxiety, an individual's pupil reaction time is usually shorter, indicating a faster response to stimuli and possibly reflecting the intensity and tension of emotional arousal. Conversely, in a stable or relaxed state, pupil reaction time is relatively longer, showing lower physiological tension and a more stable response. The calculation formula is derived as follows: in It is the baseline pupil diameter before stimulation, and the median pupil diameter at a certain point in time before stimulation. It is the first Pupil diameter at each time point. When If the value is greater than the threshold (0.2), the pupil is considered to have responded effectively. It refers to the reaction time. It is the number of sampling points from the occurrence of the stimulus to the start of the response. The size of the sliding window is used to smooth out changes in pupil diameter and reduce noise interference in reaction time calculations. This indicates the minimum number of points that must exceed a threshold for a response to be considered valid, and is used to ensure the significance of the response.

[0039] Facial Expression Feature Analysis: Facial video modality processing consists of three steps: face detection, keypoint localization, and expression recognition, aiming to extract facial features related to emotional states from video sequences. First, the MTCNN multi-task cascaded convolutional network is used for face detection and keypoint prediction. MTCNN can quickly locate the face region in each frame and accurately obtain the positions of key feature points such as the corners of the eyes, the tip of the nose, and the corners of the mouth. Because its cascaded structure simultaneously includes candidate region generation, bounding box regression, and facial landmark prediction, it is highly robust to changes in illumination, pose shifts, and local occlusion, and can stably output the facial feature points of subjects under various experimental conditions.

[0040] After obtaining stable facial regions, the captured facial images are input into an emotion analysis model to identify expression categories. This invention uses the EmotEffLibRecognizer deep learning model to classify emotions frame by frame in the video, recognizing common emotions such as happiness, anger, surprise, sadness, and fear. The system statistically analyzes the classification results of the entire video, calculates the proportion of each emotion, and generates a time-based emotion distribution histogram. According to the three-classification system of this invention, the Happiness, Surprise, and Surprise multimodal emotion recognition systems based on facial expression and eye tracking are merged into positive emotions. The Anger, Contempt, Disgust, Fear, and Sadness multimodal emotion recognition systems are classified as negative emotions. Segments without obvious facial expression features are classified as neutral emotions. The resulting proportions of positive, neutral, and negative emotions were used as key features of facial modality to model the facial expressions in conjunction with eye-tracking and image features, in order to analyze the multimodal consistency and differences of different emotional states.

[0041] Image content feature analysis: Feature extraction of image modalities is based on the spatial partitioning structure of six-grid emotional images to characterize the multidimensional visual differences in color, texture, edges, and shape of emotional images. To preserve overall content information while reflecting spatial differences in local areas, each image is divided into six sub-regions in a 2x3 grid, and independent features are extracted for each region. These features are then combined into an image feature vector according to region numbers, providing a foundation for subsequent image-gaze correspondence analysis.

[0042] Regarding color features, each sub-region image was first converted to several common color spaces, including RGB, HSV, and Lab, all based on a multimodal emotion recognition system using facial expressions and eye tracking. Subsequently, the mean, standard deviation, median, minimum, and maximum values ​​were calculated for each channel to describe brightness levels, color saturation, and overall tonal distribution. These statistical indicators can sensitively reflect the differences in color design between images depicting different emotions.

[0043] Regarding texture features, the sub-region image is first converted to grayscale format, and the local structure of the image is encoded using Local Binary Pattern (LBP). Histogram statistics are used to characterize the roughness and repetition patterns of the texture. Simultaneously, a gray-level co-occurrence matrix is ​​constructed, from which features such as contrast, dissimilarity, homogeneity, energy, and correlation are extracted to describe the joint variation of grayscale values ​​in the neighborhood. Differences in shadows, wrinkles, and repeating textures in the image can be reflected in these metrics.

[0044] Regarding edge features, contour structures in sub-region images are extracted using various edge detection operators. On one hand, the proportion of non-zero pixels after edge detection is calculated to represent the richness of contours and lines within the region. On the other hand, the edge intensity distribution is described based on the mean, standard deviation, and maximum value of the gradient magnitude map; and the second-order amplitude value of grayscale changes is obtained through the second-order derivative operator, with variance reflecting the severity of high-frequency structural changes.

[0045] Regarding shape features, the sub-region image is converted into a binary image using an adaptive thresholding method, and the largest connected contour in the foreground region is extracted. Based on this contour, its area and perimeter are calculated to characterize the size and boundary complexity of the main structure. Furthermore, circularity, convex hull area, and the ratio of contour area to convex hull area are calculated to describe the compactness and boundary regularity of the target shape. When salient objects are present in the image, these geometric features can reflect the differences in object appearance in different emotional scenarios.

[0046] The aforementioned color, texture, edge, and shape features are calculated independently within each sub-region, ultimately forming a structured feature file containing image name, category label, and region number. This provides quantifiable image modal input for subsequent joint modeling with eye-tracking features and emotion labels.

[0047] In this embodiment, the result feedback module is used to convert the analysis results of multimodal features into visual display content and generate the final emotion classification output based on eye movement signals.

[0048] For example, the specific operation process of the result feedback module includes: The role of this module in the overall process is to integrate the information obtained from feature calculation and model inference, and to present the subjects' gaze behavior, pupil changes and emotional state distribution during the viewing of emotional pictures in a graphical way, providing researchers with intuitive interpretation of the results and a basis for subsequent statistical analysis.

[0049] Regarding eye-tracking visualization results, the system can generate various types of visualizations related to gaze behavior, such as gaze heatmaps, line graphs showing pupil diameter changes over time, and spatial two-dimensional projections of gaze trajectories. The gaze heatmap, by statistically analyzing the time proportion of the gaze point in each region of interest of the stimulus image, demonstrates the participants' visual attention allocation patterns under different emotional stimuli. The pupil curve reflects the dynamic changes in pupil diameter during the task and can be used to observe physiological regulatory responses during emotional induction. The gaze trajectory graph outlines the participants' eye movement paths and spatial distribution characteristics on the image plane.

[0050] Regarding the presentation of emotion results, the system statistically analyzes the frame-level emotion results output by the facial expression recognition model, calculates the cumulative dwell time ratio of positive, neutral, and negative emotions during the observation task, and presents the distribution of emotion states in a bar chart for easy comparison of the proportion of different emotion categories. The emotion proportions shown in the chart can serve as important input information for subsequent pattern recognition, statistical testing, or individual difference analysis.

[0051] The visualization results described above, together with the emotion classification results, constitute the final output of the toolbox, allowing users to observe the correspondence between multimodal information within a unified interface. For example, users can simultaneously view gaze heatmaps and image region content, or compare the synchronous changes in pupil curves and emotion labels over time. This feedback provides a structured and intuitive interpretive approach for emotion recognition research, aiding in the analysis of differences in attentional behavior and physiological response characteristics under different emotional stimuli.

[0052] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0053] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal emotion recognition system based on facial expression and eye tracking, characterized in that: It includes a data acquisition module, a data cleaning and quality control module, a visualization analysis module, a multi-dimensional feature extraction module, and a result feedback module; The data acquisition module is used to collect multimodal raw data of the subjects in the emotion task; The multimodal raw data includes fixation point coordinates, pupil diameter, timestamp sequence, and the subject's corresponding emotion scale score. The data cleaning and quality control module is used to perform structured processing on the input multimodal raw data; The visualization analysis module is used to generate a gaze distribution map, a gaze heatmap, and a pupil change trend map based on the cleaned eye movement and pupil data, presenting the visual behavior characteristics of the subject during the stimulation period. The multi-dimensional feature extraction module is used to extract multimodal features describing gaze behavior, physiological responses, and image content from eye-tracking data, stimulus images, and facial videos. The result feedback module is used to transform the analysis results of multimodal features into visual display content and generate the final emotion classification output based on eye movement signals.

2. The multimodal emotion recognition system based on facial expression and eye tracking according to claim 1, characterized in that: The operation of the data acquisition module includes: Real-time recording of binocular fixation coordinates, left and right pupil diameters, and eye movement trajectory changes during emotional tasks; The original sequence, including fixation point coordinates, pupil values, and eye movement validity markers, is output using an eye tracker. The timestamps of the original records are synchronized to align the eye movement track points, pupil points, and facial frame sequences on the same time reference.

3. A multimodal emotion recognition system based on facial expression and eye tracking according to claim 2, characterized in that: The operation process of the data cleaning and quality control module includes: Abnormal samples in the raw eye-tracking recordings whose data frames and pupil diameters exceed a set threshold are marked as missing, and long-term missing segments are excluded. Linear interpolation and forward padding are used to recover short-term interruptions for gaze point coordinates, and coordinate points that exceed the screen resolution are uniformly set to missing. Savitzky-Golay smoothing was applied to the eye movement velocity sequences; In the preprocessing stage, facial videos undergo face region detection, cropping, illumination normalization, and key point localization initialization.

4. A multimodal emotion recognition system based on facial expression and eye tracking according to claim 3, characterized in that: The visualization analysis module also includes: Generate gaze-related visualizations based on preprocessed eye-tracking data, including visualizations showing gaze point distribution, gaze point ratio, and changes in viewing patterns. Simultaneously, a trend graph of pupil diameter changes over time and the fixation ratio corresponding to the emotional stimulus category are generated to visualize the visual attention and physiological response of the subjects during the task.

5. A multimodal emotion recognition system based on facial expression and eye tracking according to claim 4, characterized in that: The multimodal features extracted by the multidimensional feature extraction module include eye movement behavior features, facial expression features, and image content features. These three types of features are used to construct a feature matrix for emotion recognition.

6. A multimodal emotion recognition system based on facial expression and eye tracking according to claim 5, characterized in that: The process of extracting the eye movement behavior features includes: Based on the fixation point coordinates and time sequence, the number of fixations and saccades are calculated, and valid fixation events are determined based on the start and end points of the fixation time period. Sagging events are determined based on whether the displacement of adjacent fixation points reaches the saccade threshold. The standard deviation of pupil difference was calculated using the pupil diameter sequences of the left and right eyes, and the physiological fluctuation amplitude related to emotional arousal was characterized based on the standard deviation of pupil difference. Calculate fixation entropy and saccade entropy from fixation probability distribution and shift probability, respectively; Trajectory smoothness is calculated based on the change in direction vector between consecutive gaze points; Pupil reaction time is calculated by comparing baseline pupil values ​​and instantaneous pupil dilation amplitude before and after stimulus presentation to determine the onset of emotional arousal.

7. A multimodal emotion recognition system based on facial expression and eye tracking according to claim 6, characterized in that: The process of extracting facial expression features includes: Face detection and key point prediction using MTCNN multi-task cascaded convolutional network; The facial region is input into a deep emotion recognition model to classify each frame into basic emotions such as happiness, anger, sadness, and surprise. Based on the emotion system, the identification results are converted into the proportions of three emotion categories: positive, neutral, and negative, and then aggregated over time.

8. A multimodal emotion recognition system based on facial expression and eye tracking according to claim 7, characterized in that: The extraction of image content features includes a spatial division structure based on a six-grid emotional image, which characterizes the multidimensional visual differences in color, texture, edge, and shape of the emotional image.

9. A multimodal emotion recognition system based on facial expression and eye tracking according to claim 8, characterized in that: The process of extracting the image content features includes: Based on the six-grid layout, the mood image is divided into six sub-regions, and the mean and variance of the RGB / HSV / LAB colors of each region are calculated. Extract LBP texture histogram, GLCM features including contrast, energy, and correlation, and shape features such as contour area and roundness. The visual attributes of the stimulus images are described using the extracted LBP texture histogram, GLCM features, and shape features, and a correlation is established with gaze behavior.

10. A multimodal emotion recognition system based on facial expression and eye tracking according to claim 9, characterized in that: The operation of the result feedback module includes: The eye-tracking features, image visual features, and facial emotion features are combined into a multimodal feature matrix; After quantile standardization and PCA dimensionality reduction, the data are input into a multimodal fusion model for sentiment classification. The system outputs the probabilities of three types of emotions: positive, neutral, and negative. It generates visualizations including gaze trajectory, pupil change trends, and emotion distribution.

Images