A dynamic emotion recognition method and system based on EEG signals and eye movement signals
By combining the time-frequency feature extraction and regression model of EEG signals and eye movement signals, high-precision and real-time emotion recognition are achieved, solving the problem of insufficient accuracy and real-timeness of traditional methods, and providing more objective dynamic emotion recognition result.
Patent Information
- Application Number
- CN202410961563.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-18
AI Technical Summary
Traditional emotion recognition methods are not very accurate, are susceptible to environmental interference and are greatly affected by subjective factors, making it difficult to achieve high-precision and real-time emotion recognition.
Dynamic emotions recognition method based on EEG signals and eye movement signals is adopted, and the EEG and eye movement characteristics are obtained through time-frequency feature extraction technology, and the dynamic emotions category is determined using regression models, and the recognition is achieved in combination with computer systems.
It improves the accuracy and real-time nature of emotion recognition, overcomes the shortcomings of traditional methods, and provides more objective and unbiased dynamic emotion recognition results.
Smart Images

Figure CN118490233B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of electroencephalogram (EEG) and eye movement signal processing, and in particular to a method and system for dynamic emotion recognition based on electroencephalogram (EEG) signals and eye movement signals. Background Art
[0002] With the continuous advancement of science and technology, the demand for emotion recognition technology in fields such as emotional intelligent interaction and mental health monitoring is growing. The development of emotion recognition technology has been valued and supported by fields such as artificial intelligence, biomedical engineering, and psychology. Traditional emotion recognition methods mainly rely on visible signals such as facial expressions, voice features, and behavioral performance. However, these methods have problems such as low accuracy, susceptibility to environmental interference, and subjective factors of the subjects. Summary of the invention
[0003] The purpose of the present invention is to provide a method and system for dynamic emotion recognition based on electroencephalogram signals and eye movement signals, which can improve the accuracy and real-time performance of dynamic emotion recognition.
[0004] To achieve the above object, the present invention provides a dynamic emotion recognition method based on EEG signals and eye movement signals, comprising:
[0005] Acquiring EEG signal data and eye movement signal data of a target subject; the eye movement signal data includes pupil diameter and fixation duration;
[0006] The time-frequency feature extraction technology is used to extract EEG features and eye movement features from the preprocessed EEG signal data and eye movement signal data, and a regression model is used to determine the dynamic emotion category of the target subject based on the EEG features and the eye movement features; the dynamic emotion categories include calm and happy, happy and calm, sad and calm, calm and sad, nervous and calm, and calm and nervous; the regression model is obtained by pre-training with a training sample set, and each training sample in the training sample set includes an EEG sample signal, an eye movement sample signal and a corresponding dynamic emotion label.
[0007] To achieve the above objectives, the present invention also provides a computer system, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned dynamic emotion recognition method based on EEG signals and eye movement signals.
[0008] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention collects the EEG signal data and eye movement signal data of the target subject, uses the time-frequency domain feature extraction technology to extract features from the preprocessed EEG signal data and eye movement signal data, and fuses the extracted features, and uses a regression model to determine the dynamic emotion category of the target subject, thereby realizing the recognition of dynamic emotions and improving the accuracy and real-time performance of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0010] Figure 1 This is an overall flow chart of the dynamic emotion recognition method based on EEG signals and eye movement signals provided by the present invention.
[0011] Figure 2 A detailed flow chart of the dynamic emotion recognition method based on EEG signals and eye movement signals provided by the present invention.
[0012] Figure 3 This is a schematic diagram of the process of dynamic emotion recognition based on EEG signals and eye movement signals of the present invention.
[0013] Figure 4 Schematic diagram of the process of acquiring EEG signal data and eye movement signal data. DETAILED DESCRIPTION
[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0015] Electroencephalogram (EEG) and eye-tracking signals are important representatives of human physiological signals. They are unbiased and objective. EEG signals reflect the changes in electrical signals of brain activity, while eye-tracking signals record the trajectory and frequency of eye movements. Both signals can be used to reflect a person's emotional state.
[0016] The purpose of the present invention is to provide a method and system for dynamic emotion recognition based on EEG signals and eye movement signals, which can realize dynamic emotion recognition by combining EEG signals and eye movement signals, and has high accuracy and real-time performance.
[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0018] Example 1: Figure 1 and Figure 2 As shown, the dynamic emotion recognition method based on EEG signals and eye movement signals provided by the present invention includes step 1 and step 2.
[0019] Step 1: Obtain EEG signal data and eye movement signal data of the target subject. The eye movement signal data includes pupil diameter and fixation duration.
[0020] Specifically, step 1 includes steps 11 to 15.
[0021] Step 11: Collect the original EEG signals and original eye movement signals of the target subject based on the set experimental paradigm. In this embodiment, the original EEG signals and original eye movement signals are collected synchronously in real time and stored in the database. The original EEG signals include: EEG signal data in normal mode and EEG signal data in target task state mode. The original eye movement signals include: eye movement signal data in normal mode and eye movement signal data in target task state mode.
[0022] Among them, during the training process of the regression model, the experimental paradigm is: 48 movie clips are selected from the material library as stimuli used in the experiment. Each movie clip is carefully edited to produce coherent emotions. Each video should cause two desired target emotions, including six types of stimulus movie clips: happy calm and calm happy, sad calm and calm sad, and nervous calm and calm nervous. There are eight videos of each type, and the duration is about 3 minutes. Figure 3 As shown in the figure, we first create an experimental paradigm for EEG signals and eye movement signals, and synchronously obtain the original EEG signal data, original eye movement signal data, and dynamic emotion labels of the target subjects through acquisition equipment (EEG cap and eye tracker). In actual application, the experimental paradigm is similar to the training process of the regression model, the only difference is that the dynamic emotion labels are not collected.
[0023] Experts were organized to analyze application scenarios and possible situations, and to design a relatively complete experimental paradigm that can accurately reflect EEG and eye movement signals in this cognitive state. In isolation from external interference, the discharge signals of the cerebral cortex of the subjects under different cognitive activities were recorded and collected. At the same time, for comparison, the EEG and eye movement signals under normal conditions were recorded.
[0024] The quality of the collected EEG signals and eye movement signals is evaluated based on the performance of the subjects at the scene of the collection. The original EEG signals and original eye movement signals with poor performance and that cannot accurately reflect the process are eliminated, and the data in the database are fed back and updated.
[0025] like Figure 4 As shown, as a specific implementation, the process of acquiring EEG signal data and eye movement signal data is as follows.
[0026] 1) In a quiet room with appropriate temperature and brightness, the subjects were asked to wear EEG and eye movement signal acquisition equipment, and 5 seconds of prompts and precautions were played to the subjects. The experiment was divided into 3 sections, each of which played a video that induced two emotions. There were six videos that induced different types of emotional changes, namely happy-calm and calm-happy, calm-sad and sad-calm, calm-tense and tense-calm. Each video lasted about 180 seconds, with 90 seconds for each emotion.
[0027] 2) The subjects were placed in a dedicated environment that could block external interference. The brightness was always set at a comfortable level so that the subjects could clearly see the movie content. To ensure the smooth progress of the experiment, each subject was required to fill out a mental health assessment questionnaire before the experiment officially began, and each subject was trained using additional movie clips.
[0028] 3) After the experiment started, the order of presentation of each movie clip was fixed. The subjects first adjusted their emotions to be in a relatively relaxed state. When the subjects' brain waves were stable, the movie clips were played, and 180 seconds of brain wave signals and eye movement signals were collected in the cognitive activity state.
[0029] 4) Set a rest time, and rest for 15 to 30 seconds after watching each movie clip to allow the subject's brain waves to be in a stable state. Then repeat the above process until the subject completes all the processes of the entire experimental paradigm. After an interval of 5 seconds, the subject evaluates his or her self-state and completes the experiment.
[0030] Five experts were selected to participate in the selection of film clips. All of them had received professional film training and had experience in rehearsing films. Twenty physically and mentally healthy subjects were selected to participate in the experiment, including 11 males and 9 females with an average age of (20±1.5) years old.
[0031] Step 12: Filter the original EEG signal to obtain a filtered EEG signal.
[0032] Specifically, the EEG signal of the required frequency band is filtered out from the original EEG signal through a bandpass filter to obtain a filtered EEG signal. Irrelevant information outside the effective EEG frequency band is removed. The required frequency bands include 1Hz~3Hz frequency band, 4Hz~7Hz frequency band, 8Hz~13Hz frequency band, 14Hz~30Hz frequency band and 31Hz~50Hz frequency band.
[0033] Step 13: Perform artifact removal processing on the filtered EEG signal to obtain EEG signal processing data. Remove power frequency interference and artifacts of the filtered EEG signal, that is, remove disturbances such as eyelid twitching, electrocardiogram, electromyography, limb shaking, and 50Hz power frequency interference that interfere with the EEG signal.
[0034] Step 14: resample the EEG signal processing data to obtain the EEG signal data of the target subject. Specifically, the EEG signal processing data is downsampled, and the sampling frequency is set to 128 Hz. This can improve the computational efficiency while retaining information related to the cognitive state.
[0035] The original EEG signal was imported into ASA, and the electrode positioning, electrode selection, re-reference, filtering, bad segment removal, bad conductor interpolation, independent component analysis, segmentation and baseline correction were carried out in sequence, and then the data was saved.
[0036] Step 15: Filter the original eye movement signal, and derive the pupil diameter and fixation duration to obtain the eye movement signal data of the target subject.
[0037] The original eye movement signal is imported into Tobii Pro Lab for filtering. Then, the event is divided according to the created experimental paradigm, and the time of interest and area of interest are selected. The eye movement indicator data of pupil diameter and fixation duration are exported.
[0038] After collecting the original EEG signal data and eye movement signal data, the subjects are required to score the stimulus video every five seconds in the expected stimulus two-dimensional space, with a value range of [1,9], and perform normalization processing on the dynamic emotion label.
[0039] Step 2: Use time-frequency feature extraction technology to extract EEG features and eye movement features from the preprocessed EEG signal data and eye movement signal data, and use a regression model to determine the dynamic emotion category of the target subject based on the EEG features and the eye movement features. The present invention displays the dynamic emotion category of the target subject in real time.
[0040] The dynamic emotion categories include calm-happy (characterizing the dynamic emotion change from calm to happy), happy-calm (characterizing the dynamic emotion change from happy to calm), sad-calm (characterizing the dynamic emotion change from sad to calm), calm-sad (characterizing the dynamic emotion change from calm to sad), tense-calm (characterizing the dynamic emotion change from tense to calm) and calm-tense (characterizing the dynamic emotion change from calm to tense).
[0041] The regression model is obtained by training in advance using a training sample set, and each training sample in the training sample set includes an EEG sample signal, an eye movement sample signal and a corresponding dynamic emotion label.
[0042] In this embodiment, the EEG features include differential entropy features and power spectral density features of EEG signal data; the eye movement features include differential entropy features of pupil diameter, mean features of pupil diameter, standard deviation features of pupil diameter, mean features of fixation duration, and standard deviation features of fixation duration. The differential entropy feature is used to describe the complexity or uncertainty of the signal, and the power spectral density feature describes the energy distribution of the signal at different frequencies.
[0043] Furthermore, step 2 specifically includes steps 21 to 26.
[0044] Step 21: Determine the differential entropy features of the EEG signal data according to the time series information of the EEG signal data. Calculate the differential entropy features of the EEG signal data in the time domain.
[0045] The present invention adds a 5-second non-overlapping window to the 64-channel information every 5 seconds. Extract features.
[0046] Specifically, the extraction formula of the differential entropy feature of the EEG signal data is: ;in, is the differential entropy feature of the EEG signal data, is the total number of samples, For the EEG signal data n The value of the sampling point, For the EEG signal data n +1 sample value.
[0047] Step 22: Perform Fourier transform on the EEG signal data, convert the EEG signal data from the time domain to the frequency domain, and extract the power spectrum density characteristics of the EEG signal data. Convert the EEG signal data from the time domain to the frequency domain, and calculate the power spectrum density on the frequency band recognized in the field of brain science.
[0048] Specifically, the power spectrum density feature extraction formula of the EEG signal data is: ;in, is the power spectral density feature of the EEG signal data, is the total number of samples, For the EEG signal data n The value of the sampling point, j is the imaginary unit, is the angular frequency, which is used to describe the characteristics of the signal in the frequency domain. is the average power, and it has been proven that EEG signals approximately obey Gaussian distribution in a short period of time.
[0049] Step 23: Determine the differential entropy feature of the pupil diameter, the mean feature of the pupil diameter, and the standard deviation feature of the pupil diameter based on the time series information of the pupil diameter.
[0050] Step 24: Determine the mean characteristics of the gaze duration and the standard deviation characteristics of the gaze duration according to the time series information of the gaze duration.
[0051] The power spectral density characteristics of the EEG signal data, the differential entropy characteristics of the EEG signal data, the differential entropy characteristics of the pupil diameter, the mean characteristics of the pupil diameter, the standard deviation characteristics of the pupil diameter, the mean characteristics of the fixation duration, and the standard deviation characteristics of the fixation duration are standardized to ensure the scale consistency between different features to avoid the impact of excessively large or small feature values on the regression model training.
[0052] Step 25: Cascade feature fusion is performed on the power spectral density features of the EEG signal data, the differential entropy features of the EEG signal data, the differential entropy features of the pupil diameter, the mean features of the pupil diameter, the standard deviation features of the pupil diameter, the mean features of the fixation duration, and the standard deviation features of the fixation duration to obtain a fused feature vector.
[0053] The present invention has multiple combinations of different feature fusions, and different combinations of feature fusions obtain different recognition results, that is, EEG differential entropy feature + pupil diameter feature, EEG power spectrum density feature + pupil diameter feature, EEG differential entropy feature + fixation duration feature, EEG power spectrum density feature + fixation duration feature, EEG differential entropy feature + pupil diameter feature + fixation duration feature, EEG power spectrum density feature + fixation duration feature + pupil diameter feature, several different combinations.
[0054] That is, the differential entropy features of the EEG signal data and the differential entropy features, mean features and standard deviation features of the pupil diameter are input into the regression model for cascade feature fusion to obtain a longer feature vector; the differential entropy features of the EEG signal data and the mean features and standard deviation features of the gaze duration are input into the regression model for cascade feature fusion to obtain a longer feature vector.
[0055] The specific steps of cascade feature fusion are: directly attaching the EEG feature to the end of the eye movement feature to form a longer feature vector. The fused feature vector contains both EEG signal information and eye movement information, thus being able to more comprehensively reflect the user's status.
[0056] Step 26: Classify the fused feature vector to obtain the dynamic emotion category of the target subject.
[0057] Specifically, regression prediction is performed on the fused feature vector to obtain a recognition result of a preset dynamic emotion.
[0058] Furthermore, the present invention can also pre-process the online multimodal data to obtain online multimodal information, perform fusion classification processing based on the online multimodal information, and determine the online psychological state and online analysis results of the online multimodal data.
[0059] In the present invention, the regression models include radial basis function support vector regression (Support Vector Regression Radial basis function, SVRrbf) model, linear support vector regression (Support Vector Regression Linear, SVRlinear) model, random forest regression (Random Forest Regressor, RF) model, decision tree regression (Decision Tree Regressor, DT) model, K-nearest Neighbors Regressor (KNN) model, elastic network regression (ElasticNet Regressor) model, ridge regression (Ridge Regression) model and least absolute shrinkage and selection operator Regressor (Least absolute shrinkage and selection operator Regressor) model.
[0060] The regression model is any regression model selected from a pre-trained radial basis function support vector regression model, a pre-trained linear support vector regression model, a pre-trained random forest regression model, a pre-trained decision tree regression model, a pre-trained K-nearest neighbor regression model, a pre-trained elastic network regression model and a pre-trained minimum absolute shrinkage and selection operation regression model according to actual needs.
[0061] In the process of training the regression model, the parameters of each regression model are tuned using methods such as cross-validation. For each regression model, the training set is used for model training, and the validation set is used for model performance evaluation. The optimal model parameters are selected based on the evaluation results.
[0062] According to actual needs and performance evaluation results, the most suitable regression model is selected and deployed and used in practical applications.
[0063] (1) The SVRrbf model maps the input EEG signal data and eye movement signal data to a high-dimensional feature space through a radial basis function. In the high-dimensional feature space, the SVR model constructs a regression hyperplane that maximizes the interval between the training data and the hyperplane and violates the training data as little as possible within the allowable error range. This process can be formalized as an optimization problem, that is, minimizing the loss function: ;in, is the normal vector of the hyperplane, b is the bias term, and is the slack variable, C is the regularization parameter, which is used to control the size of the interval and the degree of error penalty. I is the total number of data points in the space, represents the kernel function of the radial basis function, For the first i data points, is the Lagrange multiplier, is the target value.
[0064] The SVR model calculates the predicted value based on the mapping of the input data in the high-dimensional feature space and the position of the hyperplane. The prediction process is: extract the corresponding features through the input EEG signal data and eye movement signal data, map them to the corresponding emotional space, and the regression target is the dynamic label of pleasure and arousal. This predicted value is the model's estimate of the emotional state.
[0065] (2) The SVRlinear model directly maps the input EEG signal data and eye movement signal data to the high-dimensional feature space through linear mapping. The SVRlinear model calculates the predicted value based on the linear mapping of the input data in the high-dimensional feature space and the position of the hyperplane. This predicted value is the model's estimate of the emotional state.
[0066] Compared with the SVRrbf model, the prediction process of the SVRlinear model is more direct and does not involve the calculation of the radial basis function. The SVRlinear model constructs a regression hyperplane that maximizes the interval between the training data and the hyperplane and violates the training data as little as possible within the allowable error range. This process can be formalized as an optimization problem, that is, minimizing the loss function: .
[0067] (3) The random forest regression model is an integrated learning model composed of multiple decision trees. Each decision tree randomly selects a portion of training data and features for training to reduce the risk of overfitting. In the process of building each tree, the model determines the order of node division based on the importance of the features.
[0068] After training is completed, the random forest regression model will use the collective wisdom of multiple decision trees to make predictions. For new EEG signal data and eye movement signal data, the random forest regression model will input them into each decision tree and perform weighted average or voting based on the prediction results of each tree to obtain the final emotion recognition result. Its loss function is expressed as: ;in, is the loss value of the random forest regression model, M is the number of training samples, is the dynamic emotion label of the mth training sample, is the predicted emotion category of the random forest regression model for the mth training sample. The loss function represents the average of the squared difference between the model prediction value and the true value. The training goal is to minimize .
[0069] (4) The decision tree regression model starts from the root node of the tree. Based on the feature value of the data, the decision tree regression model selects a branch and moves to the next node. This process continues until a leaf node is reached. When a leaf node is reached, the leaf node contains the predicted value of the corresponding emotional state. This predicted value is usually calculated based on the average or other statistics of the dynamic emotional labels of all training samples in the leaf node. The predicted value is the estimated value of the emotional state of the EEG signal data and eye movement signal data by the decision tree regression model. The predicted value can be directly used as the output result of the decision tree regression model.
[0070] (5) The KNN model calculates the distance between a training sample and all training samples in the training sample set. The distance is calculated using the Euclidean distance formula in the feature space: ;in, is the first m training samples, is the first o training samples, for and The distance is the number of features, is the first m The first training sample k Features, is the first o The first training sample k features. Based on the calculated distance, select the K training samples with the closest distance as the nearest neighbors of the new training sample. Perform a weighted average on the dynamic emotion labels of these K nearest neighbors to obtain an estimate of the emotional state of the new data. Usually, neighbors with closer distances are given higher weights when calculating the weighted average. A common way to calculate the weighted average is to use the reciprocal distance as the weight, that is, the closer the neighbor, the higher the weight. The weighted average can be expressed as: ;in, represents the predicted emotional state, Indicates Dynamic emotion labels of neighbors, Indicates The distance between the neighbors and the new data point.
[0071] The value obtained by weighted average is the KNN model's estimate of the emotional state of the EEG signal data and eye movement signal data. This estimate is directly used as the output result of the KNN model to characterize the emotional state of the new data.
[0072] (6) The elastic network regression model needs to convert the EEG signal data and eye movement signal data into the form of feature vectors. Next, the data is standardized to ensure that the scales between features are consistent. The standardization process uses the z-score standardization method, and its formula is: ;in, is the original eigenvalue, is the standardized eigenvalue, is the mean of the feature, is the standard deviation of the feature.
[0073] The normalized feature vector is input into the trained elastic network regression model for sentiment prediction. The prediction process of the elastic network regression model can be expressed as: ;in, is the predicted emotional state, is the input feature matrix, is the coefficient vector of the elastic net regression model.
[0074] Interpret the emotional state based on the predictions of the elastic net regression model. If the prediction is a continuous value, the predicted emotion intensity is used directly; if the prediction is a discrete value, it may be mapped to the corresponding emotion category.
[0075] (7) The minimum absolute shrinkage and selection operation regression model utilizes the characteristics of Lasso regression, that is, by constraining the L1 norm of the coefficient, it realizes the sparse selection of features. The minimum absolute shrinkage and selection operation regression model predicts new data based on the relationship between the learned features and emotions and generates corresponding emotion prediction results. This prediction result can be a continuous value reflecting the intensity of the emotion, or a discrete value representing the category of the emotion. The emotional state is interpreted based on the prediction results of the minimum absolute shrinkage and selection operation regression model. If the prediction result is a continuous value, the predicted emotion intensity can be used directly; if the prediction result is a discrete value, it needs to be mapped to the corresponding emotion category.
[0076] The present invention collects EEG signals and eye movement signals through a set experimental paradigm, uses time-frequency feature extraction technology to extract EEG features and eye movement features from the preprocessed EEG signal data and eye movement signal data, and fuses the EEG features and eye movement features, inputs them into the regression model to obtain the corresponding recognition results, and achieves recognition processing by collecting two recognition signals, thereby realizing dynamic emotion recognition. It improves the accuracy and real-time performance of emotion recognition and makes up for the shortcomings of traditional emotion recognition methods in terms of real-time performance.
[0077] Embodiment 2: A computer system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dynamic emotion recognition method based on EEG signals and eye movement signals in Embodiment 1.
[0078] Embodiment 3: A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dynamic emotion recognition method based on EEG signals and eye movement signals in Embodiment 1.
[0079] Embodiment 4: A computer program product, comprising a computer program, which, when executed by a processor, implements the dynamic emotion recognition method based on EEG signals and eye movement signals in Embodiment 1.
[0080] Embodiment 5: A computer device, which may be a database, includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the dynamic emotion recognition method based on EEG signals and eye movement signals in Embodiment 1 is implemented.
[0081] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties.
[0082] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided by the present invention may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0083] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0084] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A dynamic emotion recognition method based on EEG signals and eye movement signals, characterized in that: The dynamic emotion recognition method based on EEG signals and eye movement signals includes: Acquiring EEG signal data and eye movement signal data of a target subject; the eye movement signal data includes pupil diameter and fixation duration; The time-frequency feature extraction technology is used to extract the EEG features and eye movement features from the preprocessed EEG signal data and eye movement signal data, and a regression model is used to determine the dynamic emotion category of the target subject based on the EEG features and the eye movement features, specifically including: Determining the differential entropy characteristics of the EEG signal data according to the time series information of the EEG signal data; Performing Fourier transform on the EEG signal data, converting the EEG signal data from the time domain to the frequency domain, and extracting power spectrum density features of the EEG signal data; Determining, according to the time series information of the pupil diameter, a differential entropy feature of the pupil diameter, a mean feature of the pupil diameter, and a standard deviation feature of the pupil diameter; Determining, according to the time series information of the fixation duration, a mean characteristic of the fixation duration and a standard deviation characteristic of the fixation duration; The power spectrum density feature of the EEG signal data, the differential entropy feature of the EEG signal data, the differential entropy feature of the pupil diameter, the mean feature of the pupil diameter, the standard deviation feature of the pupil diameter, the mean feature of the fixation duration, and the standard deviation feature of the fixation duration are cascaded to fuse features to obtain a fused feature vector; wherein the cascade feature fusion step is specifically as follows: the EEG feature is directly attached to the end of the eye movement feature to form a longer feature vector; Classifying the fused feature vector to obtain a dynamic emotion category of the target subject; The dynamic emotion categories include calm and happy, happy and calm, sad and calm, calm and sad, nervous and calm, and calm and nervous; the regression model is obtained by pre-training using a training sample set, and each training sample in the training sample set includes an EEG sample signal, an eye movement sample signal and a corresponding dynamic emotion label; the EEG features include differential entropy features and power spectral density features of EEG signal data; the eye movement features include differential entropy features of pupil diameter, mean features of pupil diameter, standard deviation features of pupil diameter, mean features of gaze duration, and standard deviation features of gaze duration.
2. The dynamic emotion recognition method based on EEG signals and eye movement signals according to claim 1, characterized in that: Obtain the EEG signal data and eye movement signal data of the target subject, including: Collect the original EEG signals and original eye movement signals of the target subjects based on the set experimental paradigm; Filtering the original EEG signal to obtain a filtered EEG signal; Performing artifact removal processing on the filtered EEG signal to obtain EEG signal processing data; Resampling the EEG signal processing data to obtain EEG signal data of a target subject; The original eye movement signal is filtered and processed, and the pupil diameter and fixation duration are derived to obtain the eye movement signal data of the target subject.
3. The dynamic emotion recognition method based on EEG signals and eye movement signals according to claim 2, characterized in that: Filtering the original EEG signal to obtain a filtered EEG signal specifically includes: The EEG signal of the required frequency band is filtered out from the original EEG signal through a bandpass filter to obtain a filtered EEG signal; wherein the required frequency bands include 1Hz~3Hz frequency band, 4Hz~7Hz frequency band, 8Hz~13Hz frequency band, 14Hz~30Hz frequency band and 31Hz~50Hz frequency band.
4. The dynamic emotion recognition method based on EEG signals and eye movement signals according to claim 1, characterized in that: The extraction formula of the differential entropy feature of the EEG signal data is: ; in, is the differential entropy feature of the EEG signal data, is the total number of samples, For the EEG signal data n The value of the sampling point, For the EEG signal data n +1 sample value.
5. The dynamic emotion recognition method based on EEG signals and eye movement signals according to claim 1, characterized in that: The extraction formula of the power spectrum density feature of the EEG signal data is: ; in, is the power spectral density feature of the EEG signal data, is the total number of samples, For the EEG signal data n The value of the sampling point, j is the imaginary unit, is the angular frequency, which is used to describe the characteristics of the signal in the frequency domain. is the average power.
6. The method for dynamic emotion recognition based on EEG signals and eye movement signals according to claim 1, characterized in that: Before cascading feature fusion of the power spectrum density feature of the EEG signal data, the differential entropy feature of the EEG signal data, the differential entropy feature of the pupil diameter, the mean feature of the pupil diameter, the standard deviation feature of the pupil diameter, the mean feature of the fixation duration, and the standard deviation feature of the fixation duration, the preprocessed EEG signal data and eye movement signal data are extracted using a time-frequency feature extraction technique, and based on the EEG features and the eye movement features, a regression model is used to determine the dynamic emotion category of the target subject, which also includes: The power spectral density characteristics of the EEG signal data, the differential entropy characteristics of the EEG signal data, the differential entropy characteristics of the pupil diameter, the mean characteristics of the pupil diameter, the standard deviation characteristics of the pupil diameter, the mean characteristics of the fixation duration, and the standard deviation characteristics of the fixation duration are standardized.
7. The dynamic emotion recognition method based on EEG signals and eye movement signals according to claim 1, characterized in that: The regression models include radial basis function support vector regression model, linear support vector regression model, random forest regression model, decision tree regression model, K nearest neighbor regression model, elastic network regression model, ridge regression model and least absolute shrinkage and selection operation regression model; The regression model is any regression model selected from a pre-trained radial basis function support vector regression model, a pre-trained linear support vector regression model, a pre-trained random forest regression model, a pre-trained decision tree regression model, a pre-trained K-nearest neighbor regression model, a pre-trained elastic network regression model and a pre-trained minimum absolute shrinkage and selection operation regression model according to actual needs.
8. A computer system comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dynamic emotion recognition method based on EEG signals and eye movement signals as described in any one of claims 1 to 7.
