An emotion recognition method based on dynamic brain network features

Abstract

The application relates to the technical field of emotion recognition, and discloses an emotion recognition method based on dynamic brain network features, which is characterized by the following steps: pre-processing collected brain wave signals; using a dyPLM dynamic phase linear measurement method to extract dynamic brain function network features from the pre-processed brain wave signals; and inputting the extracted dynamic brain function network features into a trained neural model for emotion recognition. The constructed brain network features are not easily affected by noise, can more accurately capture dynamic features of emotions, can effectively improve the accuracy of emotion recognition, and have a wider application range.

Description

Technical Field

[0001] This invention relates to the technical field of emotion recognition, specifically, to an emotion recognition method based on dynamic brain network features. Background Technology

[0002] Emotion recognition is a crucial frontier research topic in the field of human-computer interaction (HCI) emotional intelligence, aiming to simulate human-computer interaction through measured human emotions. Electroencephalogram (EEG) signals are widely used in emotion recognition research due to their high temporal resolution, portability, and non-invasiveness. However, current emotion recognition rates based on EEG signals are not high, hindering their application in real-world HCI scenarios. This is primarily because they extract static features, while emotions are continuously and dynamically changing. Current feature extraction methods fail to capture the dynamic characteristics of different emotions, impeding further improvements in emotion recognition capabilities. Therefore, there is an urgent need to develop new dynamic feature extraction methods for emotion recognition, which will promote the advancement of emotion recognition-based HCI emotional intelligence.

[0003] Cognitive neuroscience research shows that the brain's cognitive processing of emotions involves the joint participation of multiple different brain regions, which collaborate to form networks with different brain functions. Brain network features, compared to traditional features such as power spectrum and differential entropy, contain richer emotion-related information, enabling more effective emotion recognition. Currently, the brain network features extracted for emotion recognition are all static, such as magnitude squared coherence (MSC), phase locking value (PLV), and Pearson correlation analysis. However, emotions are continuously and dynamically changing, and current feature extraction methods cannot capture the dynamic characteristics of different emotions, hindering further improvements in emotion recognition capabilities.

[0004] EEG-based dynamic network estimation methods still have many problems and limitations. A common method for constructing dynamic functional networks is the sliding window method, which typically selects a fixed-length time window and then calculates functional connections using signals within the window. After calculation, the time window slides forward a fixed number of time points before calculating the functional connections for the next time window. This method yields a functional network that dynamically changes over time. However, the fixed-window sliding window method has certain limitations, the most prominent being the window length. If the window length is too small, it is easily affected by noise; if the window length is too large, it is difficult to capture the dynamic characteristics of emotions, and it requires subjective parameter setting and manual tuning. Furthermore, the connections obtained by the fixed-window method are actually connections of signals within each window time segment, and the time span is affected by the window length, making it impossible to extract the brain network features of emotions at each time point, resulting in a serious lack of functional connection information. Summary of the Invention

[0005] This invention provides an emotion recognition method based on dynamic brain network features. The method uses the dyPLM algorithm to extract dynamic brain network features related to emotions, and finally inputs the extracted features into a trained neural model CNGRU for emotion recognition, which can improve the accuracy of emotion recognition.

[0006] This invention can be achieved through the following technical solutions:

[0007] An emotion recognition method based on dynamic brain network features.

[0008] The collected EEG signals are preprocessed;

[0009] Dynamic brain functional network features were extracted from the preprocessed EEG signals using the dyPLM dynamic phase linearity measurement method.

[0010] The extracted dynamic brain function network features are input into the trained neural model for emotion recognition.

[0011] Furthermore, an EEG cap with multiple acquisition electrodes was used to collect brainwave signals from different subjects. Then, the brainwave signals of each subject were preprocessed, and the dyPLM dynamic phase linear measurement method was used to extract features from the data at each acquisition time point of the preprocessed brainwave signals to obtain dynamic brain function network features. Finally, the dynamic brain function network features corresponding to each subject were input into the trained neural model for emotion recognition.

[0012] Furthermore, the preprocessed EEG signals were recorded as follows: C represents the total number of signals in the EEG cap, and T represents the total number of time points in the data collection process.

[0013] First, perform Hilbert transform on the preprocessed EEG signal to obtain the transformed EEG signal.

[0014] Then analyze the transformed EEG signals Any two lead signals and calculate in, express The conjugate of complex numbers;

[0015] Calculate the phase Δφ(t) of z(t), and obtain z(t,f) by performing a time-frequency transform on z(t). The window function is a Gaussian function. [0,T] represents the observation interval of the signal, and the corresponding energy spectral density s is then calculated. z (t,f)=|z(t,f)| 2

[0016] Finally, calculate the functional connectivity values ​​of any two leads at acquisition time point t. This is used to construct a brain functional network matrix;

[0017] Repeat the above process to obtain the functional connectivity values ​​corresponding to all collection time points t for each subject, thereby obtaining dynamic brain functional network characteristics.

[0018] Furthermore, the neural network model adopts the CNGRU model, which has two convolutional layers, each using the ReLU activation function, and the dropout method is used after pooling in each layer to reduce overfitting; after two convolutional layers, the output feature map is then input into the two-layer GRU model, and finally a softmax regression model is used for classification.

[0019] Furthermore, the emotion recognition categories are set as high-valence high-arousal, high-valence low-arousal, low-valence high-arousal, and low-valence low-arousal.

[0020] The beneficial technical effects of this invention are as follows:

[0021] This invention provides a novel method for constructing dynamic brain networks based on the dyPLM algorithm for feature extraction. It can construct a brain network for each sampling time point signal, thereby forming a dynamic brain network within a given sampling time period. Compared with the existing technology that uses the sliding window method to construct dynamic networks, it is less affected by noise and can more accurately capture the dynamic features of emotions, effectively improving the accuracy of emotion recognition and having a wider range of applications. Attached Figure Description

[0022] Figure 1This is a flowchart of the emotion recognition method based on EEG dynamic brain functional network features of the present invention;

[0023] Figure 2 This is a schematic diagram illustrating the process of constructing a dynamic brain functional network based on dyPLM according to the present invention;

[0024] Figure 3 This is a structural diagram of the CNGRU network model of the present invention;

[0025] Figure 4 The dynamic brain network features extracted by the dyPLM method and the sliding window-based Pearson correlation coefficient method provided in this invention are used to perform emotion recognition on the DEAP dataset for all subjects.

[0026] Figure 5 This is a schematic diagram illustrating the convergence of the loss function values ​​in an embodiment of the present invention on the DEAP dataset;

[0027] Figure 6 This is a schematic diagram of an emotion recognition system based on the dynamic brain function network features of electroencephalography (EEG) provided in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0029] like Figure 1 As shown, since the phase difference between any two EEG signal sources is a linear relationship that changes with time, the slope of this linear relationship depends on the difference in the center frequencies of the two signals. The dyPLM dynamic phase linear measurement algorithm is a pure phase-related measurement method. Using this algorithm to estimate brain connectivity based on signal phase, constructing dynamic brain network features, and finally inputting them into the neural network CNGRU model for emotion recognition can improve the accuracy of emotion recognition.

[0030] For ease of understanding, this invention uses an emotional EEG dataset from the DEAP public database for illustration. This DEAP database was collected experimentally by Koelstra et al. from Queen Mary University of London, the University of Twente, the University of Geneva, and ETH Zurich, and is used to study multi-channel data of human emotional states. The DEAP database records physiological signals generated under the stimulation of music video materials. It records the physiological signals of 32 subjects watching 40 minutes of music videos, with each music video watched for 1 minute. The physiological signal sampling rate was 512 Hz, and the resampling rate was 128 Hz. Each subject underwent 40 trials, and after each trial, a timely self-emotional category assessment was performed, including valence and arousal. In the following embodiments, the present invention uses 32-lead EEG data with a resampling frequency of 128Hz as an example for illustration. The emotion categories in each test are divided into four categories: High Valence High Arousal (HVHA), High Valence Low Arousal (HVLA), Low Valence High Arousal (LVHA), and Low Valence Low Arousal (LVLA).

[0031] Specifically as follows:

[0032] Step 1: Preprocess the collected EEG signals;

[0033] All EEG signal data are preprocessed to remove noise such as blinking, eye movement, and electromyography, so as to obtain noise-free and clean EEG data.

[0034] Step 2: Extract dynamic brain functional network features from the preprocessed EEG signals using the dyPLM dynamic phase linear measurement method. This involves using dyPLM to extract features from data at each acquisition time point of the preprocessed EEG signals, thereby obtaining dynamic brain functional network features. Figure 2 As shown.

[0035] The preprocessed EEG signal is recorded as C represents the total number of signals in the EEG cap, and T represents the total number of time points in the data collection process.

[0036] First, perform Hilbert transform on the preprocessed EEG signal to obtain the transformed EEG signal.

[0037] Then analyze the transformed EEG signals Any two lead signals and ,calculate in, express The conjugate of complex numbers;

[0038] Then, the phase Δφ(t) of z(t) is calculated, and the z(t) is subjected to a time-frequency transform to obtain the desired result. The window function is a Gaussian function. [0,T] represents the observation interval of the signal. The time-frequency transformation used in this invention is the S-transform, and then the corresponding energy spectral density s is calculated. z (t,f)=|z(t,f)| 2 ;

[0039] Finally, calculate the functional connectivity values ​​of any two leads at acquisition time point t. Therefore, the dyPLM algorithm calculates the percentage of spectral energy within a narrow 2B band centered at 0 at acquisition time t relative to the total signal energy at that acquisition time, thereby constructing a 32×32 brain functional network matrix corresponding to that acquisition time t.

[0040] Repeat the above process to obtain the functional connectivity values ​​corresponding to all data collection time points t for each subject, thereby obtaining the dynamic brain functional network characteristics. And its value is between 0 and 1.

[0041] Step 3: Input the extracted dynamic brain functional network features into the trained neural model for emotion recognition, such as... Figure 3 As shown. The input to this neural model is a 32×32 brain functional network matrix, where 32 represents the total number of lead signals in the electroencephalogram (EEG) data;

[0042] For each brain functional network matrix, two one-dimensional convolutional layers are input sequentially. Each convolutional layer has a kernel size of 3, a number of kernels of 128, a stride of 1, and an activation function of tanh. The pooling layer following each convolutional layer has a size of 2, and the dropout is set to 0.2. After the features pass through two convolutional layers, pooling layers, and dropout layers, a feature map of size 6×128 is obtained.

[0043] Then, the 6×128 feature map is input into a two-layer GRU model in chronological order. The first layer has a dimension of 6×256, the second layer has a dimension of 6×32, and the output feature dimension is 32×1.

[0044] Finally, the 32×1 features are connected to a fully connected layer of length 128, and then connected to a softmax layer to achieve the four categories of emotion classification.

[0045] Figure 4 The figures demonstrate the classification results of the dyPLM method and the Pearson correlation coefficient method based on the sliding window approach provided in this invention, based on feature selection for each subject on the DEAP dataset. As can be seen from the figures, the method proposed in this invention achieves a four-class classification accuracy of over 99% for the emotional EEG data of most subjects, and even the lowest classification accuracy of subject number 22 is over 92%. The accuracy of emotion recognition based on the dyPLM dynamic brain network feature extraction method is much higher than that of the Pearson correlation coefficient method based on the sliding window approach for most subjects. Figure 5 This invention demonstrates the results of applying the loss function of the method provided in this embodiment to a bracelet on the DEAP dataset. Figure 4 and Figure 5 As can be seen, the method provided by this invention performs very well and has a certain degree of robustness.

[0046] Figure 6 This invention demonstrates an emotion recognition system based on dynamic brain functional network features, comprising an EEG acquisition module, an EEG data storage module, an EEG data preprocessing module, a dynamic brain network feature extraction module, and an emotion recognition module. The EEG data preprocessing module, the dynamic brain network feature extraction module, and the emotion recognition module are implemented on a server. It should be noted that the division of these modules is merely a logical functional division; in actual implementation, all or part of them can be integrated into a single physical entity. Each module can be implemented by a software program through the calling of component functions.

[0047] The EEG acquisition module and the EEG data storage module are connected and used to acquire emotional EEG data and transmit it to the memory for storage; the emotional EEG acquisition module is used to acquire emotional EEG data, which includes multiple categories, such as positive, neutral and negative emotional EEG data.

[0048] The EEG data preprocessing module is used to preprocess the emotional EEG data to remove noise such as blinking, eye movement, and electromyography, and obtain clean EEG data without noise.

[0049] The dynamic brain function network feature extraction module is used to extract dynamic brain networks of different emotion categories from preprocessed EEG data.

[0050] The emotion recognition module is used to predict and recognize different emotion categories based on the dynamic brain network features of the different emotion categories.

[0051] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples. Various changes or modifications can be made to these embodiments without departing from the principles and essence of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims.

Claims

1. An emotion recognition method based on dynamic brain network features, characterized in that: the collected brain wave signals are preprocessed; the dyPLM dynamic phase linear measurement method is used to extract dynamic brain function network features from the preprocessed brain wave signals; the extracted dynamic brain function network features are input into a trained neural network model for emotion recognition; the multi-lead EEG cap is used to collect brain wave signals of different subjects, then the brain wave signals of each subject are preprocessed, the dyPLM dynamic phase linear measurement method is used to extract features from the data of each collection time point of the preprocessed brain wave signals, thereby obtaining dynamic brain function network features, and finally the dynamic brain function network features corresponding to each subject are input into a trained neural network model for emotion recognition; the above process is repeated to obtain the functional connection value corresponding to each collection time point t of each subject, thereby obtaining the dynamic brain function network features. The neural network model adopts a CNGRU model, which is provided with two convolutional layers, each of which adopts a Relu activation function, and a dropout method is used after each layer of pooling to reduce overfitting; after two convolutional layers, a feature map is output, then input into two GRU models, and finally a softmax regression model is used for classification. The emotion recognition categories are set as high valence high arousal, high valence low arousal, low valence high arousal, and low valence low arousal. ​ ​ The pre-processed brain wave signal is denoted as S , C is the total number of lead signals in the brain cap, T is the total time point of data collection, the pre-processed brain wave signal is subjected to Hilbert transform to obtain a transformed brain wave signal ; The transformed electroencephalogram signal Any two of the lead signals And Calculate Wherein Is The conjugate complex of Then the phase of is calculated , and the time-frequency transform is performed on to obtain , where the window function is Gaussian function , is the observation interval of the signal, and the corresponding energy spectrum density is calculated ; Finally, the functional connectivity value of any two leads at the acquisition time point t is calculated In this way, the brain function network matrix is constructed ​ 2. The emotion recognition method based on dynamic brain network features according to claim 1, characterized in that: ​ 3. The emotion recognition method based on dynamic brain network features according to claim 2, characterized in that: ​

Images