An electroencephalogram emotion recognition system based on deep reinforcement learning and double-patch residual network

Abstract

The application discloses a kind of electroencephalogram emotion recognition systems based on deep reinforcement learning and double lobe residual network, comprising the following steps: first, agent receives each frame of electroencephalogram signal sample as environment state, then using double lobe residual network to fit Q value function, give the action (i.e. classification) corresponding to maximum Q value;Afterwards, introduce step penalty factor, compare predicted category and existing label to calculate reward;Finally, state, action, reward and next state are stored in experience replay pool to form transition sequence, agent is subsequently sampled and sent into network to update parameters until optimal parameters are found.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electroencephalogram emotion recognition, in particular, the present application relates to an electroencephalogram emotion recognition system for emotion classification of electroencephalogram signals based on deep reinforcement learning and double frontal lobe residual network. BACKGROUND

[0002] Emotion, as an important physiological information, is a key factor affecting human daily interaction, action decision and other important behaviors. In human-computer interaction, emotion is crucial to the assessment of the situation and the cognition of information. The machine calculates and makes decisions based on the current emotional or behavioral state of the user, thereby providing a more friendly interaction interface for the user. How to more intelligently and accurately identify emotions is an urgent problem for researchers. Emotion recognition is one of the hottest research topics in the field of affective computing and brain-computer interface. In the field of emotion recognition, it has been studied for many years to accurately identify emotions through external reactions such as facial expressions and speech intonation. However, emotion is a spontaneous physiological phenomenon, and its internal reaction cannot be ignored. From the perspective of neuroscience, emotion is closely related to many brain regions, such as the amygdala, the prefrontal cortex and the orbitofrontal cortex. Therefore, it is a more comprehensive and more convincing method to identify emotions by placing electroencephalogram (EEG) electrodes on the scalp to record physiological signals.

[0003] The human brain is the center and core of emotional consciousness and cognitive processing, logical reasoning, enabling people to quickly generate and feedback emotions from external emotional stimuli, which is undoubtedly difficult for machines. With the continuous research and in-depth study of brain nerves and cortical anatomy in the medical field, researchers have also established a certain theoretical basis for simulating the seemingly abstract problems of how emotions are generated and how things are distinguished using mathematical models. Although the structure and principles of the brain's complex emotional and perceptual system are not yet fully understood, the physiological electrical data obtained from extensive experiments can be summarized and theoretically summarized to inspire researchers to establish biological mathematical models representing the activity mechanisms of different neurons and brain cortex, such as constructing artificial neural networks or spiking neural networks that can simulate the local neural network structure of the human brain by studying the mechanism of neuron transmission of electrical signals. Therefore, starting from the emotional and cognitive system of the human brain, by learning from the mechanism of emotion generation and information processing, exploring the processing flow of emotional stimuli, and thus constructing a more bionic intelligent emotion recognition system that is closer to the cognitive laws of the human brain.

[0004] Emotion is a complex physiological activity generated and processed by different regions of the brain. It is necessary to explore the physiological characteristics and emotional mechanisms of the brain before modeling. The process of learning rewards in the human brain is simulated using a reinforcement learning framework, and the process of processing emotional stimuli in the human brain emotional circuit is simulated using a dual frontal lobe residual network. The fusion of the two can fully simulate the emotional structure and function of the human brain, obtain rich physiological information contained in the EEG emotional signal, and further improve the accuracy of EEG emotion recognition. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a more effective EEG emotion recognition system, which can further improve the accuracy of EEG emotion recognition. Inspired by the role of the frontal lobe in emotion and the Papez emotional circuit, the purpose of the present application is to propose an EEG emotion recognition system based on deep reinforcement learning and dual frontal lobe residual network, and to construct a brain emotion mechanism learning model based on the frontal lobe-Papez circuit. The model embeds the emotional network into the framework of deep reinforcement learning, allowing the agent to iteratively learn rewards. It fully utilizes the physiological mechanisms and characteristics of EEG emotion recognition to model the generation and regulation of human emotions, provides a new method for EEG emotion recognition simulation calculation, and further improves the performance of EEG emotion recognition.

[0006] 1. An EEG emotion recognition system based on deep reinforcement learning and dual frontal lobe residual network, characterized by the following steps:

[0007] S1, converting the EEG emotion recognition task into a Markov decision process problem, constructing a reinforcement learning framework, and the agent receiving each frame of EEG sample as the environment state;

[0008] S2, using a dual frontal lobe residual network to fit the Q value function for the environment state obtained in step S1, and giving the action corresponding to the maximum Q value (i.e. classification);

[0009] S3, introducing a step penalty factor to the action obtained in step S2, comparing the predicted class and the existing label to calculate the reward;

[0010] S4, storing the state, action, reward and next state obtained in steps S1, S2 and S3 in the experience replay pool, and then sampling the agent and sending it to the dual competitive deep Q network to update the parameters until the optimal parameters are found;

[0011] 2. Further, S1 converts the EEG emotion recognition task into a Markov decision process problem, and the purpose of the agent is to assign the most likely emotional label to each EEG sample. Given EEG data X=x1,...,x nAnd the corresponding sentiment labels L = l1,...,l n , where n represents the total number of samples. In this invention, the classifier can be viewed as an intelligent agent, defined as follows:

[0012] State: In this model, an EEG sample is considered a state s. At the beginning of each phase, all training samples are shuffled, and the first sample is selected as the initial state. Similarly, the state s at time step t... t Indicates the corresponding sample x t In each time step of an iteration, the current state transitions to the next state s. t+1 , with the next sample x t+1 Consistent.

[0013] Action: An action 'a' of an agent is to predict a sentiment label, which has a value of 0 or 1.

[0014] Policy: π(a) t |s t ) is the state s that the agent chooses at step t. t Action a in t The strategy employed at that time. The agent's goal is to find the optimal policy π that maximizes the cumulative reward by assigning the best action to each state. * That is, to search for the optimal classifier for emotion recognition.

[0015] Reward: Unlike traditional reinforcement learning, the reward value r of this invention is... t Depending on the action taken by the subject a t Is it equal to the sentiment label? t The formula for calculating the reward is defined as follows:

[0016]

[0017] Termination: At the end of the t-th time step, the agent decides whether to continue exploring, using the terminal function. t This indicates that the process ends when all samples are correctly classified or when there are classification errors.

[0018]

[0019] Thus, the sentiment classification task is ultimately transformed into a reinforcement learning problem, with the goal of finding an optimal agent (classifier).

[0020] 3. Further, S2 is a network that is influenced by the Papez emotional circuit and the brain's hemispheric asymmetry response, and it contains the residual network of the two hemispheres of the frontal lobe. The network is divided into two hemispheric flows, and the original EEG data containing all the electrodes are first divided into left and right hemisphere blocks. In each hemisphere block, the EEG signals from the frontal lobe region are input into the network as prior information. Then, the features obtained by the network are merged and input into the softmax layer to obtain the probability of all actions. Finally, the final Q value is output using the competitive architecture with advantage value and state value.

[0021] Further, S3 is a step penalty factor defined as the inverse of the time step experienced, and is controlled using a coefficient λ,

[0022]

[0023] where R t represents the cumulative reward at the t-th time step. Intuitively, this term means that the agent will be given a greater penalty in earlier time steps, and only works when the agent makes a classification error, at which time the reward function is rewritten as:

[0024]

[0025] As shown above, if the agent predicts an incorrect class at the beginning of the iteration, it will get the least reward (the largest penalty). Conversely, when the agent completes the classification of all samples (i.e., there is no classification error), it will get a minimum penalty value close to zero.

[0026] Further, S4 is a double competitive deep Q network that is divided into a current Q network and a target Q network to update the parameters. The agent uses the current Q network to continuously explore the environment, and the corresponding reward is calculated. The target Q network copies the parameters from the current Q network at a set update frequency to reduce the correlation between the two networks. At the same time, the state value V and the advantage value A are added to obtain the state-action value Q, and the best action is performed according to the state s corresponding to the maximum Q value, and the state, action, reward and next state are stored in the experience replay pool. Then, the agent samples from it, calculates the loss function according to the Q values obtained from the two networks, and performs gradient descent algorithm to update the current Q network parameters.

[0027] The beneficial effects of this invention are as follows: The EEG emotion recognition system based on deep reinforcement learning and a bifrontal residual network simulates the generation and regulation of human emotions by embedding an emotion network into a reinforcement learning framework. The deep reinforcement learning framework simulates the dopamine reward learning process from the frontal lobe to the thalamus, while the bifrontal residual network simulates the collaborative operation mechanism of the Papez emotional circuit and the frontal lobe. Residual connections are used to simulate the important pathway from the thalamus to the hypothalamus in the Papez circuit. However, we use a residual neural network to simulate this process. Furthermore, this invention designs a step-size penalty factor for the original reward function. If the agent adopts an incorrect strategy earlier, it will be penalized more severely to limit the number of errors. Therefore, compared to other methods that only focus on mathematical or statistical models, this method is more biologically convincing and improves the performance of emotion recognition. Attached Figure Description

[0028] Figure 1 This is a flowchart of the overall framework of this invention;

[0029] Figure 2 This is a flowchart of the dual frontal lobe residual network in this invention;

[0030] Figure 3 This is a structural diagram of the dual-competition deep Q-network in this invention; Detailed Implementation

[0031] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments: The method of the present invention is divided into three parts.

[0032] Part 1: Deep Reinforcement Learning for Sentiment Classification

[0033] like Figure 1 As shown, information from the frontal lobe is fed into the Papez circuit for processing and then returned to the frontal lobe for recognition. This is similar to a classification task where samples are fed into a neural network for fitting and prediction. Therefore, this method compares EEG samples and the fitted Q-network with the frontal lobe and Papez circuit, respectively, thereby transforming the EEG emotion recognition task into a Markov decision process problem. A reinforcement learning process based on EEG emotion recognition is constructed, and its basic steps are as follows:

[0034] 1: Select the best action through the Q learning network;

[0035] 2: Perform action a t (i.e., classification) to obtain the corresponding reward r t ;

[0036] 3: Proceed to the next state s t+1 ;

[0037] 4: Decide whether to terminate terminal t ;

[0038] 5: Store the transition sequence (s t , a t , r t , s t+1 , terminal t ) into the replay experience pool;

[0039] 6: Update the network parameters using the transition sequence randomly sampled from the replay experience pool;

[0040] 7: Until terminal t =true, the step ends;

[0041] Second part: Dual frontal lobe residual network

[0042] Relevant neuroscience theories show that the left and right hemispheres of the human brain are asymmetric in some emotional responses, and the frontal cortex plays an indispensable role in action decision-making and emotional regulation. Therefore, the dual-hemisphere structure is proposed to explore the differences between the left and right hemispheres, and the frontal electroencephalogram data is integrated into the network as prior information to assist the left and right hemispheres in identifying emotions. In addition, the direct neural pathway from the thalamus to the hypothalamus in the emotional circuit is simulated by the residual neural network.

[0043] Let l, r, f represent the left hemisphere, the right hemisphere, and the frontal lobe region, respectively. The original electroencephalogram data matrix from different regions is represented by X i , where i∈{l,fl,r,fr} represents the left hemisphere, the frontal lobe region of the left hemisphere, the right hemisphere, and the frontal lobe region of the right hemisphere, respectively. When i∈{l,r}, When i∈{fl,fr}, Here, d represents the dimension of each sample, N is the total number of electrodes, and N f is the number of frontal lobe electrodes in the hemisphere. The approximate matrix S i for each hemisphere and its frontal lobe region is obtained through the residual neural network, defined as f(·), i.e.:

[0044]

[0045] After the residual neural network, a fully connected layer with weight matrix and bias is used to map to Here d s and d f are the dimensions of S i and F i , respectively. The mapping process is as follows:

[0046] F i = σ(W f S i + b f ) (6)

[0047] For each hemisphere, the embedding of the hemisphere and the frontal lobe are concatenated through a fully connected layer as the final output of a hemisphere stream, i.e.,

[0048]

[0049]

[0050] Here, "Concatentate" means concatenating two matrices of the same shape to construct a larger matrix. If two input feature matrices have the same dimension (d f ,N), then the two matrices are concatenated on the 0th dimension to form a matrix of dimension (2xd f ,N), indicating that it combines the information of the hemisphere and the frontal lobe.

[0051] Then, the outputs of the two hemispheres are merged, and the process is represented as:

[0052]

[0053] Finally, the probability matrix P of each action (i.e., emotion label classification) is obtained using the softmax function, and the calculation formula is as follows:

[0054]

[0055] where, and b p ∈R C×N is the parameter matrix of the softmax layer, and C is the number of emotion categories.

[0056] Part III: Double Deep Q Network

[0057] As shown in Fig. Figure 3 , the double deep Q network contains two Q networks, the current Q network is initialized with a random parameter θ, and the target network is initialized with another random parameter θ'. The goal of the agent is to find an optimal policy that maximizes the cumulative reward in an iteration. According to the definition of advantage learning, the original Q estimation network is divided into state value function V(·) and advantage value function A(·). The former outputs a scalar representing the value in the current state, and the latter outputs a vector of dimension ∣A∣ representing the value corresponding to each action, as follows:

[0058] Q(s t ,at ; θ q , θ a , θ v ) = V(s t ; θ q , θ v ) + A(s t , a t ; θ q , θ a ) (11)

[0059] where θ q is the parameter of the shared network, θ a and θ v are the parameters of the advantage function subnetwork and the value function subnetwork.

[0060] To constrain the feasible solution space of the advantage function, a mean constraint is added in the original calculation. Therefore, the improved Q value function is defined as:

[0061]

[0062] where a' is one of the actions in the current set.

[0063] In addition to using the same value for estimation and action selection, the double Q-learning algorithm also decouples these two parts. It uses two networks, where the target Q network evaluates the value corresponding to the best action selected by the current Q network. Thus, the final state-action value Q t is calculated by the following formula:

[0064] Q t = r t + γQ(s t+1 , argmax a″ Q(s t+1 , a"; θ); θ') (13)

[0065] where a" is one of the set of all actions, and the best action is selected by maximizing the Q value of the current Q network. γ ∈ [0, 1] is the discount factor, which weighs the weight of the current reward and the future reward.

[0066] Let a * (s t+1 ; θ) = argmax a″ Q(s t+1 , a"; θ), the parameters of the current Q network are updated by minimizing the value of the following loss function:

[0067] L(θ) = ||[r t + γQ(s t+1 , a * (s t+1; θ); θ') ] - Q(s t ,a t ; θ) | 2 (14)

[0068] Then, a gradient descent operation is performed to optimize the loss function as follows:

[0069]

[0070] where a is the learning rate.

[0071] Experimental design

[0072] Experimental dataset: The DEAP dataset used in the experiment is a large-scale multi-modal affective dataset, which records a variety of bioelectric signals using several electrodes, including electroencephalogram, electrooculogram, electromyogram, etc. In the DEAP dataset, 16 female volunteers and 16 male volunteers were asked to rate four affective dimensions (valence, arousal, dominance, and liking) from 1 to 9 after each time watching a one-minute music video. The dataset contains a total of 40 videos, and the volunteers recorded electroencephalogram and other physiological signals under affective stimulation while watching the videos by wearing an electrode cap. The electrodes are placed according to the international standard 10-20 system, and the first 32 electrodes are used to record electroencephalogram signals, and the remaining electrodes are used to record other bioelectric signals. Each subject contains 40 electroencephalogram recording data, each of which is a time series of 63 seconds in length.

[0073] Experimental results

[0074] Experimental results under different classification models: To verify the effectiveness of the proposed invention, the model is compared with different methods on the DEAP dataset, including traditional machine learning models, deep learning models, and reinforcement learning models. Among the machine learning models, three shallow model emotion classifiers are used: decision tree (Decision Tree, DT), k-nearest neighbor (k-Nearest Neighbor, KNN), and support vector machine (Support Vector Machine, SVM), and differential entropy (Differential Entropy, DE) is extracted as a feature; four deep learning models: convolutional neural network (Convolutional Neural Network, CNN) and long short-term memory (Long Short-Term Memory, LSTM) network, and two deep neural networks that recently solve the uncertainty in data, including rough autoencoder (Rough Autoencoder, RAE) and deep belief network (Deep Belief Network, DBN); three reinforcement learning models: deep Q network (Deep Q Network, DQN), double DQN (Double DQN), and competing DQN (Dueling DQN). Among them, CNN uses the same parameter settings as the dual lobe residual network, and LSTM has 256 neurons. All reinforcement learning models use the same parameter configuration as the reinforcement learning framework of the method. Table 1 compares the average accuracy of the baseline method and the proposed invention. It can be seen that the proposed invention is superior to all baseline methods in the valence and arousal dimensions of the DEAP dataset, proving that the invention can successfully learn the differences between the asymmetric frontal lobe regions of the two hemispheres when exploring electroencephalogram sequences, and does not need to manually extract any features.

[0075] Table 1 Average accuracies compared with different classificationmethods in Valence and Arousaldimensions on DEAP.

[0076] Table 1 Average accuracies compared with different classificationmethods in Valence and Arousaldimensions on DEAP.

[0077]

[0078] Experimental results of different innovative modules: The invention is built based on the deep reinforcement learning framework, and there are three innovative modules: residual connection between network layers, prior information from the frontal lobe, and the additional punishment imposed on the agent by the penalty factor λ. In order to evaluate the contribution of each innovative module, three variant models that delete part of the modules are designed here, so as to compare with the proposed invention:

[0079] M1: The basic model in the network only has several convolutional layers, without frontal lobe residual blocks and prior information;

[0080] M2: The model with neural network residual blocks and residual connections, but without electroencephalogram data from the frontal lobe region;

[0081] M3: The model without the penalty factor, i.e. λ = 0.

[0082] Table 2 shows the comparison results of the binary classification accuracy of the variant models and the proposed invention in the valence and arousal dimensions in the DEAP dataset.

[0083] Table 2 Contribution comparison of different novel parts in improvingperformance.

[0084] Table 2 Contribution comparison of different novel parts in improvingperformance.

[0085]

[0086]

[0087] From the table, it can be clearly seen that M2 with a double frontal lobe residual network simulating the brain emotional circuit has the greatest improvement on the model among all variant models, which verifies that the residual connection in the neural network successfully simulates the role of the direct neural pathway between the thalamus and hypothalamus in the Papez circuit. In addition, the addition of frontal lobe prior information and the penalty factor also improves the accuracy by 1.91% and 0.5% in the valence dimension, and the accuracy in the arousal dimension is improved by 1.28% and 0.63%, which also proves the important contribution of electroencephalogram data from the frontal lobe to the electroencephalogram emotion recognition process, and the penalty factor effectively constrains the probability of the agent's mistakes.

Claims

1. A deep reinforcement learning and dual-hemisphere residual network based electroencephalogram emotion recognition system, characterized in that, Comprising the following steps: S1, the electroencephalogram emotion recognition task is converted into a Markov decision process problem, a reinforcement learning framework is constructed, an agent receives each frame of EEG sample, which is regarded as an environment state, and the purpose of the agent is to assign the most possible emotion label to each EEG sample. Given the EEG data X=x1,...,x n and the corresponding emotion label L=l1,...,l n where n represents the total number of samples, and the classifier is regarded as an agent, which is defined as follows: State: In this system, an EEG sample is considered as a state s, at the beginning of each episode, all training samples are shuffled and the first sample is chosen as the initial state, similarly, the state s t at the t-th time step is the state corresponding to the sample x t At each time step of an iteration, the current state goes to the next state s t+1 consistent with the next sample x t+1 , Action: An action a of the agent is to predict a sentiment label, whose value is 0 or 1, Policy: π(a t | s t ) is the policy employed by the agent in choosing an action a t in state s t at time t, and the goal of the agent is to find the optimal policy π * that maximizes the cumulative reward by assigning the best action to each state, i.e., to search for the optimal classifier for emotion recognition, Reward: Unlike traditional reinforcement learning, the reward value r t depends on the behavior a t taken by the subject t , the reward calculation formula is defined as: Terminal: At the end of the t-th time step, the agent decides whether to continue exploration or not, using the terminal t denotes that at the end of the process, when all samples have been classified or there is a classification error, it represents the end of this phase, i.e.: In this way, the final emotion classification task is converted into a reinforcement learning problem, whose goal is to find an optimal agent (classifier); S2, using the double frontal residual network to fit the Q value function to the environment state obtained in step S1, giving the action corresponding to the maximum Q value, i.e. classification, the double frontal residual network is a network affected by the Papez emotional circuit and the brain's asymmetric response, which contains the residual network of the asymmetric information of the two frontal lobes of the brain hemisphere, the network is divided into two hemispheric flows, first, the original EEG data containing all the electrodes are divided into left and right hemisphere blocks, in each hemisphere block, the electroencephalogram from the frontal lobe region is input into the network as prior information, then the obtained representation is input into the softmax layer to obtain the probability of all actions, finally, the final Q value is output by the competitive architecture with advantage value and state value; S3, introducing a step penalty factor to the action obtained in step S2, comparing the predicted category and the existing label to calculate the reward; S4, storing the state, action, reward and next state obtained in steps S1, S2 and S3 in the experience replay pool, and then sampling and sending them into the double competitive deep Q network to update the parameters until the optimal parameters are found.

2. The electroencephalogram emotion recognition system based on deep reinforcement learning and dual lobe residual network architecture according to claim 1, characterized in that: The step penalty factor in S3 is defined as the inverse of the time step experienced, and is controlled using a coefficient λ, where R t represents the cumulative reward at the t-th time step, which intuitively means that the agent will be given a larger penalty in earlier time steps and only works when the agent makes a classification error, in which case the reward function is rewritten as: As shown above, if the agent predicts a wrong category at the beginning of the iteration, it will get the minimum reward, i.e. the maximum penalty, on the contrary, when the agent completes the classification of all samples, i.e. there is no classification error, it will get a minimum penalty value close to zero.

3. The electroencephalogram emotion recognition system based on deep reinforcement learning and dual lobe residual network architecture according to claim 1, characterized in that: The double competitive deep Q network in S4 is divided into current Q network and target Q network to update the parameters, the agent uses the current Q network to constantly explore the environment, and calculates the corresponding reward, the target Q network copies the parameters from the current Q network at a set update frequency to reduce the correlation between the two networks, at the same time, the state value V and the advantage value A are added to get the state action value Q, the best action is executed according to the state s corresponding to the maximum Q value, at the same time, the state, action, reward and next state are stored in the experience replay pool, then the agent samples from it, calculates the loss function according to the Q value obtained from the two networks and executes the gradient descent algorithm to update the current Q network parameters.

Images