An emotion recognition system based on multi-modal physiological and social information

Abstract

The application discloses an emotion recognition system based on multi-modal physiological and social information, utilizes EEG to realize modeling of a local to global central nervous system network, utilizes EEG in different brain regions to obtain a local brain neural network, utilizes spatial information between different brain regions to obtain a global brain neural network, realizes modeling of a local central nervous system to a global neural network, utilizes other modal physiological signals to supplement information of the central nervous system network on the basis of constructing the central nervous system network, utilizes ECG, EMG and other signals to model a peripheral nervous system network, realizes modeling of the central nervous system network to a global nervous system network of the human body, and further utilizes personal social information to construct a social network after obtaining the nervous system network of the human body, realizes network model construction from a local nervous system network to a global nervous system network and from an individual to a society.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to an emotion recognition system based on multimodal physiological and social information. Background Art

[0002] Emotions are a part of everyday human life, influencing cognition, behavior, and decision-making, and are indispensable in social interactions. Emotions influence nearly every aspect of human communication, including tone of voice, facial expressions, word choice, breathing, muscle tension, and more. They have the potential to fundamentally alter messages, as the way we speak can be more important than what we say. Researchers from diverse fields, including psychology, biology, sociology, and neuroscience, have defined, explained, analyzed, and categorized human emotions. In recent years, the increasing availability of various electronic devices in our lives has led people to spend increasing amounts of time on social media, online shopping, and online video games. However, most current human-computer interaction (HCI) systems have shortcomings in interpreting and understanding emotional information and lack emotional intelligence. They are unable to recognize human emotional states and apply this information to decision-making and action. Addressing this lack of rapport between humans and machines is crucial for advanced intelligent HCI. With the continuous advancement of the HCI field, it is imperative to empower machines with the ability to understand human emotions. To this end, emotion recognition technology provides a viable approach to capturing human emotional states.

[0003] Emotion recognition research primarily encompasses two areas: sentiment analysis and emotion recognition. The former performs coarse-grained emotion recognition (typically binary positive vs. negative or three-category classification tasks of positive, negative, and neutral emotions), while the latter involves fine-grained analysis (typically multi-class classification of large data sets into larger sets of emotion labels, such as more than four). The number of emotion categories has been controversial in psychology. Historically, psychologists have employed two distinct approaches to modeling emotions: basic emotion theory, which labels emotions as discrete categories, and multidimensional theory, which categorizes emotions into multiple dimensions or scales. Basic emotion theory posits that humans have several basic emotions, such as happiness, sadness, fear, anger, disgust, and surprise. Other non-basic emotional states, such as fatigue, anxiety, satisfaction, confusion, and frustration, are composed of these. Each emotion category has unique internal experiences, external manifestations, and physiological patterns.

[0004] Existing emotion recognition methods can be divided into two categories: subjective methods (such as those based on body movements, facial expressions, or speech) and objective methods (such as those based on different types of physiological signals). Compared to subjective methods, objective methods based on physiological signals, such as electroencephalogram (EEG), electrocardiogram (ECG), and electromyogram (EMG), can eliminate differences between subjects and eliminate the influence of subjective consciousness. Therefore, they have been widely used in emotion recognition in recent years.

[0005] Over the past few years, deep learning has achieved tremendous success in many fields, such as signal processing, artificial intelligence, and emotion detection, leveraging state-of-the-art systems. Convolutional neural networks (CNNs), recurrent neural networks (RNNs), and graph neural networks (GNNs) are common methods used in deep learning. Applying deep learning to emotion recognition is currently a major research area.

[0006] Li et al. proposed a new EEG emotion recognition method inspired by neuroscience, which involves the brain's response to different emotions. The method uses the R2G-STNN method, which consists of a spatiotemporal neural network model with a regional to global hierarchical feature learning process to learn discriminative spatiotemporal EEG features.

[0007] Although a variety of EEG emotion recognition methods have been proposed in recent years, several key issues remain to be further explored in order to further improve emotion recognition performance. The first is how to leverage EEG signals from different brain regions to improve emotion recognition. Recent neuroscience research has shown that human emotions are closely associated with various cerebral cortical regions, such as the orbitofrontal cortex, ventromedial prefrontal cortex, and amygdala. Therefore, EEG signals from different brain regions contribute differently to emotion recognition. The second is how to leverage peripheral neural networks to supplement brain neural networks. Peripheral neural networks, including ECG and EMG, also contain rich emotional information. Mining and utilizing this information to create higher-dimensional neural networks can help improve emotion recognition. The third is how to leverage information about a person's social environment to enhance emotion recognition capabilities. It is well known that the social environment in which people live also influences their emotions, and people in the same environment often experience the same emotions. How to link this social information with emotions is also one of the issues that this invention aims to address. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to provide an emotion recognition system based on multimodal physiological and social information. The system can use multimodal physiological signals to model brain neural networks and peripheral neural networks, explore the representation effect of brain neural networks and peripheral neural networks on emotions, and at the same time use social information to model social network models, and explore the interaction between emotions and social environment.

[0009] An emotion recognition system based on multimodal physiological and social information, including a brain network building module, a neural system network building module, a social network building module, and an emotion recognition module:

[0010] The brain network building module is used to:

[0011] The EEG is divided according to the location of the collection, and the EEG signals of different brain regions are input into the bidirectional long short-term memory network BiLSTM of different scales to learn the spatiotemporal features within different brain regions and build a local brain network model. The spatial features learned from different brain regions are input into the GNN network model to learn the spatial features between brain regions and obtain the feature tensor X corresponding to each brain region. fs The brain network building module introduces a regional attention layer to learn a set of X-related information based on the different roles played by different brain regions in EEG emotion recognition. fs The weight matrix W fs , used to represent the contribution of different brain regions; then the feature tensor X of each brain region whose spatial and temporal features have been extracted is fs Concatenate into feature tensor X r , and X r With W fs Multiply to get X after introducing weights r 'Input into the global GNN_global to get the feature tensor X gCNS , used to characterize the temporal and spatial characteristics within and between different brain regions, reflect the information contained in the central nervous system, and realize the construction of brain network models from local to global;

[0012] The neural network construction module is used to input the pre-processed physiological signals of the peripheral neural network into different BiLSTMs to extract temporal features, and then input them into different GNNs to extract spatial features, and then compare the obtained spatiotemporal features with the feature tensor X obtained by the brain network construction module. gCNS Splicing is performed to obtain the feature X containing the time and space information of the central nervous system and the peripheral nervous system p ;

[0013] The social network building module is used to: pAs node features, we use the social information of different people as the basis for the nodes’ connection edges and input them into the hypergraph neural network, thus obtaining the HGNN’s association matrix H. p Perform hypergraph convolution operation to obtain feature vector X e ;

[0014] The emotion recognition module is used to: transform the final feature tensor X e The input is sent to the emotion recognition module, which uses a fully connected layer and activation function softmax to obtain the final emotion recognition result.

[0015] Preferably, the brain network construction module divides the EEG signal into 9 brain regions.

[0016] Preferably, the social information includes gender, age, and weight information.

[0017] Preferably, the feature vector X is obtained e The methods include:

[0018] The social information is represented by a one-dimensional vector X sd Indicates; the one-dimensional vector X sd As the attribute of the point in HGNN, the feature tensor X p As the feature of the midpoint in HGNN, the hyperedge is generated by using the attribute, and the X sd The points with the same corresponding bit values ​​are connected by hyperedges, thus obtaining the HGNN association matrix H, and the feature tensor X p Perform hypergraph convolution operation, and finally obtain the feature tensor X e .

[0019] Preferably, for the feature tensor X p Perform multiple hypergraph convolution operations continuously, and the final feature tensor X e .

[0020] Preferably, the obtained feature tensor can also repeat the above operation. In this embodiment, three hypergraph convolution operations are performed. The specific formula of the hypergraph convolution operation is:

[0021] Among them, D v and D e Represents the diagonal matrix of vertex degree and edge degree respectively, W is the weight matrix, Θ p is a learnable parameter and σ is a nonlinear activation function.

[0022] Preferably, the emotion recognition module is also used to train the network, using cross entropy to calculate the loss function during the training process, and using Adam optimization as the optimization algorithm.

[0023] The present invention has the following beneficial effects:

[0024] 1、The present application realizes modeling from local to global central nervous system network by using EEG, obtains local brain neural network by using EEG in different brain regions, and obtains global brain neural network by using spatial information between different brain regions, so as to realize modeling from local central nervous system to global neural network.

[0025] 2、The present application realizes modeling from central nervous network to global nervous network by using other physiological signals to supplement information of central nervous network based on the construction of central nervous network, and modeling peripheral nervous network by using ECG, EMG and other signals.

[0026] 3、The present application realizes modeling from local nervous network to global nervous network, and modeling network from individual to society by using personal social information to construct social network after obtaining nervous network of human body. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A flow chart of an emotion recognition system based on multi-modal physiological and social information provided by the embodiment of the present application.

[0028] Figure 2 A data preprocessing flow chart for different physiological signals provided by the embodiment of the present application.

[0029] Figure 3 A model construction module schematic diagram for constructing central nervous network model provided by the embodiment of the present application. DETAILED DESCRIPTION

[0030] The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0031] EEG is a technology for recording electrical activity of brain cells on the surface of cerebral cortex or scalp. It captures electrical signals of neurons by placing electrodes on the scalp, and these signals reflect the activity state of brain, so that the electrical signals can be recorded and emotion can be recognized by analyzing EEG.

[0032] Similarly, ECG is also a technology for recording physiological signals, which captures electrical signals of heart by placing electrodes on the surface of skin, and these signals reflect the electrical physiological state and function of heart.

[0033] Similarly, EMG is a technology for recording electrical activity of muscle, which captures electrical signals generated by muscle cells by placing electrodes on the surface of skin or directly inserting into muscle, and these signals reflect the activity state of muscle.

[0034] This system mainly uses the deep learning network architecture to build each part of the network model. The specific network architectures used include: BiLSTM, GNN, HGNN, and the PyTorch framework is used to implement the construction of the above networks. The specific usage is described below:

[0035] BiLSTM is a special recurrent neural network that can process sequence data and maintain long-term memory. BiLSTM considers both past and future information, enabling the model to better capture the contextual relationships in sequence data. For physiological signals, BiLSTM can effectively capture the temporal characteristics of physiological signals.

[0036] GNNs are neural networks specifically designed for processing graph-structured data. Compared to traditional neural networks, GNNs can operate directly on graphs, learning representations of their nodes and edges. This system treats each channel / individual as a node, effectively learning the connections between them. For physiological signals, GNNs can effectively capture the spatial characteristics between different channels.

[0037] HGNN is a neural network specifically designed for processing hypergraph-structured data. A hypergraph is a more complex data structure than a traditional graph, allowing multiple nodes (vertices) to be simultaneously connected to the same hyperedge. This structure is very natural when representing many-to-many relationships. Compared to GNNs, it can more effectively capture higher-dimensional feature relationships and has considerable advantages in processing multimodal data. This system uses HGNN to construct the social network during the final generation. It can effectively utilize individual social information and physiological signals to learn higher-dimensional features for emotion recognition.

[0038] The emotion recognition system based on multi-modal physiological and social information of the application is divided into four parts; the first part mainly considers the comprehensive utilization of EEG information of different brain regions to realize the construction of brain network from local to global. The EEG is divided according to the position of collection (in this embodiment, the EEG of 32 channels is divided into 9 different regions), the EEG signals of different brain regions are respectively input into bidirectional long short-term memory (BiLSTM) networks of different scales to learn the space-time features in different brain regions and construct the local brain network model, and the space-time features learned from different brain regions are input into the GNN network model to learn the space features between brain regions, so as to realize the construction of brain network model from local to global. At the same time, considering that different brain regions play different roles in electroencephalogram emotion recognition, this part also introduces a regional attention layer to learn a set of weights representing the contribution of brain regions. This part comprehensively utilizes the space-time information in the brain region and the space information between the brain regions, effectively utilizes the EEG signals between different brain regions, and realizes the construction of brain network from local to global.

[0039] The second part mainly considers supplementing the emotion recognition system with physiological signals of peripheral nervous network. Similar to the first part, the physiological signals of peripheral nervous network such as ECG and EMG are preprocessed, and after preprocessing, the processed ECG and EMG signals are respectively input into different BiLSTM to extract time features, and then the obtained time features are input into GNN to obtain the peripheral nervous network model. The obtained peripheral nervous network and the brain nervous network of the first part are integrated to realize the modeling of human nervous network from local brain nervous network to global.

[0040] The third part mainly considers modeling the social network between people by using the social information of people. In this part, the individual is regarded as a node in the social network, and the social information of different people is used as the basis for connecting the edges of the nodes. The features extracted from the personal nervous system network in the second part are used as node information and input into the hypergraph neural network (HGNN) to realize the social network modeling from individual to whole group. By learning the topological structure and semantic information of the social network, a feature representation with individual attributes and social attributes is obtained, and the feature representation is used to realize good emotion recognition.

[0041] The systems described in this paper build a comprehensive emotion recognition system from different perspectives. The EEG-based neural network model focuses on capturing the temporal and spatial characteristics of EEG signals within and between different brain regions; the multimodal neural network model improves recognition accuracy by fusing information from different modalities; and the social network model based on social information further incorporates social factors to achieve more comprehensive emotion recognition.

[0042] In this embodiment, the system first obtains information from multiple modalities, including EEG, ECG, EMG, and social information, and then preprocesses the EEG, ECG, and EMG data and extracts features; then, through the network construction module, the features extracted from the data of different modalities are integrated to establish a social network model; finally, emotion recognition is performed on the emotion recognition module.

[0043] The present invention will be described in detail below with reference to the accompanying drawings and embodiments:

[0044] Figure 1 This is a flow chart of an emotion recognition system based on multimodal physiological and social information, provided by an embodiment of the present invention. The system comprises three parts, each of which includes several feature extraction modules and a network construction module, and finally the emotion recognition module performs final emotion recognition.

[0045] In the first part, the system first acquires EEG signals. This embodiment takes the DEAP dataset as an example. The DEAP dataset collects EEG and EMG signals from 32 healthy participants while they watch 40 one-minute music videos. EEG is collected in 32 channels according to the international standard 10-20 system, with an acquisition frequency of 512Hz and downsampled to 128Hz. Therefore, for each subject, the EEG is a tensor of size X1 = (40, 32, 60*128). The obtained data is then input into the feature extraction module for feature extraction. The feature extraction module is as follows: Figure 2 shown.

[0046] Figure 2This is a general feature extraction flow chart for different physiological signals provided by an embodiment of the present invention. Taking EEG signals as an example, first, a high-pass filter with a cutoff frequency of 0.5Hz and a low-pass filter with a cutoff frequency of 45Hz are used to remove high and low frequency noise in the EEG signal; then a reference electrode is selected to re-reference the data to remove the baseline drift of the signal; the independent component analysis method is used to screen and eliminate parts that do not belong to the EEG signal, such as eye movements, electrocardiogram, etc.; a suitable window size (such as 1s) is selected to divide the data segments for time analysis; finally, the segmented EEG data is band-pass filtered, and the corresponding cutoff frequency of the band-pass filter is set to obtain the differential entropy features of five different frequency bands: delta (1-4Hz), theta (4-8Hz), alpha (8-13Hz), beta (13-30Hz) and gamma (31-45Hz). For the DEAP dataset, the data size of each subject is processed from (40, 32, 60*128) to X fCNS =(40, 32, 60, 5) tensor.

[0047] Input the feature-extracted tensor into the model building module, Figure 3 Schematic diagram of a model building module for constructing a central neural network model provided by an embodiment of the present invention. This module first divides the feature tensor and divides the 32 channels into 9 brain regions. The specific division method is shown in Table 1, which is a table of channels included in different brain region divisions.

[0048] Table 1. Channels included in different brain regions

[0049]

[0050]

[0051] The different channel signals of the same brain region are input into the corresponding BiLSTM to extract the temporal features within the nine brain regions. Different BiLSTMs are labeled according to the brain region they are in. Next, taking the frontal lobe as an example, we will describe how to extract the temporal and spatial features within the brain region. The BiLSTM of the frontal lobe is labeled as BiLSTM_PF. In this embodiment of the present invention, the hidden layer of BiLSTM_PF is set to 60, so the tensor size after BiLSTM_PF is X ft =(40, 4, 120), and then input the 9 tensors after extracting the time features into the new GNN to extract the spatial features of the brain area. Also taking the frontal lobe as an example, the hidden layer of GNN_PF is set to 64, and the tensor size obtained is X fs =(40, 64), considering that different brain regions play different roles in EEG emotion recognition, this embodiment introduces the attention mechanism at the end to learn a set of X fsThe weight matrix W fs , used to represent the contribution of different brain regions, the 9 tensors X that have extracted spatial and temporal features fs Splice to size X r =(40, 9, 64) tensor, and X r With W fs Multiply to get X after introducing weights r ' is input into the global GNN_global, and the resulting central neural network model is the GNN_global with 9 vertices. The GNN_global hidden layer is set to 128, so the final size is X gCNS =(40, 128)-sized tensor, which contains the temporal and spatial characteristics within and between the nine brain regions, and can well reflect the information contained in the central nervous system.

[0052] The data acquisition in the second part is the same as that in the first part, and peripheral neural network signals such as ECG and EMG are obtained. Taking the DEAP dataset as an example, the DEAP dataset contains the EMG of two channels of the above 32 participants, which are also downsampled to 128Hz. This embodiment adopts a feature extraction module process similar to that in the first part, but does not use bandpass filtering in the last step. The differential entropy feature of EMG is directly obtained, and the size of X is obtained. fPNS The tensor (40, 2, 60) is input into the model construction module in the second part. Similar to the module in the first part, this module first uses BiLSTM_PNS to extract the temporal features of the peripheral neural network, and then uses GNN_PNS to construct the peripheral neural network, which can effectively capture the spatial features of the peripheral nervous system. The hidden layer of BiLSTM_PNS is also set to 60, and the hidden layer of GNN_PNS is set to 128, so X can also be obtained. gPNS =(40, 128) size tensor, and finally combine this tensor with the tensor X obtained in the first part gCNS Splice and get a size of X p =(40, 2, 128) tensor, which contains both the time and space information of the central nervous system and the peripheral nervous system.

[0053] The third part first obtains the social information of the participants, including gender, age, weight, and other information. For example, the DEAP dataset collected information on 32 participants, including 16 boys and 16 girls, with an age range of 19 to 37 years old and an average age of 26.9 years old. All participants were in good physical and mental condition, without any mental illness or brain damage, and were right-handed. Non-data information is digitized and arranged in a certain order as a one-dimensional vector, such as male is marked as 0, female is marked as 1; students are marked as 0, and social personnel are marked as 1. These one-dimensional vectors can represent the social information of the participants. For example, a 19-year-old male student weighing 70kg can be represented by a one-dimensional vector X. sd :(0,0,19,70). These one-dimensional vectors are used as the attributes of the HGNN midpoint, and the tensor obtained in the second part is used as the feature of the HGNN midpoint. This embodiment uses attributes to generate hyperedges. Taking the above participants as an example, the X of 32 participants is used as the feature of the HGNN midpoint. sd The points with the same corresponding bit values ​​are connected by hyperedges, thus obtaining the HGNN association matrix H, and the feature tensor X p Perform hypergraph convolution operation, the specific formula is as follows: Among them D v and D e Represents the diagonal matrix of vertex degree and edge degree respectively, W is the weight matrix, which defaults to a diagonal matrix of 1, Θ p is a learnable parameter, and σ is a nonlinear activation function. The obtained feature tensor can also repeat the above operation. In this embodiment, after three hypergraph convolution operations, the final feature tensor is a one-dimensional vector X of size 64. e .

[0054] In the emotion recognition module in the fourth part, the final feature tensor X e The input is fed into the emotion recognition module, which passes the feature tensor through a fully connected layer and a softmax activation function to produce the final emotion recognition result. The result is used to classify emotions using "0" for low arousal and "1" for high arousal.

[0055] After obtaining a complete social network, the network needs to be trained to achieve optimal results. This example divides the DEAP dataset into training, validation, and test sets using a subject-independent approach. The data in the training and validation sets contain labels of "0" or "1" as the basis for network optimization. Similar to the above, this example uses "0" to represent low arousal and "1" to represent high arousal. Five-fold cross-validation is also performed to ensure the accuracy of the results. During training, the loss function is calculated using cross-entropy, and the Adam optimization algorithm is used.

[0056] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An emotion recognition system based on multimodal physiological and social information, characterized by: Including brain network building module, neural system network building module, social network building module and emotion recognition module: The brain network building module is used to: The EEG is divided according to the location of the collection, and the EEG signals of different brain regions are input into the bidirectional long short-term memory network BiLSTM of different scales to learn the spatiotemporal features within different brain regions and build a local brain network model. The spatial features learned from different brain regions are input into the GNN network model to learn the spatial features between brain regions and obtain the feature tensor X corresponding to each brain region. fs The brain network building module introduces a regional attention layer to learn a set of X-related information based on the different roles played by different brain regions in EEG emotion recognition. fs The weight matrix W fs , used to represent the contribution of different brain regions; then the feature tensor X of each brain region whose spatial and temporal features have been extracted is fs Concatenate into feature tensor X r , and X r With W fs Multiply to get X after introducing weights r 'Input into the global GNN_global to get the feature tensor X gCNS , used to characterize the temporal and spatial characteristics within and between different brain regions, reflect the information contained in the central nervous system, and realize the construction of brain network models from local to global; The neural network construction module is used to input the pre-processed physiological signals of the peripheral neural network into different BiLSTMs to extract temporal features, and then input them into different GNNs to extract spatial features, and then compare the obtained spatiotemporal features with the feature tensor X obtained by the brain network construction module. gCNS Splicing is performed to obtain the feature X containing the time and space information of the central nervous system and the peripheral nervous system p ; The social network building module is used to: p As node features, we use the social information of different people as the basis for the nodes’ connection edges and input them into the hypergraph neural network, thus obtaining the HGNN’s association matrix H. p Perform hypergraph convolution operation to obtain feature vector X e ; The emotion recognition module is used to: transform the final feature tensor X e The input is sent to the emotion recognition module, which uses a fully connected layer and activation function softmax to obtain the final emotion recognition result.

2. The emotion recognition system based on multimodal physiological and social information according to claim 1, characterized in that: The brain network building module divides EEG signals into 9 brain regions.

3. The emotion recognition system based on multimodal physiological and social information according to claim 1, characterized in that: The social information includes gender, age, and weight information.

4. The emotion recognition system based on multimodal physiological and social information according to claim 1, characterized in that: Get the eigenvector X e The methods include: The social information is represented by a one-dimensional vector X sd Indicates; the one-dimensional vector X sd As the attribute of the point in HGNN, the feature tensor X p As the feature of the midpoint in HGNN, the hyperedge is generated by using the attribute, and the X sd The points with the same corresponding bit values ​​are connected by hyperedges, thus obtaining the HGNN association matrix H, and the feature tensor X p Perform hypergraph convolution operation, and finally obtain the feature tensor X e .

5. The emotion recognition system based on multimodal physiological and social information according to claim 4, characterized in that: For the feature tensor X p Perform multiple hypergraph convolution operations continuously, and the final feature tensor X e .

6. The emotion recognition system based on multimodal physiological and social information according to claim 4 or 5, characterized in that: The obtained feature tensor X p After three hypergraph convolution operations, the specific formula of the hypergraph convolution operation is: Among them, D v and D e Represents the diagonal matrix of vertex degree and edge degree respectively, W is the weight matrix, Θ p is a learnable parameter and σ is a nonlinear activation function.

7. The emotion recognition system based on multimodal physiological and social information according to claim 1, characterized in that: The emotion recognition module is also used to train the network. During the training process, cross entropy is used to calculate the loss function, and Adam optimization is used as the optimization algorithm.

Images