Emotion recognition method and system based on electroencephalogram signals, storage medium and electronic equipment

Abstract

The invention provides an emotion recognition method and system based on electroencephalogram signals, a storage medium and electronic equipment. The method comprises the following steps that the electroencephalogram signals and corresponding label information are obtained; the electroencephalogram signals are preprocessed based on the electrode positions of the electroencephalogram signals, and a three-dimensional brain feature map is obtained; forming an electroencephalogram signal data set based on the three-dimensional brain feature map and the label information; and training an emotion recognition depth model based on the electroencephalogram data set so as to realize emotion recognition based on the trained emotion recognition depth model. According to the emotion recognition method and system based on the electroencephalogram signals, the storage medium and the electronic equipment, accurate emotion recognition is achieved through the electroencephalogram signals on the basis of a mixed structure of the convolutional neural network and the Transform.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-computer signal analysis, and in particular to an emotion recognition method and system based on electroencephalogram signals, a storage medium and an electronic device. Background Art

[0002] Brain Computer Interface (BCI) refers to a new type of human-computer interface technology that creates a direct connection between the human or animal brain and external devices to achieve information exchange between the brain and external devices. It can be seen that the brain-computer interface is a special communication device that connects the human brain and external terminals, so it also has the structure of a general communication device.

[0003] Emotion detection is an important task to achieve comprehensive artificial intelligence. How to make external devices ready to recognize the user's emotional state is one of the important ways to define a new paradigm of human-computer interaction. Many categories of human emotional states, such as happiness, anger, surprise, fear, disgust, and sadness, can reflect people's attitudes and subconsciousness towards external stimuli, which is also the mechanism for applying it to practice. In virtual reality, games, and education, emotion recognition systems can adjust the interaction mode according to the user's emotional state to support personalized experience and learning, effectively improving user engagement and satisfaction. In biomedicine, by analyzing physiological data such as EEG signals, the system can assist doctors in more accurately diagnosing and treating diseases, improving the work efficiency of medical staff and the level of medical services. In human-computer interaction, including but not limited to virtual assistants, social robots, and intelligent customer service, accurately identifying user emotions is one of the key factors in achieving natural intelligent interaction. Therefore, the development of emotion recognition technology has important application potential in understanding and processing individual emotional states, improving human-computer interaction experience, and even in the medical field.

[0004] The earliest research on emotion recognition can be traced back to Professor Rosalind Pickard of the MIT Media Lab. In her research, she proposed a concept called "affective computing", emphasizing the key role of emotion in human-computer interaction. At present, in addition to analyzing morphological features such as facial expressions, tone of voice, and demeanor, physiological signals, especially electroencephalogram (EEG), have attracted widespread attention from researchers in emotion recognition solutions due to their objectivity, authenticity, simplicity, non-invasiveness, and effectiveness. By combining wireless communication technology and high-end chip design and integration technology, new wearable, low-cost, easy-to-use and portable devices can be created.

[0005] In recent years, with the rapid development of computer vision and natural language processing, deep learning has attracted widespread attention for processing EEG data. It has achieved good results by learning the inherent characteristics of samples with its powerful computational learning ability, training neural networks with a large number of example labels, and adjusting the weights of the connections between neurons according to the detected results. However, most structures based on convolutional neural networks (CNN) and recurrent neural networks (RNN) or their hybrid structures still do not fully utilize the ability of deep learning to automatically extract features. When extracting local features, CNN fails to fully consider spatial domain features, and usually needs to adopt strategies such as fusion spatial feature extraction methods or introduce spatial attention mechanisms. However, these additional processing methods introduce greater computational complexity to the entire model and increase the training and inference costs of the model. In addition, the Long Short Term Memory (LSTM) neural network faces the problem of gradient vanishing or exploding when processing long sequence signals, which also poses challenges to its application. Summary of the invention

[0006] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide an emotion recognition method and system based on EEG signals, a storage medium and an electronic device, which realizes accurate emotion recognition by using EEG signals based on a hybrid structure of convolutional neural network and Transformer.

[0007] In a first aspect, the present invention provides an emotion recognition method based on EEG signals, the method comprising the following steps: obtaining EEG signals and corresponding label information; preprocessing the EEG signals based on the electrode positions of the EEG signals to obtain a three-dimensional brain feature map; constructing an EEG signal data set based on the three-dimensional brain feature map and the label information; training an emotion recognition deep model based on the EEG signal data set to realize emotion recognition based on the trained emotion recognition deep model; wherein the emotion recognition deep model comprises a convolutional neural network module, a Transformer module and a classification module, the convolutional neural network module is used to extract the temporal features and spatial features of the EEG signals; the Transformer module is used to obtain a global feature sequence of the EEG signals based on the temporal features and the spatial features, and the classification module is used to obtain an emotion classification result of the EEG signals based on the global feature sequence.

[0008] In an implementation of the first aspect, obtaining label information of an electroencephalogram signal includes the following steps:

[0009] Acquiring the valence level and arousal level of the EEG signal based on a valence arousal model;

[0010] When the valence level is greater than a first preset threshold, the valence label of the EEG signal is determined to be high valence; when the valence level is less than the first preset threshold, the valence label of the EEG signal is determined to be low valence;

[0011] When the awakening level is greater than a second preset threshold, the awakening label of the EEG signal is determined to be high awakening; when the awakening level is less than the second preset threshold, the awakening label of the EEG signal is determined to be low awakening;

[0012] The valence label and the arousal label of the electroencephalogram signal are combined into the label information.

[0013] In an implementation of the first aspect, preprocessing the EEG signal based on the electrode position of the EEG signal to obtain a three-dimensional brain feature map includes the following steps:

[0014] Performing bandpass filtering on the EEG signal;

[0015] The bandpass filtered EEG signal is subjected to zero mean processing to obtain a normalized EEG signal;

[0016] For the normalized EEG signal in a specific frequency band, a one-dimensional EEG vector is obtained by using a sliding window;

[0017] Mapping the electrode position information to the matrix composed of the one-dimensional EEG vectors under each channel to obtain a two-dimensional matrix;

[0018] The three-dimensional brain feature map is constructed based on the two-dimensional matrix on multiple channels.

[0019] In an implementation of the first aspect, the convolutional neural network module includes three layers of continuous three-dimensional spatiotemporal convolution kernels, and the spatiotemporal convolution kernels include a temporal convolution kernel and a spatial convolution kernel.

[0020] In an implementation of the first aspect, the Transformer module adopts a self-attention mechanism and obtains a global feature sequence of the EEG signal through input position encoding, the temporal feature and the spatial feature.

[0021] In an implementation of the first aspect, the position code Where pos represents the position information of the EEG signal, i represents the dimension index of the position encoding, and d represents the dimension of the position encoding.

[0022] In an implementation of the first aspect, the classification module includes a connected fully connected layer and a softmax function.

[0023] In a second aspect, the present invention provides an emotion recognition system based on EEG signals, the system comprising a first acquisition module, a second acquisition module, a composition module and a training recognition module;

[0024] The first acquisition module is used to acquire EEG signals and corresponding label information;

[0025] The second acquisition module is used to pre-process the EEG signal based on the electrode position of the EEG signal to obtain a three-dimensional brain feature map;

[0026] The construction module is used to construct an EEG signal data set based on the three-dimensional brain feature map and the label information;

[0027] The training and recognition module is used to train the emotion recognition deep model based on the EEG signal data set, so as to realize emotion recognition based on the trained emotion recognition deep model; wherein the emotion recognition deep model includes a convolutional neural network module, a Transformer module and a classification module, and the convolutional neural network module is used to extract the temporal features and spatial features of the EEG signal; the Transformer module is used to obtain the global feature sequence of the EEG signal based on the temporal features and the spatial features, and the classification module is used to obtain the emotion classification result of the EEG signal based on the global feature sequence.

[0028] In a third aspect, the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for emotion recognition based on EEG signals.

[0029] In a fourth aspect, the present invention provides an electronic device, comprising: a processor and a memory;

[0030] The memory is used to store computer programs;

[0031] The processor is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned emotion recognition method based on EEG signals.

[0032] As described above, the method and system for emotion recognition based on EEG signals, the storage medium and the electronic device of the present invention have the following beneficial effects:

[0033] (1) By performing three-dimensional convolution processing on EEG signals and mapping the spatial features retained in EEG signals according to the positions of brain electrodes, the temporal and spatial domain features in EEG data can be better captured. By introducing the self-attention mechanism of Transformer, the feature sequences are fused together, effectively solving the problem of LSTM memory weakening caused by gradient vanishing or gradient exploding in long time series signals.

[0034] (2) By setting batch normalization layers, activation layers, and loss layers, the training convergence process can be accelerated and overfitting can be prevented;

[0035] (3) It effectively combines the advantages of CNN and Transformer, and improves the emotion classification and recognition effect of EEG signals compared with traditional structures;

[0036] (4) It solves the defects and limitations of traditional CNN and RNN network structures and their hybrid structure feature extraction, and is suitable for extracting multi-dimensional features of nonlinear long sequence signals such as EEG signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Shown is a flow chart of an embodiment of an emotion recognition method based on EEG signals of the present invention;

[0038] Figure 2 Shown is a schematic diagram of a framework of an embodiment of an emotion recognition method based on EEG signals of the present invention;

[0039] Figure 3 Shown is a schematic diagram of label information of the present invention in one embodiment;

[0040] Figure 4 Shown is a schematic diagram of generating a three-dimensional brain feature map in one embodiment of the present invention;

[0041] Figure 5 Shown is a schematic diagram of the structure of an embodiment of an emotion recognition system based on EEG signals of the present invention;

[0042] Figure 6 It is a schematic structural diagram of an electronic device of the present invention in one embodiment. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0045] The following embodiments of the present invention provide an emotion recognition method based on EEG signals, which can be applied to electronic devices. The electronic devices described in the present invention may include mobile phones with wireless charging function, tablet computers, laptop computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present invention do not impose any restrictions on the specific types of electronic devices.

[0046] For example, the electronic device may be a station (STAION, ST) in a WLAN with a wireless charging function, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with a wireless charging function, a computing device or other processing device, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next generation communication system, such as a mobile terminal in a 5G network, a mobile terminal in a future evolved Public Land Mobile Network (PLMN), or a mobile terminal in a future evolved Non-terrestrial Network (NTN).

[0047] For example, the electronic device can communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobilecommunication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation Systems (SBAS).

[0048] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0049] like Figure 1 and Figure 2 As shown, in one embodiment, the emotion recognition method based on EEG signals of the present invention includes steps S1 to S4.

[0050] Step S1, obtaining EEG signals and corresponding label information.

[0051] Specifically, different EEG signals are generated according to the subjects watching different videos as stimulation, and the EEG signals of different frequency bands represent that the subjects are in different emotional states. In the present invention, the average age of the subjects should be in the approximate range of 20-40 years old, and can be a normal reaction to external stimuli. Watch a given video clip as an emotional stimulus source and record the subject's emotional EEG signal. The electrode distribution of the brain electrode cap conforms to the 10-20 international standard lead, corresponding to the following 32 electrode channels (64 channels and 256 channels as needed): FP1, FP2, AF3, AF4, F7, F3, FZ, F4, F8, FC5, FC1, FC2, FC6, T7, C3, Cz, C4, T8, CP5, CP1, CP2, CP6, P7, P3, PZ, P4, P8, PO3, PO4, O1, OZ, O2.

[0052] In addition, if Figure 3 As shown, the valence level and arousal level of the EEG signal are obtained based on the valence arousal model. Wherein. Valence and arousal are divided into 9 levels from 1 to 9, reflecting the valence level and arousal degree. When the valence level is greater than the first preset threshold (such as level 5), the valence label of the EEG signal is determined to be high valence; when the valence level is less than the first preset threshold, the valence label of the EEG signal is determined to be low valence; when the arousal level is greater than the second preset threshold (such as level 5), the arousal label of the EEG signal is determined to be high arousal; when the arousal level is less than the second preset threshold, the arousal label of the EEG signal is determined to be low arousal. The valence label and arousal label of the EEG signal are combined into the label information. Therefore, the label information includes high valence and high arousal, high valence and low arousal, low valence and high arousal, and low valence and low arousal.

[0053] Step S2: pre-processing the EEG signal based on the electrode position of the EEG signal to obtain a three-dimensional brain feature map.

[0054] Specifically, preprocessing the EEG signal based on the electrode position of the EEG signal to obtain a three-dimensional brain feature map includes the following steps:

[0055] 21) performing band-pass filtering on the EEG signal.

[0056] 22) The bandpass filtered EEG signal is processed to zero mean to obtain a normalized EEG signal.

[0057] 23) For the normalized EEG signal in a specific frequency band, a one-dimensional EEG vector is obtained using a sliding window.

[0058] Among them, five frequency bands were selected, namely δ (1-3Hz), θ (4-7Hz), α (8-12Hz), β (13-30Hz) and γ (31-50Hz). Since the frequency band of δ is too low, according to relevant studies, it has a low correlation with emotional brain activity. In order to optimize the model structure, the EEG signals of the last four frequency bands were selected for processing.

[0059] The normalized EEG signal X i ={x1,x2,…,x n}Data segmentation is performed by setting a sliding window of length L, and the step size of the sliding window is ts, to obtain a series of sliding window data, namely the one-dimensional EEG vector. At this time, the EEG signal in a specific frequency band is converted into a one-dimensional vector {s1, s2, …, s n}, using 32 channels, that is, n = 32. The data format can be recorded as trials×channels×m×L, where m×L means that the EEG signal is divided into m small blocks of length L, trials means the number of experiments, and channels means the number of channels.

[0060] 24) Mapping the electrode position information to the matrix composed of the one-dimensional EEG vectors under each channel to obtain a two-dimensional matrix.

[0061] Among them, Figure 4 As shown, the electrode position relationship is mapped to the EEG signal to construct an H×W matrix. The positions of unused electrodes are supplemented with 0, so that the EEG signal and the electrode position information are combined through the matrix shown, and the spatial characteristics of the EEG signal are retained. The sliding window is used to form a matrix (t,n) composed of the one-dimensional EEG vector under each channel collected at one time, where t represents the time window interval and n represents the number of features extracted in the time domain and frequency domain.

[0062] 25) Constructing the three-dimensional brain feature map based on the two-dimensional matrix on multiple channels.

[0063] In this case, multiple two-dimensional matrices are stacked to form a three-dimensional tensor (t,n,k), where k represents an EEG signal containing k channels or leads, that is, each channel generates a two-dimensional feature matrix. The specific form of the three-dimensional tensor is determined by the batch size, the number of features of the EEG signal, and the number of features at each time step, (t,n,k)∈R B×H×W Therefore, the data format is recorded as trials×channels×m×B×H×W.

[0064] Step S3: constructing an EEG signal data set based on the three-dimensional brain feature map and the label information.

[0065] Specifically, the EEG signal data set is divided into a training set, a validation set and a test set for model training. Specifically, 60% of the data set is used as a training set, 20% as a validation set, and 20% as a test set.

[0066] Step S4: training an emotion recognition deep model based on the EEG signal data set to realize emotion recognition based on the trained emotion recognition deep model; wherein the emotion recognition deep model includes a convolutional neural network module, a Transformer module and a classification module, and the convolutional neural network module is used to extract the temporal features and spatial features of the EEG signal; the Transformer module is used to obtain a global feature sequence of the EEG signal based on the temporal features and the spatial features, and the classification module is used to obtain an emotion classification result of the EEG signal based on the global feature sequence.

[0067] Specifically, the convolutional neural network module includes three layers of continuous three-dimensional spatiotemporal convolution kernels, and the spatiotemporal convolution kernels include time convolution kernels and space convolution kernels, which are respectively used to realize the extraction of time features and spatial features of EEG signals. The input of the convolutional neural network module is the three-dimensional brain feature map, and the output is a two-dimensional tensor form suitable for flattening according to the input needs of the Transformer module. In the invention, the input data shape of the convolutional neural network module is 4×9×9, the size of the first layer convolution kernel is 4×4, the number is 32, the size of the second layer convolution kernel is 1×1, the number is 64, the size of the third layer convolution kernel is 3×3, the number is 128, and finally the convolution output is flattened by the flattern layer.

[0068] The Transformer module relies on the self-attention mechanism, but it does not contain position information itself, so it needs to input position encoding to preserve the order information of time steps. The position encoding is determined by sine or cosine functions. In one embodiment, the position encoding Where pos represents the position information of the EEG signal, i represents the dimension index of the position encoding, and d represents the dimension of the position encoding.

[0069] The position encoding matrix is ​​used as the input of the multi-head self-attention layer, and the input matrix It can be represented as a nested feature of a sequence, with each position having a corresponding feature vector. Multiply them with the preset linear weight matrix respectively, then divide each Q, K, V vector matrix into multiple heads, and then calculate the attention score and attention weight of each head respectively to obtain the attention score matrix and weight matrix. The multi-head attention output matrix is ​​given by the weighted sum of the attention weight and the V vector matrix, that is,

[0070] In the present invention, the encoder layer of the Transformer module uses a multi-head self-attention mechanism, and the model can perform self-attention calculations on different subspaces to better capture global features. The feedforward neural network consists of two layers of fully connected networks for nonlinear transformation features. Each sublayer is followed by a residual connection and a normalization layer to help the model train deeper networks.

[0071] Finally, the input of the Transformer module passes through the classification module to obtain the emotion recognition result. The classification module includes a connected full connection layer and a softmax function. The classification process is as follows:

[0072] MHSA(Y)=FC(Concat[A1,A2,…,A h ]·W o )

[0073] Where W o It is the weight matrix in the multi-head attention layer. During model training, after random initialization and back-propagation calculation, the weight matrix will be continuously updated and corrected to minimize the predicted emotion label and original label through the network's loss function.

[0074] The loss function is as follows:

[0075]

[0076] Among them, c represents the number of label categories, n represents the number of samples, y represents the true label, and a represents the model prediction probability of the category after Softmax activation. The Adam algorithm is used to dynamically adjust the learning rate of each parameter to complete the classification and recognition of the labeled EEG signals.

[0077] Preferably, the emotion recognition deep model training of the present invention is performed on an NVIDIA RTX 4060Ti GPU, the hybrid model is implemented using the PyTorch framework, Adam is selected as the optimizer, the learning rate is set to 10-5, and 10-fold cross-validation training results are used, the batch size is set to 64, the epoch is selected to be 200, and the dropout is set to 0.7.

[0078] Therefore, the emotion recognition method based on EEG signals of the present invention adopts the hybrid structure of CNN-Transformer, and effectively extracts the local key features of EEG signals through the local perception and parameter sharing characteristics of CNN, while Transformer introduces a multi-head attention mechanism to construct the spatiotemporal dependency of EEG activity information from a global perspective, so as to better capture the spatiotemporal correlation in EEG signals. At the same time, according to the mapping relationship of the brain electrode position, the spatial characteristics of EEG are effectively retained, the EEG signal is better modeled, and the effect of feature extraction is improved.

[0079] The protection scope of the method for emotion recognition based on EEG signals described in the embodiment of the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.

[0080] An embodiment of the present invention also provides an emotion recognition system based on EEG signals, which can implement the emotion recognition method based on EEG signals described in the present invention. However, the implementation device of the emotion recognition system based on EEG signals described in the present invention includes but is not limited to the structure of the emotion recognition system based on EEG signals listed in this embodiment. All structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.

[0081] like Figure 5 As shown, in one embodiment, the emotion recognition system based on EEG signals of the present invention includes a first acquisition module 51 , a second acquisition module 52 , a construction module 53 and a training recognition module 54 .

[0082] The first acquisition module 51 is used to acquire EEG signals and corresponding label information.

[0083] The second acquisition module 52 is connected to the first acquisition module 51, and is used to pre-process the EEG signal based on the electrode position of the EEG signal to obtain a three-dimensional brain feature map.

[0084] The constructing module 53 is connected to the second acquiring module 52 and is used to construct an EEG signal data set based on the three-dimensional brain feature map and the label information.

[0085] The training recognition module 54 is connected to the composition module 53, and is used to train the emotion recognition deep model based on the EEG signal data set, so as to realize emotion recognition based on the trained emotion recognition deep model; wherein the emotion recognition deep model includes a convolutional neural network module, a Transformer module and a classification module, and the convolutional neural network module is used to extract the temporal features and spatial features of the EEG signal; the Transformer module is used to obtain the global feature sequence of the EEG signal based on the temporal features and the spatial features, and the classification module is used to obtain the emotion classification result of the EEG signal based on the global feature sequence.

[0086] Among them, the structures and principles of the first acquisition module 51, the second acquisition module 52, the composition module 53 and the training recognition module 54 correspond one by one to the steps in the above-mentioned emotion recognition method based on EEG signals, so they are not repeated here.

[0087] In the several embodiments provided by the present invention, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules or units, which can be electrical, mechanical or other forms.

[0088] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0089] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0090] The embodiment of the present invention also provides a computer-readable storage medium. A person skilled in the art can understand that all or part of the steps in the method for emotion recognition based on EEG signals of the above embodiment can be completed by instructing a processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0091] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory.

[0092] The memory is used to store computer programs.

[0093] The memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0094] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned emotion recognition method based on EEG signals.

[0095] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0096] like Figure 6 As shown, the electronic device of the present invention is in the form of a general computing device. The components of the electronic device may include but are not limited to: one or more processors or processing units 61, a memory 62, and a bus 63 connecting different system components (including the memory 62 and the processing unit 61).

[0097] Bus 63 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0098] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0099] The memory 62 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 621 and / or cache memory 622. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 623 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 63 via one or more data medium interfaces. The memory 62 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0100] A program / utility 624 having a set (at least one) of program modules 6241 may be stored, for example, in the memory 62, such program modules 6241 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 6241 generally perform the functions and / or methods of the embodiments described herein.

[0101] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 64. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 65. Figure 6 As shown, the network adapter 65 communicates with other modules of the electronic device via the bus 63. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0102] In summary, the method and system for emotion recognition based on EEG signals, the storage medium and the electronic device of the present invention perform three-dimensional convolution processing on EEG signals, and map the spatial features retained in the EEG signals according to the positions of brain electrodes, so as to better capture the time domain and spatial domain features in the EEG data, and integrate the feature sequences by introducing the self-attention mechanism of Transformer, which effectively solves the problem of memory weakening caused by gradient vanishing or gradient explosion that still exists in LSTM in long time series signals; by setting batch normalization layer, activation layer and loss layer, the training convergence process can be accelerated and overfitting can be prevented; the advantages of CNN and Transformer are effectively combined, and the emotion classification and recognition effect of EEG signals is improved compared with the traditional structure; the defects and limitations of the traditional CNN and RNN network structures and their hybrid structure feature extraction are solved, and it is suitable for extracting multi-dimensional features of nonlinear long sequence signals such as EEG signals. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0103] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An emotion recognition method based on EEG signals, characterized in that: The method comprises the following steps: Obtain EEG signals and corresponding label information; Preprocessing the EEG signal based on the electrode position of the EEG signal to obtain a three-dimensional brain feature map; Constructing an EEG signal dataset based on the three-dimensional brain feature map and the label information; An emotion recognition deep model is trained based on the EEG signal data set to realize emotion recognition based on the trained emotion recognition deep model; wherein the emotion recognition deep model includes a convolutional neural network module, a Transformer module and a classification module, and the convolutional neural network module is used to extract the temporal features and spatial features of the EEG signal; the Transformer module is used to obtain the global feature sequence of the EEG signal based on the temporal features and the spatial features, and the classification module is used to obtain the emotion classification result of the EEG signal based on the global feature sequence.

2. The method for emotion recognition based on EEG signals according to claim 1, characterized in that: Obtaining label information of EEG signals includes the following steps: Acquiring the valence level and arousal level of the EEG signal based on a valence arousal model; When the valence level is greater than a first preset threshold, the valence label of the EEG signal is determined to be high valence; when the valence level is less than the first preset threshold, the valence label of the EEG signal is determined to be low valence; When the awakening level is greater than a second preset threshold, the awakening label of the EEG signal is determined to be high awakening; when the awakening level is less than the second preset threshold, the awakening label of the EEG signal is determined to be low awakening; The valence label and the arousal label of the electroencephalogram signal are combined into the label information.

3. The method for emotion recognition based on EEG signals according to claim 1, characterized in that: Preprocessing the EEG signal based on the electrode position of the EEG signal to obtain a three-dimensional brain feature map includes the following steps: Performing bandpass filtering on the EEG signal; The bandpass filtered EEG signal is subjected to zero mean processing to obtain a normalized EEG signal; For the normalized EEG signal in a specific frequency band, a one-dimensional EEG vector is obtained by using a sliding window; Mapping the electrode position information to the matrix composed of the one-dimensional EEG vectors under each channel to obtain a two-dimensional matrix; The three-dimensional brain feature map is constructed based on the two-dimensional matrix on multiple channels.

4. The method for emotion recognition based on EEG signals according to claim 1, characterized in that: The convolutional neural network module includes three layers of continuous three-dimensional spatiotemporal convolution kernels, and the spatiotemporal convolution kernels include time convolution kernels and space convolution kernels.

5. The method for emotion recognition based on EEG signals according to claim 1, characterized in that: The Transformer module adopts a self-attention mechanism and obtains a global feature sequence of the EEG signal through input position encoding, the temporal feature and the spatial feature.

6. The method for emotion recognition based on EEG signals according to claim 5, characterized in that: The position code Where pos represents the position information of the EEG signal, i represents the dimension index of the position encoding, and d represents the dimension of the position encoding.

7. The method for emotion recognition based on EEG signals according to claim 1, characterized in that: The classification module includes a connected fully connected layer and a softmax function.

8. An emotion recognition system based on EEG signals, characterized in that: The system includes a first acquisition module, a second acquisition module, a composition module and a training recognition module; The first acquisition module is used to acquire EEG signals and corresponding label information; The second acquisition module is used to pre-process the EEG signal based on the electrode position of the EEG signal to obtain a three-dimensional brain feature map; The construction module is used to construct an EEG signal data set based on the three-dimensional brain feature map and the label information; The training recognition module is used to train the emotion recognition deep model based on the EEG signal data set, so as to realize emotion recognition based on the trained emotion recognition deep model; wherein the emotion recognition deep model includes a convolutional neural network module, a Transformer module and a classification module, and the convolutional neural network module is used to extract the temporal features and spatial features of the EEG signal; The Transformer module is used to obtain a global feature sequence of the EEG signal based on the temporal feature and the spatial feature, and the classification module is used to obtain an emotion classification result of the EEG signal based on the global feature sequence.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for emotion recognition based on electroencephalogram signals described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the electronic device executes the emotion recognition method based on electroencephalogram signals according to any one of claims 1 to 7.

Images