Emotional eeg feature representation method, system, electronic device and storage medium

Abstract

Embodiments of the present application provide a kind of emotional eeg feature representation method, system, electronic equipment and storage medium.The method comprises: obtaining the eeg signal during the subject watches emotional stimulus material, extracts the eeg differential entropy feature of eeg signal;The eeg differential entropy feature is input to the graph Transformer network for emotion recognition;In graph Transformer network, the eeg differential entropy feature is modeled by graph convolution network unit, determine the spatial topology structure of eeg differential entropy feature under adjacency matrix, obtain the eeg feature with spatial information;The eeg feature with spatial information is processed by the self-attention layer and layer normalization about time sequence by Transformer unit, obtain the eeg feature with spatial information and time information.Embodiments of the present application utilize the structure of Transformer to capture the time information of eeg signal, reduce complexity and computational cost by avoiding recursion and overall ingestion once long sequential input, obtain more accurate emotion recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of emotion recognition, and in particular to an emotion electroencephalogram feature representation method, system, electronic device and storage medium. BACKGROUND

[0002] Emotion recognition is a prominent milestone in realizing artificial intelligence emotion intelligence. It is the basis for a wide range of potential applications in daily life, such as medical diagnosis, intelligent education, entertainment, etc. Many physiological signals are adopted as inputs to measure emotions, including language, facial expressions, eye movements, and electroencephalogram signals. Among all these ways, emotion recognition based on non-invasive electroencephalogram has gained increasing attention due to the evidence from cognitive neuroscience that the brain is effective in emotional processing. The prior art usually uses deep learning methods such as convolutional neural networks, recurrent neural networks, and long short-term memory networks for emotion recognition based on electroencephalogram.

[0003] In the process of implementing the present application, the inventors found that at least the following problems exist in the related art:

[0004] Although the deep learning methods such as convolutional neural networks, recurrent neural networks, and long short-term memory networks have achieved good results, they lack parallelization and are not efficient. These technologies are all dedicated to time-domain research, and the spatial information of electroencephalogram signal transmission has not been fully utilized. Some studies apply CNN to extract spatial connections. However, CNN can only capture local information based on Euclidean distance, and the performance is not ideal. SUMMARY

[0005] To at least solve the problem that the prior art cannot fully utilize emotion electroencephalogram information for emotion recognition.

[0006] In a first aspect, an embodiment of the present application provides an emotion electroencephalogram feature representation method, comprising:

[0007] obtaining an electroencephalogram signal of a subject during viewing of an emotional stimulus material, and extracting an electroencephalogram differential entropy feature of the electroencephalogram signal;

[0008] inputting the electroencephalogram differential entropy feature into a graph Transformer network for emotion recognition, wherein the graph Transformer network comprises a graph convolution network unit and a Transformer unit;

[0009] In the graph Transformer network, the electroencephalogram differential entropy feature is modeled by the graph convolution network unit to determine the spatial topology structure of the electroencephalogram differential entropy feature under an adjacency matrix, and an electroencephalogram feature with spatial information is obtained;

[0010] The brain electrical feature with spatial information is subjected to self-attention layer and layer normalization processing on a time sequence by the Transformer unit, to obtain the brain electrical feature with spatial information and time information for emotion recognition.

[0011] In a second aspect, an embodiment of the present application provides an emotion brain electrical feature representation system, comprising:

[0012] The feature determination program module is configured to acquire brain electrical signals of a subject during watching of emotional stimulus materials, and extract brain electrical differential entropy features of the brain electrical signals.

[0013] The feature input program module is configured to input the brain electrical differential entropy features into a graph Transformer network for emotion recognition, wherein the graph Transformer network comprises a graph convolution network unit and a Transformer unit.

[0014] The spatial information determination program module is configured to model the brain electrical differential entropy features by the graph convolution network unit in the graph Transformer network, to determine a spatial topology structure of the brain electrical differential entropy features under an adjacency matrix, and to obtain brain electrical features with spatial information.

[0015] The feature determination program module is configured to perform self-attention layer and layer normalization processing on a time sequence on the brain electrical features with spatial information by the Transformer unit, to obtain the brain electrical features with spatial information and time information for emotion recognition.

[0016] In a third aspect, an electronic device is provided, comprising at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the emotion brain electrical feature representation method of any embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps of the emotion brain electrical feature representation method of any embodiment of the present application.

[0018] The method of the embodiment of the present application has the beneficial effects that the Transformer structure is used to capture the time information of the brain electrical signals, and the complexity and the computation cost are reduced by avoiding recursion and overall ingestion of a long sequential input. Further, the graph convolution network is combined with the Transformer, the potential in time sequence classification is considered, and a more accurate emotion recognition result is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0020] Figure 1 is a flow chart of an emotional electroencephalogram feature representation method provided by an embodiment of the present application;

[0021] Figure 2 is an EmoGT model structure schematic diagram of an emotional electroencephalogram feature representation method provided by an embodiment of the present application;

[0022] Figure 3 is an EmoGTs model structure schematic diagram of an emotional electroencephalogram feature representation method provided by an embodiment of the present application;

[0023] Figure 4 is a cross-modal attention structure schematic diagram of an emotional electroencephalogram feature representation method provided by an embodiment of the present application;

[0024] Figure 5 is a performance schematic diagram of the present method compared with a plurality of prior art methods on three data sets of an emotional electroencephalogram feature representation method provided by an embodiment of the present application;

[0025] Figure 6 is a confusion matrix of EmoGT on three data sets of an emotional electroencephalogram feature representation method provided by an embodiment of the present application (from left to right, SEED, SEED-IV, SEED-V, the vertical axis represents the true label; the horizontal axis represents the predicted label);

[0026] Figure 7 is a schematic diagram of the first 10 connections in the learned adjacency matrix of an emotional electroencephalogram feature representation method provided by an embodiment of the present application (the darker the line, the higher the edge weight, and the tighter the connection between brain regions);

[0027] Figure 8 is a structure schematic diagram of an emotional electroencephalogram feature representation system provided by an embodiment of the present application;

[0028] Figure 9 is a structure schematic diagram of an electronic device for emotional electroencephalogram feature representation provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0030] As Figure 1 Fig. 1 shows a flowchart of a method for representing emotional electroencephalogram features according to an embodiment of the present application, including the following steps:

[0031] S11: obtaining electroencephalogram signals of a subject during watching emotional stimulus materials, and extracting electroencephalogram differential entropy features of the electroencephalogram signals;

[0032] S12: inputting the electroencephalogram differential entropy features into a graph Transformer network for emotion recognition, wherein the graph Transformer network includes a graph convolution network unit and a Transformer unit;

[0033] S13: modeling the electroencephalogram differential entropy features by the graph convolution network unit in the graph Transformer network, determining a spatial topology structure of the electroencephalogram differential entropy features under an adjacency matrix, and obtaining electroencephalogram features with spatial information;

[0034] S14: performing self-attention layers and layer normalization processing on the electroencephalogram features with spatial information by the Transformer unit with respect to time series, and obtaining electroencephalogram features with spatial information and time information for emotion recognition.

[0035] In the present embodiment, in order to solve the problem of parallelization, the present method uses a Transformer structure to capture the time information of the electroencephalogram signals, and reduces complexity and computational cost by avoiding recursion and overall ingestion of long sequential input. Although the above steps can obtain time information, how to simultaneously extract spatial and temporal information has not been fully explored.

[0036] The present method proposes an elastic graph Transformer framework, which combines graph convolution networks with Transformers, considers its potential in time series classification, and fully utilizes the spatial and temporal information in ER (Emotion recognition, expression recognition). The elastic graph Transformer network can be used as a separate component for multi-modal emotion, and the present method designs two special structures, which increases its robustness in real-world applications.

[0037] For step S11, the method is EmoGT (Graph Transformer network for emotion recognition) designed for electroencephalogram data.

[0038] In order to identify the emotions of the subjects, the subjects are shown emotional stimulus materials, for example, a display is placed in front of the subjects, and horror movie clips, comedy clips, oil painting appreciation clips, etc. are played to the subjects through the display. At the same time, the subjects wear electroencephalogram acquisition equipment, and the electroencephalogram acquisition equipment acquires the electroencephalogram signals of the subjects during the viewing of the emotional stimulus materials.

[0039] As an implementation manner, after obtaining the electroencephalogram signals, the method further includes: performing baseline correction on the electroencephalogram signals, performing filtering processing on the baseline corrected electroencephalogram signals, and performing time window segmentation to obtain preprocessed electroencephalogram signals.

[0040] In the embodiment, the collected electroencephalogram data is preprocessed by baseline correction, artifact removal, etc., the electroencephalogram signals are filtered, and the electroencephalogram signals are divided into C frequency bands through a filter. Then the electroencephalogram signals are segmented according to T-second non-overlapping time windows, and each sample obtained in this way can be composed of N electroencephalogram segments of C frequency bands with a time length of t seconds. The feature extraction of the electroencephalogram includes: dividing the electroencephalogram signals into C frequency bands through short-time Fourier transform, extracting the electroencephalogram differential entropy frequency domain features on each frequency band, and denoted as X=(X1, X2,... X T )∈R T×N×C .

[0041] For step S12, the structure of the EmoGT designed for electroencephalogram data is as shown in Figure 2 It consists of two basic units: GCN (graph convolutional networks) unit and Transformer unit, and the electroencephalogram differential entropy frequency domain features are input into the EmoGT.

[0042] For step S13, in the GCN graph Transformer network, the electrodes on the brain can be modeled, and the EEG (Electroencephalogram) signal is constructed as a graph G={V, E}, where V represents the vertex set, E represents the edge set, and represents the adjacency matrix which means the connection between the EEG channels, and N is the number of EEG channels, that is, N=|V|. The input EEG features are represented by where T is the number of samples in the time series, and C is the dimension of each channel.

[0043] The formula of GCN modeling is as follows:

[0044]

[0045] wherein, I is a unit matrix, denotes a diagonal degree matrix, and H l denotes the input feature of the l-th layer, while W l is a weight matrix, The output of the l-th GCN layer is It should be noted that H 0 = X.

[0046] The adjacency matrix A is crucial for learning graph representation, which describes the topology of the EEG signal. The graph Transformer network model used in this method learns the spatial topology from the data, so each A ij shows the connection weight between channel i and channel j. In addition, A is defined as a symmetric matrix to avoid overfitting. The same shared adjacency matrix is used for all GCN layers.

[0047] After the GCN layer, layer normalization is applied, ReLU is selected as the activation function, and then dropout layer. Through the above graph Transformer network processing, the electroencephalogram features with spatial information are obtained.

[0048] For step S14: The Transformer unit has achieved great success in the fields of natural language processing, computer vision, etc. Before the Transformer unit, the output of the GCN unit H-l-g∈RT×N×Cout is reshaped as input feature is converted into query Q i , key K i and value V i :

[0049]

[0050] wherein, and Self-attention calculation is:

[0051]

[0052] This method uses h heads of self-attention, each of which can be used as head i = attention (Q i , K i , Vi ) is denoted. The output of the multi-head attention is:

[0053]

[0054] wherein, and

[0055] The same as the GCN unit, there is a layer normalization after the self-attention layer, followed by a ReLU and a dropout layer.

[0056] wherein, the input emotion category token is also shown in the figure, which is used to better classify the model after training in training. By adding a category token in the input. It is learnable and has the same shape as the input EEG features. At the end of all EmoGT blocks, a linear layer and a softmax layer are applied to the class token for classification. Since the input EEG features are a continuous time series, a learnable position embedding is added to the input sequence to enable the sequence to carry time information.

[0057] After processing by the above-mentioned Transformer unit, the EEG features with parallelized time information and spatial information are obtained.

[0058] As an implementation, the graph Transformer network for emotion recognition further comprises: a linear layer and a softmax layer, wherein the linear layer and the softmax layer are used to determine the emotion category of the subject from the EEG features with spatial information and time information, or the EEG emotion fusion features with spatial information and time information, wherein the emotion category includes: negative, neutral, and positive.

[0059] In this embodiment, the EEG features with time information and spatial information obtained by the above-mentioned steps can perform more accurate emotion recognition.

[0060] As can be seen from this embodiment, the method uses the Transformer structure to capture the time information of the EEG signal, and reduces the complexity and computational cost by avoiding recursion and overall ingestion of a long sequential input. Further, the graph convolution network is combined with the Transformer, considering its potential in time series classification, to obtain more accurate emotion recognition results.

[0061] As an implementation, the method further comprises:

[0062] obtaining an eye movement signal of the subject during viewing of the emotion stimulus material, and extracting eye movement features of the eye movement signal;

[0063] input the electroencephalogram differential entropy features and the eye movement features into a graph Transformer network for emotion recognition, wherein the graph Transformer network comprises an electroencephalogram graph convolution network unit, an eye movement graph convolution network unit, and a cross-modal Transformer unit;

[0064] In the graph Transformer network, the electroencephalogram differential entropy features are modeled by the electroencephalogram graph convolution network unit, the spatial topological structure of the electroencephalogram differential entropy features under an adjacency matrix is determined, and electroencephalogram features with spatial information are obtained.

[0065] The eye movement features are modeled by the eye movement graph convolution network unit, the spatial topological structure of the eye movement features under an adjacency matrix is determined, and eye movement features with spatial information are obtained.

[0066] The electroencephalogram features with spatial information and the eye movement features with spatial information are cross-modally fused by the cross-modal Transformer unit, and electroencephalogram emotion fusion features with spatial information and temporal information for emotion recognition are obtained.

[0067] In the present embodiment, a second structure of the method is designed, and EmoGTs are designed for electroencephalogram data and eye movement data.

[0068] The eye movement signal of the subject is collected while collecting the electroencephalogram signal of the subject watching the emotional stimulus material.

[0069] The feature extraction of the eye movement includes: dividing the pupil diameter into 4 frequency bands through short-time Fourier transform, and extracting the differential entropy features of the pupil diameters of the left and right eyes. The mean, standard deviation, frequency, and other statistical features of the pupil diameter blink duration, saccade duration, and fixation duration are calculated.

[0070] After obtaining the eye movement features, the same preprocessing is performed: the collected eye movement data is subjected to pupil diameter illumination influence removal, the missing pupil diameters are completed using linear interpolation, and then principal component analysis is used to filter out the principal components.

[0071] The electroencephalogram differential entropy frequency domain features and the eye movement features are input into the EmoGTs. Compared with the EmoGT, the EmoGTs has a second GCN unit for processing the eye movement features, and the Transformer unit is cross-modal, and the specific structure is as shown in Figure 3 .

[0072] In addition to the electroencephalogram, another modality used by the present method is eye movement. Similar to the EEG, the input eye movement features can be represented as where T is the number of samples in the time series that is the same as the EEG sequence, and N' is the dimension of the eye movement features. The eye movements are also modeled as graphs, as there is correlation between them. This method proposes a cross-modal Transformer unit to fuse the modalities so that the model can leverage the complementary characteristics of different modalities. As shown in Figure 4 Figure 2, the core of the cross-modal Transformer unit is that the self-attention layer is replaced by a cross-modal attention layer.

[0073] The cross-modal attention layer considers two modalities, a and b (i.e., the electroencephalogram modality and the eye movement modality), and the input to the cross-modal attention layer is represented as and The query, key, and value can be:

[0074]

[0075] where and The cross-modal attention from a to b is defined as:

[0076]

[0077] Similarly, the cross-modal attention from b to a can be computed by CMA β:α (H β , H α ). Different modalities interact with each other in this way to learn complementary information. Note that this method also uses multi-head attention in the cross-modal Transformer unit.

[0078] Attention fusion This method uses O α and O β to represent the class labels of CMA α：β and CMA β：α , which are the outputs of modalities a and b after all EmoGTs blocks, where Transform O α and O β to the same dimension:

[0079]

[0080] where, and Then the attention weights m α and m β are calculated by the following formula:

[0081]

[0082]

[0083] where and <, > denote dot product. Thus, the fused features are extracted by:

[0084]

[0085] In this way, the EEG emotion fusion features with spatial information and temporal information for emotion recognition are obtained. Similarly, the EEG emotion fusion features are recognized by using a linear layer and a softmax layer.

[0086] As an implementation form, the graph Transformer network for emotion recognition is obtained by training an EEG emotion training set, wherein the EEG emotion training set comprises EEG training data, eye movement training data, and a reference emotion category.

[0087] The training process of the graph Transformer network for emotion recognition comprises:

[0088] The EEG emotion training set is input into the graph Transformer network for emotion recognition, and EEG emotion features with spatial information and temporal information for emotion recognition are output.

[0089] The EEG emotion features are input into a linear layer and a softmax layer to obtain an estimated emotion category, and the graph Transformer network for emotion recognition is trained based on an error between the estimated emotion category and the reference emotion category.

[0090] In this embodiment, the structure of Figure 2 In the input, a cls token (i.e., an emotion category) is additionally added, and a predicted emotion category is obtained through the processing of the model. A loss function determined based on the error between the predicted emotion category and the reference emotion category is used to train the graph Transformer network until the predicted emotion category approaches the reference emotion category.

[0091] Experiments are performed to illustrate the method. Several models of the method are tested on three public datasets (SEED, SEED-IV, and SEED-V) for emotion recognition to thoroughly verify their performance. For EEG, differential entropy is extracted from 5 frequency bands in each sample in a non-overlapping 1s time window. For eye movement data, multiple features (such as pupil diameter, dispersion, and gaze duration) are extracted. The input features are obtained by a sliding size T=5 overlapping window to make the results more comparable. The batch size is 32, the dropout rate is 0.5, the number of blocks is 4, and the learning rate is in the range of 3e-5 to 1e-3. The number of heads h and the embedding dimension C outRespectively from {2, 4} and {16, 32, 64} adjustments. Cross-entropy is chosen as the loss function, and Adam is used to optimize the parameters. For data division, the first 9 experiments in SEED are used as training data, and the remaining 6 experiments are used as test data. As for SEED-IV, the first 16 experiments are training data, and the remaining 8 experiments are test data. However, for SEED-V, 3-fold cross-validation is adopted.

[0092] For experimental results, comparison with single-modal methods, in order to verify the efficiency of the method EmoGT, it is compared with other representative models, from the baseline method DGCNN (Dynamic Graph Convolutional Neural Networks) to the most advanced method in the prior art, as shown in Figure 5 Various existing models in the prior art are listed in the figure, and in general, the performance of EmoGT is better than all other models. For the existing best result obtained by MD-AGCN (Multi-domain adaptive graph convolutional network), the average improvement rate of the model of the present method is still 1.5% on all three data sets. It is worth noting that in SEED-IV, EmoGT greatly exceeds the current best result by about 3.5%, even reaching 91.2%, which is a relatively high accuracy rate in four-class classification. These numbers undoubtedly prove the strong information extraction and processing capability of EmoGT.

[0093] The confusion matrix of EmoGT as Figure 6 described in the above table describes its ability to distinguish emotions on different data sets. For SEED and SEED-IV, the model of the present method is relatively good at identifying neutral states, but performs poorly in positive emotions. Interestingly, in SEED-V, disgust is the biggest obstacle for EmoGT classification, only 72.22%. In summary, EmoGT is likely to misclassify positive emotions and disgust, which is the same trend as other models.

[0094] Comparison with multi-modal methods The present method selects the three most advanced methods in emotion recognition to compare with the second EmoGTs of the present method. As Figure 5The second part presents the results on three datasets. Although the state-of-the-art results achieved by DCCA are very good, EmoGTs still outperform them on all datasets. It is worth noting that in the four-class classification task, the model of the present method has an amazing accuracy of 94%, which is about 6.5% higher than the state-of-the-art performance. Moreover, EmoGTs are also 2% higher than DCCA in the five-class task. In the multi-class classification, the accuracy is competitive. In addition, the considerable standard deviation proves that the model has relatively stable performance. In summary, EmoGTs represent the state-of-the-art level of multi-modal emotion recognition, and these satisfactory results inspire the solution of EmoGTs for similar multi-modal classification tasks.

[0095] Ablation studies such as Figure 5 are shown in italic lines in Table 2. The average performance gap reaches more than 5%, with better stability. By comparing the LSTM-based variant G+LSTM and EmoGT, the effectiveness of using Transformer to extract temporal information is demonstrated. The performance of the latter is much better than the former, with an accuracy of 3% to 5%. The gap between E-SMA and EmoGTs confirms that the introduced cross-modal attention mechanism is very useful for multi-modal learning, which indeed takes advantage of the complementary characteristics of different modalities to achieve better performance.

[0096] Visualization of the learned spatial connections in Figure 7 In this work, the present method demonstrates the brain connections learned by EmoGTs in the EEG channels during emotion elicitation for all three datasets. The 10 strongest connections of SEED and SEED-IV are similar to each other, while the connections of SEED-V are quite different. As can be seen from the figures, in SEED and SEED-IV, the connections are mainly concentrated in the temporal lobe on both sides of the brain and the frontal lobe, such as the connections of FT8 to T8 and FP1 to AF3, however, the lines in the SEED-V figure illustrate strong connections between the central region and the temporal lobe, such as CZ to T8, and within the central region of the brain, such as CP1 to CPZ and CZ to CPZ. From the perspective of cognitive neuroscience, a reasonable explanation is that because the emotional process with more emotion categories involves highly distributed neural circuits in the brain. As the number of emotions increases, more brain regions are activated to respond, resulting in global inter-channel connections.

[0097] Overall, the present method develops an elastic emotion recognition framework EmoGT to better utilize the EEG information in both the time domain and the spatial domain. It is achieved by integrating GCN for spatial relationship extraction and Transformer for time series information processing. By activating the cross-modal attention mechanism, EmoGT can be used as a building block for single modal to process multi-modal input. The model of the present method is superior to all existing technologies. The proposed framework clarifies the EEG-based emotion recognition task by utilizing the space and time in electroencephalogram.

[0098] As Figure 8 A structural schematic diagram of an emotion electroencephalogram feature representation system provided by an embodiment of the present application is shown. The system can execute the emotion electroencephalogram feature representation method described in any of the above embodiments and is configured in a terminal.

[0099] The emotion electroencephalogram feature representation system 10 provided by the present embodiment includes a feature determination program module 11, a feature input program module 12, a spatial information determination program module 13, and a feature determination program module 14.

[0100] The feature determination program module 11 is configured to obtain an electroencephalogram signal of a subject during viewing of an emotion stimulus material and extract an electroencephalogram differential entropy feature of the electroencephalogram signal. The feature input program module 12 is configured to input the electroencephalogram differential entropy feature into a graph Transformer network for emotion recognition, wherein the graph Transformer network includes a graph convolution network unit and a Transformer unit. The spatial information determination program module 13 is configured to model the electroencephalogram differential entropy feature in the graph Transformer network through the graph convolution network unit, determine a spatial topology structure of the electroencephalogram differential entropy feature under an adjacency matrix, and obtain an electroencephalogram feature with spatial information. The feature determination program module 14 is configured to perform self-attention layers and layer normalization processing on the electroencephalogram feature with spatial information through the Transformer unit with respect to time series, and obtain an electroencephalogram feature with spatial information and time information for emotion recognition.

[0101] Further, the feature determination program module is further configured to obtain an eye movement signal of a subject during viewing of an emotion stimulus material and extract an eye movement feature of the eye movement signal.

[0102] The feature input program module is configured to input the electroencephalogram differential entropy feature and the eye movement feature into a graph Transformer network for emotion recognition, wherein the graph Transformer network includes an electroencephalogram graph convolution network unit, an eye movement graph convolution network unit, and a cross-modal Transformer unit.

[0103] The spatial information determination program module is configured to determine the spatial topology of the brain electrical differential entropy features under an adjacency matrix by modeling the brain electrical differential entropy features through the electroencephalogram convolution network unit in the graph Transformer network, to obtain brain electrical features with spatial information, and determine the spatial topology of the eye movement features under an adjacency matrix by modeling the eye movement features through the electrooculogram convolution network unit, to obtain eye movement features with spatial information.

[0104] The feature determination program module is configured to perform cross-modal fusion on the brain electrical features with spatial information and the eye movement features with spatial information through the cross-modal Transformer unit, to obtain brain electrical emotion fusion features with spatial information and time information for emotion recognition.

[0105] The embodiment of the present application also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to execute the emotion brain electrical feature representation method in any method embodiment.

[0106] As an implementation mode, the non-volatile computer storage medium of the present application stores computer executable instructions, and the computer executable instructions are configured to:

[0107] Obtaining brain electrical signals of a subject during watching of emotional stimulus materials, and extracting brain electrical differential entropy features of the brain electrical signals;

[0108] Inputting the brain electrical differential entropy features into a graph Transformer network for emotion recognition, wherein the graph Transformer network comprises a graph convolution network unit and a Transformer unit.

[0109] In the graph Transformer network, the brain electrical differential entropy features are modeled through the graph convolution network unit to determine the spatial topology of the brain electrical differential entropy features under an adjacency matrix, to obtain brain electrical features with spatial information.

[0110] The brain electrical features with spatial information are subjected to self-attention layers and layer normalization processing with respect to time sequences through the Transformer unit, to obtain brain electrical features with spatial information and time information for emotion recognition.

[0111] As a non-volatile computer readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present application. One or more program instructions are stored in the non-volatile computer readable storage medium, and when executed by a processor, the emotion brain electrical feature representation method in any method embodiment is executed.

[0112] Figure 9 is a hardware structure schematic diagram of an electronic device of an emotion electroencephalogram feature representation method provided by another embodiment of the present application, as shown in the figure, the device comprises: Figure 9

[0113] one or more processors 910 and a memory 920, Figure 9 Take one processor 910 as an example. The device of the emotion electroencephalogram feature representation method can also include an input device 930 and an output device 940.

[0114] The processor 910, the memory 920, the input device 930 and the output device 940 can be connected through a bus or other means, Figure 9 Take the connection through the bus as an example.

[0115] The memory 920 is a kind of nonvolatile computer readable storage medium, which can be used to store nonvolatile software programs, nonvolatile computer executable programs and modules, such as program instructions / modules corresponding to the emotion electroencephalogram feature representation method in the embodiment of the present application. The processor 910 executes the various functions of the server and data processing by running the nonvolatile software program, instruction and module stored in the memory 920, that is, the emotion electroencephalogram feature representation method of the above method embodiment is realized.

[0116] The memory 920 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; The data storage area can store data and the like. In addition, the memory 920 can include a high-speed random access memory, and can also include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, or other nonvolatile solid-state storage device. In some embodiments, the memory 920 can optionally include a memory remotely arranged with respect to the processor 910, and these remote memories can be connected to the mobile device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0117] The input device 930 can receive input digital or character information. The output device 940 can include a display device such as a display screen.

[0118] The one or more modules are stored in the memory 920, and when executed by the one or more processors 910, the emotion electroencephalogram feature representation method in any of the above method embodiments is executed.

[0119] The above product can execute the method provided by the embodiment of the present application, has the function module and beneficial effect corresponding to the execution method. Technical details not described in detail in the embodiment can be referred to the method provided by the embodiment of the present application.​

[0120] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the emotional EEG feature representation method of any embodiment of this invention.

[0122] The electronic devices described in this application exist in various forms, including but not limited to:

[0123] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0124] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0125] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0126] (4) Other electronic devices with data processing functions.

[0127] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", "includes", "including" and the like can be used herein to indicate either an inclusion of a few elements or a plurality of elements without there being any limitation on the number of elements, and such term modification is not exclusive or restrictive, but shall be interpreted to allow for an inclusion of any additional elements in a given group. Additionally, terms such as "first" and "second" are used herein only to describe one entity from another without necessarily requiring or implying any actual such relationship or order between such entities.

[0128] The above-described apparatus embodiments are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0129] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An emotional electroencephalogram feature representation method, comprising: obtaining an electroencephalogram signal of a subject during watching of emotional stimulus materials, and extracting an electroencephalogram differential entropy feature of the electroencephalogram signal; obtaining an eye movement signal of the subject during watching of the emotional stimulus materials, and extracting an eye movement feature of the eye movement signal; inputting the electroencephalogram differential entropy feature and the eye movement feature into a graph Transformer network for emotion recognition, wherein the graph Transformer network comprises an electroencephalogram graph convolution network unit, an eye movement graph convolution network unit, and a cross-modal Transformer unit; in the graph Transformer network, modeling the electroencephalogram differential entropy feature by the electroencephalogram graph convolution network unit, determining a spatial topology structure of the electroencephalogram differential entropy feature under an adjacency matrix based on a learnable symmetric adjacency matrix, and obtaining an electroencephalogram feature with spatial information; wherein the learnable symmetric adjacency matrix is shared in all layers of the electroencephalogram graph convolution network unit, and is used for modeling the spatial topology structure for electroencephalogram differential entropy frequency domain features divided into multiple frequency bands; modeling the eye movement feature by the eye movement graph convolution network unit, determining a spatial topology structure of the eye movement feature under an adjacency matrix, and obtaining an eye movement feature with spatial information; performing cross-modal fusion of the electroencephalogram feature with spatial information and the eye movement feature with spatial information by the cross-modal Transformer unit based on a cross-modal attention layer, wherein the cross-modal attention layer calculates cross-modal attention from an electroencephalogram modality to an eye movement modality and from the eye movement modality to the electroencephalogram modality for the electroencephalogram modality and the eye movement modality respectively to obtain corresponding class labels, and extracts fusion features based on an attention fusion manner of the class labels to obtain an electroencephalogram emotion fusion feature with spatial information and temporal information for emotion recognition.

2. The method of claim 1, wherein, The graph Transformer network for emotion recognition further comprises a linear layer and a softmax layer, wherein the linear layer and the softmax layer are used to determine an emotion category of the subject from the electroencephalogram feature with spatial information and temporal information, or the electroencephalogram emotion fusion feature with spatial information and temporal information, wherein the emotion category comprises negative, neutral, and positive.

3. The method of claim 1, wherein, The graph Transformer network for emotion recognition is obtained by training based on an electroencephalogram emotion training set, wherein the electroencephalogram emotion training set comprises electroencephalogram training data, eye movement training data, and a reference emotion category; the training process of the graph Transformer network for emotion recognition comprises: inputting the electroencephalogram emotion training set into the graph Transformer network for emotion recognition, and outputting an electroencephalogram emotion feature with spatial information and temporal information for emotion recognition; inputting the electroencephalogram emotion feature into the linear layer and the softmax layer to obtain an estimated emotion category, and training the graph Transformer network for emotion recognition based on an error of the estimated emotion category and the reference emotion category.

4. The method of claim 1, wherein, After acquiring the electroencephalogram signal of a subject during watching emotional stimulus materials and acquiring the eye movement signal of the subject during watching the emotional stimulus materials, the method further comprises: baseline correction on the electroencephalogram signal, filtering processing and time window segmentation on the baseline-corrected electroencephalogram signal to obtain a preprocessed electroencephalogram signal; pupil diameter de-illumination processing on the eye movement signal, and linear interpolation to complete the empty pupil diameter to obtain a preprocessed eye movement signal.

5. An emotional electroencephalogram feature representation system for implementing the method of any one of claims 1-4; the system comprising: a feature determination program module for acquiring the electroencephalogram signal of a subject during watching emotional stimulus materials, and extracting the electroencephalogram differential entropy feature of the electroencephalogram signal; a feature input program module for inputting the electroencephalogram differential entropy feature into a graph Transformer network for emotion recognition, wherein the graph Transformer network comprises a graph convolution network unit and a Transformer unit; a spatial information determination program module for modeling the electroencephalogram differential entropy feature through the graph convolution network unit in the graph Transformer network, determining the spatial topology structure of the electroencephalogram differential entropy feature under the adjacency matrix based on a learnable symmetric adjacency matrix, and obtaining the electroencephalogram feature with spatial information; a feature determination program module for inputting the electroencephalogram feature with spatial information into the Transformer unit, and performing self-attention layer and layer normalization processing on the electroencephalogram feature with spatial information through the Transformer unit to obtain the electroencephalogram feature with spatial information and time information for emotion recognition.

6. The system of claim 5, wherein, The graph Transformer network for emotion recognition further comprises a linear layer and a softmax layer, wherein the linear layer and the softmax layer are used to determine the emotion category of the subject from the electroencephalogram feature with spatial information and time information, or the electroencephalogram emotional fusion feature with spatial information and time information, and the emotion category comprises negative, neutral, and positive.

7. An electronic device comprising: At least one processor and a memory in communication with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method of any one of claims 1-4.

8. A storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the steps of the method of any one of claims 1-4.

Images