A Method for EEG Signal Emotion Recognition Based on Attention and Spatiotemporal Convolutional Networks
By employing an attention-based and spatiotemporal convolutional network approach, target channels related to emotions are selected and a self-organizing graph structure is constructed. By combining graph convolution and long short-term memory networks, the computational resource consumption and noise issues caused by using all channels are resolved, thereby improving the accuracy and efficiency of emotion recognition from EEG signals.
Patent Information
- Application Number
- CN202410476937.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-19
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-04-19
Smart Images

Figure CN118383764B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of emotion classification, and more specifically, relates to a method for emotion recognition of EEG signals based on attention and spatiotemporal convolutional networks. Background Technology
[0002] Human emotions are complex psychological and physiological responses, a natural reaction to external stimuli that can influence various physiological changes in the body, such as increased heart rate and sweating. Emotions play a vital role in human cognition, particularly in rational decision-making, perception, and interpersonal communication.
[0003] Currently, emotion recognition mainly falls into two categories: one relies on non-physiological signals such as facial expressions, voice tone, and body posture. Because these non-physiological signals can be artificially controlled through spoofing, they sometimes fail to capture the true emotional signals, thus hindering accurate identification of the actual emotional state. The other category utilizes physiological signals such as electroencephalograms (EEGs), electrooculograms (EOGs), electrocardiograms (ECGs), and electromyograms (EMGs). Since physiological signals are difficult to fake and more accurately reflect human emotional states, emotion recognition based on physiological signals yields more objective and realistic results. EEG, a spatially discrete, non-stationary random signal generated by the central nervous system, can directly record changes in scalp electrical activity, providing a more reliable and accurate reflection of a person's emotional state compared to other physiological signals.
[0004] With the tremendous success of deep learning methods in various recognition tasks, multiple deep learning frameworks have begun to be applied to EEG emotion recognition. Compared to traditional machine learning methods, deep learning has a stronger ability to describe and fit problems. Current research focuses on the impact of different brain functional regions, frequency band combinations, and temporal features on emotion classification performance. Most studies are based on full-channel emotion classification; however, full-channel data contains both emotion-related information and potential interference or noise, and the independence between frequency bands is not considered when constructing the brain channel map structure. Some researchers have found that using combinations of certain frequency bands can indeed save computational costs, but the full frequency band may contain more emotional information.
[0005] Therefore, current research still has the following problems: using the entire channel not only consumes computational resources but may also introduce noisy data; directly constructing the brain channel map ignores the independent spatial relationships between frequency bands; and there are still some deficiencies in the combination of the temporal and spatial domains. All of the above problems can lead to inaccurate emotion recognition results. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this application aims to provide an emotion recognition method for EEG signals based on attention and spatiotemporal convolutional networks. This method is intended to solve the problems of inaccurate emotion recognition results caused by the use of all channels, neglect of independent spatial relationships between frequency bands, and inability to combine the temporal and spatial domains.
[0007] To achieve the above objectives, this application provides a method for emotion recognition based on EEG signals using attention and spatiotemporal convolutional networks, comprising:
[0008] Obtain an EEG emotion dataset, extract features from the EEG emotion dataset, and obtain differential entropy feature data;
[0009] Multiple target channels related to emotions are selected from the channels of the EEG emotion dataset, and the frequency band of each target channel is determined to obtain feature data of a new organizational form;
[0010] The feature data of the new organizational form is input into the recognition network. The spatial features of each target channel are obtained based on the graph convolution module of the recognition network, and the temporal features of the feature data are extracted based on the temporal feature extraction module of the recognition network.
[0011] The long short-term memory sub-network module of the recognition network identifies the spatial and temporal features to obtain the classification results corresponding to the EEG emotion dataset.
[0012] In some embodiments, the graph convolution module based on the recognition network acquires the spatial features of each target channel, and the temporal feature extraction module based on the recognition network extracts the temporal features of the feature data, including:
[0013] For each frequency band, a self-organizing graph structure is constructed to obtain the relevant channels of each current channel in each frequency band. Based on the graph convolution module, the spatial features between channels in each frequency band are extracted.
[0014] The differential entropy feature data of each frequency band are recombined, and the combined data is input into the time feature extraction module to obtain the time domain features.
[0015] In some embodiments, the construction process of the self-organizing adjacency matrix of the self-organizing graph structure includes:
[0016] The output matrix is determined using the tanh activation function based on the node input features and weight matrix of each frequency band.
[0017] The self-organizing adjacency matrix is obtained using the softmax function, based on the output matrix and its transpose.
[0018] In some embodiments, the selection process of the multiple target channels includes:
[0019] The index of the top k largest values of each vector in the self-organizing adjacency matrix is obtained according to the indexing function, where k is a positive integer;
[0020] In each row vector of the self-organizing adjacency matrix, k values belonging to the maximum value index are retained, and values not belonging to the maximum value index are set to 0. The channels corresponding to the k values are used as the target channels.
[0021] In some embodiments, the temporal feature extraction module includes a convolutional layer and an attention layer;
[0022] The step of inputting the combined data into the time feature extraction module to obtain the time-domain features includes:
[0023] The combined data is input into a convolutional layer for one 3D convolution and two pseudo-3D convolutions. The 3D convolution has a kernel size of 3×3×5, a stride size of 1×1×3, and uses zero-padding and a ReLU activation function. The pseudo-3D convolution has a kernel size of 3×3×1, a stride size of 1×1×3, and uses zero-padding and a ReLU activation function.
[0024] The output data of each pseudo-3D convolution is input into the attention layer for average pooling to obtain the temporal features.
[0025] In some embodiments, the output data of the pseudo-3D convolution is obtained by the following formula:
[0026] χ l =Φ 1×1×R (Φ P×Q×1 (χ l-1 ))
[0027] Where, χ l-1 For the output of layer (l-1), Φ P×Q×1 and Φ 1×1×R These represent two-dimensional convolutions with kernels P×Q in the spatial domain and one-dimensional convolutions with kernels R in the temporal domain, respectively.
[0028] In some embodiments, the output data of the attention layer is obtained using the following formula:
[0029]
[0030] Where M1∈R T×M M2∈R M×t b∈R N×M×T All are learnable parameters, where N is the batch size, M and T are the data sizes, and σ is the softmax function.
[0031] In a second aspect, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.
[0032] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.
[0033] Fourthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.
[0034] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.
[0035] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:
[0036] (1) Selecting multiple target channels related to emotion and determining the frequency band of each channel helps to focus on the signal part containing emotional information, improve the sensitivity of the classification model, and reduce the complexity of the model, thus saving a lot of computing resources.
[0037] (2) Combining graph convolutional neural networks and long short-term memory networks can fully utilize spatial and temporal features to improve the representation and classification of EEG signals. The graph convolutional module helps capture the relationships between different channels and provides more comprehensive spatial information, while the long short-term memory subnetwork can process temporal information and effectively utilize temporal dependencies.
[0038] (3) For each frequency band, a self-organizing graph structure is constructed to retain the most relevant channels, and graph convolution is used to obtain the spatial features between channels, thus preserving the relatively independent spatial features between each frequency band.
[0039] (4) By inputting differential entropy feature data into the recognition network and combining the extraction of spatial and temporal features, emotional information in EEG signals can be comprehensively mined from different perspectives, thereby improving classification effect and classification accuracy.
[0040] (5) It can directly extract features from raw EEG emotion data and classify emotions, simplifying the whole process and improving efficiency. Attached Figure Description
[0041] Figure 1 A flowchart illustrating the EEG signal emotion recognition method based on attention and spatiotemporal convolutional networks provided in this application embodiment;
[0042] Figure 2 This is a channel filtering diagram of an embodiment of this application;
[0043] Figure 3 This is a comparison chart of time consumption and accuracy in an embodiment of this application;
[0044] Figure 4 This is a graph showing the impact of top-k selection on classification performance in this application example;
[0045] Figure 5 This is a schematic diagram of the modules in an embodiment of this application;
[0046] Figure 6 This is a classification effect diagram of an embodiment of this application;
[0047] Figure 7 This is a diagram showing the test accuracy analysis results of an embodiment of this application.
[0048] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0050] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.
[0051] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.
[0052] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0053] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.
[0054] Next, the technical solutions provided in the embodiments of this application will be described.
[0055] Reference Figure 1 This application provides a method for emotion recognition based on EEG signals using attention and spatiotemporal convolutional networks, comprising:
[0056] S101. Obtain the EEG emotion dataset, extract features from the EEG emotion dataset, and obtain differential entropy feature data;
[0057] S102. Select multiple target channels related to emotions from the channels of the EEG emotion dataset, and determine the frequency band of each target channel to obtain feature data of the new organizational form;
[0058] S103. Input the feature data of the new organizational form into the recognition network, obtain the spatial features of each target channel based on the graph convolution module of the recognition network, and extract the temporal features of the feature data based on the temporal feature extraction module of the recognition network.
[0059] S104. Based on the long short-term memory sub-network module of the recognition network, the spatial features and temporal features are identified to obtain the classification results corresponding to the EEG emotion dataset.
[0060] Specifically, in S101 above, the EEG emotion dataset SEED is obtained, which contains two folders: Preprocessed_EEG and Extracted Features, which are the original data and the data from which DE features are extracted, respectively.
[0061] It should be noted that DE features usually refer to Differential Entropy features. Differential entropy is used to measure the uncertainty or information content of continuous random variables. In signal processing and data analysis, differential entropy is widely used in feature extraction and data modeling.
[0062] DE features are frequency domain features calculated using a 512-point short-time Fourier transform, with a 1-second non-overlapping Hanning window, and averaged across 5 frequency bands. Therefore, the output DE features can be represented as a 5×T matrix, where T represents the time window dependent on the stimulated film clip. The time window T for the SEED dataset is 185–265. For the normalized reference SOGNN, features with short time windows are zero-padded to a length of 265 for the SEED dataset. The model input EEG signal features are in the form of electrode × frequency band × time frame, and in the SEED dataset, the input format is 12×5×265.
[0063] Then, in step S102 above, channel filtering and frequency band division are performed. Since SEED collects 62 channels of EEG data, in order to save computing resources with almost no impact on the results, the data of emotion-related channels are filtered out from the 62 channels. In this embodiment, 12 target channels are selected. These target channels are then divided into 5 frequency bands to obtain a new data organization format.
[0064] Reference Figure 2 During EEG data acquisition, as many electrodes as possible are typically used to ensure a more comprehensive collection of signals. These signals include emotion-related information, but may also contain interference signals or noise. The frontal lobe of the brain is rich in emotional information, while the temporal lobe is associated with emotional processes. Therefore, 12 channels—FT7, T7, TP7, P7, C5, CP5, FT8, T8, TP8, P8, C6, and CP6—are used for emotion recognition. Furthermore, decomposing EEG signals into frequency bands helps in understanding and analyzing the characteristics of EEG signals and the frequency ranges associated with specific brain activities. Therefore, this embodiment decomposes the EEG signal into five frequency bands: δ, θ, α, β, and γ, with frequencies increasing from low to high.
[0065] Reference Figure 3 Furthermore, when combined with 4 frequency bands and 12 channels, the use of full-band and 12-channel combination improves classification accuracy with minimal increase in time consumption.
[0066] Furthermore, through the above steps S103 and S104, the data obtained by differential entropy feature extraction is input into the recognition network. These feature data contain dynamic information of the EEG signal, which helps to reveal the complex characteristics of the signal.
[0067] The graph convolution module used in the recognition network can effectively learn the spatial relationships between different target channels. The temporal feature extraction module can extract temporal features from the differential entropy feature data. Temporal features reflect the temporal variation patterns and trends of the signal, helping to capture the dynamics and temporal information of the signal.
[0068] The extracted spatial and temporal features are identified and processed using a Long Short-Term Memory (LSTM) sub-network module. LSTM networks effectively model sequential data and consider long-term dependencies, making them suitable for processing time-series data or feature data with temporal characteristics. The spatial features obtained through the graph convolution module and the temporal features extracted by the temporal feature extraction module are then integrated and identified by the LSTM sub-network module, ultimately yielding the classification result corresponding to the EEG emotion dataset to indicate the emotional state expressed by the EEG signals.
[0069] This application's embodiments, by selecting multiple target channels related to emotion and determining the frequency band of each channel, help focus on the signal portion containing emotional information, improve the sensitivity of the classification model, reduce model complexity, and significantly save computational resources. A self-organizing graph structure is constructed for each frequency band to retain the most relevant channels, and graph convolution is used to obtain the spatial features between channels, preserving the relatively independent spatial features between each frequency band. By inputting differential entropy feature data into the recognition network and combining the extraction of spatial and temporal features, emotional information in EEG signals can be comprehensively mined from different perspectives, improving classification performance and accuracy.
[0070] In some embodiments, the graph convolution module based on the recognition network acquires the spatial features of each target channel, and the temporal feature extraction module based on the recognition network extracts the temporal features of the feature data, including:
[0071] For each frequency band, a self-organizing graph structure is constructed to obtain the relevant channels of each current channel in each frequency band. Based on the graph convolution module, the spatial features between channels in each frequency band are extracted.
[0072] The differential entropy feature data of each frequency band are recombined, and the combined data is input into the time feature extraction module to obtain the time domain features.
[0073] Specifically, in this embodiment, a self-organizing graph structure is first constructed for the EEG signal data within each frequency band. Then, through a graph convolution module, spatial features between channels can be extracted from the self-organizing graph structure within each frequency band. Graph convolutional networks can effectively learn the relationships between nodes in graph structure data, thereby obtaining a more accurate and comprehensive spatial feature representation, which helps to reveal the spatial correlation between EEG signal channels in different frequency bands.
[0074] The recombined data is then input into the time feature extraction module to obtain the temporal features of each frequency band. This provides information for subsequent classification or analysis tasks.
[0075] Through the above process, EEG signal data in different frequency bands were analyzed and features were extracted in depth. By using self-organizing graph structure, graph convolutional network and temporal feature extraction module, the spatial relationship and temporal characteristics of EEG signals in each frequency band can be understood in more detail. This provides a foundation and support for further data processing and application, and helps to fully explore the information in multi-frequency band EEG signal data, so as to better understand the dynamic characteristics and changing patterns of brain activity.
[0076] In some embodiments, the construction process of the self-organizing adjacency matrix of the self-organizing graph structure includes:
[0077] The output matrix is determined using the tanh activation function based on the node input features and weight matrix of each frequency band.
[0078] The self-organizing adjacency matrix is obtained using the softmax function, based on the output matrix and its transpose.
[0079] The specific formula is as follows:
[0080] The formula for calculating the self-organizing adjacency matrix is:
[0081]
[0082] A = Softmax(GG) T )
[0083] Where v is the input feature of the node, W is the weight matrix using the Tanh activation function, and G is the output matrix of the linear layer. T Given the transpose of the matrix, the self-organizing adjacency matrix A is obtained by applying the softmax function.
[0084] Furthermore, the selection process for the multiple target channels includes:
[0085] The index of the top k largest values of each vector in the self-organizing adjacency matrix is obtained according to the indexing function, where k is a positive integer;
[0086] In each row vector of the self-organizing adjacency matrix, k values belonging to the maximum value index are retained, and values not belonging to the maximum value index are set to 0. The channels corresponding to the k values are taken as the target channels.
[0087] The specific formula is as follows:
[0088]
[0089] The argtopk(·) function finds the indices of the top-k largest values for each vector A[i,:] in the adjacency matrix A. This represents the index of a value that is not among the top-k largest values. In each row vector of the adjacency matrix A, only the k largest values are retained, and the rest are set to 0.
[0090] like Figure 4 As shown, the classification accuracy of this method is highest when top-k is set to 10, therefore the top-k of this method is set to 10.
[0091] In some embodiments, the temporal feature extraction module includes a convolutional layer and an attention layer;
[0092] The step of inputting the combined data into the time feature extraction module to obtain the time-domain features includes:
[0093] The combined data is input into a convolutional layer for one 3D convolution and two pseudo-3D convolutions. The 3D convolution has a kernel size of 3×3×5, a stride size of 1×1×3, and uses zero-padding and a ReLU activation function. The pseudo-3D convolution has a kernel size of 3×3×1, a stride size of 1×1×3, and uses zero-padding and a ReLU activation function.
[0094] The output data of each pseudo-3D convolution is input into the attention layer for average pooling to obtain the temporal features.
[0095] Specifically, refer to Figure 5 , Figure 5 The overall structure of an embodiment of this application is shown, which mainly consists of a spatial feature extraction part (1) and temporal feature extraction parts (2) and (3). The temporal feature extraction part, namely the aforementioned temporal feature extraction module, includes convolutional layers and attention layers.
[0096] The spatial feature extraction module, also known as the graph convolution module, uses DE features from the SEED dataset as input, with an input format of 12×5×265. First, the data from the original dataset is separated into five frequency bands. For each frequency band, a self-organizing graph structure is constructed, and graph convolution is used to extract the spatial relationships between electrodes within each band. This ensures that spatial features from different frequency bands are not confused, thus preserving the corresponding graph structure. Then, the data from the five frequency bands are combined and reorganized into a batch_size×channels×5×64×1 format before being fed into the temporal feature extraction module. In the temporal feature extraction module, the input data undergoes one 3D convolution and two pseudo-3D convolutions. The 3D convolution uses a 3×3×5 kernel with a stride of 1×1×3, zero-padding, and a ReLU activation function. The two pseudo-3D convolutions operate similarly, using one 3×3×1 kernel and one 1×1×3 kernel, both with zero-padding and a ReLU activation function. After each pseudo-3D convolution, a partial dot product attention and a 2×2×2 average pooling function are used. After temporal feature extraction, the data is converted into a one-dimensional tensor, LSTM is used to learn contextual relationships, and finally, a softmax function is used for classification.
[0097] Furthermore, the output data of the pseudo-3D convolution is obtained using the following formula:
[0098] χ 1 =Φ 1×1×R (Φ P×Q×1 (χ 1-1 ))
[0099] Where, χ l-1 For the output of layer (l-1), Φ P×Q×1 and Φ 1×1×R These represent two-dimensional convolutions with kernels P×Q in the spatial domain and one-dimensional convolutions with kernels R in the temporal domain, respectively.
[0100] Furthermore, the output data of the attention layer is obtained using the following formula:
[0101]
[0102] Where M1∈R T×M M2∈R M×T b∈R N×M×T All are learnable parameters, where N is the batch size, M and T are the data sizes, and σ is the softmax function.
[0103] Reference Figure 6 The present application embodiment presents a confusion matrix for classifying the SEED dataset using 12 channels and 5 frequency bands with a top-k value of 10, where 1 represents positive, 0 represents neutral, and -1 represents negative.
[0104] The following is a general overview of the thought process involved in each step of this application:
[0105] First, a self-organizing graph is constructed for each channel in each frequency band to obtain the 10 channels most relevant to the current channel. Then, graph convolution is used to capture the spatial relationships between all channels in the self-organizing graph constructed for each channel in each frequency band. Next, pseudo-3D convolution with partial dot product attention is used to obtain the temporal features of the EEG sequence. Finally, LSTM is used to learn the contextual information between adjacent time series and the Softmax function is used for classification.
[0106] In summary, compared with traditional emotion classification methods based on electroencephalogram (EEG) signals, this application has the following advantages:
[0107] Twelve channels suitable for emotion classification were selected to reduce model complexity and save significant computational resources. A self-organizing graph structure was constructed for each frequency band to retain the 10 most relevant channels, and graph convolution was used to obtain spatial features between channels, preserving the relatively independent spatial features between frequency bands. Since EEG signals are a time series, temporal features are also crucial. Combining pseudo-3D convolution with graph convolution not only extracted spatial features but also temporal features. Furthermore, combining the extraction of temporal features with partial dot product attention assigned higher weights to emotion-related information in the EEG sequence, improving classification results.
[0108] Results Comparison: This application achieved superior performance on public datasets. As shown in Tables 1 and 2, it achieved better classification results than the baseline model under both subject-independent and subject-dependent experimental strategies.
[0109] Reference Figure 7 As can be seen from the figure, except for the fourth subject, the accuracy of the other subjects was above 90%, which proves that it has a certain stability on different experimental subjects and has practical application significance.
[0110] Table 1. Subjects' independent classification accuracy on SEED
[0111] Model All Bands SVM 56.73 / 16.29 TCA 63.64 / 14.88 SA 69.00 / 10.89 BiHDM 85.40 / 07.53 BiDANN-S 84.14 / 06.87 DGCNN 79.95 / 09.02 RGNN 85.30 / 06.72 SOGNN 86.81 / 05.79 SOGPCN (This article) 94.22 / 03.42
[0112] Table 2. Classification accuracy of subject dependencies on SEED
[0113] Model All Bands SVM 83.99 / 09.92 GSVCCA 82.96 / 09.95 DBN 86.08 / 08.34 STRNN 89.50 / 07.63 DGCNN 90.40 / 08.49 BiDANN 92.38 / 07.04 RGNN 94.24 / 05.95 SOGPCN (This article) 95.26 / 03.52
[0114] Ablation experiment:
[0115] To investigate the necessity of various optimization methods in the proposed EEG signal emotion recognition method based on attention and spatiotemporal convolutional networks, an ablation experiment was conducted on the following components: 1. Self-organizing graph component; 2. Pseudo-3D convolutional component; 3. Partial dot-product attention component. In the ablation experiment, to verify the effectiveness of the spatial and temporal feature extraction modules in SOGPCN, an ablation experiment was performed on the SEED dataset using 12 channels and 5 frequency bands under a leave-one-out experiment strategy. The results are shown in Table 3. After removing the self-organizing graph convolutional module, the average accuracy decreased by 1.92%. After removing the pseudo-3D convolutional and partial dot-product attention modules, the average accuracy decreased by 24.15% and 1.19%, respectively, with improvements in the standard deviation. The results indicate that, as a multi-channel sequence data, both spatial and temporal features of EEG data contribute to improving the model's classification results and stability. Furthermore, temporal features make a significant contribution to EEG emotion recognition and are an indispensable component.
[0116] Table 3 Ablation Experiment
[0117]
[0118]
[0119] Reference Figure 8 Based on the methods in the above embodiments, this application provides an electronic device that may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.
[0120] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0121] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.
[0122] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.
[0123] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.
[0124] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.
[0125] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0126] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.
[0127] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for emotion recognition based on EEG signals using attention and spatiotemporal convolutional networks, characterized in that, include: Obtain an EEG emotion dataset, extract features from the EEG emotion dataset, and obtain differential entropy feature data; Multiple target channels related to emotions are selected from the channels of the EEG emotion dataset, and the frequency band of each target channel is determined to obtain feature data of a new organizational form; The feature data of the new organizational form is input into the recognition network. The spatial features of each target channel are obtained based on the graph convolution module of the recognition network, and the temporal features of the feature data are extracted based on the temporal feature extraction module of the recognition network. The spatial and temporal features are identified based on the long short-term memory sub-network module of the recognition network to obtain the classification results corresponding to the EEG emotion dataset; The graph convolution module based on the recognition network acquires the spatial features of each target channel, and the temporal feature extraction module based on the recognition network extracts the temporal features of the feature data, including: For each frequency band, a self-organizing graph structure is constructed to obtain the relevant channels of each current channel in each frequency band. Based on the graph convolution module, the spatial features between channels in each frequency band are extracted. The differential entropy feature data of each frequency band are recombined, and the combined data is input into the time feature extraction module to obtain the time domain features; The construction process of the self-organizing adjacency matrix of the self-organizing graph structure includes: The output matrix is determined using the tanh activation function based on the node input features and weight matrix of each frequency band. Using the softmax function, the self-organizing adjacency matrix is obtained based on the output matrix and its transpose. The selection process for the multiple target channels includes: The index of the top k largest values of each vector in the self-organizing adjacency matrix is obtained according to the indexing function, where k is a positive integer; In each row vector of the self-organizing adjacency matrix, k values belonging to the maximum value index are retained, and values not belonging to the maximum value index are set to 0. The channels corresponding to the k values are used as the target channels.
2. The EEG signal emotion recognition method based on attention and spatiotemporal convolutional networks according to claim 1, characterized in that, The temporal feature extraction module includes convolutional layers and attention layers; The step of inputting the combined data into the time feature extraction module to obtain the time-domain features includes: The combined data is input into a convolutional layer for one 3D convolution and two pseudo-3D convolutions. The 3D convolution has a kernel size of 3×3×5, a stride size of 1×1×3, and uses zero-padding and a ReLU activation function. The pseudo-3D convolution has a kernel size of 3×3×1, a stride size of 1×1×3, and uses zero-padding and a ReLU activation function. The output data of each pseudo-3D convolution is input into the attention layer for average pooling to obtain the temporal features.
3. The EEG signal emotion recognition method based on attention and spatiotemporal convolutional networks according to claim 2, characterized in that, The output data of the pseudo-3D convolution is obtained through the following formula: in, For the first The output of the layer, and These represent two-dimensional convolutions with kernels P×Q in the spatial domain and one-dimensional convolutions with kernels R in the temporal domain, respectively.
4. The EEG signal emotion recognition method based on attention and spatiotemporal convolutional networks according to claim 3, characterized in that, The output data of the attention layer is obtained using the following formula: in, , , All are learnable parameters, where N is the batch size, and M and T are the data sizes. This refers to the softmax function.
5. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-4.
6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-4.
7. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-4.
Citation Information
Patent Citations
Electroencephalogram emotion recognition method and system
CN115659207A
Emotional electroencephalogram feature representation method and system, electronic equipment and storage medium
CN116671917A
Emotion recognition method and device based on electroencephalogram signals, electronic equipment and medium
CN116671918A
