Emotion recognition result visualization method and system based on EEG data

Abstract

The present application discloses a method and system for visualizing emotion recognition results based on EEG data, which belongs to the field of emotion recognition technology. The method includes: preprocessing the original EEG data to obtain preprocessed EEG data; performing spatial transformation on the preprocessed EEG data to obtain time domain feature data; obtaining frequency domain feature data based on the time domain feature data; obtaining time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data; performing feature interaction and fusion on the time / space feature tokens and the frequency / space feature tokens to obtain fused feature tokens; performing linear transformation on the fused feature tokens to obtain the final emotion classification results; and visualizing the emotion classification results. The present application can effectively solve the problem that the traditional EEG signal emotion recognition method ignores a large amount of potential and deep feature information in the EEG data, and the visualization of the emotion results is conducive to users viewing the emotion classification results.

Description

Technical Field

[0001] The present application belongs to the field of emotion recognition technology, and specifically relates to a method and system for visualizing emotion recognition results based on electroencephalogram (EEG) data. Background Art

[0002] EEG data is a record of electrical signals generated by neuronal activity in the brain. The brain contains hundreds of millions of neurons, which transmit information through electrical activity. When neurons become excited, ions flow across their membranes, creating a potential difference. The combined potential differences across these numerous neurons form an EEG signal detectable on the surface of the scalp. This signal is essentially a time series data set that changes continuously over time, reflecting the brain's emotional state at different moments.

[0003] Traditional EEG emotion recognition methods or systems fuse the time, frequency, and spatial domain features of EEG data to comprehensively characterize the EEG signal characteristics corresponding to emotions through feature fusion from different perspectives. However, these traditional methods have serious limitations in feature extraction. Despite integrating time, frequency, and spatial domain features, they only utilize common, basic features and ignore a wealth of potential, deeper features in EEG signals. For example, these include the complex nonlinear coupling relationships between EEG frequency bands, the dynamically changing functional connectivity patterns between different brain regions, and time-varying characteristics under different cognitive tasks and emotional states. These potential features are crucial for more accurate emotion recognition, yet they are often overlooked by traditional methods. Traditional emotion recognition methods or systems often produce unintuitive predictions, making it difficult for users to observe and analyze the results. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method and system for visualizing emotion recognition results based on EEG data, which can solve the problem that traditional EEG signal emotion recognition methods ignore a large amount of potential and deep feature information in EEG data.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for visualizing emotion recognition results based on EEG data, the method comprising:

[0007] Preprocessing the raw EEG data collected by the EEG device to obtain preprocessed EEG data;

[0008] Performing spatial transformation on the pre-processed EEG data to obtain time domain feature data, wherein the time domain feature data is a vector sequence including the time features of the pre-processed EEG data;

[0009] Obtain frequency domain feature data according to time domain feature data;

[0010] Based on the time domain feature data and the frequency domain feature data, a time / space feature token and a frequency / space feature token are respectively obtained, wherein the time / space feature token includes the time domain feature and the spatial domain feature of the pre-processed EEG data, and the frequency / space feature token includes the frequency feature and the spatial domain feature of the pre-processed EEG data;

[0011] Perform feature interaction and fusion on the time / space feature tokens and the frequency / space feature tokens to obtain fused feature tokens;

[0012] Perform linear transformation on the fused feature tokens to obtain the final emotion classification result;

[0013] Visualize the emotion classification results.

[0014] Furthermore, the step of preprocessing the raw EEG data collected by the EEG device to obtain preprocessed EEG data specifically includes:

[0015] Filtering the original EEG data to obtain filtered EEG data;

[0016] Normalize the filtered EEG data to obtain preprocessed EEG data;

[0017] Among them, the normalization function expression in the normalization process is:

[0018] ,

[0019] in, represents the filtered EEG data, represents the mean value of the filtered EEG data, represents the variance of the filtered EEG data, represents the normalized EEG data.

[0020] Furthermore, in the step of respectively obtaining time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data, the steps specifically include:

[0021] Obtaining time / space feature tokens through time-space branches based on time-domain feature data;

[0022] Input the time domain feature data into the multi-scale temporal convolution layer of the spatiotemporal branch to obtain multi-scale temporal feature data;

[0023] Input the multi-scale temporal feature data into the first spatial convolution layer of the spatiotemporal branch to obtain fused spatiotemporal feature data;

[0024] Perform convolution mapping on the fused spatiotemporal feature data to obtain time / space feature tokens;

[0025] Obtain frequency / space feature tokens based on frequency domain feature data through frequency-space branches;

[0026] Input the frequency domain feature data into the multi-scale frequency convolution layer of the frequency-space branch to obtain multi-scale frequency feature data;

[0027] Input the multi-scale frequency feature data into the second spatial convolution layer of the frequency-space branch to obtain fused frequency-space feature data;

[0028] Perform convolution mapping on the fused frequency-space feature data to obtain frequency / space feature tokens.

[0029] Furthermore, in the step of performing feature interaction and fusion on the time / space feature token and the frequency / space feature token to obtain the fused feature token, the following steps are specifically included:

[0030] Perform feature interaction on the time / space feature token and the frequency / space feature token through a self-attention block and a cross-covariance attention block to obtain a first multi-feature token and a second multi-feature token, respectively, wherein the first multi-feature token and the second multi-feature token both include time domain features, frequency domain features, and spatial domain features of the preprocessed EEG data;

[0031] The first multi-feature token and the second multi-feature token are respectively concatenated based on the sample quantity dimension and the feature vector dimension, and are respectively mapped to the original dimension of the pre-processed EEG data to obtain a point dimension token and a vector dimension token;

[0032] Through the cross-covariance attention block, the point dimension tokens and vector dimension tokens are fused to obtain the fused feature tokens.

[0033] Furthermore, the self-attention block and the cross-covariance attention block also include:

[0034] Self-attention blocks and cross-covariance attention blocks introduce median mask and multi-head computation methods;

[0035] Among them, the median mask is used to reduce the impact of extreme values ​​in the pre-processed EEG data;

[0036] The long position calculation method is:

[0037] ，

[0038] ,

[0039] in, head represents the head of each attention sequence, Concat Indicates multiple heads are spliced ​​together. Q 、 K 、V Represent query vector, key vector, and value vector respectively, W o represents the output weight matrix, Represents the mapping result of linear transformation after multi-head splicing.

[0040] Furthermore, the self-attention block and the cross-covariance attention block also include:

[0041] The self-attention module acts as an encoder to convert the input feature sequence into a feature representation vector;

[0042] The cross-covariance attention block is used as a decoder to generate the target sequence based on the feature representation vector;

[0043] The output expression of the cross-covariance attention block is:

[0044] ,

[0045] ,

[0046] ( )]V，

[0047] Among them, Q represents the query vector sequence, K represents the key vector sequence, and V represents the value vector sequence. represents the dimension of the key vector sequence, Restricts each element of the vector sequence to be between -1 and 1. represents the attention score of the cross-covariance mechanism, Indicates median mask processing. Softmax Indicates normalization. Represents the output of the cross-covariance attention block.

[0048] Furthermore, in the step of performing linear transformation on the fused feature tokens to obtain the final emotion classification result, the following steps are specifically included:

[0049] Input the fused feature tokens into the fully connected layer for linear transformation;

[0050] The final emotion classification result is obtained based on the fused feature tokens after activation function and linear transformation.

[0051] In a second aspect, an embodiment of the present application provides a system for visualizing emotion recognition results based on EEG data, the system comprising:

[0052] A spatial transformation module is used to preprocess the raw EEG data collected by the EEG device to obtain preprocessed EEG data, and to perform spatial transformation on the preprocessed EEG data to obtain time domain feature data; wherein the time domain feature data is a vector sequence including the spatial features of the preprocessed EEG data;

[0053] A multi-scale convolution mapping module is used to obtain frequency domain feature data based on time domain feature data, and to obtain time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data, respectively; wherein the time / space feature tokens include the time domain features and spatial domain features of the pre-processed EEG data, and the frequency / space feature tokens include the frequency features and spatial domain features of the pre-processed EEG data;

[0054] A feature interaction and fusion module is used to perform feature interaction and fusion on time / space feature tokens and frequency / space feature tokens to obtain fused feature tokens;

[0055] The classification module is used to perform linear transformation on the fused feature tokens to obtain the final emotion classification results;

[0056] The visualization module is used to visualize the final emotion classification results.

[0057] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method described in the first aspect are implemented.

[0058] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0059] In an embodiment of the present application, the raw EEG data collected by the EEG device is first preprocessed to obtain preprocessed EEG data, thereby avoiding the inconvenience caused by interference and noise in the raw EEG data to subsequent feature fusion; the preprocessed EEG data is spatially transformed to obtain time domain feature data, and frequency domain feature data is obtained based on the time domain feature data, and time / space feature tokens and frequency / space feature tokens are obtained based on the time domain feature data and the frequency domain feature data, thereby realizing the fusion of spatial feature data, time domain feature data and frequency domain feature data, and being able to capture more comprehensive features; feature interaction and fusion are performed on the time / space feature tokens and the frequency / space tokens to obtain fused feature tokens, and median masking and multi-head calculation are adopted, as well as feature interaction and fusion through self-attention blocks and cross-covariance attention blocks to reduce the influence of outliers in the EEG data and effectively reduce the complexity of the calculation; the fused feature tokens are linearly transformed to obtain the final emotion classification result, and the emotion classification result is visualized. Displaying the emotion classification result helps to intuitively present the emotion distribution of different subjects and clearly observe the emotional changes of the subjects. This application can effectively solve the problem that traditional EEG signal emotion recognition methods ignore a large amount of potential and deep feature information in EEG data. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flowchart showing an implementation method of the method for visualizing emotion recognition results based on EEG data provided by an embodiment of the present invention is shown;

[0061] Figure 2 An implementation method of the method for visualizing emotion recognition results based on EEG data provided by an embodiment of the present invention is shown;

[0062] Figure 3 The original EEG data acquisition interface provided by the embodiment of the present invention is shown;

[0063] Figure 4 The diagram shows a visualization interface of the final emotion recognition result provided by an embodiment of the present invention;

[0064] Figure 5 A schematic diagram of the structure of a system for visualizing emotion recognition results based on EEG data provided by an embodiment of the present invention is shown;

[0065] Figure 6 A schematic structural diagram of an electronic device applying a method for visualizing emotion recognition results based on EEG data provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0067] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of this application can be implemented in an order other than those illustrated or described herein. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0068] The following describes in detail the method and system for visualizing emotion recognition results based on EEG data provided by the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0069] It is worth noting that the method of EEG emotion recognition is widely used in multiple industries. In the medical field, it assists doctors in intuitively diagnosing mental illnesses, accurately grasping patients' emotional fluctuations, and evaluating treatment effects in real time, thereby formulating and adjusting more effective treatment plans. In educational scenarios, teachers can use this emotion recognition technology to understand students' classroom emotions in real time, gain insight into students' learning status, and promptly optimize teaching strategies. It can also provide personalized support for special education students. In the field of human-computer interaction, it helps intelligent product designers optimize product design and enhance user experience based on user EEG emotional feedback, while making the interaction between emotional robots and humans more natural and more compatible.

[0070] The method disclosed in this application for visualizing emotion recognition results based on EEG data can capture a large amount of potential, deep-level characteristic information in EEG data, thereby enabling an objective assessment of an individual's emotional state. By monitoring and analyzing the brain's electrical activity signals in real time, it can quickly identify three emotions: excitement, concentration, and fear. This method is unaffected by subjective factors and has high accuracy and reliability.

[0071] See also Figures 1 to 4 , which shows an implementation of a method for visualizing emotion recognition results based on EEG data according to an embodiment of the first aspect of the present invention, the method includes steps S101 to S107, wherein:

[0072] S101 , preprocessing the original EEG data collected by the EEG device to obtain preprocessed EEG data.

[0073] In this embodiment, the raw EEG data is collected and recorded by a brain-computer interface device, such as Figure 3 Shown is the raw EEG data acquisition interface provided in an embodiment of the present application.

[0074] It is worth noting that the brain-computer device used in the embodiment of the present application can simultaneously collect and record EEG data of multiple subjects.

[0075] In some embodiments of the present application, the original EEG data is filtered to obtain filtered EEG data, and the filtered EEG data is normalized.

[0076] Among them, the normalization function expression in the normalization process is:

[0077] ,

[0078] in, represents the filtered EEG data, represents the mean value of the filtered EEG data, represents the variance of the filtered EEG data, represents the normalized EEG data.

[0079] It is worth noting that the dimensions of the filtered EEG data and the preprocessed EEG data are both C eeg ×T, where C eeg represents the number of channels, and T represents the number of time points.

[0080] In this embodiment, the raw EEG data is filtered using a sixth-order Chebyshev filter to remove high-frequency and low-frequency components in the raw EEG data.

[0081] It's worth noting that the sixth-order Chebyshev filter is a high-performance filter. Built based on Chebyshev polynomials, the Chebyshev filter exhibits equiripple characteristics within the passband and a high attenuation rate in the stopband, effectively suppressing unwanted frequency components. The higher the order, the greater the stopband attenuation capability. Furthermore, compared to some other filters, the Chebyshev filter has a narrower transition band, enabling rapid attenuation changes between the passband and stopband, thereby accurately filtering out desired frequency signals.

[0082] In this embodiment, the Z-score normalization function is used to reduce the non-stationarity and variability of the original EEG data.

[0083] It's worth noting that the Z-score normalization function, also known as standardization, is a data preprocessing technique. It effectively eliminates the dimensionality effects between different features, bringing data of varying units and magnitudes to the same scale. This prevents overemphasis or underestimation of certain features due to dimensional differences during data analysis and model training, significantly improving the performance of machine learning models. Furthermore, normalized data is kept at the same scale, making it easier to compare different features and effectively displaying information about each feature in data visualization.

[0084] Step S102: Perform spatial transformation on the pre-processed EEG data to obtain time domain feature data.

[0085] In some embodiments of the present application, the time domain feature data is a vector sequence including the time features of the pre-processed EEG data.

[0086] In this embodiment, Figure 2 As shown in the figure, the Multi-CSP method is used to perform spatial transformation on the preprocessed EEG data to obtain spatial domain feature data. After completing the preprocessing steps of filtering, artifact removal, and normalization on the EEG data, the Multi-CSP method constructs a spatial filter to effectively distinguish the spatial distribution of EEG signals in different states, thereby maximizing the variance difference. It is worth noting that after the spatial transformation, the spatial domain feature data not only improves the discrimination between EEG data in different states, but also preserves the temporal characteristics of the EEG data. Next, the spatial domain feature data is further subjected to feature extraction to obtain time domain feature data such as mean, variance, number of zero crossings, peak amplitude, and waveform slope. These feature data are of great significance for in-depth exploration of the laws of brain electrical activity and assisting in brain state classification.

[0087] It's worth noting that the Multi-CSP method is an extension and improvement of the traditional constraint satisfaction problem method (hereafter referred to as the CSP method). In multi-channel scenarios, the traditional CSP method often struggles to account for the complex correlations and characteristic differences between different channels. However, the Multi-CSP method effectively integrates information from each channel and can independently extract spatial features for each frequency band or time period in the multi-channel EEG signal, overcoming the limitations of the original CSP method. Through segmentation, frequency division, and feature integration, the Multi-CSP method can better extract discriminative features from EEG signals.

[0088] In this embodiment, a one-versus-rest strategy (OVR strategy) is introduced to extend the performance of Multi-CSP, thereby using a multi-category spatial filter to perform dimensionality reduction on EEG data.

[0089] It is worth noting that the OVR strategy is a method for multi-classification problems. Its core idea is to transform the multi-classification problem into multiple binary classification problems, treating one class as the positive class and all other classes as negative classes, thus forming a binary classification problem for each feature.

[0090] In this embodiment, for each binary classification problem, a corresponding binary classification spatial filter is generated using the Multi-CSP method, and then different binary classification spatial filters are spliced ​​to obtain a multi-classification spatial filter.

[0091] Step S103: Obtain frequency domain feature data according to the time domain feature data.

[0092] In this embodiment, the time domain feature data is converted into frequency domain feature data by using the Fourier transform principle.

[0093] It is worth noting that in the process of data analysis and processing, for time domain feature data recorded with time as the variable, the Fourier transform can decompose the time domain data that originally shows the characteristics of time dimension changes into a form composed of sine waves and cosine waves of different frequencies, and then clearly present the various frequency components contained in these data and their corresponding amplitude, phase and other information, and then convert the time domain feature data into frequency domain feature data.

[0094] Step S104: Obtain time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data respectively.

[0095] It is worth noting that the time / space feature token includes the time domain features and spatial domain features of the pre-processed EEG data, and the frequency / space feature token includes the frequency features and spatial domain features of the pre-processed EEG data.

[0096] In some embodiments of the present application, a time / space feature token is obtained through a space-time branch based on the time domain feature data, the time domain feature data is input into the multi-scale time convolution layer of the space-time branch to obtain multi-scale time feature data, the multi-scale time feature data is input into the first spatial convolution layer of the space-time branch to obtain fused space-time feature data, the fused space-time feature data is convolutionally mapped, and the time / space feature token is obtained.

[0097] In this embodiment, for the time / space feature token, multi-scale one-dimensional time convolution is first applied to the time domain feature data, and then the space-time features are fused through one-dimensional space convolution, and then the fused space-time feature data is mapped into the time / space feature token through convolution.

[0098] In some embodiments of the present application, frequency / space feature tokens are obtained based on frequency domain feature data through the frequency-space branch, the frequency domain feature data is input into the multi-scale frequency convolution layer of the frequency-space branch to obtain multi-scale frequency feature data, the multi-scale frequency feature data is input into the second spatial convolution layer of the frequency-space branch to obtain fused frequency-space feature data, the fused frequency-space feature data is convolutionally mapped, and the frequency / space feature tokens are obtained.

[0099] In this embodiment, for frequency / space feature tokens, multi-scale one-dimensional frequency convolution is first applied to the frequency domain feature data, and then the frequency-space feature fusion is realized through one-dimensional spatial convolution, and then the fused frequency-space feature data is mapped into frequency / space feature tokens through convolution.

[0100] It is worth noting that in the space-time branch and the space-frequency branch, due to the differences in EEG data of different subjects, the setting of the convolution kernel size will have a certain impact on the EEG feature extraction of different subjects. The embodiment of the present application selects a more general convolution kernel size. The data after multi-scale time convolution and multi-scale frequency convolution are:

[0101] = + + + ,

[0102] = + + + ,

[0103] in, represents the temporal convolution kernel, Represents the frequency domain convolution kernel, and the size of both is (1 , nf s ), f s represents the sampling rate, n represents the convolution kernel size factor, Represents the data after the multi-scale temporal convolution layer, Represents the data after the multi-scale frequency convolution layer.

[0104] In this embodiment, the time / space feature tokens and the frequency / space feature tokens are position-encoded, which is expressed as:

[0105] ,

[0106] .

[0107] in, i Represents the position index, POS Indicates the position in the sequence, represents the hidden layer dimension, PE Indicates positional encoding.

[0108] S105: Perform feature interaction and fusion on the time / space feature token and the frequency / space token to obtain a fused feature token.

[0109] In some embodiments of the present application, Figure 2 As shown, feature interaction is performed on the time / space feature token and the frequency / space feature token through the self-attention block and the cross-covariance attention block to obtain the first multi-feature token and the second multi-feature token respectively, and the first multi-feature token and the second multi-feature token are spliced ​​based on the sample quantity dimension and the feature vector dimension respectively, and mapped to the original dimension of the preprocessed EEG data respectively to obtain the point dimension token and the vector dimension token, and the point dimension token and the vector dimension token are fused through the cross-covariance attention block to obtain the fused feature token.

[0110] In this example, given two input sequences and , where X and Y represent two different sequences, n Indicates the length of the sequence, d Represents the feature dimension. The input sequence is mapped into three different sequence vectors through the linear projection layer, namely the query vector Q, the key vector K and the value vector V. The transformation is as follows:

[0111] , ,

[0112] ,

[0113] in, represents the dimension of query vector and key vector, Represents the dimension of the value vector.

[0114] It is worth noting that when the query vector, key vector and value vector are all projected from the same sequence, it is the self-attention mechanism in the self-attention block adopted in the embodiment of the present application.

[0115] In some embodiments of the present application, median masking and multi-head calculation are introduced in the self-attention block, replacing the conventional masking operation with a median mask to reduce the impact of certain extreme values ​​in the EEG data. When the median is used as a threshold mask to calculate the attention weight of the query vector and the corresponding key vector, the positions below the median threshold are masked and the output weight is updated. The output of the self-attention block is represented as follows:

[0116] ( )]V，

[0117] Among them, Q represents the query vector sequence, K represents the key vector sequence, and V represents the value vector sequence. represents the dimension of the key vector sequence, Indicates median mask processing. Softmax Indicates normalization. Represents the output of the self-attention block.

[0118] It is worth noting that the attention weight is obtained by calculating the dot product of the query vector and the corresponding key vector. In the attention weight matrix obtained based on the query vector and the key vector, each element is compared with the median threshold calculated from the elements in that row. If the element is judged to be less than the threshold, the element is set to negative infinity. In this way, tokens with a weaker relationship with the query token can be shielded, so that tokens that are more relevant to the query occupy a larger proportion in the attention mechanism. In this way, the weights between related tokens can be enhanced.

[0119] In this embodiment, a self-attention block with a median mask is used as an encoder to convert the input feature sequence into a feature representation vector.

[0120] It's worth noting that the median mask effectively filters out key information from the input feature sequence and removes any noise or redundancy. By sorting the feature values ​​and selecting the median as the mask, the model focuses on representative features, improving the accuracy and efficiency of information processing. Compared to traditional encoder architectures, the self-attention block encoder with median masking not only excels in feature extraction and representation, but also boasts faster computation and improved scalability due to its unique masking mechanism and parallel computing capabilities. It demonstrates greater efficiency and stability when processing large-scale data or complex feature sequences.

[0121] In some embodiments of the present application, median masking and multi-head computation are introduced in the cross-covariance attention block.

[0122] In this embodiment, in the calculation of the covariance matrix and the eigenvector, a Euclidean normalization operation is used to ensure that each element of the query vector and the key vector falls within the range of [-1, 1].

[0123] ,

[0124] ,

[0125] Among them, Q represents the query vector sequence, K represents the key vector sequence, Represents the Euclidean normalization operation, which constrains each element of a vector sequence to be between -1 and 1.

[0126] The attention score is obtained by calculating the dot product of the normalized query vector sequence and the key vector sequence:

[0127] Attention scores= ,

[0128] ( )]V，

[0129] in, 、 They represent the query vector sequence and key vector sequence after Euclidean normalization, represents the dimension of the key vector sequence, Indicates median mask processing. Softmax Indicates normalization. Represents the output of the cross-covariance attention block.

[0130] It is worth noting that in the cross-covariance attention block, a median mask and scaling factor are used to keep the attention weights balanced and stable.

[0131] In some embodiments of the present application, a multi-head calculation method is also introduced:

[0132] ，

[0133] ,

[0134] in, head represents the head of each attention sequence, Concat Indicates multiple heads are spliced ​​together. Q 、 K 、 V Represent query vector, key vector, and value vector respectively, W o represents the output weight matrix, Represents the mapping result of linear transformation after multi-head splicing.

[0135] In this embodiment, the cross-covariance attention block with median mask and multi-head computation method is used as a decoder to generate a target sequence based on the feature representation vector generated by the encoder.

[0136] It's worth noting that multi-head computing effectively avoids information redundancy and insufficient information capture. Each independent head processes input data, then aggregates and maps the output information back to its original dimensions. This reduces the complexity of a single head and enables the capture of multiple features and relationships, improving the robustness and stability of the model. The multi-head attention mechanism further enhances feature fusion. By computing attention across multiple heads in parallel, each head can focus on different feature subspaces, extracting feature information from different perspectives. These features are then fused together, enriching the diversity of feature representations.

[0137] In this embodiment, in order to adapt to the small sample problem brought by EEG data, a parameter sharing mechanism is utilized when using parallel cross-attention feature interaction. This not only ensures the flow of information when different branches interact and captures the common features between different features, but also can significantly reduce the number of model parameters, reduce high memory requirements and the risk of overfitting.

[0138] It is worth noting that in some embodiments of the present application, in order to achieve information interaction between features, multiple cross-covariance attention blocks are applied.

[0139] In this embodiment, feature interaction is performed on the time / space feature tokens and the frequency / space feature tokens through the cross-covariance attention block to obtain the first multi-feature token and the second multi-feature token respectively. Each token obtains feature information in different domains, greatly expanding the dimension and diversity of the features.

[0140] In some embodiments of the present application, the first multi-feature token and the second multi-feature token are spliced ​​based on the sample quantity dimension and the feature vector dimension, respectively, and mapped to the original dimension of the preprocessed EEG data to obtain point dimension tokens and vector dimension tokens.

[0141] In this embodiment, by splicing in the sample quantity dimension, it is possible to integrate the information of feature data from different sources at the sample level, increase the richness of the data, provide more comprehensive sample information, and thus improve the ability to distinguish different samples. In the feature vector dimension splicing, it is possible to deeply fuse various feature vectors, making the feature expression richer and more powerful, and capable of capturing more subtle and complex feature combinations. The spliced ​​tokens are mapped back to the original dimension of the pre-processed EEG data. This process ensures the compatibility of the data with the original EEG data, so that the data can be efficiently processed in a familiar dimensional space, avoiding information loss or distortion due to dimensional differences.

[0142] In some embodiments of the present application, point dimension tokens and vector dimension tokens are interactively fused through a cross-covariance attention block to obtain a fused feature token.

[0143] It is worth noting that in this embodiment, a single-layer convolution is used instead of a linear mapping to aggregate the fused feature information, which can further increase the expressive power of the features. The global dependencies between features at different positions are captured by the self-attention block and the cross-covariance attention block. When fusing features, all features can be calculated in parallel across various positions in the sequence, rather than sequentially processed like traditional recurrent neural networks. This enables the model to quickly obtain the correlation between different features, thereby better fusing information.

[0144] Step S106: Perform a linear transformation on the fused feature tokens to obtain the final emotion classification result.

[0145] In some embodiments of the present application, the fused feature token is input into a fully connected layer for linear transformation, and the final emotion classification result is obtained based on the activation function and the fused feature token after linear transformation.

[0146] In this embodiment, three fully connected layers are used to perform linear transformation on the fused feature tokens, and after the linear transformation, the SoftMax activation function is used to obtain the weight matrix, where the weight matrix can be visualized.

[0147] It is worth noting that the three fully connected layers can gradually and deeply explore the potential information of the fused feature tokens. Through continuous linear transformations, the weights and combinations of features are constantly adjusted, allowing the correlation between features to be more fully reflected, thereby greatly enriching the expressive power of features. The SoftMax activation function converts the results of the linear transformation into a weight matrix in the form of a probability distribution. This approach gives the weight matrix a clear probabilistic meaning and makes the output more interpretable. The probability value represented by each element in the weight matrix clearly reflects the relative importance of each feature in the decision-making process.

[0148] Step S107: Visually display the emotion classification results.

[0149] In this embodiment, the visualization technology utilizes HTML+CSS+JavaScript technology, which can accept the prediction results of different ports and visualize the emotions of different subjects. Figure 4As shown, the method for visualizing emotion recognition results based on EEG data provided by the embodiment of the present application can be adapted to a variety of application scenarios, such as game scenarios, sports scenarios, etc. In the game scenario or sports scenario, there are a total of five dials in the figure, and each dial has a total of three rings, which visualize the data incoming per second. The dial has a total of 60 scales representing a total of 60 seconds of predicted results. After the dial displays the recognition results incoming for one minute, it will be refreshed and cleared. In order to facilitate observation, the scale size of the dial is only displayed at the visualization starting point. As shown in the large dial in the figure, the scale value range from 60 to 100 is displayed from the second ring to the third ring, and the scale range of 0-60 from the first ring to the second ring is hidden. Among the four small dials in the figure, driver1 represents the emotion recognition results of e-sports players and athletes, viewer1, viewer2, and viewer3 represent the emotion recognition results of the audience, and the large dial displays the visualization results of the four small dials together, which is more convenient for users to observe. For visualization, the color of the lines generated each second is determined by the colors represented by the three emotions in the diagram: excitement, focus, and fear. The color of the circles at the end of the lines matches the color of the emotion category, and the size of the circles is determined by the weight of the predicted emotion. It is worth noting that the emotional visualization of the five dials is synchronized, making it easier for observers to observe the emotional changes of each subject in real time. Displaying the results in the above application scenario helps esports players and athletes optimize their training plans and improve their performance to achieve optimal performance, ultimately providing a better viewing experience for the audience.

[0150] In an embodiment of the present application, the raw EEG data collected by the EEG device is first preprocessed to obtain preprocessed EEG data, thereby avoiding interference and noise in the raw EEG data that cause inconvenience to subsequent feature fusion; the preprocessed EEG data is spatially transformed to obtain time domain feature data, and frequency domain feature data is obtained based on the time domain feature data, and time / space feature tokens and frequency / space feature tokens are obtained based on the time domain feature data and the frequency domain feature data, thereby realizing the fusion of spatial feature data, time domain feature data and frequency domain feature data, and being able to capture more comprehensive features; feature interaction and fusion are performed on the time / space feature tokens and the frequency / space tokens to obtain fused feature tokens, median masking and multi-head computing are used, as well as feature interaction and fusion through self-attention blocks and cross-covariance attention blocks, thereby reducing the influence of outliers in the EEG data and effectively reducing the complexity of the calculation; the fused feature tokens are linearly transformed to obtain the final emotion classification result, and the emotion classification result is visualized. Displaying the emotion classification result helps to intuitively present the emotion distribution of different subjects and clearly observe the emotional changes of the subjects. This application can effectively solve the problem that traditional EEG signal emotion recognition methods ignore a large amount of potential and deep feature information in EEG data.

[0151] See also Figure 5 , shown is a structural diagram of the emotion recognition result visualization system based on EEG data provided in an embodiment of the second aspect of the present invention.

[0152] It should be noted that the method for visualizing emotion recognition results based on EEG data provided in the embodiments of the present application can be executed by a system for visualizing emotion recognition results based on EEG data, or by a control module in the system for executing and loading the method for visualizing emotion recognition results based on EEG data. In the embodiments of the present application, the method for visualizing emotion recognition results based on EEG data provided in the embodiments of the present application is explained by taking an example of a system for visualizing emotion recognition results based on EEG data executing the method for visualizing emotion recognition results based on EEG data.

[0153] The EEG-based emotion recognition result visualization system in the embodiments of the present application can be a system, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the non-mobile electronic device can be a server, a network attached storage (NAS), or a personal computer (PC), etc., which are not specifically limited in the embodiments of the present application.

[0154] The EEG-based emotion recognition result visualization system in the embodiments of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0155] The emotion recognition result visualization system based on EEG data provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the method embodiment will not be described again here.

[0156] In some embodiments of the present application, the above-mentioned EEG data-based emotion recognition result visualization system includes:

[0157] The spatial transformation module 201 is used to preprocess the original EEG data collected by the EEG device to obtain preprocessed EEG data, and perform spatial transformation on the preprocessed EEG data to obtain time domain feature data; wherein the time domain feature data is a vector sequence including the spatial features of the preprocessed EEG data.

[0158] The multi-scale convolution mapping module 202 is used to obtain frequency domain feature data based on time domain feature data, and to obtain time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data respectively; wherein the time / space feature tokens include the time domain features and spatial domain features of the preprocessed EEG data, and the frequency / space feature tokens include the frequency features and spatial domain features of the preprocessed EEG data.

[0159] The feature interaction and fusion module 203 is used to perform feature interaction and fusion on the time / space feature tokens and the frequency / space feature tokens to obtain fused feature tokens.

[0160] The classification module 204 is used to perform linear transformation on the fused feature tokens to obtain the final emotion classification result.

[0161] The visualization module 205 is used to visualize the final emotion classification result.

[0162] It is worth noting that the EEG data-based emotion recognition result visualization system provided in the embodiment of the present application can also be trained to obtain the optimal system.

[0163] In this embodiment, a dataset is obtained and rationally divided into a training set and a validation set. The training set is input into an EEG-based emotion recognition visualization system. Based on the data provided by the training set, the system parameters are adjusted. A backpropagation algorithm is used to calculate the error between the predicted results and the true labels. Based on this error, the system parameters are optimized, gradually improving the system's ability to recognize the emotional information contained in the EEG data. The validation set data is input into the trained system and the predicted results are output. An evaluation metric is used to determine the accuracy of the trained system's predictions, and the system with the highest accuracy is selected as the optimal system.

[0164] It's worth noting that the dataset used to train the EEG-based emotion recognition visualization system in this embodiment of the present invention is publicly available. This publicly available dataset undergoes a rigorous collection and annotation process, ensuring data quality and effectively improving the accuracy and reliability of the training model. This also reduces research costs to a certain extent.

[0165] In an embodiment of the present application, the raw EEG data collected by the EEG device is first preprocessed to obtain preprocessed EEG data, thereby avoiding the inconvenience caused by interference and noise in the raw EEG data to subsequent feature fusion; the preprocessed EEG data is spatially transformed to obtain time domain feature data, and frequency domain feature data is obtained based on the time domain feature data, and time / space feature tokens and frequency / space feature tokens are obtained based on the time domain feature data and the frequency domain feature data, thereby realizing the fusion of spatial feature data, time domain feature data and frequency domain feature data, and being able to capture more comprehensive features; feature interaction and fusion are performed on the time / space feature tokens and the frequency / space tokens to obtain fused feature tokens, and median masking and multi-head calculation are adopted, as well as feature interaction and fusion through self-attention blocks and cross-covariance attention blocks to reduce the influence of outliers in the EEG data and effectively reduce the complexity of the calculation; the fused feature tokens are linearly transformed to obtain the final emotion classification result, and the emotion classification result is visualized. Displaying the emotion classification result helps to intuitively present the emotion distribution of different subjects and clearly observe the emotional changes of the subjects. This application can effectively solve the problem that traditional EEG signal emotion recognition methods ignore a large amount of potential and deep feature information in EEG data.

[0166] See also Figure 6, shown is an embodiment of the third aspect of the present application, which provides an electronic device, including a processor 301, a memory 302, and a computer program 303 stored in the memory and capable of running on the processor. When the computer program 303 is executed by the processor, the various processes of the embodiment of the above-mentioned method for visualizing emotion recognition results based on EEG data are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0167] An embodiment of the fourth aspect of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned embodiment of the method for visualizing emotion recognition results based on EEG data are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0168] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0169] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0170] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0171] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0172] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for visualizing emotion recognition results based on EEG data, characterized in that: The method comprises: Preprocessing the raw EEG data collected by the EEG device to obtain preprocessed EEG data; Performing spatial transformation on the pre-processed EEG data to obtain time domain feature data, wherein the time domain feature data is a vector sequence including the time features of the pre-processed EEG data; Acquiring frequency domain feature data according to the time domain feature data; Based on the time domain feature data and the frequency domain feature data, a time-space feature token and a frequency-space feature token are respectively obtained; wherein the time-space feature token includes the time domain feature and the spatial domain feature of the pre-processed EEG data, and the frequency-space feature token includes the frequency feature and the spatial domain feature of the pre-processed EEG data; Performing feature interaction on the time-space feature token and the frequency-space feature token through a self-attention block and a cross-covariance attention block to obtain a first multi-feature token and a second multi-feature token, respectively, wherein the first multi-feature token and the second multi-feature token both include the time domain features, frequency domain features, and spatial domain features of the preprocessed EEG data; The first multi-feature token and the second multi-feature token are respectively concatenated based on the sample quantity dimension and the feature vector dimension, and are respectively mapped to the original dimension of the preprocessed EEG data to obtain a point dimension token and a vector dimension token; Fusing the point dimension token and the vector dimension token through the cross-covariance attention block to obtain a fused feature token; The fused feature tokens are input into the fully connected layer for linear transformation, and the final emotion classification results are obtained based on the activation function and the fused feature tokens after linear transformation; The final emotion classification result is visualized.

2. The method for visualizing emotion recognition results based on EEG data according to claim 1, characterized in that: The step of preprocessing the raw EEG data collected by the EEG device to obtain preprocessed EEG data specifically includes: Filtering the raw EEG data to obtain filtered EEG data; Normalizing the filtered EEG data to obtain the pre-processed EEG data; The normalization function expression in the normalization process is: ， in, represents the filtered EEG data, represents the mean value of the filtered EEG data, represents the variance of the filtered EEG data, represents the normalized EEG data.

3. The method for visualizing emotion recognition results based on EEG data according to claim 1, characterized in that: The step of obtaining a time-space feature token and a frequency-space feature token based on the time domain feature data and the frequency domain feature data specifically includes: Acquire the time-space feature token through time-space branching based on the time domain feature data; Inputting the time domain feature data into the multi-scale time convolution layer of the space-time branch to obtain multi-scale time feature data; Inputting the multi-scale temporal feature data into the first spatial convolution layer of the spatiotemporal branch to obtain fused spatiotemporal feature data; Performing convolution mapping on the fused spatiotemporal feature data to obtain the time-space feature token; Acquire the frequency-space feature token based on the frequency domain feature data through the frequency-space branch; Inputting the frequency domain feature data into the multi-scale frequency convolution layer of the frequency-space branch to obtain multi-scale frequency feature data; Inputting the multi-scale frequency feature data into the second spatial convolution layer of the frequency-space branch to obtain fused frequency-space feature data; Convolution mapping is performed on the fused frequency-space feature data to obtain the frequency-space feature token.

4. The method for visualizing emotion recognition results based on EEG data according to claim 1, characterized in that: The self-attention block and the cross-covariance attention block also include: The self-attention block and the cross-covariance attention block introduce median mask and multi-head calculation methods; The median mask is used to reduce the influence of extreme values ​​in the pre-processed EEG data; The long calculation method is: ， ， in, represents the head of each attention sequence, Indicates multiple heads are spliced ​​together. represents the output weight matrix, Represents the mapping result of linear transformation after multi-head splicing.

5. The method for visualizing emotion recognition results based on EEG data according to claim 4, characterized in that: The self-attention block and the cross-covariance attention block also include: The self-attention block acts as an encoder to convert the input feature sequence into a feature representation vector; The cross-covariance attention block acts as a decoder to generate a target sequence based on the feature representation vector; The output expression of the cross-covariance attention block is: , , , in, represents the query vector sequence, represents a sequence of key vectors, represents a sequence of value vectors, represents the dimension of the key vector sequence, Restricts each element of the vector sequence to be between -1 and 1. represents the attention score of the cross-covariance mechanism, Indicates median mask processing. Indicates normalization. Represents the output of the cross-covariance attention block.

6. A visualization system for emotion recognition results based on EEG data, characterized in that: The system comprises: A spatial transformation module is used to preprocess the raw EEG data collected by the EEG device to obtain preprocessed EEG data, and to perform spatial transformation on the preprocessed EEG data to obtain time domain feature data; wherein the time domain feature data is a vector sequence including the spatial features of the preprocessed EEG data; a multi-scale convolution mapping module, configured to obtain frequency domain feature data according to the time domain feature data, and to obtain time-space feature tokens and frequency-space feature tokens based on the time domain feature data and the frequency domain feature data; wherein the time-space feature tokens include the time domain features and spatial domain features of the pre-processed EEG data, and the frequency-space feature tokens include the frequency features and spatial domain features of the pre-processed EEG data; A feature interaction and fusion module is used to perform feature interaction on the time-space feature token and the frequency-space feature token through a self-attention block and a cross-covariance attention block to obtain a first multi-feature token and a second multi-feature token respectively, wherein the first multi-feature token and the second multi-feature token both include the time domain features, frequency domain features and spatial domain features of the pre-processed EEG data; the first multi-feature token and the second multi-feature token are respectively spliced ​​based on the sample quantity dimension and the feature vector dimension, and are respectively mapped to the original dimension of the pre-processed EEG data to obtain a point dimension token and a vector dimension token; the point dimension token and the vector dimension token are fused through the cross-covariance attention block to obtain a fused feature token; The classification module is used to input the fused feature tokens into the fully connected layer for linear transformation, and obtain the final emotion classification results based on the activation function and the fused feature tokens after linear transformation; The visualization module is used to visualize the final emotion classification result.

7. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method for visualizing emotion recognition results based on EEG data are implemented as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed by the processor, the steps of the method for visualizing emotion recognition results based on EEG data as described in any one of claims 1 to 5 are implemented.

Images