Emotion recognition result visualization method and system based on electroencephalogram data

Abstract

The invention discloses an emotion recognition result visualization method and system based on electroencephalogram data, and belongs to the technical field of emotion recognition. The method comprises the following steps: preprocessing original electroencephalogram data to obtain preprocessed electroencephalogram data; performing spatial transformation on the preprocessed electroencephalogram data to obtain time domain feature data; acquiring frequency domain feature data according to the time domain feature data; acquiring a time / space feature token and a frequency / space feature token based on the time domain feature data and the frequency domain feature data; performing feature interaction and fusion on the time / space feature token and the frequency / space feature token to obtain a fused feature token; performing linear transformation on the fusion feature token to obtain a final emotion classification result; and performing visual display on the emotion classification result. According to the method, the problem that a large amount of potential and deep feature information in electroencephalogram data is ignored in a traditional electroencephalogram signal emotion recognition method can be effectively solved, and emotion result visualization is beneficial to watching an emotion classification result by a user.

Description

Technical Field

[0001] The present application belongs to the technical field of emotion recognition, and specifically relates to a method and system for visualizing emotion recognition results based on electroencephalogram data. Background Art

[0002] EEG data is a record of electrical signals generated by the activity of brain neurons. There are hundreds of millions of neurons in the brain, which transmit information through electrical activity. When neurons are excited, ions flow across the membrane, forming a potential difference. The potential difference of many neurons is combined to form an EEG signal that can be detected on the surface of the scalp. This signal is essentially a time series data that changes over time and reflects the emotional state of the brain at different times.

[0003] In traditional EEG signal emotion recognition methods or systems, the time domain features, frequency domain features, and spatial domain features of EEG data are fused to achieve a comprehensive description of the EEG signal characteristics corresponding to emotions through feature fusion from different perspectives. However, traditional methods have serious limitations in feature extraction. Although the time domain, frequency domain, and spatial domain features are fused, they only stay at the utilization of common and basic features, ignoring a large amount of potential and deep-level feature information in EEG signals, such as the complex nonlinear coupling relationship between frequency bands in EEG signals, the dynamically changing functional connection patterns between different brain regions, and the time-varying features under different cognitive tasks and emotional states. These potential features are of great significance for more accurate emotion recognition, but they are ignored by traditional methods. The results predicted by traditional emotion recognition methods or systems are often not intuitive, which makes it inconvenient for users to observe and analyze the recognition results. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and system for visualizing emotion recognition results based on EEG data, which can solve the problem that traditional EEG signal emotion recognition methods ignore a large amount of potential and deep feature information in EEG data.

[0005] In order to solve the above technical problems, this application is implemented as follows: In a first aspect, an embodiment of the present application provides a method for visualizing emotion recognition results based on EEG data, the method comprising: Preprocessing the raw EEG data collected by the EEG device to obtain preprocessed EEG data; Performing spatial transformation on the pre-processed EEG data to obtain time domain feature data, wherein the time domain feature data is a vector sequence including the time features of the pre-processed EEG data; Obtain frequency domain feature data according to time domain feature data; Based on the time domain feature data and the frequency domain feature data, respectively, a time / space feature token and a frequency / space feature token are obtained, wherein the time / space feature token includes the time domain feature and the spatial domain feature of the preprocessed EEG data, and the frequency / space feature token includes the frequency feature and the spatial domain feature of the preprocessed EEG data; Perform feature interaction and fusion on the time / space feature token and the frequency / space feature token to obtain a fused feature token; Perform linear transformation on the fused feature tokens to obtain the final emotion classification result; Visualize the emotion classification results.

[0006] Furthermore, the step of preprocessing the raw EEG data collected by the EEG device to obtain preprocessed EEG data specifically includes: Filtering the original EEG data to obtain filtered EEG data; Normalize the filtered EEG data to obtain pre-processed EEG data; Among them, the normalization function expression in the normalization process is: , in, represents the filtered EEG data, represents the mean value of the filtered EEG data, represents the variance of the filtered EEG data, Represents the normalized EEG data.

[0007] Further, in the step of obtaining time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data, it specifically includes: Obtaining time / space feature tokens through time-space branches based on time-domain feature data; Input the time domain feature data into the multi-scale time convolution layer of the space-time branch to obtain multi-scale time feature data; Input the multi-scale time feature data into the first spatial convolution layer of the space-time branch to obtain fused space-time feature data; Perform convolution mapping on the fused spatiotemporal feature data to obtain time / space feature tokens; Obtain frequency / space feature tokens through frequency-space branches based on frequency-domain feature data; Input the frequency domain feature data into the multi-scale frequency convolution layer of the frequency-space branch to obtain multi-scale frequency feature data; Input the multi-scale frequency feature data into the second spatial convolution layer of the frequency-space branch to obtain fused frequency-space feature data; Perform convolution mapping on the fused frequency-space feature data to obtain frequency / space feature tokens.

[0008] Furthermore, in the step of performing feature interaction and fusion on the time / space feature token and the frequency / space feature token to obtain the fused feature token, the following steps are specifically included: Perform feature interaction on the time / space feature token and the frequency / space feature token through a self-attention block and a cross-covariance attention block to obtain a first multi-feature token and a second multi-feature token, respectively, wherein the first multi-feature token and the second multi-feature token both include time domain features, frequency domain features, and spatial domain features of the pre-processed EEG data; The first multi-feature token and the second multi-feature token are concatenated based on the sample quantity dimension and the feature vector dimension, respectively, and are mapped to the original dimension of the preprocessed EEG data to obtain a point dimension token and a vector dimension token; Through the cross-covariance attention block, the point dimension tokens and the vector dimension tokens are fused to obtain the fused feature tokens.

[0009] Furthermore, the self-attention block and the cross-covariance attention block also include: Self-attention block and cross-covariance attention block introduce median mask and multi-head calculation methods; Among them, the median mask is used to reduce the impact of extreme values ​​in the pre-processed EEG data; The long position calculation method is: ， , in, head represents the head of each attention sequence, Concat It means multiple heads are connected together. Q , K , V Represent the query vector, key vector, and value vector respectively. W o represents the output weight matrix, It represents the mapping result after linear transformation of multiple heads.

[0010] Furthermore, the self-attention block and the cross-covariance attention block also include: The self-attention module acts as an encoder to convert the input feature sequence into a feature representation vector; The cross-covariance attention block is used as a decoder to generate the target sequence based on the feature representation vector; The output expression of the cross-covariance attention block is: , , ( )]V， Among them, Q represents the query vector sequence, K represents the key vector sequence, and V represents the value vector sequence. represents the dimension of the key vector sequence, It means that each element of the vector sequence is restricted to be between -1 and 1. represents the attention score of the cross-covariance mechanism, Indicates median mask processing. Softmax Indicates normalization. Represents the output of the cross-covariance attention block.

[0011] Furthermore, in the step of performing linear transformation on the fused feature tokens to obtain the final emotion classification result, the following steps are specifically included: Input the fused feature tokens into the fully connected layer for linear transformation; The final emotion classification result is obtained based on the fused feature tokens after activation function and linear transformation.

[0012] In a second aspect, an embodiment of the present application provides a system for visualizing emotion recognition results based on EEG data, the system comprising: A spatial transformation module is used to pre-process the raw EEG data collected by the EEG device to obtain pre-processed EEG data, and to perform spatial transformation on the pre-processed EEG data to obtain time domain feature data; wherein the time domain feature data is a vector sequence including the spatial features of the pre-processed EEG data; A multi-scale convolution mapping module, used to obtain frequency domain feature data according to time domain feature data, and to obtain time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data, respectively; wherein the time / space feature tokens include the time domain features and the spatial domain features of the pre-processed EEG data, and the frequency / space feature tokens include the frequency features and the spatial domain features of the pre-processed EEG data; A feature interaction and fusion module is used to perform feature interaction and fusion on time / space feature tokens and frequency / space feature tokens to obtain fused feature tokens; The classification module is used to linearly transform the fused feature tokens to obtain the final emotion classification results; The visualization module is used to visualize the final emotion classification results.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In an embodiment of the present application, the raw EEG data collected by the EEG device is first preprocessed to obtain preprocessed EEG data, thereby avoiding interference and noise in the raw EEG data from causing inconvenience to subsequent feature fusion; the preprocessed EEG data is spatially transformed to obtain time domain feature data, and frequency domain feature data is obtained based on the time domain feature data, and time / space feature tokens and frequency / space feature tokens are obtained based on the time domain feature data and the frequency domain feature data, thereby realizing the fusion of spatial domain feature data, time domain feature data and frequency domain feature data, and being able to capture more comprehensive features; feature interaction and fusion are performed on the time / space feature tokens and the frequency / space tokens to obtain fused feature tokens, median masking and multi-head calculation are used, as well as feature interaction and fusion through self-attention blocks and cross-covariance attention blocks, thereby reducing the influence of outliers in the EEG data and effectively reducing the complexity of the calculation; the fused feature tokens are linearly transformed to obtain the final emotion classification result, and the emotion classification result is visualized. Displaying the emotion classification result helps to intuitively present the emotion distribution of different subjects and clearly observe the emotional changes of the subjects. This application can effectively solve the problem that traditional EEG signal emotion recognition methods ignore a large amount of potential and deep feature information in EEG data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart showing an implementation method of the method for visualizing emotion recognition results based on EEG data provided by an embodiment of the present invention; Figure 2 An implementation method of the method for visualizing emotion recognition results based on EEG data provided by an embodiment of the present invention is shown; Figure 3 The original EEG data acquisition interface provided by the embodiment of the present invention is shown; Figure 4 The visualization interface of the final emotion recognition result provided by the embodiment of the present invention is shown; Figure 5 A schematic diagram showing the structure of a system for visualizing emotion recognition results based on EEG data provided by an embodiment of the present invention is shown; Figure 6 A schematic diagram of the structure of an electronic device using a method for visualizing emotion recognition results based on electroencephalogram data provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.

[0019] In conjunction with the accompanying drawings, the method and system for visualizing emotion recognition results based on EEG data provided by the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0020] It is worth mentioning that the method of EEG emotion recognition is widely used in many industries. In the medical field, it assists doctors to intuitively diagnose mental illnesses, accurately grasp patients' emotional fluctuations, and evaluate treatment effects in real time, so as to formulate and adjust more effective treatment plans. In educational scenarios, teachers can use this emotion recognition technology to understand students' classroom emotions in real time, gain insight into students' learning status, optimize teaching strategies in a timely manner, and provide personalized support for special education students. In the field of human-computer interaction, it helps intelligent product designers optimize product design and improve user experience based on user EEG emotional feedback, while making the interaction between emotional robots and humans more natural and more in tune.

[0021] The method for visualizing emotion recognition results based on EEG data disclosed in this application can grasp a large amount of potential and deep-level feature information in EEG data, thereby achieving an objective assessment of the individual's emotional state. By real-time monitoring and analyzing the brain's electrical activity signals, the three emotions of excitement, concentration, and fear can be quickly identified without being affected by subjective factors, and have high accuracy and reliability.

[0022] See also Figures 1 to 4 , which shows an implementation of a method for visualizing emotion recognition results based on EEG data provided by an embodiment of the first aspect of the present invention, the method comprises steps S101 to S107, wherein: S101, preprocessing the original EEG data collected by the EEG device to obtain preprocessed EEG data.

[0023] In this embodiment, the raw EEG data is collected and recorded by a brain-computer device, such as Figure 3 The figure shows the raw EEG data acquisition interface provided by the embodiment of the present application.

[0024] It is worth noting that the brain-computer device used in the embodiment of the present application can simultaneously collect and record the EEG data of multiple subjects.

[0025] In some embodiments of the present application, the original EEG data is filtered to obtain filtered EEG data, and the filtered EEG data is normalized.

[0026] Among them, the normalization function expression in the normalization process is: , in, represents the filtered EEG data, represents the mean value of the filtered EEG data, represents the variance of the filtered EEG data, Represents the normalized EEG data.

[0027] It is worth noting that the dimensions of the filtered EEG data and the preprocessed EEG data are both C eeg ×T, where C eeg represents the number of channels, and T represents the number of time points.

[0028] In this embodiment, the original EEG data is filtered by a sixth-order Chebyshev filter to remove high-frequency and low-frequency components in the original EEG data.

[0029] It is worth mentioning that the sixth-order Chebyshev filter is a filter with excellent performance. The Chebyshev filter is constructed based on Chebyshev polynomials, presents equiripple characteristics in the passband, and has a high attenuation rate in the stopband, which can effectively suppress unwanted frequency components, and the higher the order, the stronger the stopband attenuation ability. At the same time, compared with some other filters, the Chebyshev filter has a narrower transition band, which can achieve rapid attenuation changes between the passband and the stopband, thereby accurately screening out the required frequency signal.

[0030] In this embodiment, the Z-score normalization function is used to reduce the non-stationarity and variability of the original EEG data.

[0031] It is worth noting that the Z-score normalization function, also known as the standardization function, is a data preprocessing technology. The Z-score normalization function can effectively eliminate the dimensional effects between different features, unify data of different units and orders of magnitude to the same scale, and avoid over-emphasis or neglect of certain features due to dimensional differences in data analysis and model training, thereby significantly improving the performance of machine learning models. In addition, the normalized data is at the same scale, which facilitates the comparison of different features and can reasonably display the information of each feature when visualizing the data.

[0032] Step S102: perform spatial transformation on the pre-processed EEG data to obtain time domain feature data.

[0033] In some embodiments of the present application, the time domain feature data is a vector sequence including the time features of the pre-processed EEG data.

[0034] In this embodiment, if Figure 2 As shown in the figure, the pre-processed EEG data is spatially transformed by the Multi-CSP method to obtain spatial domain feature data. After completing the pre-processing steps of filtering, artifact removal, and normalization of the EEG data, the Multi-CSP method constructs a spatial filter to effectively distinguish the EEG signals in different states in terms of spatial distribution, thereby maximizing the variance difference. It is worth noting that after the spatial transformation, the spatial domain feature data can not only improve the discrimination between EEG data in different states, but also retain the temporal characteristics of the EEG data. Then, the spatial domain feature data is further feature extracted to obtain time domain feature data such as mean, variance, number of zero crossings, peak amplitude, and waveform slope. These feature data are of great significance for in-depth exploration of the laws of brain electrical activity and assisting in brain state classification.

[0035] It is worth mentioning that the Multi-CSP method is an extension and improvement of the traditional constraint satisfaction problem method (hereinafter referred to as the CSP method). In multi-channel scenarios, the traditional CSP method often finds it difficult to take into account the complex correlations and characteristic differences between different channels, while the Multi-CSP method can effectively integrate the information of each channel and independently extract the spatial features of each frequency band or time period in the multi-channel EEG signal, thus solving the limitations of the original CSP method. The Multi-CSP method can better extract the discriminative features in EEG signals through segmentation, frequency division and feature integration.

[0036] In this embodiment, a one-versus-rest strategy (OVR strategy for short) is introduced to expand the performance of Multi-CSP, thereby using a multi-category spatial filter to perform dimensionality reduction processing on EEG data.

[0037] It is worth mentioning that the OVR strategy is a method for multi-classification problems. Its core idea is to transform the multi-classification problem into multiple binary classification problems, taking one of the categories as the positive class and all the other categories as the negative class, thus forming a binary classification problem for each feature.

[0038] In this embodiment, for each binary classification problem, a corresponding binary classification spatial filter is generated using the Multi-CSP method, and then different binary classification spatial filters are spliced ​​to obtain a multi-classification spatial filter.

[0039] Step S103: Obtain frequency domain feature data according to the time domain feature data.

[0040] In this embodiment, the time domain feature data is converted into frequency domain feature data by using the Fourier transform principle.

[0041] It is worth mentioning that in the process of data analysis and processing, for time domain feature data recorded with time as the variable, Fourier transform can decompose the time domain data that originally shows the characteristics of time dimension changes into a form composed of sine waves and cosine waves of different frequencies, and then clearly present the various frequency components contained in these data and their corresponding amplitude, phase and other information, and then convert the time domain feature data into frequency domain feature data.

[0042] Step S104: Obtain time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data respectively.

[0043] It is worth noting that the time / space feature token includes the time domain features and spatial domain features of the pre-processed EEG data, and the frequency / space feature token includes the frequency features and spatial domain features of the pre-processed EEG data.

[0044] In some embodiments of the present application, a time / space feature token is obtained through a space-time branch based on time domain feature data, the time domain feature data is input into a multi-scale time convolution layer of the space-time branch to obtain multi-scale time feature data, the multi-scale time feature data is input into the first spatial convolution layer of the space-time branch to obtain fused space-time feature data, convolution mapping is performed on the fused space-time feature data, and the time / space feature token is obtained.

[0045] In this embodiment, for the time / space feature token, a multi-scale one-dimensional time convolution is first applied to the time domain feature data, and then the space-time features are fused through one-dimensional space convolution, and then the fused space-time feature data is mapped into the time / space feature token through convolution.

[0046] In some embodiments of the present application, frequency / space feature tokens are obtained based on frequency domain feature data through the frequency-space branch, the frequency domain feature data is input into the multi-scale frequency convolution layer of the frequency-space branch to obtain multi-scale frequency feature data, the multi-scale frequency feature data is input into the second spatial convolution layer of the frequency-space branch to obtain fused frequency-space feature data, convolution mapping is performed on the fused frequency-space feature data, and frequency / space feature tokens are obtained.

[0047] In this embodiment, for frequency / space feature tokens, multi-scale one-dimensional frequency convolution is first applied to the frequency domain feature data, and then the frequency-space feature fusion is realized through one-dimensional spatial convolution, and then the fused frequency-space feature data is mapped into frequency / space feature tokens through convolution.

[0048] It is worth noting that in the space-time branch and the frequency-space branch, due to the differences in EEG data of different subjects, the setting of the convolution kernel size will have a certain impact on the EEG feature extraction of different subjects. The embodiment of the present application selects a more general convolution kernel size. The data after multi-scale time convolution and multi-scale frequency convolution are: = + + + , = + + + , in, represents the temporal convolution kernel, represents the frequency domain convolution kernel, and the size of both is (1 , nf s ), f s represents the sampling rate, n represents the convolution kernel size factor, Represents the data after the multi-scale temporal convolution layer, Represents the data after the multi-scale frequency convolution layer.

[0049] In this embodiment, the time / space feature tokens and the frequency / space feature tokens are position-encoded, which is expressed as: , .

[0050] in, i Represents the position index, POS represents the position in the sequence, represents the hidden layer dimension, PE Represents positional encoding.

[0051] S105: Perform feature interaction and fusion on the time / space feature token and the frequency / space token to obtain a fused feature token.

[0052] In some embodiments of the present application, Figure 2 As shown, feature interaction is performed on the time / space feature tokens and the frequency / space feature tokens through the self-attention block and the cross-covariance attention block to obtain the first multi-feature token and the second multi-feature token respectively, the first multi-feature token and the second multi-feature token are spliced ​​based on the sample quantity dimension and the feature vector dimension respectively, and are mapped to the original dimension of the preprocessed EEG data respectively to obtain the point dimension token and the vector dimension token, and the point dimension token and the vector dimension token are fused through the cross-covariance attention block to obtain the fused feature token.

[0053] In this example, given two input sequences and , where X and Y represent two different sequences, n represents the length of the sequence, d Represents the feature dimension. The input sequence is mapped into three different sequence vectors through the linear projection layer, namely the query vector Q, the key vector K and the value vector V. The transformation is as follows: , , , in, represents the dimension of query vector and key vector, Represents the dimension of the value vector.

[0054] It is worth noting that when the query vector, key vector and value vector are all projected from the same sequence, it is the self-attention mechanism in the self-attention block adopted in the embodiment of the present application.

[0055] In some embodiments of the present application, median masking and multi-head calculation are introduced in the self-attention block, and the conventional masking operation is replaced by the median masking to reduce the impact of certain extreme values ​​in the EEG data. When the median is used as a threshold mask to calculate the attention weight of the query vector and the corresponding key vector, the positions below the median threshold are masked and the output weight is updated. The output of the self-attention block is represented as follows: ( )]V， Among them, Q represents the query vector sequence, K represents the key vector sequence, and V represents the value vector sequence. represents the dimension of the key vector sequence, Indicates median mask processing. Softmax Indicates normalization. Represents the output of the self-attention block.

[0056] It is worth noting that the attention weight is obtained by calculating the dot product of the query vector and the corresponding key vector. In the attention weight matrix obtained based on the query vector and the key vector, each element is compared with the median threshold calculated from the elements of the row. If the element is judged to be less than the threshold, the element is set to negative infinity. In this way, tokens with a weaker relationship with the query token can be shielded, so that tokens that are more relevant to the query occupy a larger proportion in the attention mechanism. In this way, the weights between related tokens can be enhanced.

[0057] In this embodiment, a self-attention block with a median mask is used as an encoder to convert an input feature sequence into a feature representation vector.

[0058] It is worth noting that the existence of the median mask can effectively filter out the key information in the input feature sequence and remove possible noise or redundant parts. It sorts the feature values ​​and selects the median as the basis for the mask, so that the model focuses more on representative features and improves the accuracy and efficiency of information processing. Compared with the traditional encoder structure, the self-attention block encoder with median mask is not only better in feature extraction and representation capabilities, but also has faster computing speed and better scalability due to its unique masking mechanism and parallel computing characteristics. When processing large-scale data or complex feature sequences, it can show higher efficiency and stability.

[0059] In some embodiments of the present application, median mask and multi-head computation are introduced in the cross-covariance attention block.

[0060] In this embodiment, in the calculation of the covariance matrix and the characteristic vector, the Euclidean normalization operation is used to make each element of the query vector and the key vector fall within the range of [-1, 1].

[0061] , , Among them, Q represents the query vector sequence, K represents the key vector sequence, Represents the Euclidean normalization operation, which constrains each element of the vector sequence to be between -1 and 1.

[0062] The attention score is obtained by calculating the dot product of the normalized query vector sequence and the key vector sequence: Attention scores= , ( )]V， in, , They represent the query vector sequence and key vector sequence after Euclidean normalization, represents the dimension of the key vector sequence, Indicates median mask processing. Softmax Indicates normalization. Represents the output of the cross-covariance attention block.

[0063] It is worth mentioning that in the cross-covariance attention block, the median mask and scaling factor are used to keep the attention weights balanced and stable.

[0064] In some embodiments of the present application, a multi-head calculation method is also introduced: ， , in, head represents the head of each attention sequence, Concat It means multiple heads are connected together. Q , K , V Represent the query vector, key vector, and value vector respectively. W o represents the output weight matrix, It represents the mapping result after linear transformation of multiple heads.

[0065] In this embodiment, the cross-covariance attention block with median mask and multi-head computation method is used as a decoder to generate a target sequence based on the feature representation vector generated by the encoder.

[0066] It is worth noting that multi-head computing can effectively avoid information redundancy and insufficient information capture. The input data is processed in each independent head, and the output information of each head is finally aggregated and mapped into the original dimension. This reduces the complexity of a single head and can capture multiple features and relationships to improve the robustness and stability of the model. The multi-head attention mechanism further enhances the effect of feature fusion. By calculating attention in parallel with multiple heads, each head can focus on different feature subspaces, extract feature information from different angles, and then fuse the features from different perspectives, enriching the diversity of feature representation.

[0067] In this embodiment, in order to adapt to the small sample problem brought by EEG data, a parameter sharing mechanism is utilized when using parallel cross-attention feature interactions, which not only ensures the flow of information when different branches interact and captures the common features between different features, but also can significantly reduce the number of model parameters, reduce high memory requirements and the risk of overfitting.

[0068] It is worth noting that in some embodiments of the present application, in order to achieve the interaction of information between features, multiple cross-covariance attention blocks are applied.

[0069] In this embodiment, feature interaction is performed on the time / space feature tokens and the frequency / space feature tokens through the cross-covariance attention block to obtain the first multi-feature token and the second multi-feature token respectively. Each token obtains feature information in different domains, greatly expanding the dimension and diversity of the features.

[0070] In some embodiments of the present application, the first multi-feature token and the second multi-feature token are concatenated based on the sample quantity dimension and the feature vector dimension, respectively, and mapped to the original dimension of the preprocessed EEG data to obtain point dimension tokens and vector dimension tokens.

[0071] In this embodiment, by splicing in the sample quantity dimension, it is possible to integrate the information of feature data from different sources at the sample level, increase the richness of the data, provide more comprehensive sample information, and thus enhance the ability to distinguish different samples. When splicing in the feature vector dimension, various feature vectors can be deeply integrated to make the feature expression richer and more powerful, and to capture more subtle and complex feature combinations. The spliced ​​tokens are mapped back to the original dimensions of the preprocessed EEG data. This process ensures the compatibility of the data with the original EEG data, so that the data can be efficiently processed in a familiar dimensional space, avoiding information loss or distortion due to dimensional differences.

[0072] In some embodiments of the present application, point dimension tokens and vector dimension tokens are interactively fused through a cross-covariance attention block to obtain a fused feature token.

[0073] It is worth noting that in this embodiment, a single-layer convolution is used instead of a linear mapping to aggregate the fused feature information, which can further increase the expressiveness of the features. The global dependencies between features at different positions are captured by self-attention blocks and cross-covariance attention blocks. When fusing features, all features can be calculated in parallel across various positions of the sequence, rather than sequentially processed like traditional recurrent neural networks, which enables the model to quickly obtain the correlation between different features, thereby better fusing information.

[0074] Step S106: linearly transform the fused feature tokens to obtain the final emotion classification result.

[0075] In some embodiments of the present application, the fused feature token is input into the fully connected layer for linear transformation, and the final emotion classification result is obtained based on the activation function and the fused feature token after linear transformation.

[0076] In this embodiment, three fully connected layers are used to perform linear transformation on the fused feature tokens, and after the linear transformation, the SoftMax activation function is used to obtain the weight matrix, wherein the weight matrix can be visualized.

[0077] It is worth noting that the three fully connected layers can gradually dig deeper into the potential information of the fused feature tokens, and through continuous linear transformation, constantly adjust the weights and combinations of features, so that the association between features can be more fully reflected, thereby greatly enriching the expressive power of features. The SoftMax activation function converts the result after linear transformation into a weight matrix in the form of probability distribution. This method gives the weight matrix a clear probability meaning, making the output more interpretable. The probability value represented by each element in the weight matrix clearly reflects the relative importance of each feature in the decision-making process.

[0078] Step S107: Visually display the emotion classification results.

[0079] In this embodiment, the visualization technology uses HTML+CSS+JavaScript technology, which can accept the prediction results of different ports and visualize the emotions of different subjects. Figure 4As shown, the visualization method of emotion recognition results based on EEG data provided by the embodiment of the present application can be adapted to a variety of application scenarios, such as game scenarios, sports scenarios, etc. In the game scenario or sports scenario, there are five dials in the figure, each dial has three rings, and the data transmitted per second is visualized. The dial has a total of 60 scales representing a total of 60 seconds of prediction results. After the dial displays the recognition results transmitted for one minute, it will be refreshed and cleared. In order to facilitate observation, the scale size of the dial is only displayed at the visualization starting point. As shown in the large dial in the figure, the scale value range from 60 to 100 is displayed from the second ring to the third ring, and the scale range from 0 to 60 from the first ring to the second ring is hidden. Among the four small dials in the figure, driver1 represents the emotion recognition results of e-sports players and athletes, viewer1, viewer2, and viewer3 represent the emotion recognition results of the audience, and the large dial displays the visualization results of the four small dials together, which is more convenient for users to observe. For the visualization effect, the color of the line generated every second is determined by the colors represented by the three emotions in the figure: excitement, focus, and fear. The color of the circle at the end of the line is consistent with the color of the emotion category, and the size of the circle is determined by the weight of the predicted emotion. It is worth noting that the emotional visualization display of the five dials is synchronized, which makes it easier for observers to observe the emotional changes of each subject in real time. Displaying the results in the above application scenarios makes it easier for e-sports players and athletes to optimize their training plans and improve their status to achieve the best performance, which can ultimately bring a better viewing experience to the audience.

[0080] In an embodiment of the present application, the raw EEG data collected by the EEG device is first preprocessed to obtain preprocessed EEG data, thereby avoiding interference and noise in the raw EEG data that may cause inconvenience to subsequent feature fusion; the preprocessed EEG data is spatially transformed to obtain time domain feature data, and frequency domain feature data is obtained based on the time domain feature data, and time / space feature tokens and frequency / space feature tokens are obtained based on the time domain feature data and the frequency domain feature data, thereby realizing the fusion of spatial domain feature data, time domain feature data and frequency domain feature data, and being able to capture more comprehensive features; feature interaction and fusion are performed on the time / space feature tokens and the frequency / space tokens to obtain fused feature tokens, median masking and multi-head calculation are used, as well as feature interaction and fusion through self-attention blocks and cross-covariance attention blocks, thereby reducing the influence of outliers in the EEG data and effectively reducing the complexity of the calculation; the fused feature tokens are linearly transformed to obtain the final emotion classification result, and the emotion classification result is visualized. Displaying the emotion classification result helps to intuitively present the emotion distribution of different subjects and clearly observe the emotional changes of the subjects. This application can effectively solve the problem that traditional EEG signal emotion recognition methods ignore a large amount of potential and deep feature information in EEG data.

[0081] See also Figure 5 , shown is a structural schematic diagram of the emotion recognition result visualization system based on EEG data provided in an embodiment of the second aspect of the present invention.

[0082] It should be noted that the method for visualizing emotion recognition results based on EEG data provided in the embodiment of the present application can be executed by a system for visualizing emotion recognition results based on EEG data, or a control module in the system for executing and loading the method for visualizing emotion recognition results based on EEG data. In the embodiment of the present application, the method for visualizing emotion recognition results based on EEG data provided in the embodiment of the present application is explained by taking an example of a system for visualizing emotion recognition results based on EEG data executing a method for visualizing emotion recognition results based on EEG data.

[0083] The visualization system of emotion recognition results based on EEG data in the embodiment of the present application can be a system, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), etc., which is not specifically limited in the embodiment of the present application.

[0084] The visualization system of emotion recognition results based on EEG data in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0085] The emotion recognition result visualization system based on EEG data provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0086] In some embodiments of the present application, the above-mentioned emotion recognition result visualization system based on EEG data includes: The spatial transformation module 201 is used to preprocess the original EEG data collected by the EEG device to obtain preprocessed EEG data, and to perform spatial transformation on the preprocessed EEG data to obtain time domain feature data; wherein the time domain feature data is a vector sequence including the spatial features of the preprocessed EEG data.

[0087] The multi-scale convolution mapping module 202 is used to obtain frequency domain feature data based on time domain feature data, and to obtain time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data, respectively; wherein the time / space feature tokens include the time domain features and spatial domain features of the pre-processed EEG data, and the frequency / space feature tokens include the frequency features and spatial domain features of the pre-processed EEG data.

[0088] The feature interaction and fusion module 203 is used to perform feature interaction and fusion on the time / space feature tokens and the frequency / space feature tokens to obtain fused feature tokens.

[0089] The classification module 204 is used to perform linear transformation on the fused feature tokens to obtain a final emotion classification result.

[0090] The visualization module 205 is used to visualize the final emotion classification result.

[0091] It is worth noting that the emotion recognition result visualization system based on EEG data provided in the embodiment of the present application can also be trained to obtain the optimal system.

[0092] In this embodiment, a data set is obtained and reasonably divided into a training set and a validation set. The training set is input into the visualization system of emotion recognition results based on EEG data. The system parameters are adjusted according to the data provided by the training set. The error between the predicted result and the true label is calculated through the back propagation algorithm. Based on this error, the system parameters are optimized to gradually improve the system's ability to recognize the emotional information contained in the EEG data. The validation set data is input into the trained system and the predicted result is output. The evaluation index is used to judge the accuracy of the predicted result of the trained system, and the system with the highest accuracy is selected as the optimal system.

[0093] It is worth noting that the data set used in the training of the EEG-based emotion recognition result visualization system in the embodiment of the present invention is already public. The public data set has gone through a strict collection and labeling process, and the data quality is guaranteed, which can effectively improve the accuracy and reliability of the training model. At the same time, it also reduces the research cost to a certain extent.

[0094] In an embodiment of the present application, the raw EEG data collected by the EEG device is first preprocessed to obtain preprocessed EEG data, thereby avoiding interference and noise in the raw EEG data from causing inconvenience to subsequent feature fusion; the preprocessed EEG data is spatially transformed to obtain time domain feature data, and frequency domain feature data is obtained based on the time domain feature data, and time / space feature tokens and frequency / space feature tokens are obtained based on the time domain feature data and the frequency domain feature data, thereby realizing the fusion of spatial domain feature data, time domain feature data and frequency domain feature data, and being able to capture more comprehensive features; feature interaction and fusion are performed on the time / space feature tokens and the frequency / space tokens to obtain fused feature tokens, median masking and multi-head calculation are used, as well as feature interaction and fusion through self-attention blocks and cross-covariance attention blocks, thereby reducing the influence of outliers in the EEG data and effectively reducing the complexity of the calculation; the fused feature tokens are linearly transformed to obtain the final emotion classification result, and the emotion classification result is visualized. Displaying the emotion classification result helps to intuitively present the emotion distribution of different subjects and clearly observe the emotional changes of the subjects. This application can effectively solve the problem that traditional EEG signal emotion recognition methods ignore a large amount of potential and deep feature information in EEG data.

[0095] See also Figure 6, shown is an embodiment of the third aspect of the present application, providing an electronic device, including a processor 301, a memory 302, and a computer program 303 stored in the memory and executable on the processor. When the computer program 303 is executed by the processor, each process of the embodiment of the method for visualizing emotion recognition results based on EEG data is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0096] An embodiment of the fourth aspect of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned embodiment of the method for visualizing emotion recognition results based on EEG data is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0097] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0098] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0099] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0101] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for visualizing emotion recognition results based on EEG data, characterized in that: The method comprises: Preprocessing the raw EEG data collected by the EEG device to obtain preprocessed EEG data; Performing spatial transformation on the pre-processed EEG data to obtain time domain feature data, wherein the time domain feature data is a vector sequence including the time features of the pre-processed EEG data; Acquire frequency domain feature data according to the time domain feature data; Based on the time domain feature data and the frequency domain feature data, respectively, a time / space feature token and a frequency / space feature token are obtained; wherein the time / space feature token includes the time domain feature and the spatial domain feature of the pre-processed EEG data, and the frequency / space feature token includes the frequency feature and the spatial domain feature of the pre-processed EEG data; Performing feature interaction and fusion on the time / space feature token and the frequency / space feature token to obtain a fused feature token; Linearly transforming the fused feature tokens to obtain a final emotion classification result; The final emotion classification result is visualized.

2. The method for visualizing emotion recognition results based on EEG data according to claim 1, characterized in that: The step of preprocessing the raw EEG data collected by the EEG device to obtain preprocessed EEG data specifically includes: Filtering the raw EEG data to obtain filtered EEG data; Normalizing the filtered EEG data to obtain the pre-processed EEG data; Among them, the normalization function expression in the normalization process is: ， in, represents the filtered EEG data, represents the mean value of the filtered EEG data, represents the variance of the filtered EEG data, Represents the normalized EEG data.

3. The method for visualizing emotion recognition results based on EEG data according to claim 1, characterized in that: The step of acquiring the time / space feature token and the frequency / space feature token based on the time domain feature data and the frequency domain feature data specifically includes: Acquire the time / space feature token through time-space branching based on the time-domain feature data; Inputting the time domain feature data into the multi-scale time convolution layer of the space-time branch to obtain multi-scale time feature data; Inputting the multi-scale time feature data into the first spatial convolution layer of the space-time branch to obtain fused space-time feature data; Performing convolution mapping on the fused spatiotemporal feature data to obtain the time / space feature token; Acquire the frequency / space feature token based on the frequency domain feature data through frequency-space branching; Inputting the frequency domain feature data into the multi-scale frequency convolution layer of the frequency-space branch to obtain multi-scale frequency feature data; Inputting the multi-scale frequency feature data into the second spatial convolution layer of the frequency-space branch to obtain fused frequency-space feature data; Convolution mapping is performed on the fused frequency-space feature data to obtain the frequency / space feature token.

4. The method for visualizing emotion recognition results based on EEG data according to claim 1, characterized in that: The step of performing feature interaction and fusion on the time / space feature token and the frequency / space feature token to obtain a fused feature token specifically includes: Perform feature interaction on the time / space feature token and the frequency / space feature token through a self-attention block and a cross-covariance attention block to obtain a first multi-feature token and a second multi-feature token respectively, wherein the first multi-feature token and the second multi-feature token both include the time domain features, frequency domain features and spatial domain features of the pre-processed EEG data; The first multi-feature token and the second multi-feature token are concatenated based on the sample quantity dimension and the feature vector dimension respectively, and are mapped to the original dimension of the preprocessed EEG data respectively to obtain a point dimension token and a vector dimension token; The point dimension token and the vector dimension token are fused through the cross-covariance attention block to obtain the fused feature token.

5. The method for visualizing emotion recognition results based on EEG data according to claim 4, characterized in that: The self-attention block and the cross-covariance attention block also include: The self-attention block and the cross-covariance attention block introduce a median mask and a multi-head calculation method; Wherein, the median mask is used to reduce the influence of extreme values ​​in the pre-processed EEG data; The long calculation method is: ， ， in, head represents the head of each attention sequence, Concat It means multiple heads are connected together. Q , K , V Represent the query vector, key vector, and value vector respectively. W o represents the output weight matrix, It represents the mapping result after linear transformation of multiple heads.

6. The method for visualizing emotion recognition results based on EEG data according to claim 5, characterized in that: The self-attention block and the cross-covariance attention block also include: The self-attention module is used as an encoder to convert the input feature sequence into a feature representation vector; The cross-covariance attention block is used as a decoder to generate a target sequence based on the feature representation vector; The output expression of the cross-covariance attention block is: , , ( )]V， Among them, Q represents the query vector sequence, K represents the key vector sequence, and V represents the value vector sequence. represents the dimension of the key vector sequence, It means that each element of the vector sequence is restricted to be between -1 and 1. represents the attention score of the cross-covariance mechanism, Indicates median mask processing. Softmax Indicates normalization. Represents the output of the cross-covariance attention block.

7. The method for visualizing emotion recognition results based on EEG data according to claim 1, characterized in that: The step of performing a linear transformation on the fused feature token to obtain a final emotion classification result specifically includes: Inputting the fused feature token into a fully connected layer for linear transformation; The final emotion classification result is obtained based on the activation function and the fused feature token after the linear transformation.

8. An emotion recognition result visualization system based on EEG data, characterized in that: The system comprises: A spatial transformation module, used to pre-process the raw EEG data collected by the EEG device to obtain pre-processed EEG data, and to perform spatial transformation on the pre-processed EEG data to obtain time domain feature data; wherein the time domain feature data is a vector sequence including the spatial features of the pre-processed EEG data; A multi-scale convolution mapping module, used to obtain frequency domain feature data according to the time domain feature data, and to obtain time / space feature tokens and frequency / space feature tokens based on the time domain feature data and the frequency domain feature data; wherein the time / space feature tokens include the time domain features and spatial domain features of the pre-processed EEG data, and the frequency / space feature tokens include the frequency features and spatial domain features of the pre-processed EEG data; A feature interaction and fusion module, used to perform feature interaction and fusion on the time / space feature token and the frequency / space feature token to obtain a fused feature token; A classification module, used for performing a linear transformation on the fused feature tokens to obtain a final emotion classification result; The visualization module is used to visualize the final emotion classification result.

9. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the steps of the brain-computer interface emotion recognition visualization method as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the brain-computer interface emotion recognition visualization method as described in any one of claims 1-7 are implemented.

Images