Multi-feature-based cross-scale attention convolution electroencephalogram emotion recognition method
Through the multi-feature cross-scale convolution method, the temporal and spatial characteristics of EEG signals were extracted, and the problem of insufficient classification accuracy of EEG emotion recognition in the existing technology was solved, especially in the performance of few-electrode devices, which achieved significant improvements in performance. Efficient emotion recognition effect.
Patent Information
- Application Number
- CN202410952343.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing EEG emotion recognition methods need to be improved in classification accuracy, especially the performance of few-electrode devices is not good, and the temporal continuity and spatial information of EEG signals are not fully utilized.
The multi-feature-based cross-scale attention convolution method is adopted to extract the feature of differential entropy, power spectral density, nonlinear energy and fractal dimensions of the EEG signal, and combine the cross-scale attention convolution module, frequency-space attention force Module and time feature extraction module, build a deep classification model, and perform emotion recognition.
High classification accuracy was achieved on the DEAP dataset, the accuracy of the titer dimension and the wake-up dimension reached 99.70% and 99.74%, respectively, and good results were also achieved under the condition of few electrodes. The classification accuracy of the titer awakening dimension reached 97.27 %, proves the effectiveness and applicability of the method.
Smart Images

Figure CN118902474A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of electroencephalogram emotion recognition, and in particular to a multi-feature based cross-scale attention convolution electroencephalogram emotion recognition method. Background Art
[0002] Emotion recognition has been applied in many fields, such as mental health, medical diagnosis, intelligent driving and distance education. Early emotion recognition methods mainly relied on non-physiological signals such as voice intonation, facial expressions and body movements, but these methods have the disadvantages of subjectivity and being easily controlled. In order to overcome the shortcomings of non-physiological signals, people have proposed emotion recognition methods based on physiological electrical signals. Physiological electrical signals include electroencephalogram, electromyography, electrooculography and electrocardiogram. Electroencephalogram contains rich information in the time domain, frequency domain and space domain, which can more objectively reflect the real emotional state and avoid emotional disguise. It is currently the preferred physiological signal for emotion recognition.
[0003] Most of the studies on emotion recognition based on EEG use all electrodes that completely cover the entire scalp. However, with the development of science and technology, more and more portable, miniaturized EEG devices that only require a few electrodes have emerged. Some studies on emotion recognition with few electrodes have not achieved good classification results, and there is still a lot of room for improvement. For example, Mert et al. used empirical mode decomposition (EMD) and multivariate expansion (MEMD) to process EEG signals, using 18 electrodes in the DEAP dataset, and the binary classification accuracy in the arousal dimension and valence dimension was 75% and 72.87%, respectively. This method does not take into account the temporal continuity and spatial information of EEG signals, nor does it perform channel selection, and fails to fully utilize the information in EEG. et al. used DWT as feature extraction, added dynamic electrode selection, and selected 5 electrodes that were more relevant to emotions in the DEAP dataset, achieving a maximum binary classification result of 77.14%. This method uses an electrode selection algorithm, but uses a machine learning method, fails to extract high-level features in EEG signals, and the classification accuracy needs to be improved. Topic et al. used ReliefF and Neighborhood Component Analysis (NCA) for electrode selection, and used eigenvalues to construct a holographic feature map. Using data from 10 electrodes in the DEAP dataset, they obtained a maximum binary classification accuracy of 88.58%. This method extracts nine time-frequency domain features of EEG signals and constructs a two-dimensional feature map, but ignores the positional relationship between electrodes in EEG signals and uses a shallow classifier for classification, which affects the classification effect. Summary of the invention
[0004] 1. Technical issues to be solved
[0005] In view of the shortcomings of the prior art, the present invention provides a cross-scale attention convolution EEG emotion recognition method based on multiple features, which has the advantages of accurate recognition and solves the above-mentioned technical problems.
[0006] (II) Technical solution
[0007] To achieve the above object, the present invention provides the following technical solution: a multi-feature based cross-scale attention convolution EEG emotion recognition method, comprising the following steps:
[0008] S1. Collect EEG signals, where the EEG signals are specifically data in the DEAP dataset;
[0009] S2, preprocessing and feature extraction of the EEG signal in step S1, including dividing the EEG signal into N non-overlapping equal-length segments, decomposing the data in each segment into four frequency bands, and extracting the data of each frequency band in turn, respectively extracting four features of differential entropy DE, power spectral density PSD, nonlinear energy NE and fractal dimension FD of each frequency band, and after the extraction is completed, uniformly normalizing to obtain one-dimensional EEG features;
[0010] S3: Perform feature mapping and feature fusion on the features extracted in step S2 to obtain the fused feature set X con ;
[0011] S4. Construct a cross-scale attention convolution EEG emotion recognition model based on multiple features, which includes a cross-scale attention convolution module, a feature transition module, a frequency-space attention module, a temporal feature extraction module and a deep classification module;
[0012] The cross-scale attention convolution module is used to extract spatial features of different scales from EEG signals;
[0013] The feature transition module controls memory consumption while maintaining network performance;
[0014] The frequency-space attention module gives higher weights to important channels and spatial positions;
[0015] The time feature extraction module is used to extract the time feature of the EEG signal;
[0016] The deep classification module is used for emotion classification of EEG signals;
[0017] S5. Input the EEG signal into the multi-feature based cross-scale attention convolution EEG emotion recognition model for EEG emotion recognition.
[0018] As a preferred technical solution of the present invention, the calculation expression of the differential entropy DE in step S2 is as follows:
[0019]
[0020] Among them, e represents a natural constant, σ represents the standard deviation of x, represents the square of the standard deviation of the i-th Gaussian distribution, x represents the input EEG signal, a, b represent the EEG signal of length [a, b], log(*) represents the logarithmic function, μ represents the mean of x, and DE represents the differential entropy;
[0021] The calculation expression of the power spectral density PSD is as follows:
[0022]
[0023] Among them, P(ω k ) represents the power spectral density PSD, M represents the length of the EEG signal, ω k represents the angular frequency, k represents the angular frequency index, k=-(M-1),-(M-2),···,0,1,···,(M-1),M-1. t represents the time, with a value of 0~M-1, Indicates the summation of internal data, e represents a natural constant, j represents an imaginary unit, x(t) represents the value of the EEG signal at time point t, γ(t) is the autocorrelation function of x(t), and represents the autocorrelation function value of the EEG signal at time offset t. The specific expression is as follows:
[0024] γ(t)=E[x(n)x * (n+t)]
[0025] Where E[*] represents the expectation, x(n) represents the value of the EEG signal at the nth sampling point, and x * (n+t) represents the complex conjugate value of the EEG signal at n+t, n represents the sampling point of a segment of the EEG signal, and x * represents the complex conjugate of x;
[0026] The calculation expression of the nonlinear energy NE is as follows:
[0027]
[0028] Among them, N EEG represents the number of sampling points of the EEG signal, Indicates the internal N EEG The data are summed up, Γ(x(n)) represents the estimated value of nonlinear energy, and its calculation expression is as follows:
[0029] Γ(x(n))=(y(n)) 2 +|y hibart (n)| 2
[0030] Among them, y(n) represents the first-order difference of the EEG signal x(n), y hibart (n) represents the Hibert transform of the first-order difference y(n) of the EEG signal, |y hibart (n)| 2 Represents y hibart (n) the square of the absolute value;
[0031] The calculation expression of the fractal dimension FD is as follows:
[0032]
[0033] Among them, log 10 (*) indicates a logarithmic function with base 10, N EEG Represents the number of sampling points of the EEG signal, N Δ Indicates the number of sign changes of the EEG signal.
[0034] As a preferred technical solution of the present invention, the feature mapping process in step S3 is specifically as follows: mapping the features of the 32-electrode EEG signals into a two-dimensional feature matrix to obtain a two-dimensional feature map, mapping the features of the EEG signals of each frequency band to the two-dimensional feature map, and then stacking the two-dimensional feature maps of the four frequency bands into three-dimensional features, and forming the corresponding four-dimensional features from the three-dimensional features formed in each segment. The specific expression is as follows:
[0035] X DE =[d 1 ,d 2 ,d 3 ,…,d N ]
[0036] X PSD =[p 1 ,p 2 ,p 3 ,…,p N ]
[0037] X NE =[n 1 ,n 2 ,n 3 ,…,n N ]
[0038] X FD =[f 1 ,f 2 ,f 3 ,…,f N ]
[0039] Among them, d 1 ,d 2 ,d 3 ,…,d nRepresents the three-dimensional features of 1 to N DEs, X DE represents the four-dimensional features of DE, p 1 ,p 2 ,p 3 ,…,p N Represents the three-dimensional features of 1 to N PSDs, X PSD Represents the four-dimensional features of PSD, n 1 ,n 2 ,n 3 ,…,n N Represents the three-dimensional features of 1 to N NEs, X NE represents the four-dimensional features of NE, f 1 ,f 2 ,f 3 ,…,f N Represents the three-dimensional features of 1 to N FDs, X FD represents the four-dimensional characteristics of FD, DE represents differential entropy, PSD represents power spectral density, NE represents nonlinear energy, FD represents fractal dimension, and then X DE , X PSD , X NE and X FD Perform feature fusion, the specific expression is as follows:
[0040] X con =[X DE ;X PSD ;X NE ;X FD ]=[d 1 ;p 1 ;n 1 ;f 1 ,…,d N ;p N ;n N ;f N ]
[0041] Among them, X con Represents the fused feature set.
[0042] As a preferred technical solution of the present invention, the cross-scale attention module captures features of different scales by using convolution kernels of different sizes in parallel and combining the frequency-space attention mechanism;
[0043] The frequency-space attention module is used for input features Figure X After the frequency attention mechanism F f and spatial attention mechanism F S Then the output feature map Y is obtained, and the specific expression is as follows:
[0044]
[0045]
[0046] Among them, F f (X) represents the input feature Figure X Perform frequency attention operation, X' represents the features after frequency attention operation, Indicates multiplication operation, F S (X') represents the features after the spatial attention mechanism operation, and Y represents the output feature map;
[0047] The feature transition module takes the output feature map Y of the frequency-space attention module as input and normalizes it;
[0048] The temporal feature extraction module is equipped with a bidirectional long short-term memory network. The bidirectional long short-term memory network uses two LSTM units when processing at each time step, one for processing EEG signals from front to back, and the other for processing EEG signals from back to front. The calculation results of the two LSTM units in different directions are superimposed and output;
[0049] The deep classification module takes the result of the temporal feature extraction module as input and recognizes emotions.
[0050] As a preferred technical solution of the present invention, the frequency attention mechanism F in the frequency-space attention module f Processing input features Figure X The specific process is as follows:
[0051] S4a.1. Input features Figure X Perform global adaptive average pooling operation on the spatial dimension to obtain X avg , specifically expressed as follows:
[0052] X avg =Adaptive-AvgPool(X(h,w))
[0053] Among them, X avg Representation characteristics Figure X Output after pooling operation, Adaptive-AvgPool(*) represents global adaptive average pooling, X(h,w) represents the input features Figure X , the parameters are h and w, h represents the height of the feature map, and w represents the width of the feature map;
[0054] S4a.2, X avg Input the multi-layer perceptron, and finally pass through the linear layer, and add the bias term b to get F f (X);
[0055] S4a.3, F f(X) is multiplied by X to obtain X';
[0056] The multilayer perceptron consists of two fully connected layers and ReLu and Sigmoid activation functions. The expressions of ReLu activation function and Sigmoid activation function are as follows:
[0057]
[0058]
[0059] Among them, F ReLu (x) represents the ReLu activation function, F Signod (x) represents the Sigmoid activation function, e represents a natural constant, and x represents the input EEG signal.
[0060] As a preferred technical solution of the present invention, the spatial attention mechanism F in the frequency-space attention module S The specific steps for processing X' are as follows:
[0061] S4b.1. Pass the feature X' after the frequency attention mechanism operation through the maximum pooling layer, and obtain F through two-dimensional convolution and ReLu and Sigmoid activation functions. S (X'), the specific expression is as follows:
[0062] X' max =MaxPool(X'(h,w))
[0063] Among them, X' max Representation characteristics Figure X 'Output after the maximum pooling operation, MaxPool(*) represents the maximum pooling, and X'(h,w) represents the input features Figure X ', the parameters are h and w, h represents the height of the feature map, and w represents the width of the feature map;
[0064] S4b.2, F S Multiply (X') by X' to get Y.
[0065] As a preferred technical solution of the present invention, the LSTM unit calculation expression is as follows:
[0066] k t =σ(W k ·[h t-1 ,q t ]+b k )
[0067] f t =σ(W f ·[h t-1 ,q t ]+bf )
[0068] g t =tanh(W c ·[h t-1 ,q t ]+b c )
[0069] C t =f t C t-1 +k t g t
[0070] o t =σ(W o ·[h t-1 ,q t ]+b o )
[0071] h t =o t tanh(C t )
[0072] Among them, k t represents the input gate, f t represents the forget gate, g t represents the candidate cell state, C t Indicates cell status, o t represents the output gate, h t Hidden state, h t-1 represents the hidden state of the previous time step, W k , W f , W c , W o represents the weight matrix of the corresponding gate, b k 、b f 、b c 、b o represents the bias vector of the corresponding gate, q t Represents the input of the current time step, tanh(*) represents the hyperbolic tangent function, and σ(*) represents the Sigmoid function.
[0073] Compared with the prior art, the present invention provides a cross-scale attention convolution EEG emotion recognition method based on multiple features, which has the following beneficial effects:
[0074] The present invention extracts DE, PSD, NE and FD features from EEG signals of different electrodes, removes the baseline of these features and constructs a three-dimensional feature matrix according to the relative positions of the electrodes to make full use of the frequency domain, energy, nonlinearity, spatiotemporal complexity and spatial information in the EEG signals. Then, a cross-scale attention module is used to extract spatial features of different scales in the EEG signals, and the frequency-space attention mechanism is used to assign different weights to the extracted features of different importance in the spatial position and channel to improve the classification performance of the model. In order to prevent overfitting and reduce the amount of calculation, a transition module is introduced to improve the generalization of the model, extract the time features in the EEG signals, and realize the fusion of spatiotemporal features. The experimental results of the proposed method on the DEAP dataset show that the method can effectively extract emotion-related features in EEG signals, and the classification accuracy in the valence dimension and arousal dimension reaches 99.70% and 99.74% respectively, and the classification accuracy in the valence arousal dimension reaches 97.27%. This method has good performance in the case of few electrodes.
[0075] Good results have also been achieved in this area. In the 5-electrode EEG signal experiment, the classification accuracy on the valence dimension and arousal dimension reached 97.96% and 98.11% respectively, and the classification accuracy on the valence arousal dimension reached 92.86%. In the 18-electrode experiment, the classification accuracy on the valence dimension and arousal dimension reached 99.59% and 99.53% respectively, and the classification accuracy on the valence arousal dimension reached 94.57%. BRIEF DESCRIPTION OF THE DRAWINGS
[0076] Figure 1 It is a schematic diagram of the process of the present invention;
[0077] Figure 2 It is a schematic diagram of the feature extraction and feature mapping process of the present invention;
[0078] Figure 3 It is a schematic diagram of a specific method of feature mapping of the present invention;
[0079] Figure 4 Schematic diagram of the cross-scale attention module of the present invention;
[0080] Figure 5 Schematic diagram of the frequency-space attention module framework of the present invention;
[0081] Figure 6 A schematic diagram of different weights assigned to 464 channels by the frequency attention mechanism of the present invention in CSAM;
[0082] Figure 7 This is a thermal diagram showing that different weights are assigned to different electrode positions after the features in the CSAM of the present invention are subjected to the spatial attention mechanism;
[0083] Figure 8 It is a schematic diagram of the characteristic transition module of the present invention;
[0084] Fig. 9 This is a schematic diagram of the deep classification module framework of the present invention;
[0085] Fig.10 It is a schematic diagram of the accuracy of all subjects in the DEAP dataset of the present invention;
[0086] Fig.11 A schematic diagram of the confusion matrix of the CATM experiment performed on the DEAP dataset of the present invention;
[0087] Fig.12 It is a schematic diagram of the accuracy of all subjects in different models of the present invention;
[0088] Fig.13 This is a schematic diagram of electrode mapping with few electrodes according to the present invention;
[0089] Fig.14 This is a schematic diagram of the accuracy of the DEAP dataset for all subjects in the 5-electrode experiment of the present invention;
[0090] Fig.15 Schematic diagram of the accuracy of the DEAP dataset for all subjects in the 18-electrode experiment of the present invention. DETAILED DESCRIPTION
[0091] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0092] See also Figure 1-15 The present invention provides the following technical solution: a method for cross-scale attention convolution EEG emotion recognition based on multiple features, comprising the following steps:
[0093] S1. Collect EEG signals, where the EEG signals are specifically data in the DEAP dataset;
[0094] The DEAP dataset is a physiological signal dataset generated based on music videos as induction materials. Each subject watched 40 minutes of music videos during the experiment and evaluated the emotions of each video based on their subjective feelings, including arousal, valence, liking, and dominance. A total of 32 subjects participated in the experiment. Each experimental data consists of signals from 40 electrodes. The first 32 electrodes are EEG signals, and the last 8 electrodes include other physiological signals such as electrooculography and electromyography. The duration of each experiment is 63 seconds, of which the first 3 seconds are baseline signals and the last 60 seconds are emotion-induced signals. The sampling frequency of the dataset has been downsampled from 512Hz to 128Hz, and the physiological signal matrix of each subject is 40*40*8064 (40 experimental music, 40 physiological signals, 8064 sampling points). The DEAP dataset provides rich EEG and other physiological signal data, as well as the subjective evaluation of emotions by subjects, which can help researchers gain a deeper understanding of the relationship between emotions and physiological signals.
[0095] S2, preprocessing and feature extraction of the EEG signal in step S1, including dividing the EEG signal into N non-overlapping equal-length segments, decomposing the data in each segment into four frequency bands, and extracting the data of each frequency band in turn, respectively extracting four features of differential entropy DE, power spectral density PSD, nonlinear energy NE and fractal dimension FD of each frequency band, and after the extraction is completed, uniformly normalizing to obtain one-dimensional EEG features;
[0096] The signal-to-noise ratio of the EEG signal is relatively low, there are a large number of low-frequency components, and the waveform presents constantly changing nonlinear characteristics. Based on these characteristics, first, for each subject, the original EEG data is divided into N non-overlapping and equal-length segments, and the labels of the segments are assigned. Secondly, each segment is decomposed into four frequency bands: theta (3-8Hz), alpha (8-14Hz), beta (14-31Hz) and gamma (31-45Hz) through a Butterworth filter. Next, four features, differential entropy (DE), power spectral density (PSD), nonlinear energy (NE) and fractal dimension (FD), are extracted from the data of each frequency band. The feature extraction and feature mapping process is as follows: Figure 2 Shown
[0097] The calculation expression of the differential entropy DE in step S2 is as follows:
[0098]
[0099] Among them, e represents a natural constant, σ represents the standard deviation of x, represents the square of the standard deviation of the i-th Gaussian distribution, x represents the input EEG signal, a, b represent the EEG signal of length [a, b], log(*) represents the logarithmic function, μ represents the mean of x, and DE represents the differential entropy;
[0100] The calculation expression of the power spectral density PSD is as follows:
[0101]
[0102] Among them, P(ω k ) represents the power spectral density PSD, M represents the length of the EEG signal, ω k represents the angular frequency, k represents the angular frequency index, k=-(M-1),-(M-2),···,0,1,···,(M-1),M-1. t represents the time, with a value of 0~M-1, Indicates the summation of internal data, e represents a natural constant, j represents an imaginary unit, x(t) represents the value of the EEG signal at time point t, γ(t) is the autocorrelation function of x(t), and represents the autocorrelation function value of the EEG signal at time offset t. The specific expression is as follows:
[0103] γ(t)=E[x(n)x * (n+t)]
[0104] Where E[*] represents the expectation, x(n) represents the value of the EEG signal at the nth sampling point, and x * (n+t) represents the complex conjugate value of the EEG signal at n+t, n represents the sampling point of a segment of the EEG signal, and x * represents the complex conjugate of x;
[0105] The calculation expression of the nonlinear energy NE is as follows:
[0106]
[0107] Among them, N EEG represents the number of sampling points of the EEG signal, Indicates the internal N EEG The data are summed up, Γ(x(n)) represents the estimated value of nonlinear energy, and its calculation expression is as follows:
[0108] Γ(x(n))=(y(n)) 2 +|y hibart (n)| 2
[0109] Among them, y(n) represents the first-order difference of the EEG signal x(n), y hibart (n) represents the Hibert transform of the first-order difference y(n) of the EEG signal, |y hibart(n)| 2 Represents y hibart (n) the square of the absolute value;
[0110] The calculation expression of the fractal dimension FD is as follows:
[0111]
[0112] Among them, log 10 (*) indicates a logarithmic function with base 10, N EEG Represents the number of sampling points of the EEG signal, N Δ Indicates the number of sign changes of the EEG signal
[0113] S3: Perform feature mapping and feature fusion on the features extracted in step S2 to obtain the fused feature set X con ;
[0114] EEG signals also contain important spatial position information. In order to make full use of the time, space and frequency information in EEG signals, a four-dimensional structural feature is constructed to describe EEG signals. Figure 3 As shown, the present invention maps the features of the 32-electrode EEG signals into a two-dimensional feature matrix based on the 32-electrode layout of the international 10-20 system, and fills the positions without electrodes with 0 to obtain a two-dimensional feature map of size 8×9. The features of the EEG signals in each frequency band are mapped to a two-dimensional feature map, and then the two-dimensional feature maps of the four frequency bands are stacked into a three-dimensional feature of 8×9×4, and the three-dimensional features formed in each segment constitute the corresponding four-dimensional features. The specific expression is as follows:
[0115] X DE =[d 1 ,d 2 ,d 3 ,…,d N ]
[0116] X PSD =[p 1 ,p 2 ,p 3 ,…,p N ]
[0117] X NE =[n 1 ,n 2 ,n 3 ,…,n N ]
[0118] X FD =[f 1 ,f 2 ,f 3 ,…,f N ]
[0119] Among them, d 1 ,d 2 ,d 3 ,…,d n Represents the three-dimensional features of 1 to N DEs, X DE represents the four-dimensional features of DE, p 1 ,p 2 ,p 3 ,…,p N Represents the three-dimensional features of 1 to N PSDs, X PSD Represents the four-dimensional features of PSD, n 1 ,n 2 ,n 3 ,…,n N Represents the three-dimensional features of 1 to N NEs, X NE represents the four-dimensional features of NE, f 1 ,f 2 ,f 3 ,…,f N Represents the three-dimensional features of 1 to N FDs, X FD represents the four-dimensional features of FD, DE represents differential entropy, PSD represents power spectral density, NE represents nonlinear energy, FD represents fractal dimension, and then X DE , X PSD , X NE and X FD Perform feature fusion, the specific expression is as follows:
[0120] X con =[X DE ;X PSD ;X NE ;X FD ]=[d 1 ;p 1 ;n 1 ;f 1 ,…,d N ;p N ;n N ;f N ]
[0121] Among them, X con Represents the fused feature set
[0122] S4. Construct a cross-scale attention convolution EEG emotion recognition model based on multiple features. The model includes a cross-scale attention convolution module (CSAM), a feature transition module (FTM), a frequency-space attention module (FSAM), a temporal feature extraction module and a deep classification module (DCM). First, the feature set X conInput two-dimensional convolution, the output feature shape is (128, 80, 8, 9), then input the feature transition module, which does not change the shape of the input feature, then input the cross-scale attention convolution module, the output feature shape is (128, 464, 8, 9), then pass through the feature transition module, the maximum pooling layer, and the frequency-space attention module. The above three modules do not change the shape of the input feature, and then input the feature into the time feature extraction module, the output feature shape is (128, 128), and finally input the feature into the deep classification module, the output feature shape is (128, 2) for classification to obtain the result;
[0123] The cross-scale attention convolution module is used to extract spatial features of different scales from EEG signals;
[0124] The feature transition module controls memory consumption while maintaining network performance;
[0125] The frequency-space attention module gives higher weights to important channels and spatial positions;
[0126] The time feature extraction module is used to extract the time feature of the EEG signal;
[0127] The deep classification module is used for emotion classification of EEG signals;
[0128] The cross-scale attention module captures features of different scales by using convolution kernels of different sizes in parallel and combining the frequency-space attention mechanism;
[0129] CSAM uses convolution kernels of different sizes in parallel and combines them with the frequency-space attention mechanism to capture features of different scales. Although the CSAM structure is sparse, it can produce dense feature data. The CSAM module uses three different sizes of convolution kernels (1×1, 3×3, 5×5) and a maximum pooling kernel (3×3). Convolution kernels of different sizes have different receptive fields. Using convolution kernels of different sizes can not only extract subtle EEG features, but also take into account the relative position relationship between electrodes. The maximum pooling layer is used to compress the spatial size of the feature map, and more abstract and advanced features are further extracted by reducing the width and height of the feature map. Adding the BN layer and ReLU activation function after the convolution and pooling operations effectively prevents overfitting and increases the nonlinear characteristics of the network. Finally, the CSAM module is obtained after the frequency-space attention mechanism is added to the ReLU activation function. The structure of the CSAM module is as follows: Figure 4 As shown in Figure 2, the CSAM module consists of a two-dimensional convolutional layer, a maximum pooling layer, a BN layer, a ReLU activation function, and a frequency-space attention module (FSAM). The convolution kernel sizes are (1×1, 3×3, 5×5), the maximum pooling kernel size is (3×3), and it is composed of convolution kernels of different sizes and maximum pooling kernels side by side.
[0130] The parameter settings of the CSAM module are shown in Table 1.
[0131] Table 1. Detailed parameters of CSAM
[0132]
[0133]
[0134] Figure 5 The input tensor is B×C×H×W, where B, C, H, and W represent the batch size, number of channels, height, and width of the tensor, respectively. For a given input feature map, the feature map is obtained after the frequency attention mechanism and the spatial attention mechanism are applied in sequence. The frequency-space attention module Figure X After the frequency attention mechanism F f and spatial attention mechanism F S Then the output feature map Y is obtained, and the specific expression is as follows:
[0135]
[0136]
[0137] Among them, F f (X) represents the input feature Figure X ∈R B×C×H×W Perform frequency attention operation, X' represents the features after frequency attention operation, Indicates multiplication operation, F S (X') represents the features after the spatial attention mechanism operation, and Y represents the output feature map;
[0138] The frequency attention mechanism F in the frequency-space attention module f Processing input features Figure X The specific process is as follows:
[0139] For input features Figure X Perform global adaptive average pooling operation on the spatial dimension to obtain X avg ∈R B×C×1×1 , specifically expressed as follows:
[0140] X avg =Adaptive-AvgPool(X(h,w))
[0141] Among them, X avg Representation characteristics Figure X Output after pooling operation, Adaptive-AvgPool(*) represents global adaptive average pooling, X(h,w) represents the input features Figure X, the parameters are h and w, h represents the height of the feature map, and w represents the width of the feature map;
[0142] X avg Input the multi-layer perceptron, and finally pass through the linear layer, and add the bias term b to get F f (X), the output frequency attention weight F f (X) is multiplied by the input X to obtain X' with frequency weights. Figure 6 It shows the different weights assigned by the frequency attention mechanism to 464 channels in CSAM.
[0143] S4a.3, F f (X) is multiplied by X to obtain X';
[0144] The multilayer perceptron consists of two fully connected layers and ReLu and Sigmoid activation functions. The expressions of ReLu activation function and Sigmoid activation function are as follows:
[0145]
[0146]
[0147] Among them, F ReLu (x) represents the ReLu activation function, F Signod (x) represents the Sigmoid activation function, e represents a natural constant, and x represents the input EEG signal.
[0148] S4b.1. Pass the feature X' after the frequency attention mechanism operation through the maximum pooling layer, and obtain F through two-dimensional convolution and ReLu and Sigmoid activation functions. S (X'), the specific expression is as follows:
[0149] X' max =MaxPool(X'(h,w))
[0150] Among them, X' max Representation characteristics Figure X 'Output after the maximum pooling operation, MaxPool(*) represents the maximum pooling, and X'(h,w) represents the input features Figure X ', the parameters are h and w, h represents the height of the feature map, and w represents the width of the feature map;
[0151] S4b.2, F S Multiply (X') by X' to get Y;
[0152] Figure 7This is a heat map showing that different electrode positions are given different weights after the features in CSAM are processed by the spatial attention mechanism. It is not difficult to see that the weight in the frontal lobe of the brain is higher, while the weight in the parietal lobe is lower.
[0153] The feature transition module takes the output feature map Y of the frequency-space attention module as input and normalizes it;
[0154] The present invention uses multiple features of EEG signals and performs feature fusion. In the subsequent cross-scale convolution, feature splicing is also used to fuse feature maps of different sizes. The output feature map has more channels. These operations will double the dimension of the input feature map and occupy more memory. In order to control memory consumption while maintaining network performance, a transition module is introduced. The transition module normalizes the input feature map through a batch normalization layer, so that low-dimensional features with more emotional expression ability and stability can be obtained. The network structure of the feature transition module is as follows: Figure 8 As shown, the size of the maximum pooling kernel is set to
[0155] The temporal feature extraction module is equipped with a bidirectional long short-term memory network. The bidirectional long short-term memory network uses two LSTM units when processing at each time step, one for processing EEG signals from front to back, and the other for processing EEG signals from back to front. The calculation results of the two LSTM units in different directions are superimposed and output;
[0156] The LSTM unit calculation expression is as follows:
[0157] k t =σ(W k ·[h t-1 ,q t ]+b k )
[0158] f t =σ(W f ·[h t-1 ,q t ]+b f )
[0159] g t =tanh(W c ·[h t-1 ,q t ]+b c )
[0160] C t =f t C t-1 +k t g t
[0161] o t =σ(Wo ·[h t-1 ,q t ]+b o )
[0162] h t =o t tanh(C t )
[0163] Among them, k t represents the input gate, f t represents the forget gate, g t represents the candidate cell state, C t Indicates cell status, o t represents the output gate, h t Hidden state, h t-1 represents the hidden state of the previous time step, W k , W f , W c , W o represents the weight matrix of the corresponding gate, b k 、b f 、b c 、b o represents the bias vector of the corresponding gate, q t Represents the input of the current time step, tanh(*) represents the hyperbolic tangent function, and σ(*) represents the Sigmoid function.
[0164] The deep classification module (DCM) takes the result of the temporal feature extraction module as input and recognizes emotions. Since the effect of deep classifiers is often better than that of shallow classifiers, the deep classification module is composed of a fully connected layer, a ReLU activation function and a Dropout regularization for the extracted temporal feature output. Fig. 9 As shown, the deep classification module consists of linear layer 1, ReLU activation function, Dropout regularization, linear layer 2, ReLU activation function, Dropout regularization, and linear layer 3. The specific parameters are shown in Table 2;
[0165] Table 2. Detailed parameters of DCM
[0166]
[0167] S5. Input the EEG signal into the multi-feature based cross-scale attention convolution EEG emotion recognition model for EEG emotion recognition.
[0168] The following is the experimental process and results of this paper's multi-feature-based cross-scale attention convolutional EEG emotion recognition model:
[0169] Experimental setup:
[0170] In the DEAP dataset, valence and arousal labels are selected, and labels with scores above 5 are assigned a value of 1, and labels with scores below 5 are assigned a value of 0. Two-category experiments can be classified into high valence (HV) and low valence (LV) or high arousal (HA) and low arousal (LA), and four-category experiments (VA) can be classified into high valence and high arousal (HVHA), high valence and low arousal (HVLA), low valence and high arousal (LVHA), and low valence and low arousal (LVLA). HVHA corresponds to an excited or agitated state, HVLA corresponds to a calm or relaxed state, LVHA corresponds to depression or anger, and LVLA corresponds to frustration or sadness.
[0171] All experiments use the same software environment, experimental data set division, parameter settings and evaluation indicators. The software environment is Windows 11 operating system, Python 3.9 programming language environment and Pytorch deep learning framework, and the hardware uses NVIDIA GeForce RTX 4060 GPU. Adam is used as the optimizer during model training, cross entropy and L2 regularization are used as loss functions, and the learning rate is set to 0.001. In order to reduce overfitting during training, the Dropout parameter is set to 0.5. In addition, each subject uses ten-fold cross validation, and the average of ten experiments is taken as the recognition result of each subject. Finally, the average value of the recognition of 32 subjects is taken as the data set result. In order to objectively evaluate the performance of the model, the present invention will use accuracy, precision, recall and f1 score F1_Score as evaluation indicators, and the calculation formulas are as follows:
[0172]
[0173]
[0174]
[0175]
[0176] Among them, TP means that the true label is positive and the prediction is positive, TN means that the true label is negative and the prediction is negative, FP means that the true label is negative but the prediction is positive, and FN means that the true label is positive but the prediction is negative;
[0177] Experimental results analysis:
[0178] In order to verify the effectiveness of the proposed method, the present invention conducts ablation experiments and comparative experiments on the DEAP dataset, and evaluates the model performance with evaluation indicators. Fig.10The figure shows the accuracy of 32 subjects in the DEAP dataset in the valence dimension, arousal dimension, and valence-arousal dimension. Fig.11 The confusion matrices of CATM on the DEAP dataset are shown. The accuracy of the proposed method in the Valence dimension and Arousal dimension are 99.70% and 99.74% respectively, and the accuracy of VA is 97.27%;
[0179] Ablation experiment:
[0180] The present invention conducts feature ablation experiments and module ablation experiments. The feature ablation experiments ablate four features, DE, PSD, NE and FD, respectively. The experimental results are shown in Table 3.
[0181] Table 3. Experimental results of different features on the DEAP dataset.
[0182]
[0183] From the experimental results, it can be seen that compared with single feature, double feature and triple feature, the classification accuracy of four features is the best. The reason is that the introduction of multiple features helps to improve the accuracy of emotion recognition. The experimental results also prove the effectiveness of PSD feature, FD feature and NE feature for emotion recognition.
[0184] The module ablation experiment ablates the cross-scale attention module, frequency-space attention module and temporal feature extraction module to verify the impact of the missing modules on the model. It mainly includes the model without the cross-scale attention module (ATM), the model without the frequency-space attention module (CTM), and the model without the temporal feature extraction module (CAM). The module ablation experiment results are shown in Table 4.
[0185] Table 4. Ablation experiment results of different modules.
[0186]
[0187] As shown in Table 4, after ablating the cross-scale attention module, the classification accuracy of the valence dimension and arousal dimension decreased by 8.36% and 4.68% respectively; after ablating the frequency-space attention module, the classification accuracy of the valence dimension and arousal dimension decreased by 0.75% and 0.77% respectively; after ablating the temporal feature extraction module, the classification accuracy of the valence dimension and arousal dimension decreased by 7.95% and 7.29 respectively. It can be seen that after ablating each module, the feature extraction ability of the model has decreased, and the time, space and frequency information in the EEG signal are not fully utilized. Fig.12The accuracy of each subject in the valence dimension and arousal dimension in the model ablation experiment is shown. The present invention found that after ablating certain modules, the recognition accuracy of individual subjects decreased. The module ablation experiment proves the effectiveness of each module in the model.
[0188] Few-electrode experiment:
[0189] In order to prove the effectiveness of this method on a device with few electrodes, we selected electrodes with high correlation with emotions based on previous research
[16] and used the data of the same electrodes in the DEAP dataset for experiments. We conducted experiments with 5 electrodes and 18 electrodes. The 5 electrodes were Fp1, Fp2, F7, F8, and O1, and the 18 electrodes were Fp1, Fp2, F7, F3, FZ, F4, F8, T7, C3, CZ, C4, T8, P7, P3, P4, P8, O1, and O2. The electrode mapping is shown in Figure 2. Fig.13 As shown:
[0190] The experimental results of the evaluation indicators are shown in Tables 5 and 6 respectively.
[0191] Table 5. Experimental results of 5 electrodes on DEAP dataset
[0192]
[0193]
[0194] According to Tables 5 and 6, the method of the present invention also achieved good performance in the few-electrode experiment. In the 5-electrode experiment, the accuracy of Valence and Arousal were 97.96% and 98.11%, respectively, and the accuracy of VA was 92.86%. In the 18-electrode experiment, the accuracy of Valence and Arousal were 99.59% and 99.53%, respectively, and the accuracy of VA was 94.57%. The accuracy of each subject is shown in Table 5. Fig.14 and 15 .
[0195] Comparative experiment
[0196] The proposed emotion classification model is compared with recent emotion classification models, as shown in Table 7.
[0197] Table 7. Performance comparison of different recent methods
[0198]
[0199] The characteristics of various methods are analyzed in Table 7:
[0200] (1) FSA-3D-CNN: This method constructs a three-dimensional EEG matrix containing spatiotemporal information, introduces an attention mechanism, and uses 3D-CNN for emotion classification tasks.
[0201] (2) TSFFN: This method removes the baseline of EEG and uses parallel transformers and three-dimensional convolutional neural networks (3D-CNN) to extract spatiotemporal features from EEG signals, and finally performs emotion classification tasks.
[0202] (3) Multi-aCRNN: This method proposes a multi-view feature fusion attention convolutional recurrent neural network. By smoothing labels and reducing the interference of label noise, GRU and CNN are combined to complete the emotion classification task.
[0203] (4) RA2-3DCNN: This method introduces the segmentation-transformation-merging technique, residual and attention mechanism into the shallow network to improve the accuracy of the model. Emotion recognition is performed based on two-dimensional convolutional neural network and three-dimensional convolutional neural network.
[0204] (5) MDCNA Resnet: This method extracts differential entropy features from EEG signals and constructs a three-dimensional feature matrix. It uses deformable convolution to extract high-level abstract features and combines MDCNA Resnet with a bidirectional gated recurrent unit (BiGRU) to achieve emotion recognition.
[0205] (6) BiTCAN: This method uses the bi-hemispheric difference module to extract the salient features of brain cognition, fuses the salient features and spatiotemporal features into the attention module, and inputs them into the classifier for emotion recognition.
[0206] (7) RFPN–S2D–CNN: This method uses preprocessed signals, differential entropy (DE), symmetric difference and symmetric quotient to construct four EEG signal feature matrices, and proposes a residual feature pyramid network (RFPN) to obtain the correlation between channels, effectively improving the classification accuracy of emotion recognition.
[0207] (8) FCAN–XGBoost: This method extracts DE features and PSD features of EEG and combines FCAN and XGBoost algorithms for emotion recognition, which reduces the computational cost and improves the classification accuracy.
[0208] (9) Multi-scale 3D-CRU: This method reconstructs the three-dimensional feature representation of EEG including Delta (δ) frequency and combines it with the recurrent neural network GRU for emotion classification.
[0209] (10) MES-CTNet: This method proposes a new capsule transformer network based on multi-domain features.
[0210] A variety of features and attention mechanisms are used to achieve high emotion classification accuracy.
[0211] The method proposed in the present invention uses different features of EEG and combines the cross-scale attention convolution module and the time feature extraction module to extract the time-frequency-space features in the EEG signal, thereby improving the accuracy of emotion classification. The data in Table 7 show that the method of the present invention is better than other methods in the two-classification and four-classification tasks;
[0212] The experimental results of the method proposed in this paper on the DEAP dataset show that this method can effectively extract emotion-related features in EEG signals, and the classification accuracy on the valence dimension and arousal dimension reaches 99.70% and 99.74% respectively, and the classification accuracy on the valence arousal dimension reaches 97.27%. This method has also achieved good results in terms of few electrodes. In the 5-electrode EEG signal experiment, the classification accuracy on the valence dimension and arousal dimension reached 97.96% and 98.11% respectively, and the classification accuracy on the valence arousal dimension reached 92.86%. In the 18-electrode experiment, the classification accuracy on the valence dimension and arousal dimension reached 99.59% and 99.53% respectively, and the classification accuracy on the valence arousal dimension reached 94.57%.
[0213] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A multi-feature based cross-scale attention convolution EEG emotion recognition method, characterized by: The following steps are involved: S1. Collect EEG signals, where the EEG signals are specifically data in the DEAP dataset; S2, preprocessing and feature extraction of the EEG signal in step S1, including dividing the EEG signal into N non-overlapping equal-length segments, decomposing the data in each segment into four frequency bands, and extracting the data of each frequency band in turn, respectively extracting four features of differential entropy DE, power spectral density PSD, nonlinear energy NE and fractal dimension FD of each frequency band, and after the extraction is completed, uniformly normalizing to obtain one-dimensional EEG features; S3: Perform feature mapping and feature fusion on the features extracted in step S2 to obtain the fused feature set X con ; S4. Construct a cross-scale attention convolution EEG emotion recognition model based on multiple features, which includes a cross-scale attention convolution module, a feature transition module, a frequency-space attention module, a temporal feature extraction module and a deep classification module; The cross-scale attention convolution module is used to extract spatial features of different scales from EEG signals; The feature transition module controls memory consumption while maintaining network performance; The frequency-space attention module gives higher weights to important channels and spatial positions; The time feature extraction module is used to extract the time feature of the EEG signal; The deep classification module is used for emotion classification of EEG signals; S5. Input the EEG signal into the multi-feature based cross-scale attention convolution EEG emotion recognition model for EEG emotion recognition.
2. The method for cross-scale attention convolutional EEG emotion recognition based on multiple features according to claim 1, characterized in that: The calculation expression of the differential entropy DE in step S2 is as follows: Among them, e represents a natural constant, σ represents the standard deviation of x, represents the square of the standard deviation of the i-th Gaussian distribution, x represents the input EEG signal, a, b represent the EEG signal of length [a, b], log(*) represents the logarithmic function, μ represents the mean of x, and DE represents the differential entropy; The calculation expression of the power spectral density PSD is as follows: Among them, P(ω k ) represents the power spectral density PSD, M represents the length of the EEG signal, ω k represents the angular frequency, k represents the angular frequency index, k=-(M-1),-(M-2),···,0,1,···,(M-1),M-1; t represents the time, and its value ranges from 0 to M-1. Indicates the summation of internal data, e represents a natural constant, j represents an imaginary unit, x(t) represents the value of the EEG signal at time point t, γ(t) is the autocorrelation function of the EEG signal at time offset t, and represents the autocorrelation function value of the EEG signal at time offset t. The specific expression is as follows: γ(t)=E[x(n)x * (n+t)] Where E[*] represents the expectation, x(n) represents the value of the EEG signal at the nth sampling point, and x * (n+t) represents the complex conjugate value of the EEG signal at n+t, n represents the sampling point of a segment of the EEG signal, and x * represents the complex conjugate of x; The calculation expression of the nonlinear energy NE is as follows: Among them, N EEG represents the number of sampling points of the EEG signal, Indicates the internal N EEG The data are summed up, Γ(x(n)) represents the estimated value of nonlinear energy, and its calculation expression is as follows: Γ(x(n))=(y(n)) 2 +|y hibart (n)| 2 Among them, y(n) represents the first-order difference of the EEG signal x(n), y hibart (n) represents the Hibert transform of the first-order difference y(n) of the EEG signal, |y hibart (n)| 2 Represents y hibart (n) the square of the absolute value; The calculation expression of the fractal dimension FD is as follows: Among them, log 10 (*) indicates a logarithmic function with base 10, N EEG Indicates the number of sampling points of the EEG signal, N Δ Indicates the number of sign changes of the EEG signal.
3. The method for cross-scale attention convolutional EEG emotion recognition based on multiple features according to claim 1, characterized in that: The feature mapping process in step S3 is specifically as follows: mapping the features of the 32-electrode EEG signals into a two-dimensional feature matrix to obtain a two-dimensional feature map, mapping the features of the EEG signals of each frequency band to the two-dimensional feature map, and then stacking the two-dimensional feature maps of the four frequency bands into three-dimensional features, and forming the corresponding four-dimensional features from the three-dimensional features formed in each band. The specific expression is as follows: X DE =[d1,d2,d3,…,d N ] X PSD =[p1,p2,p3,…,p N ] X NE =[n1,n2,n3,…,n N ] <h2 style=";text-align:left;direction:ltr">X<h2 style=";text-align:left;direction:ltr"> FD <h2 style=";text-align:left;direction:ltr"> (f1,f2,f3,…,f)<h2 style=";text-align:left;direction:ltr"> N <h2 style=";text-align:left;direction:ltr"> ] Among them, d1, d2, d3, …, d n Represents the three-dimensional features of 1 to N DEs, X DE Represents the four-dimensional features of DE, p1, p2, p3, ..., p N Represents the three-dimensional features of 1 to N PSDs, X PSD Represents the four-dimensional features of PSD, n1,n2,n3,…,n N Represents the three-dimensional features of 1 to N NEs, X NE Represents the four-dimensional features of NE, f1,f2,f3,…,f N Represents the three-dimensional features of 1 to N FDs, X FD represents the four-dimensional characteristics of FD, DE represents differential entropy, PSD represents power spectral density, NE represents nonlinear energy, FD represents fractal dimension, and then X DE , X PSD , X NE and X FD Perform feature fusion, the specific expression is as follows: X con =[X DE ;X PSD ;X NE ;X FD ]=[d1;p1;n1;f1,…,d N ;p N ;n N ;f N ] Among them, X con Represents the fused feature set.
4. The method for cross-scale attention convolutional EEG emotion recognition based on multiple features according to claim 1, characterized in that: The cross-scale attention module captures features of different scales by using convolution kernels of different sizes in parallel and combining the frequency-space attention mechanism; The frequency-space attention module sequentially passes the frequency attention mechanism F for the input feature map X. f and spatial attention mechanism F S Then the output feature map Y is obtained, and the specific expression is as follows: Among them, F f (X) represents the operation of the frequency attention mechanism on the input feature map X, and X' represents the feature after the frequency attention mechanism operation. Indicates multiplication operation, F S (X') represents the features after the spatial attention mechanism operation, and Y represents the output feature map; The temporal feature extraction module is equipped with a bidirectional long short-term memory network. The bidirectional long short-term memory network uses two LSTM units when processing at each time step, one for processing EEG signals from front to back, and the other for processing EEG signals from back to front. The calculation results of the two LSTM units in different directions are superimposed and output; The deep classification module takes the result of the temporal feature extraction module as input and recognizes emotions.
5. The method for cross-scale attention convolution EEG emotion recognition based on multiple features according to claim 4, characterized in that: The frequency attention mechanism F in the frequency-space attention module f The specific process of processing the input feature map X is as follows: S4a.
1. Perform global adaptive average pooling on the input feature map X in the spatial dimension to obtain X avg , specifically expressed as follows: X avg =Adaptive_AvgPool(X(h,w)) Among them, X avg Represents the output of feature map X after pooling operation, Adaptive_AvgPool(*) represents global adaptive average pooling, X(h,w) represents the input feature map X, and the parameters are h and w, where h represents the height of the feature map and w represents the width of the feature map; S4a.2, X avg Input the multi-layer perceptron, and finally pass through the linear layer, and add the bias term b to get F f (X); S4a.3, F f (X) is multiplied by X to obtain X'; The multilayer perceptron consists of two fully connected layers and ReLu and Sigmoid activation functions. The expressions of ReLu activation function and Sigmoid activation function are as follows: Among them, F ReLu (x) represents the ReLu activation function, F Signod (x) represents the Sigmoid activation function, e represents a natural constant, and x represents the input EEG signal.
6. The method for cross-scale attention convolution EEG emotion recognition based on multiple features according to claim 5, characterized in that: The spatial attention mechanism F in the frequency-space attention module S The specific steps for processing X' are as follows: S4b.
1. Pass the feature map X' after the frequency attention mechanism operation through the maximum pooling layer, and obtain F through two-dimensional convolution and ReLu and Sigmoid activation functions. S (X'), the specific expression is as follows: X' max =MaxPool(X'(h,w)) Among them, X' max Represents the output of feature map X' after the maximum pooling operation, MaxPool(*) represents maximum pooling, X'(h,w) represents the input feature map X', the parameters are h and w, h represents the height of the feature map, and w represents the width of the feature map; S4b.2, F S Multiply (X') by X' to get Y.
7. The method for cross-scale attention convolution EEG emotion recognition based on multiple features according to claim 4, characterized in that: The LSTM unit calculation expression is as follows: k t =σ(W k ·[h t-1 ,q t ]+b k ) f t =σ(W f ·[h t-1 ,q t ]+b f ) q t =tanh(W c ·[h t-1 ,q t ]+b c ) C t =f t C t-1 +k t g t o t =σ(W o ·[h t-1 ,q t ]+b o ) h t =o t fishy(C) t ) Among them, k t represents the input gate, f t represents the forget gate, g t represents the candidate cell state, C t Indicates cell status, o t represents the output gate, h t Hidden state, h t-1 represents the hidden state of the previous time step, W k , W f , W c , W o represents the weight matrix of the corresponding gate, b k , b f , b c , b o represents the bias vector of the corresponding gate, q t Represents the input of the current time step, tanh(*) represents the hyperbolic tangent function, and σ(*) represents the Sigmoid function.
Citation Information
Patent Citations
MSCRNN emotion recognition method and device based on electroencephalogram signals
CN114662547A
Fatigue classification method based on four-dimensional attention convolutional recurrent neural network
CN114781442A
Internet of Things data prediction method
CN115330013A
EEG emotion recognition method by combining attention residual network with LSTM
CN117332317A
Multi-modal feature fusion emotion recognition method based on gating cross-attention mechanism
CN117370828A
Cited By
Emotion recognition method based on multi-domain attention fusion network
CN119557728A
AI digital human interaction method and system based on emotion recognition
CN120447735A
Emotion and fatigue integrated monitoring system and method based on electroencephalogram signals
CN121154180A
An emotion and fatigue integrated monitoring system and method based on electroencephalogram signals
CN121154180B
