Lightweight brain electrical emotion recognition method and system based on convolutional neural network
By connecting the timing convolution network and the three-dimensional convolution neural network, a lightweight TCNN model is built, which solves the complexity and memory requirements of the existing EEG emotion recognition model, and achieves the effect of high accuracy and lightweight deployment.
Patent Information
- Application Number
- CN202410940703.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-29
AI Technical Summary
The existing EEG emotion recognition model is complex, requires a large amount of trainable parameters and memory, which is difficult to deploy in complex environments, and has low accuracy.
The TCNN model is constructed in series by using time-series convolutional network (TCN) and three-dimensional convolutional neural network (3DCNN), to learn the frequency domain and spatial characteristics of EEG signals, and extract the relationship between multiple frequency bands to reduce the amount of trainable parameters and memory requirements.
A lighter model is realized, which improves the accuracy of EEG signal classification and the deployment convenience of model, especially the performance is relatively balanced on the SEED and SEED-IV datasets, with an accuracy rate of 98% and above.
Smart Images

Figure CN119048795A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of electroencephalogram emotion recognition, and in particular to a lightweight electroencephalogram emotion recognition method and system based on convolutional neural network. Background Art
[0002] Emotions are people's attitudes and experiences towards objective things and their corresponding behavioral responses, which have a significant impact on human behavior and mental health. How to accurately understand and process emotions, so as to achieve more intelligent and humanized human-computer interaction, the development and application of emotional computing are of great significance; emotion recognition can identify the emotional state contained in data such as voice, posture, facial expression and physiological signals, among which physiological signals are difficult to disguise and contain more information. In addition, among physiological signals such as EEG, ECG, EMG and skin temperature, EEG directly reflects the electrical activity of the brain, providing a direct physiological indicator to evaluate emotional state, which is more objective and accurate.
[0003] However, in the field of EEG signal emotion recognition, there are still many problems and methods waiting to be solved and discovered. First of all, for EEG emotion recognition tasks, many current works try to explore inter-channel information, because when emotions change, the connection between nodes also changes. Some studies propose to use each channel as a node to construct a graph structure, and then use a variant of GCN to learn the relationship between channels, or use frequency domain features to generate a feature matrix based on the topological structure of the cerebral cortex, and then build a deep learning model to extract the features of the relationship between channels. In fact, there are also rich relationships between different frequency bands of each channel, especially multi-band features extracted using a 2Hz frequency resolution, which can provide a lot of details for EEG classification tasks.
[0004] Secondly, the models proposed for EEG classification tasks are generally more complex, because the existing models have large trainable parameters and memory requirements in the frequency domain for temporal feature learning, and most of them use Long Short-Term Memory (LSTM) or Transformer that can process time series for feature extraction. Shallow 3DCNN can learn temporal and spatial features at the same time, and compared with the serially combined space-time model, the number of trainable parameters and computational complexity are smaller, and can replace the conventional temporal and spatial serial structure model part. Summary of the invention
[0005] In order to overcome the defects and shortcomings of the prior art, the present invention provides a lightweight EEG emotion recognition method and system based on convolutional neural network. The present invention connects the temporal convolutional network TCN and the three-dimensional convolutional neural network 3DCNN in series to construct a TCNN model, which can not only learn the temporal and spatial characteristics of EEG signals in the frequency domain, but also additionally learn the relationship between the multiple frequency bands of each channel. Compared with other models for the same task, the model is more lightweight, requires a smaller amount of trainable parameters, requires less time and memory in training, and is more conducive to deployment in complex environments.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] The present invention provides a lightweight EEG emotion recognition method based on convolutional neural network, comprising the following steps:
[0008] Obtaining raw EEG data, performing feature extraction on the raw EEG data, and generating a three-dimensional feature tensor;
[0009] The temporal convolutional network (TCN) and the three-dimensional convolutional neural network (3DCNN) are connected in series to construct a TCNN model.
[0010] The three-dimensional feature tensor is input into the TCNN model, and the temporal convolutional network TCN performs multi-frequency band relationship learning and outputs a 3D tensor block. The 3D tensor block is input into the three-dimensional convolutional neural network 3DCNN to perform time and space feature learning in the frequency domain, and learn to obtain representative features of multi-frequency band-time space;
[0011] A classifier is constructed to classify the learned representative features of the multi-band-time space to obtain the probability of each emotion category, and the emotion category with the highest probability is selected as the final emotion prediction result.
[0012] As a preferred technical solution, the feature extraction of the original EEG data to generate a three-dimensional feature tensor specifically includes:
[0013] The original EEG data is divided into multiple samples using a sliding window of set length, and each sample is divided into multiple segments;
[0014] The differential entropy of each channel of each segment is extracted as the frequency domain feature, and the frequency domain features are concatenated to generate a three-dimensional feature tensor.
[0015] As a preferred technical solution, the differential entropy of each channel of each segment is extracted as a frequency domain feature, specifically including:
[0016] Each segment is band-pass filtered from 0 Hz to 50 Hz with a step size of 2 Hz, and the differential entropy features are extracted from the signal of each frequency band to obtain the frequency domain features with a frequency resolution of 2 Hz.
[0017] As a preferred technical solution, the differential entropy feature is extracted from the signal of each frequency band, which is specifically expressed as follows:
[0018] F(X)=-∫ X p(x)log(p(x))dx
[0019] Where p(x) represents the probability density function of the signal in each frequency band, and X represents the signal in each frequency band;
[0020] When the signal of each frequency band follows Gaussian distribution N(μ,σ 2 ), the differential entropy feature is extracted for the signal of each frequency band, which is specifically expressed as:
[0021]
[0022] Among them, σ 2 represents the variance of X, π and e are both constants, and μ represents the mean of X.
[0023] As a preferred technical solution, the temporal convolutional network TCN adopts a TCN-based model. The TCN-based model is composed of two TCN blocks connected in sequence, and each TCN block is sequentially provided with an expanded non-causal convolution, a weight normalization, a ReLU activation function and a discard layer.
[0024] As a preferred technical solution, the output of each layer of the TCN-based model is expressed as:
[0025]
[0026] Where X∈R K × C × B represents a three-dimensional feature tensor, K represents the time in K seconds, C represents the number of channels, and B represents the number of frequency bands. represents the output of the model layer l, W l represents the convolution kernel of the lth layer, * d represents the exponential expansion rate, and σ(x) represents the sigmoid function.
[0027] As a preferred technical solution, the 3D tensor block is input into a three-dimensional convolutional neural network 3DCNN to perform time and space feature learning in the frequency domain, specifically including:
[0028] Use the 3D convolution kernel in the 3D convolutional neural network 3DCNN to perform convolution operations on the 3D tensor features, extract features in both time and space, use multiple 3D convolution kernels to extract multiple features, and each 3D convolution kernel shares weights;
[0029] Add BatchNorm3d layer and LeakyReLu layer to the input three-dimensional convolutional neural network 3DCNN.
[0030] As a preferred technical solution, a classifier is constructed to classify the learned representative features of the multi-band-time space, specifically including:
[0031] The representative features are flattened to 1 dimension, and then transformed by the fully connected layer to output the vector z. The vector z is input into the softmax layer. The softmax function converts the output of the fully connected layer into a set of probability values s, each of which represents the probability that the input sample belongs to a certain emotion category.
[0032] As a preferred technical solution, the vector z is expressed as:
[0033] z=f(Wx)+b
[0034] Among them, f represents the activation function, W represents the weight matrix, x represents the representative feature flattened to 1 dimension, and b represents the bias vector.
[0035] The present invention also provides a lightweight EEG emotion recognition system based on convolutional neural network, which is used to implement the above-mentioned lightweight EEG emotion recognition method based on convolutional neural network, and the system includes: an original EEG data acquisition module, a feature extraction module, a TCNN model construction module, a feature learning module, and a classification module;
[0036] The raw EEG data acquisition module is used to acquire raw EEG data;
[0037] The feature extraction module is used to extract features from the original EEG data and generate a three-dimensional feature tensor;
[0038] The TCNN model construction module is used to connect the temporal convolutional network TCN and the three-dimensional convolutional neural network 3DCNN in series to construct a TCNN model;
[0039] The feature learning module is used to input the three-dimensional feature tensor into the TCNN model, the temporal convolutional network TCN performs multi-band relationship learning, outputs a 3D tensor block, and the 3D tensor block is input into the three-dimensional convolutional neural network 3DCNN to perform time and space feature learning in the frequency domain, and learn to obtain representative features of multi-band-time space;
[0040] The classification module is used to construct a classifier to classify the learned representative features of the multi-band-time space, obtain the probability of each emotion category, and select the emotion category with the largest probability as the final emotion prediction result.
[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0042] (1) The present invention performs feature extraction on the original EEG data. Specifically, the differential entropy of each channel of each original EEG data sample segment is extracted as a frequency domain feature, and the frequency domain features are spliced to generate a three-dimensional feature tensor, which contains richer and more detailed frequency band information while also including the spatial and temporal features of the frequency domain.
[0043] (2) The present invention connects the temporal convolutional network (TCN) and the three-dimensional convolutional neural network (3DCNN) in series to construct a TCNN model, which can not only learn the temporal and spatial characteristics of EEG signals in the frequency domain, but also learn the relationship between the multiple frequency bands of each channel. Compared with other models for the same task, the model is more lightweight, requires a smaller number of trainable parameters, requires less time and memory during training, and is more conducive to deployment in complex environments.
[0044] (3) The TCNN model of the present invention achieved better classification results on the SEED and SEED-IV benchmark datasets. In the three categories of the SEED dataset, negative, neutral, and positive, the accuracy rate was 98%, and the performance was relatively balanced. In the four categories of SEED-IV, the accuracy rate in the neutral, sad, and happy categories was 98%, and the accuracy rate in the fear category was 96%.
[0045] (4) The present invention reveals the learning process of different neural network components and proves that the relationship between multiple frequency bands of each channel can be used as a robust feature for emotion classification tasks, and improves the interpretability of the TCNN model by visualizing and analyzing the features.
[0046] (5) The present invention adopts a non-traditional deep learning classification technology solution for EEG data, which solves the technical problems of the current mainstream model requiring large trainable parameters and memory, as well as low model accuracy, and achieves the technical effect of making it easier to deploy the model in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a flow chart of a lightweight EEG emotion recognition method based on a convolutional neural network of the present invention;
[0048] Figure 2 A schematic diagram of the implementation process of generating a three-dimensional feature tensor according to the present invention;
[0049] Figure 3 A schematic diagram of the implementation process of the lightweight model TCNN of the present invention when learning representative features of different emotional states;
[0050] FIG4( a ) is a schematic diagram of the network structure of the non-causal dilated convolution of the present invention;
[0051] FIG4( b ) is a schematic diagram of the network structure of the TCN-based model of the present invention;
[0052] Figure 5(a) is a convolution diagram of 2DCNN;
[0053] Figure 5(b) is a schematic diagram of the convolution of 3DCNN;
[0054] Figure 6 A schematic diagram of the implementation process of the present invention of classifying representative features of different emotions learned based on a classifier;
[0055] Figure 7 (a) is a diagram of the confusion matrix drawn based on the classification results of TCNN on the SEED dataset;
[0056] Figure 7 (b) is a diagram of the confusion matrix drawn based on the classification results of TCNN on the SEED-IV dataset. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0058] Example 1
[0059] like Figure 1 As shown, this embodiment provides a lightweight EEG emotion recognition method based on convolutional neural network, comprising the following steps:
[0060] S1: Obtaining raw EEG data, performing feature extraction on the raw EEG data, and generating a three-dimensional feature tensor;
[0061] In this embodiment, the original EEG data uses the uncut original EEG signals of the SEED and SEED-IV datasets, so it is necessary to cut them into multiple samples, and extract the differential entropy (DE) of each sample as a frequency domain feature, and use the frequency domain feature to generate a three-dimensional feature tensor;
[0062] S11: First, the original EEG data is segmented into multiple samples using a non-overlapping sliding window of K seconds in length, and each sample is divided into K 1-second segments;
[0063] In this embodiment, a non-overlapping sliding window (with a duration of K seconds) is used to crop the data to obtain all samples. In order to utilize the time domain information, each sample is divided into K segments with a duration of 1 second;
[0064] S12: Then, the differential entropy (DE) of each channel of K 1-second (s) segments of each sample is extracted, and the extracted DE features are concentrated together to generate a 3-dimensional feature tensor;
[0065] In this embodiment, the differential entropy (DE) which performs better than the traditional power spectrum density (PSD) is selected as the feature of emotion classification, and the differential entropy of the frequency band with an interval of 2Hz from 0 to 50Hz of each channel of the segment is extracted;
[0066] The original calculation formula for the differential entropy (DE) feature F(X) of an EEG signal X is:
[0067] F(X)=-∫ X p(x)log(p(x))dx
[0068] Where p(x) represents the probability density function of the EEG signal;
[0069] In addition, because the EEG signal is first band-pass filtered from 0 Hz to 50 Hz with a step size of 2 Hz, and then the DE feature is calculated for the EEG signal of each sub-band, the EEG signal in each sub-band almost follows the Gaussian distribution N(μ,σ 2 ), if the EEG frequency band signal X obeys the Gaussian distribution N(μ,σ 2 ), the calculation of the differential entropy (DE) feature F(X) of X can be simplified to the following equation:
[0070]
[0071] Where μ is the mean of X, σ 2 is the variance of X, π and e are constants;
[0072] For the same total frequency band (0-50Hz), most studies extract features from five frequency bands: δ (1-4Hz), θ (4-8Hz), α (8-13Hz), β (13-30Hz) and γ (30-45Hz). However, in order to obtain more and more detailed frequency band features, the original EEG signal is band-pass filtered from 0Hz to 50Hz with a step size of 2Hz, and the differential entropy features are extracted from the signal X of each frequency band to obtain the frequency domain features with a frequency resolution of 2Hz. In addition, 2Hz performs better than the EEG features of the five frequency bands in some tasks.
[0073] like Figure 2 As shown, each time segment H of K seconds (s) time sample is calculated in turn i The DE features of each frequency band of all channels are obtained to obtain the two-dimensional feature tensor D i ∈R C × B, which is a tensor of size C×B, where the number of channels (C) = 62 and the number of frequency bands (B) = 25. Then, the DE features D of K 1-second time segments are i The three-dimensional tensor feature block X = [D1, D2, ..., D K ]∈R C×B The three-dimensional tensor feature block contains the information about how the frequency domain features of multiple frequency bands change over time, so it contains rich multi-frequency-time-space information.
[0074] S2: Construct a lightweight model TCNN, take the 3D feature tensor as input, and learn the representative features of different emotional states through a network composed of a temporal convolutional network (TCN) and a three-dimensional convolutional neural network 3DCNN in series;
[0075] like Figure 3 As shown in the figure, the structure of the lightweight model TCNN is obtained and representative features are learned. TCNN aims to learn representative features of multiple frequency bands, time and space from 3D tensor feature blocks. TCNN is composed of Temporal Convolution Network (TCN) and 3D Convolutional Neural Network (3DCNN) in series. TCN focuses on learning the relationship features between multiple frequency bands of each channel, and 3DCNN focuses on learning the features of the frequency domain in time and space, including:
[0076] S21: Temporal Convolutional Network (TCN) for multi-band relationship learning;
[0077] In order to learn the relationship between the multi-frequency bands of the channel, the multi-frequency bands of the channel are regarded as sequences that do not contain time information. For sequence learning, RNN-based models such as LSTM and GRU, and Transformer can be used. However, these models have many trainable parameters and Flops, which require more memory in training and actual deployment. In addition, RNN-based and Transformer are targeted at sequences containing time information, and cannot play their advantages for sequences that do not contain time information. Temporal Convolutional Network (TCN) can solve these problems well. TCN can not only be trained in parallel, but also has lower storage requirements during training. In this embodiment, TCN is used to learn the potential relationship between the above sequences in different categories of time-frequency bands.
[0078] In this embodiment, the output of each layer of the TCN-based derivative model used can be expressed by the following formula:
[0079]
[0080] Where X∈R K × C × B Represents the input TCN-based three-dimensional feature tensor, which is specifically a tensor of size K×C×B, where K is K seconds, C is the number of channels, B is the number of frequency bands, and σ(x) represents the sigmoid function. represents the output of the lth layer of TCN, W l is the convolution kernel of the lth layer. In order to expand the receptive field, an exponential expansion rate* is used in the convolution operation of the convolution kernel on the feature tensor. d , the expansion rate of each layer is d l =2l-1. In this process, the zero-filling strategy is also used to keep the length of the time series unchanged.
[0081] In addition, since the multi-band channel constitutes a sequence of length 25 without time information, the dilated causal convolution in the TCN-based is replaced by dilated non-causal convolution, and only two dilated non-causal convolution layers with k = 3 are used to obtain a sufficient receptive field, and a residual structure is added to enhance the convolution performance.
[0082] As shown in Figure 4(a) and Figure 4(b), TCN-based consists of two TCN blocks (TCN-Block), each of which consists of a dilated non-causal convolution, weight normalization, ReLU activation function and a dropout layer. This embodiment connects two TCN blocks in series to enable them to learn deeper features. The input 3D feature tensor containing rich multi-band-time-space information is regarded as a sequence without time information at this time, and the relationship between the elements of the sequence, that is, the relationship between the frequency bands, is learned through TCN-based.
[0083] In this embodiment, the 3D tensor block T = [T1, T2, ..., T K ] will be fed into 3DCNN to continue learning the temporal-spatial features in the frequency domain.
[0084] S22: 3D convolutional neural network 3DCNN learns representative features of multi-band-time space;
[0085] TCN-based output 3D tensor block T = [T1, T2, ..., T K ]∈R K×C×BIt can still be considered as a two-dimensional tensor that changes with time, where each two-dimensional tensor is composed of frequency domain information of multiple channels and multiple frequency bands, so it contains spatial distribution information of the frequency domain. These two-dimensional tensors change with time, so they contain rich time and space information. This embodiment uses three 3D convolution layers (3DConv layer) in 3DCNN to learn the time-space features of the frequency domain.
[0086] As shown in Figure 5(a), the common 2DCNN can only learn the spatial features of each two-dimensional tensor in the three-dimensional tensor, and cannot learn the time domain features of the two-dimensional tensor that change over time. As shown in Figure 5(b), this embodiment uses the three-dimensional convolution kernel in the 3DCNN to perform a convolution operation on the three-dimensional tensor features, thereby extracting features in both time and space of the three-dimensional tensor features. A single convolution kernel can only extract one type of feature from the three-dimensional tensor features, because the weights of the convolution kernels are the same in the entire three-dimensional tensor features, that is, they share weights and are all the same convolution kernel. Therefore, this embodiment uses multiple convolution kernels to extract multiple features and obtain multi-dimensional and richer features.
[0087] In this embodiment, BatchNorm3d and LeakyReLu are added to all 3DCNN layers. BatchNorm3d can accelerate the training process, and LeakyReLu can alleviate the problem of gradient disappearance, which is specifically expressed as follows:
[0088]
[0089] Among them, E(X) is to find the mean of X, and var(x) is to find the variance of x;
[0090] LeakyReLU(X)=max(0,x)+0.01*min(0,X)
[0091] In addition, because the convolution kernel size of 3DCNN is limited, a large enough receptive field is obtained by stacking multiple convolution layers and using a small number of parameters.
[0092] As shown in Table 1 below, the structure of 3DCNN includes the kernel size, stride, padding, number of layers, activation function and output size of each layer. The 3D tensor block T = [T1, T2, ..., T K ]∈R K×C×BAs the input layer of 3DCNN, it then passes through three 3D convolutional layers (3DConv layer) to learn the temporal-spatial features in the frequency domain. The output of each layer of 3DConv layer is 2x3x62x25, 4x2x62x25 and 6x1x62x25 respectively.
[0093] Table 1 Schematic diagram of the structure of 3DCNN
[0094]
[0095] In this embodiment, a convolution operation is performed on the three-dimensional tensor feature by the three-dimensional convolution kernel in the 3DCNN, so as to extract features in both time and space of the three-dimensional tensor feature. Finally, the representative features learned by the 3DCNN, that is, the 3D data 6x1x62x25 outputted by the last 3D convolution layer, are sent to a classifier composed of a fully connected layer and a softmax layer to obtain a classification result.
[0096] S3: Build a classifier to classify the representative features learned by TCNN;
[0097] In this embodiment, the representative features learned by TCNN are three-dimensional data and cannot be directly used for classification. Figure 6 As shown in the figure, a classifier is constructed to classify the representative features learned by TCNN. First, the representative features are flattened to 1 dimension and then converted into a vector z with a specific size by the fully connected layer (Linear layer). This size matches the number of categories to be classified. The vector z is expressed as:
[0098] z=f(Wx)+b
[0099] Among them, x is the representative feature flattened to 1 dimension, W is the weight matrix, b is the bias vector, f is the activation function, and z is the output of the fully connected layer, which is the same as the number of categories for classification;
[0100] Then, this vector z is sent to the softmax layer. The softmax function converts the output of the fully connected layer into a set of probability values s, the sum of which is 1.
[0101] The relationship between the probability value s and the vector z is:
[0102]
[0103] Each probability value represents the probability that the input sample belongs to a certain category, that is, the probability of each emotion category. Finally, the emotion category with the highest probability is selected as the final prediction result of the model.
[0104] This embodiment is tested on the public datasets SEED and SEED-IV, and the accuracy of each category is tested respectively. The trainable parameters of TCNN are calculated and compared with other methods. This embodiment has significantly improved the accuracy of emotion recognition, as follows:
[0105] In order to obtain the classification performance of TCNN in each category of SEED and SEED-IV datasets, the correct classification results of all test sets of the datasets are counted, and the confusion matrix corresponding to the SEED and SEED-IV datasets is finally obtained, as shown in Figure 7 (a) Figure 7 As shown in (b), we can see that TCNN has achieved 98% accuracy in the three categories of SEED dataset, negative, neutral, and positive, and its performance is relatively balanced. In addition, TCNN has achieved 98% accuracy in the four categories of SEED-IV, including neutral, sad, and happy, and 96% accuracy in the fear category.
[0106] In order to reflect the lightweight of the TCNN proposed in this embodiment, as shown in Table 2 below, the trainable parameter amounts of TCNN and mainstream methods in the past one or two years are summarized in the table:
[0107] Table 2 Comparison of trainable parameters of TCNN model and other mainstream models
[0108]
[0109] It can be seen that while TCNN achieves good performance on the SEED and SEED-IV datasets, the amount of trainable parameters is 0.6627M, which is much smaller than the models of other methods. HCRNN uses LSTM to extract time domain features, so the total amount of trainable parameters is large, which is 1.2224M. ACTNN uses a complex time domain coding part to extract the time features of the frequency domain, and the total amount of trainable parameters is 3.5937M. For S2D+RFPN, since the RFPN part learns features of different scales at the same time, each scale has trainable parameters, and the total amount of trainable parameters is 3.961M. In summary, the time feature learning part in the frequency domain of other baseline methods occupies most of the trainable parameters. This embodiment uses TCN to learn the relationship between multiple frequency bands as features, and then uses a shallow 3DCNN to simultaneously learn the time and space features of the frequency domain. Since TCN and shallow 3DCNN are lightweight models, the total amount of trainable parameters is less, which is more conducive to being installed on low-power wearable devices.
[0110] In general, the lightweight EEG emotion recognition method based on convolutional neural network in this embodiment not only achieves good performance in classification results, but also preserves the lightweight of the model, making it convenient to implement and install in actual complex environments.
[0111] Example 2
[0112] This embodiment provides a lightweight EEG emotion recognition system based on convolutional neural network, which is used to implement the lightweight EEG emotion recognition method based on convolutional neural network in the above embodiment 1. The system includes: an original EEG data acquisition module, a feature extraction module, a TCNN model construction module, a feature learning module, and a classification module;
[0113] In this embodiment, the raw EEG data acquisition module is used to acquire raw EEG data;
[0114] In this embodiment, the feature extraction module is used to extract features from the original EEG data and generate a three-dimensional feature tensor;
[0115] In this embodiment, the TCNN model construction module is used to connect the temporal convolutional network TCN and the three-dimensional convolutional neural network 3DCNN in series to construct a TCNN model;
[0116] In this embodiment, the feature learning module is used to input the three-dimensional feature tensor into the TCNN model, the temporal convolutional network TCN performs multi-band relationship learning, outputs a 3D tensor block, and the 3D tensor block is input into the three-dimensional convolutional neural network 3DCNN to perform time and space feature learning in the frequency domain, and learn to obtain representative features of multi-band-time space;
[0117] In this embodiment, the classification module is used to construct a classifier to classify the learned multi-band-time-space representative features, obtain the probability of each emotion category, and select the emotion category with the highest probability as the final emotion prediction result.
[0118] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A lightweight EEG emotion recognition method based on convolutional neural network, characterized in that: The steps include: Obtaining raw EEG data, performing feature extraction on the raw EEG data, and generating a three-dimensional feature tensor; The temporal convolutional network (TCN) and the three-dimensional convolutional neural network (3DCNN) are connected in series to construct a TCNN model. The three-dimensional feature tensor is input into the TCNN model, and the temporal convolutional network TCN performs multi-band relationship learning and outputs a 3D tensor block. The 3D tensor block is input into the three-dimensional convolutional neural network 3DCNN to perform time and space feature learning in the frequency domain, and learn to obtain multi-band-time-space representative features; A classifier is constructed to classify the learned representative features of the multi-band-time space to obtain the probability of each emotion category, and the emotion category with the highest probability is selected as the final emotion prediction result.
2. The lightweight EEG emotion recognition method based on convolutional neural network according to claim 1 is characterized in that: The feature extraction of the original EEG data to generate a three-dimensional feature tensor specifically includes: The original EEG data is divided into multiple samples using a sliding window of set length, and each sample is divided into multiple segments; The differential entropy of each channel of each segment is extracted as the frequency domain feature, and the frequency domain features are concatenated to generate a three-dimensional feature tensor.
3. The lightweight EEG emotion recognition method based on convolutional neural network according to claim 2 is characterized in that: The differential entropy of each channel of each fragment is extracted as the frequency domain feature, including: Each segment is band-pass filtered from 0 Hz to 50 Hz with a step size of 2 Hz, and the differential entropy features are extracted from the signal of each frequency band to obtain the frequency domain features with a frequency resolution of 2 Hz.
4. The lightweight EEG emotion recognition method based on convolutional neural network according to claim 3 is characterized in that: The differential entropy feature is extracted from the signal of each frequency band, which is specifically expressed as: F(X)=-∫ X p(x)log(p(x))dx Where p(x) represents the probability density function of the signal in each frequency band, and X represents the signal in each frequency band; When the signal in each frequency band follows the Gaussian distribution N(μ, σ 2 ), the differential entropy feature is extracted for the signal of each frequency band, which is specifically expressed as: Among them, σ 2 represents the variance of X, π and e are both constants, and μ represents the mean of X.
5. The lightweight EEG emotion recognition method based on convolutional neural network according to claim 1, characterized in that: The temporal convolutional network (TCN) adopts a TCN-based model, which consists of two TCN blocks connected in sequence. Each TCN block is equipped with an expanded non-causal convolution, weight normalization, a ReLU activation function, and a dropout layer.
6. The lightweight EEG emotion recognition method based on convolutional neural network according to claim 5 is characterized in that: The output of each layer of the TCN-based model is expressed as: Where X∈R K×C×B represents a three-dimensional feature tensor, K represents the time in K seconds, C represents the number of channels, and B represents the number of frequency bands. represents the output of the model layer l, W l represents the convolution kernel of the lth layer, * d represents the exponential expansion rate, and σ(x) represents the sigmoid function.
7. The lightweight EEG emotion recognition method based on convolutional neural network according to claim 1, characterized in that: The 3D tensor block is input into a three-dimensional convolutional neural network 3DCNN to perform time and space feature learning in the frequency domain, specifically including: Use the 3D convolution kernel in the 3D convolutional neural network 3DCNN to perform convolution operations on the 3D tensor features, extract features in both time and space, use multiple 3D convolution kernels to extract multiple features, and each 3D convolution kernel shares weights; Add BatchNorm3d layer and LeakyReLu layer to the input three-dimensional convolutional neural network 3DCNN.
8. The lightweight EEG emotion recognition method based on convolutional neural network according to claim 1, characterized in that: Construct a classifier to classify the learned representative features of multi-band-time space, including: The representative features are flattened to 1 dimension, and then transformed by the fully connected layer to output the vector z. The vector z is input into the softmax layer. The softmax function converts the output of the fully connected layer into a set of probability values s, each of which represents the probability that the input sample belongs to a certain emotion category.
9. The lightweight EEG emotion recognition method based on convolutional neural network according to claim 8, characterized in that: The vector z is represented as: z=f(Wx)+b Among them, f represents the activation function, W represents the weight matrix, x represents the representative feature flattened to 1 dimension, and b represents the bias vector.
10. A lightweight EEG emotion recognition system based on convolutional neural network, characterized in that: The system is used to implement the lightweight EEG emotion recognition method based on convolutional neural network as described in any one of claims 1 to 9, comprising: an original EEG data acquisition module, a feature extraction module, a TCNN model construction module, a feature learning module, and a classification module; The raw EEG data acquisition module is used to acquire raw EEG data; The feature extraction module is used to extract features from the original EEG data and generate a three-dimensional feature tensor; The TCNN model construction module is used to connect the temporal convolutional network TCN and the three-dimensional convolutional neural network 3DCNN in series to construct a TCNN model; The feature learning module is used to input the three-dimensional feature tensor into the TCNN model, the temporal convolutional network TCN performs multi-band relationship learning, outputs a 3D tensor block, and the 3D tensor block is input into the three-dimensional convolutional neural network 3DCNN to perform time and space feature learning in the frequency domain, and learn to obtain representative features of multi-band-time space; The classification module is used to construct a classifier to classify the learned representative features of the multi-band-time space, obtain the probability of each emotion category, and select the emotion category with the largest probability as the final emotion prediction result.
Citation Information
Cited By
Epileptic seizure detection and positioning system and method combining time convolution and Transform network
CN120203503A
Ceramic censer design method and system based on perceptual engineering
CN120562241A
