Emotion recognition method of EEG signals based on online adaptive feature parameter optimization

Abstract

The present invention discloses a method for EEG signal emotion recognition based on online adaptive feature parameter optimization, comprising: 1. preprocessing raw EEG data to divide source domain data into target domain data; 2. establishing a source domain multi-layer perceptron network, extracting source domain data features, and then inputting the features into the source domain network for training to obtain an optimal source domain model; 3. establishing a target domain MLP network containing an online feature parameter extraction module; 4. extracting features from the target domain data, and then inputting the features into the target domain model for online self-optimization and adjustment of feature parameters, thereby obtaining the optimized target domain model and completing the emotion recognition task for the target domain data. The present invention can optimize an existing model without contacting the source domain data and only contacting the target domain data once, while protecting the privacy of the subject, thereby improving recognition accuracy and completing online emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of affective computing, and in particular to an EEG emotion recognition method based on online adaptive feature parameter optimization. Background Art

[0002] Emotions are a comprehensive manifestation of human consciousness and behavior, reflecting both subjective feelings and physiological responses. Numerous research methods exist for emotion recognition, with both non-physiological and physiological signals commonly used to assess emotions. Common non-physiological signals include facial expressions, language, and body movements. However, assessing emotions based on these signals suffers from a lack of standardized criteria and low recognition accuracy. While physiological signals are more difficult to acquire than non-physiological signals, they offer higher recognition accuracy, making them essential for systematic emotion recognition. Electroencephalogram (EEG), as a physiological signal, strongly correlates with cognitive behavior and psychological activity, possesses excellent temporal resolution, and can more directly reflect emotional changes. These signals are also less susceptible to subjective manipulation. Therefore, researchers often employ EEG-based emotion recognition methods. EEG emotion recognition algorithms fall into two main categories: traditional algorithms and deep learning-based algorithms.

[0003] Deep learning-based emotion recognition methods are categorized into user-dependent and user-independent methods, depending on whether they target single or multiple users. User-dependent methods target a specific subject, using the same subject's EEG data for training and testing to ultimately achieve emotion recognition. User-independent methods aim to use a source-domain model trained on a specific subject's EEG data to achieve high recognition accuracy on target-domain EEG data from different subjects. In recent years, user-dependent EEG emotion recognition methods have made significant progress. However, due to the significant differences in the distribution of EEG data between individuals, models trained using source-domain EEG data from a single subject inevitably suffer significant drops in recognition accuracy when tested on target-domain EEG data from different subjects. Furthermore, user-independent EEG emotion recognition methods require EEG signal data from different users, transferring it between multiple users, developers, and organizations to improve cross-user and cross-dataset performance. However, by directly sharing a subject's EEG data, attackers can infer personal and private information about the subject without their consent or knowledge. The collected EEG data is highly confidential and can reflect the user's personality traits, cognitive abilities, physical and mental health, etc., thus leading to the leakage of the subject's privacy information. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention proposes an EEG signal emotion recognition method based on online adaptive feature parameter optimization, so that the existing recognition model can be tuned without contacting the source domain data and only contacting the target domain data once, while protecting the privacy of the subjects, thereby improving the recognition accuracy and completing online emotion recognition.

[0005] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0006] The method for emotion recognition based on EEG signals based on online adaptive feature parameter optimization is characterized in that it is performed according to the following steps:

[0007] Step 1: Preprocess the original EEG signal and perform data division;

[0008] Step 1.1: Obtain the raw EEG signal data of A subjects and downsample them. Then, use a bandpass filter to filter the downsampled EEG signal data. Then, use a sliding window to segment the filtered EEG signal data to obtain preprocessed EEG samples.

[0009] Step 1.2: Use the leave-one-out method to partition the preprocessed EEG samples, so that the preprocessed EEG samples of a subject are used as the target domain data X t , the remaining A-1 subjects’ pre-processed EEG samples are used as source domain data X s ;and in, Represents the source domain data X s The i-th source domain EEG sample, Represents the target domain data X t The i-th target domain EEG sample, Q represents the number of channels of the EEG sample, and P represents the number of sampling points; N s ,N t Represent the source domain data X s and target domain data X t The number of EEG samples; let the i-th source domain EEG sample The label is Let the i-th target domain EEG sample The label is and r represents a certain emotion category of the label, and E represents the total number of emotion categories of the label;

[0010] Step 2: In the source domain data X s and its label Y s A multi-layer perceptron (MLP) network is built and trained to obtain the optimal source domain model g of the subject. s ;

[0011] Step 3: Target domain data X t The multi-layer perceptron MLP network with feature parameter extraction module TFPE is established and trained in batches to obtain the optimal source domain model g of the subject. t ;

[0012] The feature of the method for EEG signal emotion recognition based on online adaptive feature parameter optimization of the present invention is that step 2 is performed as follows:

[0013] Step 2.1: Use short-time Fourier transform to extract the i-th source domain EEG sample x i The differential entropy features of several frequency bands in each channel of s, and after splicing the frequency band features of all channels, the i-th source domain EEG sample x is obtained i Differential entropy source domain characteristics after s splicing F is the number of features after splicing;

[0014] Step 2.2: Build a source domain MLP network based on a multi-layer perceptron, consisting of M linear layers, M batch normalization layers, M activation function layers, and 1 fully connected layer.

[0015] Step 2.3: The spliced ​​differential entropy source domain features Input the source domain MLP network for processing to obtain the i-th source domain EEG sample x i s The predicted label y′ i s ;

[0016] Step 2.4: Based on the i-th source domain EEG sample Tags and predicted labels Use formula (1) to construct the cross entropy loss function

[0017]

[0018] In formula (1), Represents the i-th source domain EEG sample Tags The rth emotion label, Represents the predicted label y′ i s The rth emotion label The probability value of

[0019] Step 2.5: Use SGD optimizer to train the source domain MLP network and perform cross entropy loss function Minimize the solution to the parameter θ of the source domain MLP network sOptimize and finally get the optimal source domain model g s ;

[0020] Described step 3 is carried out as follows:

[0021] Step 3.1: Build a target domain MLP network, consisting of M linear layers and their corresponding feature parameter extraction modules (TFPEs), M batch normalization layers, M activation function layers, and one fully connected layer.

[0022] The optimal source domain model g s The parameters of are shared with the target domain MLP network, and the feature parameter extraction module TFPE is initialized to obtain the initialized target domain MLP network;

[0023] Define the current number of training times as n and initialize n=1;

[0024] The initialized feature parameter extraction module TFPE is used as the feature parameter extraction module TFPE for the n-1th training; the initialized target domain MLP network is used as the target domain model for the n-1th optimization;

[0025] Step 3.2: Follow the process of step 2.1 to process the target domain data X t Processing is performed to obtain the spliced ​​differential entropy target domain features and divide them into B batches of features;

[0026] Step 3.3: Input the features of the nth batch into the initialized target domain MLP network for processing to obtain the predicted labels of the nth batch of EEG samples. in, Represents the jth target domain EEG sample in the nth batch The predicted label of

[0027] Step 3.4: Based on the j-th target domain EEG sample of the n-th batch The predicted label Use formula (2) to construct the Shannon entropy loss function for the nth training

[0028]

[0029] In formula (2), Represents the number of EEG samples contained in the nth batch, Represents the jth target domain EEG sample in the nth batch The predicted label The rth emotion label The probability value of

[0030] Step 3.5, freeze the shared parameters in the target domain MLP network; use the SGD optimizer to train the feature parameter extraction module TFPE for the n-1th training, and calculate the Shannon entropy loss function of the nth training. Perform minimization to optimize the parameters of the feature parameter extraction module TFPE corresponding to the n-th batch of EEG samples, and obtain the n-th trained feature parameter extraction module TFPE and the n-th optimized target domain model;

[0031] Step 3.6: After assigning n+1 to n, return to step 3.3 and execute sequentially until n>B, thus obtaining the final trained optimal target domain model g. t , used for emotion recognition on EEG signal data.

[0032] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the method, and the processor is configured to execute the program stored in the memory.

[0033] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the method when executed by a processor.

[0034] Compared with the prior art, the beneficial effects of the present invention are embodied in:

[0035] The present invention aims to protect the privacy of the subjects, without using the subjects' labeled EEG data during source domain training and only contacting the target domain data once, and only using the trained source domain model and target domain test data for online adaptive feature parameter optimization. The optimization of the target domain model is completed online through only one iteration for each batch of target domain data, thereby achieving real-time performance and improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the process of the present invention;

[0037] Figure 2 This is a schematic diagram of the source domain model network of the present invention;

[0038] Figure 3 is a structural diagram of the TFPE module of the present invention;

[0039] Figure 4 This is a network diagram of the target domain model of the present invention;

[0040] Figure 5 This is a flow chart of the present invention for optimizing the target domain model in batches. DETAILED DESCRIPTION

[0041] In this embodiment, a method for EEG emotion recognition based on online adaptive feature parameter optimization is mainly to add a feature parameter extraction module TFPE (Test-time Feature Parameter Extractor) to the source domain model, collect feature parameters through the TFPE module to deeply extract signal features. Then, the collected feature parameters are optimized using Shannon entropy as the loss function of the stochastic gradient descent (SGD) optimizer, and finally the target domain model after online adaptive feature parameter optimization is obtained to achieve emotion classification, such as Figure 1 As shown, the steps are as follows:

[0042] Step 1: Preprocess the original EEG signal and then divide the data;

[0043] Step 1.1: Obtain the original EEG signal data X of A subjects, where X={X1,X2,...,X i ,...,X N}, where X i ∈R Q×P represents the i-th EEG signal sample, Q represents the number of EEG signal channels, P represents the number of sampling points, N represents the number of EEG samples contained, and downsampling is performed using a sampling rate of 200 Hz, and then a bandpass filter is used to filter the downsampled EEG signal data, and then a 1s sliding window is used to segment the filtered EEG signal data to obtain the preprocessed EEG sample of the subject's EEG signal sample; the emotion label of the EEG signal is recorded as Y = {y1, y2, ..., y i ,...,y N}, determined by the labels of the movie clips watched by the subjects; the SEED dataset used in this paper is a selection of 15 clips (positive, neutral and negative emotions) from 6 movies as stimuli used in the experiment. Each movie clip has been carefully edited to produce a coherent emotion and maximize the emotional meaning; the emotion labels are divided into three categories: positive emotion, neutral emotion and negative emotion;

[0044] Step 1.2: Use the leave-one-out method to partition the preprocessed EEG samples, so that the preprocessed EEG samples of a subject are used as the target domain data X t The remaining A-1 subjects’ pre-processed EEG samples are used as the source domain data X s ;and in, Represents the source domain data X s The i-th source domain EEG sample, Represents the target domain data X t The i-th target domain EEG sample; N s ,N t Represent the source domain data X s and target domain data X t The number of EEG samples; let the i-th source domain EEG sample The label is Let the i-th target domain EEG sample The label is and r represents a certain emotion category of the label, and E represents the total number of emotion categories of the label;

[0045] Step 2: In the source domain data X s and its label Y s A multilayer perceptron (MLP) network is built and trained to obtain the optimal source domain model g of the subject. s ;

[0046] Step 2.1: Use short-time Fourier transform to extract the i-th source domain EEG sample x i s The differential entropy features of several frequency bands in each channel of , and after splicing the frequency band features of all channels, the i-th source domain EEG sample x is obtained i s Differential entropy source domain characteristics after splicing F is the number of features after splicing;

[0047] Step 2.2: Build a source domain MLP network based on a multi-layer perceptron, including: M linear layers, M batch normalization layers, M activation function layers, and 1 fully connected layer;

[0048] Specifically, the source domain network is as follows Figure 2 As shown in the figure, it consists of 4 linear layers, 4 batch normalization layers, 4 activation function layers and 1 fully connected layer: the first part consists of the linear layer linear1, the batch normalization layer BN1 (which normalizes the offset data distribution to standardize and speed up the model convergence), and the LeakyRelu activation function. The second, third, and fourth parts of the network structure are the same as the first part. The last layer is the fully connected layer fc (which maps the input features to E categories of emotions).

[0049] Step 2.3: The spliced ​​differential entropy source domain features Input the source domain MLP network for processing to obtain the i-th source domain EEG sample The predicted label

[0050] The differential entropy source domain features after splicing in step 2.1 Input into the source domain MLP network for processing, and the output features after the source domain MLP network are recorded as Will Input the external SoftMax function layer to obtain the probability values ​​of E emotional states. The calculation method is shown in formula (1) and formula (2):

[0051]

[0052]

[0053] The Softmax function can transform the output of the sample into a probability density function. The label corresponding to the value with the largest probability is used as the result of emotion recognition, thereby obtaining the i-th source domain EEG sample. The predicted label

[0054] Step 2.4: Based on the i-th source domain EEG sample Tags and predicted labels Use formula (3) to construct the cross entropy loss function of the nth training

[0055]

[0056] In formula (3), Represents the i-th source domain EEG sample x i s Label y i s The rth emotion label, Represents the predicted label y′ i s The rth emotion label The probability value of

[0057] Step 2.5: Use the stochastic gradient descent SGD optimizer to train the source domain MLP network and use the cross entropy loss function for the nth training Minimize the solution to the parameter θ of the source domain MLP network s Optimize and finally get the optimal source domain model g s ;

[0058] Step 3: Create Figure 3 The multi-layer perceptron MLP network of the feature parameter extraction module TFPE shown in the figure is used to extract the target domain data X t After batching, the network training is performed and the parameters in TFPE are optimized to obtain the optimal source domain model g of the subject. t ;

[0059] Step 3.1: Build the target domain MLP network, which consists of M linear layers and their corresponding feature parameter extraction modules (TFPE), M batch normalization layers, M activation function layers, and 1 fully connected layer.

[0060] Specifically, the source domain network is as follows Figure 4 As shown, it consists of 4 linear layers and 4 correspondingly connected feature parameter extraction modules TFPE, 4 batch normalization layers, 4 activation function layers and 1 fully connected layer: the first part consists of the linear layer linear1, the batch normalization layer BN1 (which normalizes the offset data distribution to speed up the model convergence) and the LeakyRelu activation function. The network structures of the second, third and fourth parts are the same as the first part, and the last layer is the fully connected layer fc (which maps the input features to E types of emotion classification).

[0061] The optimal source domain model g s The parameters of the target domain MLP network are shared, and the feature parameter extraction module TFPE is initialized (the initial value of the scale parameter γ contained in TFPE is set to 1, and the initial value of the shift parameter β is set to 0), thereby obtaining the initialized target domain MLP network;

[0062] Define the current number of training times as n and initialize n=1;

[0063] The initialized feature parameter extraction module TFPE is used as the feature parameter extraction module TFPE for the n-1th training; the obtained initialized target domain MLP network is used as the target domain model for the n-1th optimization;

[0064] Step 3.2: Follow the process of step 2.1 to process the target domain data X t Processing is performed to obtain the differential entropy target domain features after splicing and divide the data into B batches;

[0065] Step 3.3: Input the features of the nth batch into the initialized target domain MLP network for processing to obtain the predicted labels of the nth batch of EEG samples.

[0066] The nth batch of differential entropy target domain features after splicing in step 3.2 The input is processed in the source domain MLP network, and the output features after passing through the target domain MLP network are recorded as Will g Input the external SoftMax function layer and convert the sample output into a probability density function according to the process of step 2.3. The label corresponding to the value with the largest probability is used as the result of emotion recognition, thereby obtaining the probability value of E kinds of emotional states, and thus obtaining the target domain EEG sample of the nth batch The predicted label in, Represents the jth target domain EEG sample in the nth batch The predicted label of

[0067] Step 3.4: Based on the j-th target domain EEG sample in the n-th batch The predicted label Use formula (4) to construct the Shannon entropy loss function

[0068]

[0069] In formula (4), Represents the number of EEG samples contained in the nth batch, Represents the jth target domain EEG sample in the nth batch The predicted label The rth emotion label The probability value of

[0070] Step 3.5, freeze the shared parameters in the target domain MLP network; use the SGD optimizer to train the feature parameter extraction module TFPE for the n-1th training, and calculate the Shannon entropy loss function. Perform minimization to optimize the parameters of the feature parameter extraction module TFPE corresponding to the n-th batch of EEG samples, and obtain the n-th trained feature parameter extraction module TFPE and the n-th optimized target domain model;

[0071] Step 3.6: After assigning n+1 to n, return to step 3.3 and execute sequentially until n>B, thus obtaining the final trained optimal target domain model g. t , used for emotion recognition on EEG signal data.

[0072] In this embodiment, the batch number B is set to 4, and the process diagram of optimizing the target domain model in 4 batches is as follows: Figure 5As shown in the figure, by training only the TFPE parameters, the original parameters of the source domain model remain unchanged, while the deep features extracted by the linear layer of the source domain model are adjusted and optimized to match the distribution of the target dataset. Shannon entropy is chosen as the loss function for optimizing the target domain model because the supervised model has already learned sufficient knowledge during training on the source domain data. This allows for online adaptive feature parameter optimization on the target domain data, thereby reducing the entropy of the target domain model and minimizing prediction errors.

[0073] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0074] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0075] In the specific implementation, the target domain model obtained based on online adaptive feature parameter optimization is compared with the source domain model, and the average recognition accuracy of the target domain data is used as the evaluation indicator. The higher the recognition accuracy, the better the recognition ability of the model. The average recognition accuracy of 15 subjects in three sessions is shown in Table 1:

[0076] Table 1

[0077]

[0078]

[0079] The average recognition accuracy of all subjects is shown in Table 2:

[0080] Table 2

[0081] Accuracy (%) Source domain model 49.4 Target domain model 62.1

[0082] The experimental results in Tables 1 and 2 show that the model obtained through online adaptive feature parameter optimization has a significant improvement in recognition accuracy, verifying that the EEG emotion recognition method based on online adaptive feature parameter optimization can optimize the model using only target domain data, and this method has good recognition capabilities for different subjects.

[0083] In summary, the present invention can achieve a higher recognition accuracy by using only target domain data without using source domain data while protecting user privacy.

[0084] The present invention proposes a model obtained by the EEG signal emotion recognition method based on online adaptive feature parameter optimization. In this embodiment, the number of batches of target domain data divided for completing the online adaptive feature parameter optimization is set to 4. The target domain data is only contacted once during the testing phase, achieving the effect of online rapid processing of EEG data, and at the same time achieving good recognition results on the public dataset SEED.

Claims

1. A method for EEG emotion recognition based on online adaptive feature parameter optimization, characterized in that: The steps are as follows: Step 1: Preprocess the original EEG signal and perform data division; Step 1.1: Get The original EEG signal data of each subject is downsampled, and then the downsampled EEG signal data is filtered using a bandpass filter. Then, the filtered EEG signal data is segmented using a sliding window to obtain the preprocessed EEG samples. Step 1.2: Use the leave-one-out method to partition the preprocessed EEG samples, so that the preprocessed EEG samples of one subject are used as the target domain data. , the rest The EEG samples preprocessed by the subjects are used as source domain data ;and , ;in, Represents source domain data No. i Source domain EEG samples, Represents target domain data No. i target domain EEG samples, Indicates the number of channels of EEG samples, Indicates the number of sampling points; Represents source domain data and target domain data The number of EEG samples; i Source domain EEG samples The label is , order i Target domain EEG samples The label is ,and , r Indicates a certain emotional category of the label, E Indicates the total number of emotion categories of the label; Step 2: In the source domain data and its label A multi-layer perceptron (MLP) network is built and trained to obtain the optimal source domain model of the subject. ; Step 3: Build a target domain MLP network, which consists of M linear layers and their corresponding feature parameter extraction modules (TFPEs), M batch normalization layers, M activation function layers, and one fully connected layer. TFPE includes scale and shift parameters. Target domain data The training is carried out in batches to establish a multi-layer perceptron MLP network with a feature parameter extraction module TFPE and train it in batches to obtain the optimal target domain model of the subject. , used for emotion recognition on EEG signal data.

2. The method for EEG emotion recognition based on online adaptive feature parameter optimization according to claim 1, characterized in that: Described step 2 is carried out as follows: Step 2.1: Use short-time Fourier transform to extract the i Source domain EEG samples The differential entropy features of several frequency bands in each channel of the , and after splicing the frequency band features of all channels, we get the i Source domain EEG samples Differential entropy source domain characteristics after splicing , F is the number of features after splicing; Step 2.2: Build a source domain MLP network based on a multi-layer perceptron, consisting of M linear layers, M batch normalization layers, M activation function layers, and 1 fully connected layer. Step 2.3: The spliced ​​differential entropy source domain features Input the source domain MLP network for processing and obtain the i Source domain EEG samples The predicted label ; Step 2.4, based on i Source domain EEG samples Tags and predicted labels , use formula (1) to construct the cross entropy loss function : (1) In formula (1), Representative i Source domain EEG samples Tags Middle r Emotional labels, Represents the predicted label Middle r Emotional labels The probability value of Step 2.5: Use SGD optimizer to train the source domain MLP network and perform cross entropy loss function Minimize the parameters of the source domain MLP network Optimize and finally get the optimal source domain model .

3. The method for EEG emotion recognition based on online adaptive feature parameter optimization according to claim 2, characterized in that: Described step 3 is carried out as follows: Step 3.1: The optimal source domain model The parameters of are shared with the target domain MLP network, and the feature parameter extraction module TFPE is initialized to obtain the initialized target domain MLP network; Define the current number of training times as , and initialize ; The initialized feature parameter extraction module TFPE is used as the The feature parameter extraction module TFPE of the second training; The initialized target domain MLP network is used as the Suboptimal target domain model; Step 3.2: Follow the process of step 2.1 to analyze the target domain data. After processing, the differential entropy target domain features after splicing are obtained and divided into Characteristics of the batch; Step 3.3, The features of the batch are input into the initialized target domain MLP network for processing to obtain the Predicted labels for batches of EEG samples ;in, Indicates the The first batch Target domain EEG samples The predicted label of Step 3.4, based on Batch No. Target domain EEG samples The predicted label , using formula (2) to construct the Shannon entropy loss function for training ; (2) In formula (2), Representative The number of EEG samples contained in the batch, Representative The first batch Target domain EEG samples The predicted label Middle r Emotional labels The probability value of Step 3.5: Freeze the shared parameters in the target domain MLP network; use the SGD optimizer to optimize the The feature parameter extraction module TFPE trained for the first time training, and Shannon entropy loss function for training Minimize the solution to The parameters of the feature parameter extraction module TFPE corresponding to the batch EEG samples are optimized to obtain the The feature parameter extraction module TFPE trained in the second Suboptimal target domain model; Step 3.6, Assign to Then return to step 3.3 and execute sequentially until So far, the final trained optimal target domain model is obtained , used for emotion recognition on EEG signal data.

4. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the method according to any one of claims 1 to 3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are performed.

Images