Emotion monitoring method based on online independent component analysis

Abstract

The invention discloses an emotion monitoring method based on online independent component analysis, and the method comprises the following steps: 1, carrying out the filtering, bad conductor removal and baseline removal operation of a real-time electroencephalogram signal obtained from a signal collection device; 2, setting a 1s sliding window, setting the step length to be 0.5 s, segmenting the electroencephalogram signal, sending segmented data into a generative adversarial network model to generate an artifact signal, and inputting the artifact signal and the segmented signal into online independent component analysis to fully remove artifacts; 3, respectively extracting space and time characteristics of the electroencephalogram signals from the processed data by using a space mapping technology and a time sequence modeling mode; and step 4, carrying out deep fusion on time and space features, and inputting the fused features into a trained softmax classifier for emotion monitoring. According to the method, real-time false removal operation is performed on the real-time electroencephalogram signals, and it is guaranteed that false removal delay is low and false removal quality is kept at a high level.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time electroencephalogram (EEG) signal processing, and in particular to an emotion monitoring method based on online ICA algorithm signal processing. Background Art

[0002] In recent years, people have paid more and more attention to their own health and emotions. As an emerging biological signal processing method, emotion monitoring technology has been widely studied and applied in many fields such as psychology, medicine, and human-computer interaction. The recognition of emotional state can not only help people understand their own emotional changes, but also better understand their own health status.

[0003] As a biological signal of brain activity, EEG (Electroencephalography) can well reflect a person's emotional state. By collecting real-time EEG signals and analyzing the waveform characteristics, a person's emotional state can be accurately judged. At the same time, there are many interferences from exogenous and endogenous signals in the collection of EEG signals, such as electrooculogram (EOG) and electromyography (EMG). More and more effective processing is needed to remove these interference signals.

[0004] Independent Component Analysis (ICA) is a powerful blind source separation algorithm that can effectively remove interference signals in EEG signals and accurately judge emotions.

[0005] However, current ICA algorithms rely on offline methods, usually processing after the EEG signal is collected, which results in a slight delay and makes it impossible to achieve emotion monitoring. In practice, real-time monitoring can reflect the human body state more promptly, so that it can be used in places with higher requirements for emotion monitoring, such as medical treatment, human-computer interaction, etc. Summary of the invention

[0006] In view of the shortcomings of the above-mentioned prior arts, the present invention proposes an emotion monitoring method based on online independent component analysis, which can perform real-time signal processing and emotion monitoring while collecting EEG signals, eliminating the delay problem of traditional ICA processing methods; at the same time, combined with the ICA algorithm optimized by GAN, it can well remove artifacts and extract the required EEG signals, which can effectively improve the accuracy and robustness of emotion monitoring.

[0007] To achieve the above object, the technical solution of the present invention is: comprising the following steps:

[0008] S1, obtain the user's real-time brain signal and perform preliminary preprocessing, including filtering, removing bad signals and removing baselines;

[0009] S2, the brain signal after preprocessing is segmented according to the sliding window, and the segmented data is sent to the generative adversarial network model to generate similar artifact signals and perform independent component analysis together with the segmented data to remove artifacts;

[0010] S3, use convolutional neural network and self-attention mechanism to extract spatial and temporal features of EEG signals;

[0011] S4, deeply fuses the extracted spatial and temporal features and inputs them into the softmax classifier for sentiment classification to achieve online real-time sentiment monitoring.

[0012] Preferably, step S1, preliminary preprocessing includes the following:

[0013] (1) Filtering with a bandpass filter of 1 to 45 Hz;

[0014] (2) Based on the amplitude and waveform stability of the EEG signal, the bad conductor is marked;

[0015] (3) Select a certain static signal, calculate the baseline signal, and subtract the baseline value;

[0016]

[0017] T is the time of a certain period of resting state, and the subsequent EEG signals are subtracted from the baseline value.

[0018] Preferably, in step S2, the size of the sliding window is set to 1s, the step size is set to 0.5s, and the segmented EEG signal is:

[0019] X={x1,x2,...,x t}

[0020] T represents the tth sliding window;

[0021] Using real EEG signals, artifact signals are generated through GAN; artifact signals that are statistically similar to real signals are generated by training the generator, and the generated artifact signals are compared with the real EEG signals. ICA is used to remove artifacts from these signals, such as electrooculogram and electromyography, to improve the quality and efficiency of artifact removal.

[0022] Preferably, in step S3, first, according to the electrode position of the EEG signal acquisition device, the EEG signal X processed in step S2 is converted into {x1, x2, ..., x t}, where x t represents the tth sliding window;

[0023] The segmented EEG signal is input into the Generative Adversarial Network model, and the generator is made to generate artifact signals as much as possible. The discriminator distinguishes the artifact signals generated by the generator from the real EEG signals. On this basis, the generator is continuously optimized to generate artifact signals close to the real ones. The generated artifact signals and the segmented original EEG signals are subjected to independent component analysis together, so that the signals processed by ICA can effectively remove endogenous signals (such as electrooculogram, electromyography, etc.) and other interference signals. The independent component analysis algorithm optimized by GAN can further extract the real EEG signals and improve the robustness.

[0024] Preferably, in step S3, the one-dimensional EEG signal processed in step 2 is mapped into a two-dimensional matrix according to the electrode position of the EEG signal acquisition device:

[0025]

[0026] m represents the mth electrode;

[0027] Use convolutional neural network to process the two-dimensional signal frame X t For spatial feature extraction, use a 3×3 convolution kernel and a 2×2 convolution kernel, and do not use pooling operations to retain more spatial feature information:

[0028] F1=Conv(X,W1,b1)

[0029] F2=Conv(X,W2,b2)

[0030] W is the convolutional layer, b is the bias term, and F is the output feature map;

[0031] The self-attention mechanism is used to extract the temporal features of the EEG signal processed in step 2, and the weighted sum is used to obtain the final temporal feature vector Ft. The EEG signal is mapped to three different matrices with different weights:

[0032] Query matrix (Q): Q = x t w q ;

[0033] Key matrix (K): K = x t w k ;

[0034] Value matrix (V) V = x t w v ;

[0035] where w q , w k , w vis the weight matrix obtained through learning;

[0036] Then calculate the similarity of the sliding window, that is, the attention score:

[0037]

[0038] And perform normalization:

[0039]

[0040] The final output matrix V is weighted summed to obtain the final time feature vector:

[0041] Output=Attention Weights×V.

[0042] Preferably, in step S3, the first convolution kernel is:

[0043] F1 = ReLU (W1*X(t)+b1)

[0044] Among them, * represents the convolution operation, ReLU is the activation function, F1 is the feature matrix after convolution, and b1 is the bias term;

[0045] The second convolution kernel:

[0046] F2=ReLU(W2*F1+b2)

[0047] At the same time, the pooling operation is not used to ensure the maximum retention of spatial features. Finally, the extracted spatial feature matrix F2 is flattened into a one-dimensional vector F s .

[0048] Preferably, in step S4, the spatial and temporal features extracted from the CNN and the self-attention mechanism are F s and F t , and extracts the temporal features F of the EEG signal through the self-attention mechanism t , concatenate the spatial and temporal features to form a new fusion feature vector F f , F f =[F s ||F t ],|| represents the concatenation operation of features;

[0049] The fusion feature is expressed as:

[0050] F f =concat(F s ,F t )

[0051] Finally, F f Input softmax classifier for sentiment classification:

[0052]

[0053] θ k It is expressed as the weight of the k-th category sentiment, where k is the number of categories, and y k Represents the predicted probability of the kth category of emotion, and finally selects the emotion with the largest predicted probability as the output.

[0054] The technical principles and beneficial effects of the present invention are as follows:

[0055] The online ICA algorithm proposed in the present invention can perform real-time signal processing and emotion monitoring while collecting EEG signals, eliminating the delay problem of traditional ICA processing methods; at the same time, combined with the ICA algorithm optimized by GAN, it can well remove artifacts and extract the required EEG signals, which can effectively improve the accuracy and robustness of emotion monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is the algorithm flow chart of the present invention;

[0057] Figure 2 It is a schematic diagram of the GAN-based optimized ICA algorithm framework of the present invention. DETAILED DESCRIPTION

[0058] The technical solutions in the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only preferred embodiments of the present invention, not all embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Example

[0060] like Figure 1 As shown, the present invention has the following steps:

[0061] S1, obtain the user's real-time brain signal and perform preliminary preprocessing, including filtering, removing bad signals and removing baselines;

[0062] S2, the brain signal after preprocessing is segmented according to the sliding window, and the segmented data is sent to the generative adversarial network model to generate similar artifact signals and perform independent component analysis together with the segmented data to remove artifacts;

[0063] S3, use convolutional neural network and self-attention mechanism to extract spatial and temporal features of EEG signals;

[0064] S4, deeply fuses the extracted spatial and temporal features and inputs them into the softmax classifier for sentiment classification to achieve online real-time sentiment monitoring.

[0065] Preferably, step S1, preliminary preprocessing includes the following:

[0066] (1) Filtering with a bandpass filter of 1 to 45 Hz;

[0067] (2) Based on the amplitude and waveform stability of the EEG signal, the bad conductor is marked;

[0068] (3) Select a certain static signal, calculate the baseline signal, and subtract the baseline value;

[0069]

[0070] T is the time of a certain period of resting state, and the subsequent EEG signals are subtracted from the baseline value.

[0071] Preferably, in step S2, the size of the sliding window is set to 1s, the step size is set to 0.5s, and the segmented EEG signal is:

[0072] X={x1,x2,...,x t}

[0073] T represents the tth sliding window;

[0074] Using real EEG signals, artifact signals are generated through GAN; artifact signals that are statistically similar to real signals are generated by training the generator, and the generated artifact signals are compared with the real EEG signals. ICA is used to remove artifacts from these signals, such as electrooculogram and electromyography, to improve the quality and efficiency of artifact removal.

[0075] Preferably, in step S3, first, according to the electrode position of the EEG signal acquisition device, the EEG signal X processed in step S2 is converted into {x1, x2, ..., x t}, where x t represents the tth sliding window;

[0076] The segmented EEG signal is input into the Generative Adversarial Network model, and the generator is made to generate artifact signals as much as possible. The discriminator distinguishes the artifact signals generated by the generator from the real EEG signals. On this basis, the generator is continuously optimized to generate artifact signals close to the real ones. The generated artifact signals and the segmented original EEG signals are subjected to independent component analysis together, so that the signals processed by ICA can effectively remove endogenous signals (such as electrooculogram, electromyography, etc.) and other interference signals. The independent component analysis algorithm optimized by GAN can further extract the real EEG signals and improve the robustness.

[0077] Preferably, in step S3, the one-dimensional EEG signal processed in step 2 is mapped into a two-dimensional matrix according to the electrode position of the EEG signal acquisition device:

[0078]

[0079] m represents the mth electrode;

[0080] Use convolutional neural network to process the two-dimensional signal frame X t For spatial feature extraction, use a 3×3 convolution kernel and a 2×2 convolution kernel, and do not use pooling operations to retain more spatial feature information:

[0081] F1=Conv(X,W1,b1)

[0082] F2=Conv(x,W2,b2)

[0083] W is the convolutional layer, b is the bias term, and F is the output feature map;

[0084] Use the self-attention mechanism to extract the temporal features of the EEG signal processed in step 2, and perform weighted summation to obtain the final temporal feature vector F t , the EEG signal is mapped to three different matrices with different weights:

[0085] Query matrix (Q): Q = x t w q ;

[0086] Key matrix (K): K = x t w k ;

[0087] Value matrix (V) V = x t w v ;

[0088] where w q , w k , w v is the weight matrix obtained through learning;

[0089] Then calculate the similarity of the sliding window, that is, the attention score:

[0090]

[0091] And perform normalization:

[0092]

[0093] The final output matrix V is weighted summed to obtain the final time feature vector:

[0094] Output=AttentionWeights×V.

[0095] Preferably, in step S3, the first convolution kernel is:

[0096] F1 = ReLU (W1*X(t)+b1)

[0097] Among them, * represents the convolution operation, ReLU is the activation function, F1 is the feature matrix after convolution, and b1 is the bias term;

[0098] The second convolution kernel:

[0099] F2=ReLU(W2*F1+b2)

[0100] At the same time, the pooling operation is not used to ensure the maximum retention of spatial features. Finally, the extracted spatial feature matrix F2 is flattened into a one-dimensional vector F s .

[0101] Preferably, in step S4, the spatial and temporal features extracted from the CNN and the self-attention mechanism are F S and F t The temporal feature Ft of the EEG signal is extracted through the self-attention mechanism, and the spatial and temporal features are concatenated to form a new fusion feature vector F f , F f =[F s ||F t ],|| represents the concatenation operation of features;

[0102] The fusion feature is expressed as:

[0103] F f =concat(F S ,F t )

[0104] Finally, F f Input softmax classifier for sentiment classification:

[0105]

[0106] θ k It is expressed as the weight of the k-th category sentiment, where k is the number of categories, and y k Represents the predicted probability of the kth category of emotion, and finally selects the emotion with the largest predicted probability as the output.

[0107] Implementation Process

[0108] The specific steps of the emotion monitoring method based on the online ICA algorithm of the present invention are as follows:

[0109] Step S1: Collect real-time EEG signals and perform preliminary preprocessing. Specifically:

[0110] Real-time EEG signal acquired t , assuming that the signal amplitude is x at each time point t , where t represents the time point. Further, preliminary preprocessing is required:

[0111] S11. Use a 1-45 Hz bandpass filter H(ω), whose frequency response is defined as:

[0112] Y(ω)=X(ω)·H(ω)

[0113] Where X(ω) is the spectrum of the original EEG signal and Y(ω) is the spectrum of the filtered signal.

[0114] S12. Based on the amplitude and waveform stability of the EEG signal, the signal is marked as a bad conductor. If the amplitude of the signal is greater than the threshold ∈ for a period of time, it can be marked as a bad conductor:

[0115] ||x(t)|| ∞ ≥∈.

[0116] S13, baseline correction, collecting the baseline signal b of the user in a resting state, the corrected signal is:

[0117] x′ t , x'(t)=x(t)-b

[0118] Step S2: Generate adversarial network (GAN) to optimize ICA algorithm, see Figure 2 Schematic diagram of the GAN-based optimized ICA algorithm framework, specifically:

[0119] S21. In order to ensure efficient segmentation of real-time signal streams and the quality of the segmented signals, a sliding window is used. However, the size and step size of the sliding window are also very important. A window that is too small may not capture enough signal features, resulting in poor separation effect; a window that is too large may increase the computational burden and cause signal delay problems; if the step size is set too small, the computational complexity of the algorithm will increase; if the step size is too large, it may cause discontinuous signal segmentation, affecting the final emotion classification.

[0120] Finally, the size was chosen to be:

[0121] T win =1s

[0122] The step length is:

[0123] Δt=0.5s.

[0124] Then the final segmentation signal is:

[0125] X t = {x(t), x(t+1), ..., x(t+T win -1)},t∈Z

[0126] Where X t Represented as the signal of the t-th window.

[0127] S22. The generative adversarial network consists of a generator G and a discriminator D. The purpose of the generator is to generate an artifact signal S(t) that is close to the real one (such as electrooculogram, electromyography, etc.).

[0128] S(t)=G(z)

[0129] Where G(z) represents the output of the generator and z represents random noise.

[0130] The purpose of the discriminator is to distinguish between the generated artifact signal and the real signal. The discriminator will perform a binary judgment on each signal x(t) and output the result D(x), which represents the probability that the signal is a real signal.

[0131]

[0132] Where W D represents the weight of the discriminator, b D is the bias term.

[0133] At the same time, there are certain problems. Since the training of GAN is an adversarial process involving the continuous iteration of the generator and the discriminator, the training of GAN usually requires a lot of computing resources and time. In order to ensure the real-time performance of the algorithm, it is necessary to pre-train the GAN model on a large-scale data set in advance, so that there is no need to train from scratch. On the basis of the realization signal, it is only necessary to fine-tune the loss function to ensure that the training process of GAN can converge in a short time. The following is the loss function of GAN:

[0134] L G = -E[logD(G(z))]

[0135] L D =-E[logD(x(t)))]-E[log(1-D(G(z)))]

[0136] S23. During the GAN training process, the generated artifact signal S(t) is input into the ICA algorithm together with the original EEG signal. The artifact signal is removed by blind source separation to obtain the optimized independent component S optimizied (t).

[0137] X(t)=W optimized ·S optimizied (t)

[0138] Where W optimized It is represented as the optimized unmixing matrix. Through adversarial training of the generator and the discriminator, the blind source separation process of the ICA algorithm can be effectively optimized, the artifact interference in the EEG signal can be removed, and the accuracy and robustness of emotion monitoring can be improved.

[0139] S3. Use CNN to extract the spatial features of EEG signals, and use the self-attention mechanism to extract the temporal features of EEG signals.

[0140] S31. Spatial feature extraction. Assume that the device worn by the user has m electrodes to collect EEG signals. Map the EEG signals x(t) to the corresponding electrode positions to form a two-dimensional matrix:

[0141]

[0142] Two convolution kernels K1 = 3 × 3 and K2 = 2 × 2 are used to extract spatial features of different scales. In order to retain more spatial information, the pooling layer is not used. The ReLU activation function is used, and the final feature matrix F spatial for:

[0143] F spatial =ReLU(X conv )

[0144] Finally, a flattening operation is required to convert the two-dimensional feature vector into a one-dimensional vector F spatial

[0145] F spatial =Flatten(F spatial )

[0146] S32, time feature extraction, use the self-attention mechanism to extract time features. Map the signal x(t) into the query matrix Q, key matrix K and value matrix V:

[0147] Q = x t w q

[0148] K = x t w k

[0149] V = x t w v

[0150] where w q 、w k 、w v is the weight matrix obtained through advance learning. Then the attention score AttentionScore is calculated and normalized and weighted summed to obtain the final time feature F time ,

[0151]

[0152] F time =Attention Weights×V

[0153] S4, deeply fuse the temporal and spatial features to form the final feature vector F,

[0154] F=[F spatial ||F time ]

[0155] Among them, || represents the concatenation of features. Finally, the feature vector F is input into the Softmax classifier to obtain the predicted probability P(K) of each emotion category:

[0156]

[0157] where θ k It is expressed as the weight of the k-th category sentiment, k is the number of categories, and P(K) represents the predicted value of the K-th category sentiment.

[0158] Measure the probability, and finally select the emotion category y with the highest probability k As a result of emotion monitoring:

[0159] y k = arg max k P(K)

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. The sentiment monitoring method based on online independent component analysis is characterized by: The steps include: S1, obtain the user's real-time brain signal and perform preliminary preprocessing, including filtering, removing bad signals and removing baselines; S2, the brain signal after preprocessing is segmented according to the sliding window, and the segmented data is sent to the generative adversarial network model to generate similar artifact signals and perform independent component analysis together with the segmented data to remove artifacts; S3, use convolutional neural network and self-attention mechanism to extract spatial and temporal features of EEG signals; S4, deeply fuses the extracted spatial and temporal features and inputs them into the softmax classifier for sentiment classification to achieve online real-time sentiment monitoring.

2. The emotion monitoring method based on online independent component analysis according to claim 1, characterized in that: Step S1, preliminary preprocessing includes the following: (1) Filtering with a bandpass filter of 1 to 45 Hz; (2) Based on the amplitude and waveform stability of the EEG signal, the bad conductor is marked; (3) Select a certain section of static signal, calculate the baseline signal, and subtract the baseline value; T is the time of a certain period of resting state, and the subsequent EEG signals are subtracted from the baseline value.

3. The emotion monitoring method based on online independent component analysis according to claim 1, characterized in that: Step S2, the sliding window size is set to 1s, the step size is set to 0.5s, and the segmented EEG signal is: X={x1,x2,...,x t } T represents the tth sliding window; Using real EEG signals, artifact signals are generated through GAN; artifact signals that are statistically similar to real signals are generated by training the generator, and the generated artifact signals are compared with the real EEG signals. ICA is used to remove artifacts from these signals, such as electrooculogram and electromyography, to improve the quality and efficiency of artifact removal.

4. The emotion monitoring method based on online independent component analysis according to claim 1, characterized in that: Step S3, firstly, according to the electrode position of the EEG signal acquisition device, the EEG signal X processed in step S2 is converted into {x1, x2, ..., x t }, where x t represents the tth sliding window; The segmented EEG signals are input into the Generative Adversarial Network model, and the generator is made to generate artifact signals as much as possible. The discriminator distinguishes the artifact signals generated by the generator from the real EEG signals. On this basis, the generator is continuously optimized to generate artifact signals close to the real ones. The generated artifact signals and the segmented original EEG signals are subjected to independent component analysis together, so that the signals processed by ICA can effectively remove endogenous signals (such as electrooculogram, electromyography, etc.) and other interference signals. The independent component analysis algorithm optimized by GAN can further extract the real EEG signals and improve the robustness.

5. The emotion monitoring method based on online independent component analysis according to claim 1, characterized in that: Step S3, mapping the one-dimensional EEG signal processed in step 2 into a two-dimensional matrix according to the electrode position of the EEG signal acquisition device: m represents the mth electrode; Use convolutional neural network to process the two-dimensional signal frame X t For spatial feature extraction, use a 3×3 convolution kernel and a 2×2 convolution kernel, and do not use pooling operations to retain more spatial feature information: F1=Conv(X,W1,b1) F2=Conv(X,W2,b2) W is the convolutional layer, b is the bias term, and F is the output feature map; Use the self-attention mechanism to extract the temporal features of the EEG signal processed in step 2, and perform weighted summation to obtain the final temporal feature vector F t , the EEG signal is mapped to three different matrices with different weights: Query matrix (Q): Q = x t w q ; Key matrix (K): K = x t w k ; Value matrix (V) V = x t w v ; where w q , w k , w v is the weight matrix obtained through learning; Then calculate the similarity of the sliding window, that is, the attention score: And perform normalization: The final output matrix V is weighted summed to obtain the final time feature vector: Output=Attention Weights×V.

6. The emotion monitoring method based on online independent component analysis according to claim 1, characterized in that: Step S3, the first convolution kernel: F1 = ReLU (W1*X(t)+b1) Among them, * represents the convolution operation, ReLU is the activation function, F1 is the feature matrix after convolution, and b1 is the bias term; The second convolution kernel: F2=ReLU(W2*F1+b2) At the same time, the pooling operation is not used to ensure the maximum retention of spatial features. Finally, the extracted spatial feature matrix F2 is flattened into a one-dimensional vector F s .

7. The emotion monitoring method based on online independent component analysis according to claim 1, characterized in that: Step S4: The spatial and temporal features extracted from CNN and self-attention mechanism are F s and F t , and extracts the temporal features F of the EEG signal through the self-attention mechanism t , concatenate the spatial and temporal features to form a new fusion feature vector F f , F f =[F s ||F t ],|| represents the concatenation operation of features; The fusion feature is expressed as: F f =concat(F s ,F t ) Finally, F f Input softmax classifier for sentiment classification: θ k It is expressed as the weight of the k-th category sentiment, where k is the number of categories, and y k Represents the predicted probability of the kth category of emotion, and finally selects the emotion with the largest predicted probability as the output.

Images