Electroencephalogram recognition method and system based on nonlinear cross frequency modulation network

By using a nonlinear cross-frequency modulation network system, the problem of insufficient cross-frequency information capture in EEG signal processing is solved, achieving efficient emotion recognition and improving the classification accuracy of EEG signals.

CN118948277AActive Publication Date: 2024-11-15BEIJING INST OF TECH

Patent Information

Application Number
CN202410992403.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-15
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

Existing technologies cannot effectively capture the coherent information across frequencies of complex brain electrical dynamics in EEG signal processing, resulting in limited emotion recognition capabilities. Furthermore, traditional methods have bottlenecks in feature extraction and memory usage.

Method used

An EEG recognition system based on a nonlinear cross-frequency modulation network is adopted, including a preprocessing module, a frequency band decomposition and feature representation module, a cross-frequency coupling layer, a CNN1D-ALSTM model, and an emotion classification module. Frequency band division is performed through eMEMD, and cross-frequency coupling features are captured and emotions are classified by combining a one-dimensional convolutional neural network and a bidirectional long short-term memory network.

Benefits of technology

It achieves adaptive extraction of cross-frequency coupling features, which improves the accuracy and robustness of emotion recognition, especially performing well in binary and quaternary classification tasks on the DEAP database.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118948277A_ABST
    Figure CN118948277A_ABST
Patent Text Reader

Abstract

The invention discloses an electroencephalogram recognition method and system based on a nonlinear cross frequency modulation network, relates to the technical field of electroencephalogram signal processing, and realizes nonlinear division of a target frequency band and self-adaptive extraction of cross frequency coupling characteristics. The electroencephalogram signal preprocessing module is used for receiving and preprocessing collected original electroencephalogram signals. And the frequency band decomposition and feature representation module is used for carrying out frequency band division on the preprocessed electroencephalogram signals based on eMEMD and nonlinearly extracting signal components of a target frequency band. And the cross frequency coupling layer is used for capturing coupling information among different frequency bands and outputting cross frequency coupling characteristics. The CNN1D-ALSTM model is combined with the time sequence feature extraction capability of the one-dimensional convolutional neural network and the long sequence learning capability of the Bi-LSTM to capture the time sequence features in the cross frequency coupling features. And the space decoding module is used for carrying out space decoding on the time sequence characteristics to obtain space-time characteristics. And the emotion classification module is used for projecting the spatial-temporal features into an emotion space to be classified.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal processing, and in particular to an EEG recognition method and system based on a nonlinear cross-frequency modulation network. Background Art

[0002] Advances in brain-computer interfaces, signal processing, and artificial intelligence have driven the development of the field of affective computing. This interdisciplinary field supports enhanced decision-making, cognitive efficiency, and attention in the context of education and advertising. Among them, intervention for patients with emotional illnesses (such as depression) can be achieved through real-time emotion monitoring and assessment. High temporal resolution electroencephalogram (EEG) signals are conducive to objectively reflecting the mechanism of emotion generation and provide a convenient and cost-effective solution for large-scale data collection.

[0003] Brain rhythms are generally divided into frequency bands to represent different emotional patterns, such as the α band for stable emotions and the β band for excited emotions. Linear filters, such as wavelet filters, are used for frequency band division in some studies, which may distort nonlinear waveforms and patterns. On the other hand, empirical mode decomposition (EMD) can be used as a non-stationary and nonlinear filter to decompose the signal into intrinsic mode functions (IMFs). Multivariate empirical mode decomposition (M-EMD) extracts highly consistent IMFs from multi-source signals by jointly locating multi-channel extrema and iteratively constructing a joint envelope. However, M-EMD lacks the ability to accurately obtain the target frequency bands (such as θ and α) of EEG signals. To address this limitation, the exclusive masking empirical mode decomposition (eMEMD) method continuously divides the masking signal by the center frequency, thereby achieving IMF alignment on multiple channels and restricting it to the empirical frequency band of the EEG signal.

[0004] After that, some studies extracted features from single empirical brain rhythms to achieve emotion recognition. Although these algorithms may be useful, they ignore the coupling information across multiple frequency bands.

[0005] To solve this problem, researchers used a splicing method to fuse features from different bands. The cascaded spatial features or connectivity features of θ, α, β, and γ were input into the low-level classifier, which helped the recognition of emotional modality to a certain extent. However, the lack of cross-frequency coherent frequency semantic information may hinder the capture of complex EEG dynamics, thereby limiting its discrimination ability. In addition, due to the increase in feature dimensions in the connection operation, the increased memory usage hinders real-time emotion recognition.

[0006] Linear filtering methods, such as Butterworth filters or infinite impulse response (FIR) filters, have characteristics that deviate from the non-stationary and nonlinear characteristics of EEG signals. Empirical mode decomposition (EMD) is beneficial for correctly preserving nonlinear and non-stationary causal characteristics while filtering. Multivariate empirical mode decomposition (M-EMD) extracts highly consistent intrinsic mode functions (IMFs) from multi-source signals by jointly locating multi-channel extrema and iteratively constructing a joint envelope. However, M-EMD cannot obtain the precise frequency modulation of IMFs under the constraints of the EEG signal's empirical frequency band (such as θ, α).

[0007] Traditional M-EMD cannot constrain frequency. However, in view of the needs of biopsychological diagnosis and emotion recognition, considering the collaborative heterogeneous mechanism of complex emotional EEG signals in multi-band rhythms, using only a single-band feature recognition may lose the multi-dimensional information of biopsychological EEG rhythms. If the multi-band features are fused by the traditional splicing method, the cross-frequency coupling mechanism is ignored. Summary of the invention

[0008] In view of this, the present invention provides an EEG recognition method and system based on a nonlinear cross-frequency modulation network, which realizes nonlinear division of target frequency bands and can adaptively extract cross-frequency coupling features.

[0009] To achieve the above purpose, the technical solution of the present invention is: an EEG recognition system based on a nonlinear cross-frequency modulation network, including a preprocessing module, a frequency band decomposition and feature representation module, a cross-frequency coupling layer, a CNN1D-ALSTM model, a spatial decoding module and an emotion classification module:

[0010] The preprocessing module is used to receive the collected original EEG signals and perform preprocessing.

[0011] The frequency band decomposition and feature representation module is used to divide the preprocessed EEG signal into frequency bands based on eMEMD and extract the signal components of the target frequency band nonlinearly.

[0012] The cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output the cross-frequency coupling characteristics.

[0013] The CNN1D-ALSTM model combines the temporal feature extraction capability of the one-dimensional convolutional neural network and the long sequence learning capability of the Bi-LSTM to capture the temporal features in the cross-frequency coupling features.

[0014] The spatial decoding module is used to perform spatial decoding on the time series features to obtain spatiotemporal features.

[0015] The emotion classification module is used to project the spatiotemporal features into the emotion space to be classified.

[0016] Preferably, the preprocessing module, the specific preprocessing method is:

[0017] Get the collected original EEG signal, the sampling frequency is f s , use the blind source signal separation method to remove motion artifacts; down-sample to unify the multi-channel sampling frequency, and use a bandpass filter on the down-sampled EEG signal to remove power frequency noise and irrelevant frequency components.

[0018] Preferably, the frequency band decomposition and feature representation module is specifically implemented in the following manner:

[0019] S1. The remaining signal is set to Initialize h(t) equal to the input signal x(t); C represents the number of EEG signal channels, T represents the number of EEG signal sampling points, is a set of real numbers of dimension C×T.

[0020] S2. Generate a mask frequency sequence in f s is the sampling frequency; is the mask frequency sequence f a The qth frequency value in , q is initialized to 1.

[0021]

[0022] S3. Design mask signal set based on step strategy Where l is a self-determined constant, is the kth mask signal:

[0023]

[0024] Where std(*) is the standard deviation, θ k is the kth mask phase, θ k =2πk / l.

[0025] The first-order eigenmode function of the k-th phase E1(*) represents the operation of taking the first-order eigenmode function IMF1 of the k-th phase. When k < l, increment k by 1 and return to S3; otherwise, calculate the q-th order eigenmode function

[0026] S5. If q < 4, calculate the new residual signal as the original residual signal minus d q (t), increment q by 1, and repeat to return to S3; otherwise, stop the loop.

[0027] Thus, narrowband signals in four empirical frequency bands are obtained

[0028] S6. In a single trial, segment the narrowband signals in each empirical frequency band through a non-overlapping Hanning window of length T s The narrowband signals x γ , x β , x α , x θ The dimension becomes Reshape x γ , x β , x α , x θ into Then complete the Conv1D convolution, and the obtained T f represents the number of features after convolution,

[0029] The time series corresponding to the narrowband signals x γ , x β , x α , x θ are f γ , f β , f α , f θ .

[0030] Preferably, the cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output the cross-frequency coupling feature f CFC which is expressed as

[0031] f CFC = tanh(f β ) ⊙ σ(f γ ) + tanh(f θ ) ⊙ σ(f α ) (4)

[0032] Where ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The time series f is activated by the σ activation function. γ and f α , regularize the time series f by the tanh function β and f θ .

[0033] Preferably, the CNN1D-ALSTM model includes a Conv1D layer, a MaxPool layer, a Batch Norm layer, a first BiLSTM layer, a second BiLSTM layer and a self-attention mechanism layer.

[0034] The first stage consists of the Conv1D layer, the MaxPool layer, and the Batch Norm layer. The Conv1D layer acts as an automatic feature extractor with the help of the convolution kernel. After that, with the help of the length p o The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization.

[0035] The first and second BiLSTM layers constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

[0036] Preferably, the spatial decoding module includes using a convolutional layer - Conv1D layer to extract temporal features of different channels, recombining the temporal features between different channels into new spatiotemporal features, and converting the spatiotemporal features from multi-dimensional features to one-dimensional output.

[0037] Another embodiment of the present invention further provides an EEG recognition method based on a nonlinear cross-frequency modulation network, comprising the following steps:

[0038] Step 1: Preprocess the collected raw EEG signals.

[0039] Step 2: Perform frequency band division on the preprocessed EEG signal based on eMEMD, and extract the signal components of the target frequency band nonlinearly.

[0040] Step 3: Capture the coupling information between different frequency bands and output the cross-frequency coupling characteristics.

[0041] Step 4: Capture the timing characteristics in the cross-frequency coupling characteristics.

[0042] Step 5: Spatial decoding of time series features to obtain spatiotemporal features.

[0043] Step 6: Project the spatiotemporal features into the emotion space to be classified.

[0044] Further, in step two, the following specific method is adopted:

[0045] S1. Set the remaining signal as Initialize h(t) to be equal to the input signal x(t); C represents the number of EEG signal channels, and T represents the number of EEG signal sampling points, which is a real number set of C×T dimensions;

[0046] S2. Generate a mask frequency sequence where f s is the sampling frequency; is the q-th frequency value in the mask frequency sequence f a , and q is initialized to 1.

[0047]

[0048] S3. Design a mask signal set according to the stepping strategy where l is a self-defined constant, is the k-th mask signal:

[0049]

[0050] where std(*) is the standard deviation, and θ k is the k-th mask phase, and θ k = 2πk / l.

[0051] S4. The first-order intrinsic mode function of the k-th phase represents the operation of taking the first-order intrinsic mode function IMF1 of the k-th phase. When k < l, increment k by 1 and return to S3; otherwise, calculate the q-th order intrinsic mode function

[0052] S5. If q < 4, calculate the new remaining signal as the original remaining signal minus d q (t), increment q by 1, and repeat to return to S3; otherwise, stop the loop.

[0053] Thus, narrowband signals under four empirical frequency bands are obtained

[0054] S6. In a single trial, segment each narrowband signal under the empirical frequency band through a non-overlapping Hanning window of length T s , and the narrowband signals x γ , x β , x α , x θ become Set x γ , x β , xα , x θ Reshape Then complete the Conv1D convolution and get T f represents the number of features after convolution,

[0055] Narrowband signal x in four empirical frequency bands γ , x β , x α , x θ The corresponding time series are f γ , f β , f α , f θ .

[0056] Step 3: Output cross-frequency coupling characteristics f CFC Expressed as

[0057] f CFC =tanh(f β )⊙σ(f γ )+tanh(f θ )⊙σ(f α ) (4)

[0058] Where ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The time series f is activated by the σ activation function. γ and f α , regularize the time series f by the tanh function β and f θ .

[0059] Furthermore, in step four, the CNN1D-ALSTM model is used to capture the timing characteristics in the cross-frequency coupling features.

[0060] The CNN1D-ALSTM model includes a Conv1D layer, a MaxPool layer, a Batch Norm layer, a first BiLSTM layer, a second BiLSTM layer, and a self-attention mechanism layer.

[0061] The first stage consists of the Conv1D layer, the MaxPool layer, and the Batch Norm layer. The Conv1D layer acts as an automatic feature extractor with the help of the convolution kernel. After that, with the help of the length p o The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization.

[0062] The first and second BiLSTM layers constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

[0063] Furthermore, in step five, the convolutional layer - Conv1D layer is used to extract the temporal features of different channels, and the temporal features between different channels are recombined into new spatiotemporal features, converting the spatiotemporal features from multi-dimensional features to one-dimensional output.

[0064] Step 6: Use the softmax function to calculate the conditional probability of each emotion category, and then select the category corresponding to the maximum conditional probability as the predicted output category.

[0065] Beneficial effects:

[0066] The present invention provides an EEG recognition method and system based on a nonlinear cross frequency modulation network, and innovatively proposes a cross frequency coupling layer (CFCL), which can adaptively and robustly construct cross frequency coupling features for emotion recognition. First, the cross frequency modulation network (CFMN) performs frequency band division based on eMEMD, nonlinearly extracts the signal components of the target frequency band, and further, by bitwise multiplication of the high frequency modulation weights with the time series elements embedded in the low frequency, CFCL is established to extract the semantic information of cross frequency interaction. CFMN combines a Conv1D-BiLSTM-A layer, which includes a one-dimensional convolution neural network layer (Conv1D) and a temporally-attended bidirectional long short-term memory network (BiLSTM-A) to extract sequence information from the modulation frequency sequence. The processed features are fed to a fully connected layer to distinguish different emotions. In a non-stationary nonlinear simulation system, eMEMD is proven to effectively divide the frequency bands, that is, its independence is significantly better than that of the EMD and M-EMD algorithms. In the real data experiment based on DEAP database, the nonlinear cross-frequency features of eMEMD split frequency bands are proven to improve classification accuracy. In the valence / arousal binary classification and four-element classification verification experiments conducted on DEAP database, CFMN achieves the best under similar conditions. In summary, the proposed CFMN can adaptively extract cross-frequency coupling features, which is an effective tool for completing emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1A schematic diagram of the process flow of the EEG recognition system based on the nonlinear cross-frequency modulation network provided by the present invention;

[0068] Figure 2 Flowchart of empirical frequency band partitioning for eMEMD nonlinearity;

[0069] Figure 3 Combined two independent hidden layers for Bi-LSTM;

[0070] Figure 4 Schematic diagram of mode independence test. DETAILED DESCRIPTION

[0071] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0072] Embodiment 1:

[0073] The EEG recognition system based on nonlinear cross-frequency modulation network consists of the following components: Figure 1 As shown in the figure, the system includes a preprocessing module, a frequency band decomposition and feature representation module, a cross-frequency coupling layer, a CNN1D-ALSTM model, a spatial decoding module, and an emotion classification module:

[0074] The preprocessing module is used to receive the collected original EEG signal and perform preprocessing. In the embodiment of the present invention, the specific preprocessing method in the preprocessing module is: obtain the collected original EEG signal, the sampling frequency is f s , use the blind source signal separation method to remove motion artifacts; down-sample to unify the multi-channel sampling frequency, and use a bandpass filter on the down-sampled EEG signal to remove power frequency noise and irrelevant frequency components.

[0075] The frequency band decomposition and feature representation module is used to perform frequency band division on the preprocessed EEG signal based on eMEMD, and nonlinearly extract the signal component of the target frequency band; in the embodiment of the present invention, the frequency band decomposition and feature representation module is specifically adopted in the following manner:

[0076] S1. The remaining signal is set to Initialize h(t) equal to the input signal x(t); C represents the number of EEG signal channels, T represents the number of EEG signal sampling points, is a set of real numbers of dimension C×T;

[0077] S2. Generate a mask frequency sequence in f s is the sampling frequency; is the mask frequency sequence f a The qth frequency value in , q is initialized to 1;

[0078]

[0079] S3. Design the mask signal set according to the step strategy where l is a self - defined constant, is the k - th mask signal:

[0080]

[0081] where std(*) is the standard deviation, and θ k is the k - th mask phase, and θ k = 2πk / l;

[0082] S4. The first - order eigenmode function of the k - th phase E1(*) represents the operation of taking the first - order eigenmode function IMF1 of the k - th phase. When k < l, let k increment by 1 and return to S3; otherwise, calculate the q - th order eigenmode function

[0083] S5. If q < 4, calculate the new residual signal as the original residual signal minus d q (t), increment q by 1, and repeat to return to S3; otherwise, stop the loop;

[0084] Thus, narrow - band signals under four empirical frequency bands are obtained

[0085] S6. In a single trial, segment the narrow - band signals under each empirical frequency band through non - overlapping Hanning windows of length T s . The narrow - band signals x γ , x β , x α , x θ become of dimension Reshape x γ , x β , x α , x θ into Then complete the Conv1D convolution, and the obtained T f represents the number of features after convolution,

[0086] The time series corresponding to the narrow - band signals x γ , x β , x α , x θ under the four empirical frequency bands are f γ , f β , f α , f θ .

[0087] The cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output the cross-frequency coupling features; the present invention establishes a cross-frequency coupling layer (CFCL) for extracting cross-frequency interactions. In short, the modulated attention weights generated from the embedded high-frequency time series are used to enhance the embedded low-frequency time series by element-by-element multiplication. This process constructs cross-frequency semantic information while retaining detailed time series information. Meta-operations include element-wise multiplication and addition, thereby reducing the feature size and ensuring operation speed.

[0088] In the embodiment of the present invention, the cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output the cross-frequency coupling feature f CFC Expressed as

[0089] f CFC =tanh(f β )⊙σ(f γ )+tanh(f θ )⊙σ(f α ) (4)

[0090] Where ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The time series f is activated by the σ activation function. γ and f α , regularize the time series f by the tanh function β and f θ .

[0091] The CNN1D-ALSTM model combines the temporal feature extraction capability of the one-dimensional convolutional neural network and the long sequence learning capability of the Bi-LSTM to capture the temporal features in the cross-frequency coupling features. The CNN1D-ALSTM model includes the Conv1D layer, the MaxPool layer, the Batch Norm layer, the first BiLSTM layer, the second BiLSTM layer, and the self-attention mechanism layer.

[0092] The first stage consists of the Conv1D layer, the MaxPool layer, and the Batch Norm layer. The Conv1D layer acts as an automatic feature extractor with the help of the convolution kernel. After that, with the help of the length p o The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization;

[0093] The first and second BiLSTM layers constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

[0094] A spatial decoding module is used to spatially decode the temporal features to obtain spatiotemporal features. In an embodiment of the present invention, the spatial decoding module includes using a convolutional layer - Conv1D layer to extract the temporal features of different channels, recombining the temporal features between different channels into new spatiotemporal features, and converting the spatiotemporal features from multi-dimensional features to one-dimensional output.

[0095] The emotion classification module is used to project the spatiotemporal features into the emotion space to be classified. In the embodiment of the present invention, the conditional probability of each emotion category is calculated using the softmax function, and then the category corresponding to the maximum conditional probability is selected as the predicted output category.

[0096] Embodiment 2:

[0097] In order to solve the technical difficulty that the traditional emotion classification network cannot adaptively extract nonlinear cross-frequency coupling features, this embodiment proposes an EEG recognition method based on a nonlinear cross-frequency modulation network. This method achieves 93.2% valence accuracy and 92.5% arousal accuracy when distinguishing emotional tasks. Compared with other studies under similar conditions, it has better accuracy and can be used for emotion detection and recognition. This method can be executed in the EEG recognition system based on a nonlinear cross-frequency modulation network provided in Example 1; the EEG recognition method based on a nonlinear cross-frequency modulation network provided in this embodiment has a technical process as follows Figure 2 The detailed process is as follows:

[0098] Step 1: Collect the EEG signal of the subject, with a sampling frequency of f s , the above raw EEG signal is preprocessed, specifically including the following preprocessing:

[0099] A1. Use blind source signal separation methods, such as independent component analysis (ICA), to remove motion artifacts such as electrooculography and electromyography;

[0100] A2. Downsample to unify the multi-channel sampling frequency (such as 128Hz), and use a bandpass filter (such as 4-45Hz) on the downsampled EEG signal to avoid the influence of power frequency noise and irrelevant frequency components on the results.

[0101] Step 2: frequency band division is performed on the preprocessed EEG signal based on eMEMD, and the signal component of the target frequency band is nonlinearly extracted. The embodiment of the present invention includes the following steps:

[0102] S1. Define the remaining signal Initialize h(t) equal to the input signal x(t). C represents the number of EEG signal channels, and T represents the number of EEG signal sampling points. is a set of real numbers of dimension C×T;

[0103] S2. Generate a mask frequency sequence where f s is the sampling frequency. is the q-th frequency value in the mask frequency sequence f a ;

[0104]

[0105] q is initialized to 1;

[0106] S3. Design a mask signal set according to the stepping strategy where l is a user-defined constant, generally 6. is the k-th mask signal

[0107]

[0108] std() is the standard deviation, θ k is the k-th mask phase, θ k = 2πk / l.

[0109] S4. Calculate E1() represents the operation of taking the first-order intrinsic mode function IMF1 of the k-th phase. The first-order intrinsic mode function of the k-th phase When k < l, let k = k + 1, and repeat steps 3 - 4; otherwise, calculate the q-th order intrinsic mode function

[0110] S5. If q < 4, calculate the new residual signal h(t) = h(t) - d q (t), q = q + 1, and repeat steps 3 - 5; otherwise, stop the loop and obtain the narrowband signals under four empirical frequency bands

[0111] S6. In a single trial, segment the EEG signals of each frequency band through a non-overlapping Hanning window with a length of T s . The dimension of the narrowband signal becomes T s which can generally be set to 256. Apply a one-dimensional convolutional layer Conv1D with a convolutional kernel size k d of 3 and a stride s d of 2 to each segment to extract the features of C leads. Conv1D continuously changes the convolutional kernel through training, and then adaptively filters out the frequency band components of interest within the specified frequency band range. If the original signal is directly input, Conv1D will convolve the temporal features of multiple adjacent channels, so the correlation between different channels is extracted simultaneously. This will lead to a decrease in the attention to the extraction of the frequency of interest. Therefore, first xγ , x β , x α , x θ Reshape Then complete the Conv1D convolution. Since the reshape operation separates the channels into other dimensions, Conv1D focuses on extracting the frequency components of interest in each channel. We set 128 Conv1D convolution kernels, so the obtained T f Represents the number of features after convolution.

[0112]

[0113] Step 3: Capture the coupling information between different frequency bands and output the cross-frequency coupling characteristics; the specific steps of this step in this embodiment are as follows:

[0114] The cross-frequency coupling layer (CFCL) is used to capture the coupling information between different frequency bands and output the cross-frequency coupling feature f CFC Expressed as

[0115] f CFC =tanh(f β )⊙σ(f γ )+tanh(f θ )⊙σ(f β ) (4)

[0116] ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The high-frequency time series (f γ and f α ,) generates attention weights that enhance the low-frequency time series (f β and f θ ) because f θ and f α The resting group is more representative of stable and peaceful emotions, and f γ and f β It can better represent the more excited emotions and is recorded as the active group. γ and f α For a higher frequency time series, the σ function is used for activation, f β and f θ Since it is a lower frequency time series, the tanh function is used for regularization. Then, the high and low layers of cross-frequency semantic features are superimposed element by element to obtain CFCL captures the cross-frequency coupling characteristics while retaining the sequence information of the original signal.

[0117] Step 4: Capture the timing characteristics in the cross-frequency coupling characteristics.

[0118] The CNN1D-ALSTM model combines the temporal feature extraction capability of a one-dimensional convolutional neural network with the long-sequence learning capability of a long short-term memory network to capture the complex temporal patterns in EEG recordings. The Conv1D layer acts as an automatic feature extractor with the help of a learnable convolution kernel, where the size of the convolution kernel is k. o is 3, step length s o is 1. Therefore, the time series features extracted in the first stage

[0119]

[0120] Then, with the help of length p o The maximum pooling layer downsamples to reduce the number of parameters. And the batch norm layer is applied to feature regularization to improve generalization ability. The feature representation after pooling is

[0121] T o ′=T o |p o (6)

[0122] | represents an integer division operation. In the second stage, the Bi-LSTM layer processes features over time to capture evolving emotional states. Bi-LSTM is a bidirectional recursive recurrent neural network variant. Compared to unidirectional LSTM, it captures sequence dependencies more comprehensively by considering both past and future information at each time step. This bidirectional structure allows the network to consider both past (forward) and future (backward) EEG activity at each time step, more comprehensively capturing brain activity information related to emotional states. The Bi-LSTM structure is shown in Figure 1. Figure 3 shown. Figure 3 The figure shows that the Bi-LSTM combines two independent hidden layers, the forward layer for moving from front to back and the backward layer for moving from back to front. This allows the network to model both the forward and backward information of the input layer at each timestamp. Therefore, both past and future information are preserved in the output layer. The self-attention mechanism assigns importance weights to each time step and weighted aggregates the nodes of all time steps to extract sentiment-focused temporal features.

[0123] The number of Bi-LSTM hidden layers is set to 128. The time series of the input layer is represented from front to back as Bi-LSTM consists of two independent hidden layers (forward layer and backward layer). The hidden cell state of the forward layer is expressed as The hidden cell state of the backward layer is expressed as

[0124] The iterative update method of the forward layer and the backward layer is as follows:

[0125]

[0126] in represents the learnable weights, and is a learnable bias. r after forward update t ,

[0127] Then, the forward unit and backward unit corresponding to the temporal position are used to generate the state representation of the output layer.

[0128]

[0129] Two layers of Bi-LSTM networks are stacked to deeply memorize and encode sequence features. The hidden layer of the second Bi-LSTM is also set to 128.

[0130] In the third stage, the subject will focus at certain moments in a trial and relax at other times. Focusing on the sequence characteristics of the subject's focus period and ignoring the relaxation period can reduce the number of parameters and improve classification accuracy. The embodiment of the present invention adopts an extended self-attention mechanism to assign weights to each EEG signal sample by exploring the intrinsic importance of each sample. The structure of self-attention is as follows Figure 4 As shown in the upper part. The self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation. First, the output layer is mapped to the attention vector

[0131]

[0132] in represents the learnable weights, is a learnable bias. The attention vector is randomly initialized and jointly learned during training. The softmax activation function ensures that the sum of all calculated weights is 1 to obtain the importance weight of each time step

[0133]

[0134] When copying and expanding the importance weights into Then, the weighted sum of all EEG timestamps is calculated as the timestamp focus expressed as ⊙ represents the Hardmann product.

[0135] f Ta =∑ t α i ⊙vi (12)

[0136] Step 5: Spatial decoding of time series features to obtain spatiotemporal features.

[0137] In the last part of the proposed CFMN, we first use the convolution kernel size k d is 3, step length s d Conv1D with a value of 1 is used to extract the features of C leads. The temporal features between different channels are reorganized into new spatiotemporal features.

[0138] f st =conv1d(f Ta ) (13)

[0139] Among them, conv1d() is the convolutional layer.

[0140] The Flatten operation converts a multidimensional vector into a one-dimensional output

[0141] O = flatten(f st ) (14)

[0142] Among them, flatten() expands the function.

[0143] Step 6: Project the spatiotemporal features into the emotion space to be classified.

[0144] The fully connected layer then projects the high-dimensional spatiotemporal features into the emotion space to be classified. The softmax function calculates the conditional probability of each emotion category and then selects the category corresponding to the maximum conditional probability as the predicted output category. Then, the cross entropy error is evaluated on all labeled samples.

[0145] The present invention proposes for the first time a new cross-frequency modulation network (CFMN) method for emotional EEG recognition, which is used to adaptively extract cross-band features of EEG signals. It is suitable for the recognition and monitoring of health and disease, is efficient, reliable, and easy to software. Figure 4 Test for mode independence. (a) Three EMD-based algorithms, EMD, MEMD, and eMEMD, are compared. The three panels correspond to the frequency spectral densities of the fourth-order IMFs, IMF1 (yellow line), IMF2 (orange line), IMF3 (blue line), and IMF4 (green line), of the decomposition of the three-channel simulation signal. (b) There are significant differences among the three EMD-based algorithms in terms of the correlation between the IMFs (p < 0.0001). eMEMD shows lower IMF correlation than EMD and MEMD, and therefore has higher decomposition independence.

[0146] In order to solve the technical difficulty that the traditional emotion classification network cannot adaptively extract nonlinear cross-frequency coupling features, the present invention proposes a set of cross-frequency modulation networks (CFMN). This method achieves 93.2% valence accuracy and 92.5% arousal accuracy in the task of distinguishing emotions. Compared with other studies under similar conditions, it has better accuracy and can be used for emotion detection and recognition. The technical flow chart of the present invention is as follows Figure 2 The detailed process is as follows:

[0147] Step 1) Collect the EEG signal of the subject (emotion-induced) at a sampling frequency of f s , the above raw EEG signals are preprocessed, specifically including: 1. Using blind source signal separation methods, such as independent component analysis (ICA), to remove motion artifacts such as electrooculography and electromyography; 2. Downsampling to unify the multi-channel sampling frequency (such as 128Hz), and using a bandpass filter (such as 4-45Hz) on the downsampled EEG signals to avoid the influence of power frequency noise and irrelevant frequency components on the results.

[0148] Step 2) Feature extraction of empirical frequency band components ( Figure 3 ):

[0149] 1. Define (loop continuously) the remaining signal Initialize h(t) equal to the input signal x(t). C represents the number of EEG signal channels, and T represents the number of EEG signal sampling points. is a set of real numbers of dimension C×T.

[0150] 2. Generate a mask frequency sequence in f s is the sampling frequency. is the mask frequency sequence f a The qth frequency value in .

[0151]

[0152] q is initialized to 1.

[0153] 3. Design the mask signal set according to the step strategy Where l is a self-determined constant, usually 6. is the kth mask signal

[0154]

[0155] std() is the standard deviation, θ k is the kth mask phase, θ k =2πk / l

[0156] 4. Calculation E1() represents the operation of taking the first-order eigenmode function IMF1 of the k-th phase, and the first-order eigenmode function of the k-th phase When k < l, let k = k + 1, and repeat steps 3 - 4; otherwise, calculate the q-th order eigenmode function

[0157] 5. If q < 4, calculate the new residual signal h(t) = h(t) - d q (t), q = q + 1, and repeat steps 3 - 5; otherwise, stop the loop and obtain the narrowband signals under four empirical frequency bands

[0158] 6. In a single trial, segment the EEG signals of each frequency band through non-overlapping Hanning windows of length T s (which can generally be set to 256). The dimension of the narrowband signal becomes Apply a one-dimensional convolutional layer (Conv1D) with a convolutional kernel size k d of 3 and a stride s d of 2 to each segment to extract the features of C leads. Conv1D continuously changes the convolutional kernel through training, and then adaptively filters out the frequency band components of interest within the specified frequency band range. If the original signal is directly input, Conv1D will convolve the temporal features of adjacent multiple channels, so the correlation between different channels is extracted simultaneously. This will lead to a decrease in the attention for extracting the frequency of interest. Therefore, first reshape x γ , x β , x α , x θ into and then complete the Conv1D convolution. Since the reshaping operation separates the channels into other dimensions, Conv1D focuses on extracting the frequency components of interest for each channel. We set 128 Conv1D convolutional kernels, so the obtained T f represents the number of features after convolution.

[0159]

[0160] Step 3) Adaptive extraction of cross-frequency coupling

[0161] The cross-frequency coupling layer (CFCL) is used to capture the coupling information between different frequency bands and output the cross-frequency coupling feature f CFC which is expressed as

[0162] f CFC = tanh(f β ) ⊙ σ(f γ)+tanh(f θ )⊙σ(f α ) (4)

[0163] ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The high-frequency time series (f γ and f α ,) generates attention weights that enhance the low-frequency time series (f β and f θ ) because f θ and f α The resting group is more representative of stable and peaceful emotions, and f γ and f β It can better represent the more excited emotions and is recorded as the active group. γ and f α For a higher frequency time series, the σ function is used for activation, f β and f θ Since it is a lower frequency time series, the tanh function is used for regularization. Then, the high and low layers of cross-frequency semantic features are superimposed element by element to obtain CFCL captures the cross-frequency coupling characteristics while retaining the sequence information of the original signal.

[0164] Step 4) Time series feature extraction

[0165] The CNN1D-ALSTM model combines the temporal feature extraction capability of a one-dimensional convolutional neural network with the long-sequence learning capability of a long short-term memory network to capture the complex temporal patterns in EEG recordings. The Conv1D layer acts as an automatic feature extractor with the help of a learnable convolution kernel, where the size of the convolution kernel is k. o is 3, step length s o is 1. Therefore, the time series features extracted in the first stage

[0166]

[0167] Then, with the help of length p o The maximum pooling layer downsamples to reduce the number of parameters. And the batch norm layer is applied to feature regularization to improve generalization ability. The feature representation after pooling is

[0168]

[0169] | represents an integer division operation. In the second stage, the Bi-LSTM layer processes features over time to capture evolving emotional states. Bi-LSTM is a bidirectional recursive recurrent neural network variant. Compared to unidirectional LSTM, it captures sequence dependencies more comprehensively by considering both past and future information at each time step. This bidirectional structure allows the network to consider both past (forward) and future (backward) EEG activity at each time step, more comprehensively capturing brain activity information related to emotional states. The Bi-LSTM structure is shown in Figure 1. Figure 4 As shown in the bottom part. The Bi-LSTM hidden layer is set to 128. The time series of the input layer from front to back is represented as Bi-LSTM consists of two independent hidden layers (forward layer and backward layer). The hidden cell state of the forward layer is expressed as The hidden cell state of the backward layer is expressed as

[0170] The iterative update method of the forward layer and the backward layer is as follows:

[0171]

[0172] in represents the learnable weights, and is a learnable bias. r after forward update t ,

[0173] Then, the forward unit and backward unit corresponding to the temporal position are used to generate the state representation of the output layer.

[0174]

[0175] Two layers of Bi-LSTM networks are stacked to deeply memorize and encode sequence features. The hidden layer of the second Bi-LSTM is also set to 128.

[0176] In the third stage, the subjects will focus at certain moments in a trial and relax at other times. Focusing on the sequence characteristics of the subject's focus period and ignoring the relaxation period can reduce the number of parameters and improve classification accuracy. We use an extended self-attention mechanism to assign weights to each EEG signal sample by exploring the intrinsic importance of each sample. The structure of self-attention is shown in Figure 2. Figure 4 As shown in the upper part. The self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation. First, the output layer is mapped to the attention vector

[0177]

[0178] in represents the learnable weights, is a learnable bias. The attention vector is randomly initialized and jointly learned during training. The softmax activation function ensures that the sum of all calculated weights is 1 to obtain the importance weight of each time step

[0179]

[0180] When copying and expanding the importance weights into Then, the weighted sum of all EEG timestamps is calculated as the timestamp focus expressed as ⊙ represents the Hardmann product.

[0181] f Ta =∑ t α i ⊙v i (12)

[0182] Step 5) Spatial feature extraction and emotion classification

[0183] In the last part of the proposed CFMN, we first use the convolution kernel size k d is 3, step length s d Conv1D with a value of 1 is used to extract the features of C leads. The temporal features between different channels are reorganized into new spatiotemporal features.

[0184] f st =conv1d(f Ta ) (13)

[0185] conv1d() convolutional layer

[0186] The Flatten operation converts a multidimensional vector into a one-dimensional output

[0187] O = flatten(f st ) (14)

[0188] flatten() expansion function

[0189] Then the fully connected layer projects the high-dimensional spatiotemporal features into the emotion space to be classified. The softmax function calculates the conditional probability of each emotion category and then selects the category corresponding to the maximum conditional probability as the predicted output category.

[0190] Then, the cross entropy error is evaluated on all labeled samples.

[0191] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. The EEG recognition system based on nonlinear cross-frequency modulation network is characterized by: It includes preprocessing module, frequency band decomposition and feature representation module, cross-frequency coupling layer, CNN1D-ALSTM model, spatial decoding module and emotion classification module: The preprocessing module is used to receive the collected original EEG signals and perform preprocessing; The frequency band decomposition and feature representation module is used to perform frequency band division on the preprocessed EEG signal based on eMEMD, and nonlinearly extract the signal component of the target frequency band; The cross-frequency coupling layer is used to capture coupling information between different frequency bands and output cross-frequency coupling characteristics; The CNN1D-ALSTM model combines the temporal feature extraction capability of the one-dimensional convolutional neural network and the long sequence learning capability of the Bi-LSTM to capture the temporal features in the cross-frequency coupling features; The spatial decoding module is used to perform spatial decoding on the temporal features to obtain spatiotemporal features; The emotion classification module is used to project the spatiotemporal features into the emotion space to be classified.

2. The EEG recognition system based on nonlinear cross-frequency modulation network as claimed in claim 1, characterized in that: The specific preprocessing method of the preprocessing module is as follows: Get the collected original EEG signal, the sampling frequency is f s , use the blind source signal separation method to remove motion artifacts; down-sample to unify the multi-channel sampling frequency, and use a bandpass filter on the down-sampled EEG signal to remove power frequency noise and irrelevant frequency components.

3. The EEG recognition system based on nonlinear cross-frequency modulation network as claimed in claim 1, characterized in that: The frequency band decomposition and feature representation module is specifically implemented in the following manner: S1. The remaining signal is set to Initialize h(t) equal to the input signal x(t); C represents the number of EEG signal channels, T represents the number of EEG signal sampling points, is a set of real numbers of dimension C×T; S2. Generate a mask frequency sequence in f s is the sampling frequency; is the mask frequency sequence f a The qth frequency value in , q is initialized to 1; S3. Design mask signal set based on step strategy Among them, is a self-determined constant, is the kth mask signal: Where std(*) is the standard deviation, θ k is the kth mask phase, θ k =2πk / l; S4. The first order eigenmode function of the kth phase E1(*) represents the operation of taking the first-order intrinsic mode function IMF1 of the k-th phase. When k < l, let k increment by 1 and return to S3; otherwise, calculate the q-th order intrinsic mode function S5. If q<4, calculate the new residual signal as the original residual signal minus d q (t), q increases by 1, and the process returns to S3 repeatedly; Otherwise, stop the loop; Thus, narrowband signals in four empirical frequency bands are obtained. S6. In a single test, the length of the s The non-overlapping Hanning window of the narrowband signal in each empirical frequency band is segmented. γ , x β , x α , x θ The dimension becomes x γ , x β , x α , x θ Reshape Then complete the Conv1D convolution and get T f represents the number of features after convolution, Narrowband signal x in four empirical frequency bands γ , x β , x α , x θ The corresponding time series are f γ , f β , f α , f θ .

4. The EEG recognition system based on nonlinear cross-frequency modulation network as claimed in claim 3, characterized in that: The cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output the cross-frequency coupling feature f CFC Represented as f CFC =tanh(f β )⊙σ(f γ )+tanh(f θ )⊙σ(f α ) (4) Where ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The time series f is activated by the σ activation function. γ and f α , regularize the time series f by the tanh function β and f θ .

5. The EEG recognition system based on nonlinear cross-frequency modulation network according to any one of claims 1 to 4, characterized in that: The CNN1D-ALSTM model includes a Conv1D layer, a MaxPool layer, a Batch Norm layer, a first BiLSTM layer, a second BiLSTM layer, and a self-attention mechanism layer; The first stage consists of a convolutional layer-Conv1D layer, a maximum pooling layer-MaxPool layer, and a batch norm layer-Batch Norm layer; the Conv1D layer acts as an automatic feature extractor with the help of a convolution kernel; then, with the help of a length of p o The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization; The first BiLSTM layer and the second BiLSTM layer constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

6. The EEG recognition system based on nonlinear cross-frequency modulation network as claimed in claim 5, characterized in that: The spatial decoding module includes using a convolutional layer - Conv1D layer to extract temporal features of different channels, recombining the temporal features between different channels into new spatiotemporal features, and converting the spatiotemporal features from multi-dimensional features to one-dimensional output.

7. The EEG recognition method based on nonlinear cross-frequency modulation network is characterized in that: The steps include: Step 1: Preprocess the collected original EEG signals; Step 2: Perform frequency band division on the preprocessed EEG signal based on eMEMD, and extract the signal components of the target frequency band nonlinearly; Step 3: Capture the coupling information between different frequency bands and output the cross-frequency coupling characteristics; Step 4: capturing the timing characteristics in the cross-frequency coupling characteristics; Step 5: spatially decode the temporal features to obtain spatiotemporal features; Step 6: Project the spatiotemporal features into the emotion space to be classified.

8. The EEG recognition method based on nonlinear cross-frequency modulation network as claimed in claim 7, characterized in that: The step 2 is specifically implemented as follows: S1. The remaining signal is set to Initialize h(t) equal to the input signal x(t); C represents the number of EEG signal channels, T represents the number of EEG signal sampling points, is a set of real numbers of dimension C×T; S2. Generate a mask frequency sequence in f s is the sampling frequency; is the mask frequency sequence f a The qth frequency value in , q is initialized to 1; S3. Design mask signal set based on step strategy Where l is a self-determined constant, is the kth mask signal: Where std(*) is the standard deviation, θ k is the kth mask phase, θ k =2πk / l; The first-order eigenmode function of the k-th phase E1(*) represents the operation of taking the first-order eigenmode function IMF1 of the k-th phase. When k < l, increment k by 1 and return to S3; otherwise, calculate the q-th order eigenmode function S5. If q<4, calculate the new residual signal as the original residual signal minus d q (t), q increases by 1, and the process returns to S3 repeatedly; Otherwise, stop the loop; Thus, narrowband signals in four empirical frequency bands are obtained. S6. In a single test, the length of the s The non-overlapping Hanning window of the narrowband signal in each empirical frequency band is segmented. γ , x β , x α , x θ The dimension becomes x γ , x β , x α , x θ Reshape Then complete the Conv1D convolution and get T f represents the number of features after convolution, Narrowband signal x in four empirical frequency bands γ , x β , x α , x θ The corresponding time series are f γ , f β , f α , f θ ; In step 3, the output cross-frequency coupling characteristic f CFC Expressed as f CFC =tanh(f β )⊙σ(f γ )+tanh(f θ )⊙σ(f α ) (4) Where ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The time series f is activated by the σ activation function. γ and f α , regularize the time series f by the tanh function β and f θ .

9. The EEG recognition method based on nonlinear cross-frequency modulation network according to claim 7 or 8, characterized in that: In the step 4, a CNN1D-ALSTM model is used to capture the timing characteristics in the cross-frequency coupling characteristics; The CNN1D-ALSTM model includes a Conv1D layer, a MaxPool layer, a Batch Norm layer, a first BiLSTM layer, a second BiLSTM layer, and a self-attention mechanism layer; The first stage consists of a convolutional layer-Conv1D layer, a maximum pooling layer-MaxPool layer, and a batch norm layer-Batch Norm layer; the Conv1D layer acts as an automatic feature extractor with the help of a convolution kernel; then, with the help of a length of p o The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization; The first BiLSTM layer and the second BiLSTM layer constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

10. The EEG recognition method based on nonlinear cross-frequency modulation network according to claim 9, characterized in that: In the step 5, a convolutional layer-Conv1D layer is used to extract the temporal features of different channels, the temporal features between different channels are recombined into new spatiotemporal features, and the spatiotemporal features are converted from multi-dimensional features to one-dimensional output; In step six, the conditional probability of each emotion category is calculated using the softmax function, and then the category corresponding to the maximum conditional probability is selected as the predicted output category.

Citation Information

Patent Citations

  • System and method for monitoring neural signals

    CN112867441A

  • Motion function monitoring management method based on electroencephalogram and myoelectricity cross frequency coupling

    CN113274033A

  • Emotion recognition method and device based on variational phase amplitude coupling

    CN114343635A

  • Individualized cognitive function adjusting device based on cross frequency coupling

    CN115487418A

  • Emotion recognition method and device based on electroencephalogram phase amplitude coupling network

    CN115770044A

Cited By

  • Nerve regulation and control method based on dynamic frequency band analysis

    CN121041587A

  • Non-stationary signal cross-frequency coupling frequency domain feature acquisition method and system

    CN121101604A

  • Attention recognition method based on electroencephalogram signals

    CN121867786A