EEG recognition method and system based on nonlinear cross-frequency modulation network

Abstract

The present invention discloses an EEG recognition method and system based on a nonlinear cross-frequency modulation network, relating to the technical field of EEG signal processing. The method realizes nonlinear division of target frequency bands and adaptively extracts cross-frequency coupling features. The technical solution specifically comprises: a preprocessing module for receiving and preprocessing the collected raw EEG signals; a frequency band decomposition and feature representation module for performing frequency band division on the preprocessed EEG signals based on eMEMD and nonlinearly extracting the signal components of the target frequency band; a cross-frequency coupling layer for capturing coupling information between different frequency bands and outputting cross-frequency coupling features; a CNN1D-ALSTM model, combining the temporal feature extraction capability of a one-dimensional convolutional neural network with the long sequence learning capability of a Bi-LSTM, to capture the temporal features in the cross-frequency coupling features; a spatial decoding module for spatially decoding the temporal features to obtain spatiotemporal features; and an emotion classification module for projecting the spatiotemporal features into the emotion space to be classified.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal processing, and in particular to an EEG recognition method and system based on a nonlinear cross-frequency modulation network. Background Art

[0002] Advances in brain-computer interfaces, signal processing, and artificial intelligence have driven the development of affective computing. This interdisciplinary field supports enhanced decision-making, cognitive efficiency, and attention in the context of education and advertising. Intervention for patients with mood disorders, such as depression, can be achieved through real-time affective monitoring and assessment. High-temporal resolution electroencephalogram (EEG) signals facilitate objective reflection of emotion-generating mechanisms, providing a convenient and cost-effective solution for large-scale data collection.

[0003] Brain rhythms are often divided into frequency bands to represent different emotional patterns, such as the α band representing stable emotions and the β band representing excitement. Linear filters, such as wavelet filters, have been used in some studies for frequency band division, but these filters can distort nonlinear waveforms and patterns. Empirical mode decomposition (EMD), on the other hand, can act as a nonstationary and nonlinear filter to decompose signals into intrinsic mode functions (IMFs). Multivariate empirical mode decomposition (M-EMD) extracts highly consistent IMFs from multi-source signals by jointly locating multi-channel extrema and iteratively constructing a joint envelope. However, M-EMD lacks the ability to accurately capture target EEG frequency bands (e.g., θ and α). To address this limitation, the exclusive masking empirical mode decomposition (eMEMD) method continuously dichotomizes the masking signal by its center frequency, achieving IMF alignment across multiple channels and confining them to the EEG empirical frequency band.

[0004] Since then, some studies have extracted features from single empirical brain rhythms to achieve emotion recognition. Although these algorithms may be useful, they ignore the coupled information across multiple frequency bands.

[0005] To address this issue, researchers have employed concatenation to fuse features from different bands. Concatenated spatial features or connectivity features of θ, α, β, and γ are fed into low-level classifiers, which to some extent aid in the recognition of emotional modalities. However, the lack of cross-frequency coherence in frequency semantic information can hinder the capture of complex EEG dynamics, thereby limiting its discriminative power. Furthermore, the increased feature dimensionality during concatenation hinders real-time emotion recognition, resulting in increased memory usage.

[0006] Linear filtering methods, such as Butterworth filters or infinite impulse response (FIR) filters, have characteristics that deviate from the non-stationary and nonlinear characteristics of EEG signals. Empirical mode decomposition (EMD) is beneficial for correctly preserving nonlinear and non-stationary causal characteristics while filtering. Multivariate empirical mode decomposition (M-EMD) extracts highly consistent intrinsic mode functions (IMFs) from multi-source signals by jointly locating multi-channel extrema and iteratively constructing a joint envelope. However, M-EMD cannot obtain the precise frequency modulation of IMFs under the constraints of the EEG signal's empirical frequency band (such as θ and α).

[0007] Traditional M-EMD cannot constrain frequency. However, for the needs of biopsychological diagnosis and emotion recognition, considering the collaborative heterogeneity mechanism of complex emotional EEG signals in multi-band rhythms, using only single-band feature recognition may lose the multi-dimensional information of biopsychological EEG rhythms. If the traditional splicing method is used to fuse multi-band features, its cross-frequency coupling mechanism is ignored. Summary of the Invention

[0008] In view of this, the present invention provides an EEG recognition method and system based on a nonlinear cross-frequency modulation network, which realizes nonlinear division of target frequency bands and can adaptively extract cross-frequency coupling features.

[0009] To achieve the above objectives, the technical solution of the present invention is: an EEG recognition system based on a nonlinear cross-frequency modulation network, including a preprocessing module, a frequency band decomposition and feature representation module, a cross-frequency coupling layer, a CNN1D-ALSTM model, a spatial decoding module, and an emotion classification module:

[0010] The preprocessing module is used to receive the collected original EEG signals and perform preprocessing.

[0011] The frequency band decomposition and feature representation module is used to divide the preprocessed EEG signal into frequency bands based on eMEMD and nonlinearly extract the signal components of the target frequency band.

[0012] The cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output the cross-frequency coupling characteristics.

[0013] The CNN1D-ALSTM model combines the temporal feature extraction capability of a one-dimensional convolutional neural network with the long sequence learning capability of a Bi-LSTM to capture the temporal features in cross-frequency coupling features.

[0014] The spatial decoding module is used to perform spatial decoding on the time series features to obtain spatiotemporal features.

[0015] The emotion classification module is used to project spatiotemporal features into the emotion space to be classified.

[0016] Preferably, the preprocessing module includes the following specific preprocessing methods:

[0017] Get the collected original EEG signal with a sampling frequency of f s , use the blind source signal separation method to remove motion artifacts; downsampling is performed to unify the multi-channel sampling frequency, and a bandpass filter is used on the downsampled EEG signal to remove power frequency noise and irrelevant frequency components.

[0018] Preferably, the frequency band decomposition and feature representation module is specifically implemented as follows:

[0019] S1. The remaining signal is set to Initialize h(t) equal to the input signal x(t); C represents the number of EEG signal channels, T represents the number of EEG signal sampling points, is a set of real numbers of dimension C×T.

[0020] S2. Generate a mask frequency sequence in f s is the sampling frequency; is the mask frequency sequence f a The qth frequency value in , q is initialized to 1.

[0021]

[0022] S3. Design mask signal set based on step strategy Where l is a self-determined constant, is the kth mask signal:

[0023]

[0024] Where std(*) is the standard deviation, θ k is the kth mask phase, θ k =2πk / l.

[0025] The first-order eigenmode function of the k-th phase E1(*) represents the operation of taking the first-order eigenmode function IMF1 of the k-th phase. When k < l, increment k by 1 and return to S3; otherwise, calculate the q-th order eigenmode function

[0026] S5. If q < 4, calculate the new residual signal as the original residual signal minus d q (t), increment q by 1, and repeat to return to S3; otherwise, stop the loop.

[0027] Thus, narrowband signals in four empirical frequency bands are obtained

[0028] S6. In a single trial, segment the narrowband signal in each empirical frequency band through a non-overlapping Hanning window with length T s The narrowband signals x γ , x β , x α , x θ become in dimension Reshape x γ , x β , x α , x θ into With the Conv1D convolution completed, the obtained T f represents the number of features after convolution,

[0029] The time series corresponding to the narrowband signals x γ , x β , x α , x θ are respectively f γ , f β , f α , f θ .

[0030] Preferably, the cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output the cross-frequency coupling feature f CFC which is expressed as

[0031] f CFC = tanh(f β ) ⊙ σ(f γ ) + tanh(f θ ) ⊙ σ(f α ) (4)

[0032] Where ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The time series f is activated by the σ activation function. γ and f α , regularize the time series f by the tanh function β and f θ .

[0033] Preferably, the CNN1D-ALSTM model includes a Conv1D layer, a MaxPool layer, a Batch Norm layer, a first BiLSTM layer, a second BiLSTM layer, and a self-attention mechanism layer.

[0034] The first stage consists of the Conv1D layer, the MaxPool layer, and the Batch Norm layer. The Conv1D layer acts as an automatic feature extractor with the help of the convolution kernel. After that, with the help of the length p o The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization.

[0035] The first and second BiLSTM layers constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

[0036] Preferably, the spatial decoding module includes using a convolutional layer - Conv1D layer to extract temporal features of different channels, recombining the temporal features between different channels into new spatiotemporal features, and converting the spatiotemporal features from multi-dimensional features into one-dimensional output.

[0037] Another embodiment of the present invention further provides an EEG recognition method based on a nonlinear cross-frequency modulation network, comprising the following steps:

[0038] Step 1: Preprocess the collected raw EEG signals.

[0039] Step 2: Perform frequency band division on the preprocessed EEG signal based on eMEMD, and extract the signal components of the target frequency band nonlinearly.

[0040] Step 3: Capture the coupling information between different frequency bands and output the cross-frequency coupling characteristics.

[0041] Step 4: Capture the timing characteristics of the cross-frequency coupling characteristics.

[0042] Step 5: Spatial decoding of temporal features to obtain spatiotemporal features.

[0043] Step 6: Project the spatiotemporal features into the emotion space to be classified.

[0044] Further, in step two, the following specific method is adopted:

[0045] S1. Set the remaining signal as Initialize h(t) to be equal to the input signal x(t); C represents the number of EEG signal channels, and T represents the number of EEG signal sampling points. is a real number set of C×T dimensions;

[0046] S2. Generate a mask frequency sequence where f s is the sampling frequency; is the q-th frequency value in the mask frequency sequence f a , and q is initialized to 1.

[0047]

[0048] S3. Design a mask signal set according to the stepping strategy where l is a self-defined constant. is the k-th mask signal:

[0049] <{

[0050] where std(*) is the standard deviation, and θ k is the k-th mask phase, and θ k = 2πk / l.

[0051] S4. The first-order intrinsic mode function of the k-th phase represents the operation of taking the first-order intrinsic mode function IMF1 of the k-th phase. When k < l, let k increment by 1 and return to S3; otherwise, calculate the q-th order intrinsic mode function <000033|> <00|00332>S5. If q < 4, calculate the new remaining signal as the original remaining signal minus d q (t), increment q by 1, and repeat to return to S3; otherwise, stop the loop.

[0053] Thus, narrowband signals under four empirical frequency bands are obtained

[0054] S6. In a single trial, segment the narrowband signals under each empirical frequency band through a non-overlapping Hanning window with a length of T s The narrowband signals x γ , x β , x α , x θ The dimension becomes Make x γ , x β , xα , x θ Reshape into Then complete the Conv1D convolution and get T f represents the number of features after convolution,

[0055] Narrowband signal x in four empirical frequency bands γ , x β , x α , x θ The corresponding time series are f γ , f β , f α , f θ .

[0056] Step 3: Output cross-frequency coupling characteristics f CFC Expressed as

[0057] f CFC =tanh(f β )⊙σ(f γ )+tanh(f θ )⊙σ(f α ) (4)

[0058] Where ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The time series f is activated by the σ activation function. γ and f α , regularize the time series f by the tanh function β and f θ .

[0059] Furthermore, in step four, the CNN1D-ALSTM model is used to capture the timing features in the cross-frequency coupling features.

[0060] The CNN1D-ALSTM model includes a Conv1D layer, a MaxPool layer, a Batch Norm layer, a first BiLSTM layer, a second BiLSTM layer, and a self-attention mechanism layer.

[0061] The first stage consists of the Conv1D layer, the MaxPool layer, and the Batch Norm layer. The Conv1D layer acts as an automatic feature extractor with the help of the convolution kernel. After that, with the help of the length p o The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization.

[0062] The first and second BiLSTM layers constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

[0063] Furthermore, in step five, the convolutional layer - Conv1D layer is used to extract the temporal features of different channels, and the temporal features between different channels are recombined into new spatiotemporal features, converting the spatiotemporal features from multi-dimensional features to one-dimensional output.

[0064] Step 6: Use the softmax function to calculate the conditional probability of each emotion category, and then select the category corresponding to the maximum conditional probability as the predicted output category.

[0065] Beneficial effects:

[0066] This paper provides an EEG recognition method and system based on a nonlinear cross-frequency modulation network. The paper innovatively proposes a cross-frequency coupling layer (CFCL), which adaptively and robustly constructs cross-frequency coupling features for emotion recognition. First, the cross-frequency modulation network (CFMN) performs frequency band segmentation based on eMEMD, nonlinearly extracting signal components in the target frequency band. Furthermore, the CFCL is constructed by bitwise multiplying high-frequency modulation weights with low-frequency embedded time series elements to extract semantic information from cross-frequency interactions. The CFMN combines a Conv1D-BiLSTM-A layer, consisting of a one-dimensional convolutional neural network (Conv1D) and an attentive bidirectional long short-term memory (BiLSTM-A) network, to extract sequence information from the modulation frequency sequence. The processed features are fed into a fully connected layer to distinguish different emotions. In a nonstationary nonlinear simulation system, eMEMD is shown to effectively segment frequency bands, significantly outperforming the EMD and M-EMD algorithms in terms of independence. In real-world experiments using the DEAP database, the nonlinear cross-frequency features of the eMEMD's frequency band segmentation were shown to improve classification accuracy. In validation experiments on the DEAP database for both binary and four-category valence / arousal classification, the CFMN achieved optimal performance under similar conditions. In summary, the proposed CFMN can adaptively extract cross-frequency coupling features, making it an effective tool for emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1This is a schematic diagram of the process flow of the EEG recognition system based on the nonlinear cross-frequency modulation network provided by the present invention;

[0068] Figure 2 Flowchart for empirical frequency band partitioning for eMEMD nonlinearity;

[0069] Figure 3 Combines two independent hidden layers for Bi-LSTM;

[0070] Figure 4 Schematic diagram of mode independence test. DETAILED DESCRIPTION

[0071] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0072] Example 1:

[0073] The EEG recognition system based on nonlinear cross-frequency modulation network is composed of Figure 1 As shown in the figure, the system includes a preprocessing module, a frequency band decomposition and feature representation module, a cross-frequency coupling layer, a CNN1D-ALSTM model, a spatial decoding module, and an emotion classification module:

[0074] The preprocessing module is used to receive the collected original EEG signal and perform preprocessing. In the embodiment of the present invention, the specific preprocessing method in the preprocessing module is: obtain the collected original EEG signal, the sampling frequency is f s , use the blind source signal separation method to remove motion artifacts; downsampling is performed to unify the multi-channel sampling frequency, and a bandpass filter is used on the downsampled EEG signal to remove power frequency noise and irrelevant frequency components.

[0075] The frequency band decomposition and feature representation module is used to perform frequency band division on the preprocessed EEG signal based on eMEMD and nonlinearly extract the signal components of the target frequency band. In the embodiment of the present invention, the frequency band decomposition and feature representation module specifically adopts the following method:

[0076] S1. The remaining signal is set to Initialize h(t) equal to the input signal x(t); C represents the number of EEG signal channels, T represents the number of EEG signal sampling points, is a set of real numbers of C×T dimensions;

[0077] S2. Generate a mask frequency sequence in f s is the sampling frequency; is the mask frequency sequence f a The qth frequency value in , q is initialized to 1;

[0078]

[0079] S3. Design the mask signal set according to the step-by-step strategy where l is a self-defined constant, is the k-th mask signal:

[0080]

[0081] where std(*) is the standard deviation, θ k is the k-th mask phase, θ k = 2πk / l;

[0082] S4. The first-order eigenmode function of the k-th phase E1(*) represents the operation of taking the first-order eigenmode function IMF1 of the k-th phase. When k < l, let k increment by 1 and return to S3; otherwise, calculate the q-th order eigenmode function

[0083] S5. If q < 4, calculate the new residual signal as the original residual signal minus d q (t), increment q by 1, and repeat to return to S3; otherwise, stop the loop;

[0084] Thus, narrowband signals under four empirical frequency bands are obtained

[0085] S6. In a single trial, segment the narrowband signal under each empirical frequency band through a non-overlapping Hanning window with length T s , the narrowband signal x γ , x β , x α , x θ dimension becomes Reshape x​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​The cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output cross-frequency coupling features. The present invention establishes a cross-frequency coupling layer (CFCL) to extract cross-frequency interactions. In short, the modulated attention weights generated from the embedded high-frequency time series are used to enhance the embedded low-frequency time series through element-by-element multiplication. This process constructs cross-frequency semantic information while retaining detailed time series information. Meta-operations include element-wise multiplication and addition, thereby reducing feature size and ensuring operation speed.

[0088] In the embodiment of the present invention, the cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output the cross-frequency coupling feature f CFC Expressed as

[0089] f CFC =tanh(f β )⊙σ(f γ )+tanh(f θ )⊙σ(f α ) (4)

[0090] Where ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The time series f is activated by the σ activation function. γ and f α , regularize the time series f by the tanh function β and f θ .

[0091] The CNN1D-ALSTM model combines the temporal feature extraction capabilities of a one-dimensional convolutional neural network with the long sequence learning capabilities of a Bi-LSTM to capture temporal features in cross-frequency coupling features. The CNN1D-ALSTM model includes a Conv1D layer, a MaxPool layer, a Batch Norm layer, the first BiLSTM layer, the second BiLSTM layer, and a self-attention mechanism layer.

[0092] The first stage consists of the Conv1D layer, the MaxPool layer, and the Batch Norm layer. The Conv1D layer acts as an automatic feature extractor with the help of the convolution kernel. After that, with the help of the length p o The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization;

[0093] The first and second BiLSTM layers constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

[0094] A spatial decoding module is used to spatially decode temporal features to obtain spatiotemporal features. In an embodiment of the present invention, the spatial decoding module includes using a convolutional layer - Conv1D layer to extract temporal features of different channels, recombining the temporal features between different channels into new spatiotemporal features, and converting the spatiotemporal features from multi-dimensional features into one-dimensional output.

[0095] The emotion classification module is used to project the spatiotemporal features into the emotion space to be classified. In the embodiment of the present invention, the softmax function is used to calculate the conditional probability of each emotion category, and then the category corresponding to the maximum conditional probability is selected as the predicted output category.

[0096] Example 2:

[0097] In order to solve the technical difficulty that the traditional emotion classification network cannot adaptively extract nonlinear cross-frequency coupling features, this embodiment proposes an EEG recognition method based on a nonlinear cross-frequency modulation network. This method achieves a valence accuracy of 93.2% and an arousal accuracy of 92.5% when distinguishing emotional tasks. Compared with other studies under similar conditions, it has better accuracy and can be used for emotion detection and recognition. This method can be executed in the EEG recognition system based on a nonlinear cross-frequency modulation network provided in Example 1; the EEG recognition method based on a nonlinear cross-frequency modulation network provided in this embodiment has a technical process as follows Figure 2 The detailed process is as follows:

[0098] Step 1: Collect the EEG signal of the subject, with a sampling frequency of f s , the above raw EEG signal is preprocessed, specifically including the following preprocessing:

[0099] A1. Use blind source signal separation methods, such as independent component analysis (ICA), to remove motion artifacts such as electrooculography and electromyography.

[0100] A2. Downsample to a uniform multi-channel sampling frequency (e.g., 128 Hz). Apply a bandpass filter (e.g., 4-45 Hz) to the downsampled EEG signal to prevent power frequency noise and irrelevant frequency components from affecting the results.

[0101] Step 2: Perform frequency band division on the pre-processed EEG signal based on eMEMD, and nonlinearly extract the signal components of the target frequency band. In this embodiment of the present invention, the following steps are included:

[0102] S1. Define the residual signal Initialize h(t) equal to the input signal x(t). C represents the number of EEG signal channels, and T represents the number of EEG signal sampling points. is a set of real numbers of C×T dimensions;

[0103] S2. Generate a mask frequency sequence where f s is the sampling frequency. is the qth frequency value in the mask frequency sequence f a ;

[0104]

[0105] q is initialized to 1;

[0106] S3. Design a mask signal set according to the stepping strategy where l is a self-defined constant, usually 6. is the kth mask signal

[0107]

[0108] std() is the standard deviation, θ k is the kth mask phase, θ k = 2πk / l.

[0109] S4. Calculate E1() represents the operation of taking the first-order eigenmode function IMF1 of the kth phase. The first-order eigenmode function of the kth phase When k < l, let k = k + 1, and repeat steps 3 - 4; otherwise, calculate the qth-order eigenmode function

[0110] S5. If q < 4, calculate the new residual signal h(t) = h(t) - d q (t), q = q + 1, and repeat steps 3 - 5; otherwise, stop the loop and obtain the narrowband signals under four empirical frequency bands

[0111] S6. In a single trial, segment the EEG signals of each frequency band through a non-overlapping Hanning window with a length of T s . The dimension of the narrowband signal becomes T s which is generally set to 256. Apply a one-dimensional convolutional layer Conv1D with a convolutional kernel size k d of 3 and a stride s d of 2 to each segment to extract the features of C leads. Conv1D continuously changes the convolutional kernel through training, and then adaptively filters out the frequency band components of interest within the specified frequency band range. If the original signal is directly input, Conv1D will convolve the temporal features of multiple adjacent channels, so the correlation between different channels is extracted simultaneously. This will lead to a decrease in the attention to the extraction of the frequency of interest. Therefore, first xγ , x β , x α , x θ Reshape into Then complete the Conv1D convolution. Since the reshape operation separates the channels into other dimensions, Conv1D focuses on extracting the frequency components of interest in each channel. We set 128 Conv1D convolution kernels, so the obtained T f Represents the number of features after convolution.

[0112]

[0113] Step 3: Capture coupling information between different frequency bands and output cross-frequency coupling characteristics. The specific steps of this step in this embodiment are as follows:

[0114] The cross-frequency coupling layer (CFCL) is used to capture the coupling information between different frequency bands and output the cross-frequency coupling feature f CFC Expressed as

[0115] f CFC =tanh(f β )⊙σ(f γ )+tanh(f θ )⊙σ(f β ) (4)

[0116] ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The high-frequency time series (f γ and f α ,) generates attention weights that enhance the low-frequency time series (f β and f θ ) because f θ and f α The resting group is more representative of stable and peaceful emotions, and f γ and f β It can better represent the more excited emotions and is recorded as the active group. γ and f α For a higher frequency time series, the σ function is used for activation, f β and f θ Since it is a lower frequency time series, the tanh function is used for regularization. Then, the high and low layers of cross-frequency semantic features are superimposed element by element to obtain CFCL captures the cross-frequency coupling characteristics while retaining the sequence information of the original signal.

[0117] Step 4: Capture the timing characteristics of the cross-frequency coupling characteristics.

[0118] The CNN1D-ALSTM model combines the temporal feature extraction capability of a one-dimensional convolutional neural network with the long-term learning capability of a long short-term memory network to capture the complex temporal patterns in EEG recordings. The Conv1D layer acts as an automatic feature extractor with the help of a learnable convolution kernel, where the size of the convolution kernel is k. o is 3, step length s o is 1. Therefore, the time series features extracted in the first stage

[0119]

[0120] Then, with the help of length p o The maximum pooling layer downsamples to reduce the number of parameters. And the batch norm layer is applied to feature regularization to improve generalization ability. The feature representation after pooling is

[0121] T o ′=T o |p o (6)

[0122] | represents an integer division operation. In the second stage, the Bi-LSTM layer processes features over time to capture the evolving emotional state. Bi-LSTM is a bidirectional recursive neural network variant. Compared with the unidirectional LSTM, it captures the sequence dependency more comprehensively by considering both past and future information at each time step. This bidirectional structure enables the network to consider both past (forward) and future (backward) EEG activity at each time step, more comprehensively capturing brain activity information related to the emotional state. The Bi-LSTM structure is as follows: Figure 3 shown. Figure 3 The figure shows a Bi-LSTM model that combines two independent hidden layers: a forward layer for forward-to-backward movement and a backward layer for backward-to-front movement. This allows the network to model both forward and backward information from the input layer at each timestamp. Consequently, both past and future information are preserved in the output layer. A self-attention mechanism assigns importance weights to each time step and weightedly aggregates nodes across all time steps to extract sentiment-focused temporal features.

[0123] The number of Bi-LSTM hidden layers is set to 128. The time series of the input layer is represented from front to back as Bi-LSTM consists of two independent hidden layers (forward layer and backward layer). The hidden cell state of the forward layer is represented as The hidden cell state of the backward layer is expressed as

[0124] The iterative update method of the forward layer and the backward layer is as follows:

[0125]

[0126] in represents the learnable weights, and is a learnable bias. r after forward update t ,

[0127] Then the forward unit and backward unit corresponding to the temporal position are used to generate the state representation of the output layer

[0128]

[0129] Two layers of Bi-LSTM networks are stacked to deeply memorize and encode sequence features. The hidden layer of the second Bi-LSTM is also set to 128.

[0130] In the third stage, the subject will focus their attention at certain moments in a trial and relax at other times. Focusing on the sequence characteristics of the subject's focus period and ignoring the relaxation period can reduce the number of parameters and improve classification accuracy. The embodiment of the present invention adopts an extended self-attention mechanism to assign a weight to each EEG signal sample by exploring the intrinsic importance of each sample. The structure of self-attention is as follows Figure 4 As shown in the upper part. The self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation. First, the output layer is mapped to the attention vector

[0131]

[0132] in represents the learnable weights, is a learnable bias. The attention vector is randomly initialized and learned jointly during training. The softmax activation function ensures that the sum of all calculated weights is 1 to obtain the importance weight of each time step

[0133]

[0134] When copying and expanding the importance weights into Then, the weighted sum of all EEG timestamps is calculated as the timestamp focus expressed as ⊙ represents the Hardmann product.

[0135] f Ta =∑ t α i ⊙vi (12)

[0136] Step 5: Spatial decoding of temporal features to obtain spatiotemporal features.

[0137] In the last part of the proposed CFMN, we first use the convolution kernel size k d is 3, step length s d Conv1D with a value of 1 is used to extract the features of C leads. The temporal features between different channels are recombined into new spatiotemporal features.

[0138] f st =conv1d(f Ta ) (13)

[0139] Among them, conv1d() is the convolution layer.

[0140] The Flatten operation converts a multidimensional vector into a one-dimensional output

[0141] O=flatten(f st ) (14)

[0142] The flatten() expansion function.

[0143] Step 6: Project the spatiotemporal features into the emotion space to be classified.

[0144] The fully connected layer then projects the high-dimensional spatiotemporal features into the emotion space to be classified. The softmax function calculates the conditional probability of each emotion category and selects the category corresponding to the maximum conditional probability as the predicted output category. The cross-entropy error is then evaluated across all labeled examples.

[0145] This paper proposes for the first time a new cross-frequency modulation network (CFMN) method for emotional EEG recognition, which is used to adaptively extract the cross-band characteristics of EEG signals. It is suitable for the identification and monitoring of health and disease, is efficient, reliable, and easy to software. Figure 4 Test for mode independence. (a) Comparison of three EMD-based algorithms: EMD, MEMD, and eMEMD. The three panels correspond to the frequency spectral densities of the fourth-order IMFs (IMF1 (yellow line), IMF2 (orange line), IMF3 (blue line), and IMF4 (green line) of the decomposition of a three-channel simulation signal. (b) The three EMD-based algorithms showed significant differences in the correlation between the IMFs (p < 0.0001). eMEMD exhibited lower IMF correlation than EMD and MEMD, and therefore had higher decomposition independence.

[0146] To address the technical difficulty of traditional emotion classification networks in being unable to adaptively extract nonlinear cross-frequency coupling features, this paper proposes a cross-frequency modulation network (CFMN). This method achieves 93.2% valence accuracy and 92.5% arousal accuracy in emotion classification tasks, which is superior to other studies under similar conditions and can be used for emotion detection and recognition. The technical flow chart of the present invention is shown below. Figure 2 The detailed process is as follows:

[0147] Step 1) Collect the EEG signal of the subject (emotional induced) with a sampling frequency of f s The above raw EEG signals are preprocessed, specifically including: 1. using blind source signal separation methods, such as independent component analysis (ICA), to remove motion artifacts such as electrooculogram and electromyography; 2. downsampling to unify the multi-channel sampling frequency (such as 128Hz), and applying a bandpass filter (such as 4-45Hz) to the downsampled EEG signals to avoid the influence of power frequency noise and irrelevant frequency components on the results.

[0148] Step 2) Feature extraction of empirical frequency band components ( Figure 3 ):

[0149] 1. Define (loop continuously) the remaining signal Initialize h(t) equal to the input signal x(t). C represents the number of EEG signal channels, and T represents the number of EEG signal sampling points. is a set of real numbers of dimension C×T.

[0150] 2. Generate a mask frequency sequence in f s is the sampling frequency. is the mask frequency sequence f a The qth frequency value in .

[0151]

[0152] q is initialized to 1.

[0153] 3. Design mask signal set based on step strategy Where l is a self-determined constant, usually 6. is the kth mask signal

[0154]

[0155] std() is the standard deviation, θ k is the kth mask phase, θ k =2πk / l

[0156] 4. Calculation E1() represents the operation of taking the first-order eigenmode function IMF1 of the k-th phase, and the first-order eigenmode function of the k-th phase When k < l, let k = k + 1, and repeat steps 3 - 4; otherwise, calculate the q-th order eigenmode function

[0157] 5. If q < 4, calculate the new residual signal h(t) = h(t) - d q (t), q = q + 1, and repeat steps 3 - 5; otherwise, stop the loop and obtain the narrowband signals under four empirical frequency bands

[0158] 6. In a single trial, segment the EEG signals of each frequency band through non-overlapping Hanning windows of length T s The dimension of the narrowband signal becomes Apply a one-dimensional convolutional layer (Conv1D) with a convolutional kernel size k d of 3 and a stride s d of 2 to each segment to extract the features of C leads. Conv1D continuously changes the convolutional kernel through training, and then adaptively filters out the frequency band components of interest within the specified frequency band range. If the original signal is directly input, Conv1D will convolve the temporal features of multiple adjacent channels, so the correlation between different channels is extracted simultaneously. This will lead to a decrease in the attention for extracting the frequency of interest. Therefore, first reshape x γ , x β , x α , x θ into and then complete the Conv-1D convolution. Since the reshaping operation separates the channels into other dimensions, Conv1D focuses on extracting the frequency components of interest for each channel. We set 128 Conv1D convolutional kernels, so the obtained T f represents the number of features after convolution.

[0159]

[0160] Step 3) Adaptive extraction of cross-frequency coupling

[0161] The cross-frequency coupling layer (CFCL) is used to capture the coupling information between different frequency bands and output the cross-frequency coupling feature f CFC which is expressed as [[ID=​​​​​​)+tanh(f θ )⊙σ(f α ) (4)

[0163] ⊙ represents the Hadamard product, tanh(·) represents the tanh activation function, and σ(·) represents the σ activation function. The high-frequency time series (f γ and f α ,) generates attention weights that enhance the low-frequency time series (f β and f θ ) because f θ and f α The resting group is more representative of stable and peaceful emotions, and f γ and f β It can better represent the more excited emotions and is recorded as the active group. γ and f α For a higher frequency time series, the σ function is used for activation, f β and f θ Since it is a lower frequency time series, the tanh function is used for regularization. Then, the high and low layers of cross-frequency semantic features are superimposed element by element to obtain CFCL captures the cross-frequency coupling characteristics while retaining the sequence information of the original signal.

[0164] Step 4) Time series feature extraction

[0165] The CNN1D-ALSTM model combines the temporal feature extraction capability of a one-dimensional convolutional neural network with the long-term learning capability of a long short-term memory network to capture the complex temporal patterns in EEG recordings. The Conv1D layer acts as an automatic feature extractor with the help of a learnable convolution kernel, where the size of the convolution kernel is k. o is 3, step length s o is 1. Therefore, the time series features extracted in the first stage

[0166]

[0167] Then, with the help of length p o The maximum pooling layer downsamples to reduce the number of parameters. And the batch norm layer is applied to feature regularization to improve generalization ability. The feature representation after pooling is

[0168]

[0169] | represents an integer division operation. In the second stage, the Bi-LSTM layer processes features over time to capture the evolving emotional state. Bi-LSTM is a bidirectional recursive neural network variant. Compared with the unidirectional LSTM, it captures the sequence dependency more comprehensively by considering both past and future information at each time step. This bidirectional structure enables the network to consider both past (forward) and future (backward) EEG activity at each time step, more comprehensively capturing brain activity information related to the emotional state. The Bi-LSTM structure is as follows: Figure 4 As shown in the bottom part. The Bi-LSTM hidden layer is set to 128. The time series of the input layer is represented from front to back as Bi-LSTM consists of two independent hidden layers (forward layer and backward layer). The hidden cell state of the forward layer is represented as The hidden cell state of the backward layer is expressed as

[0170] The iterative update method of the forward layer and the backward layer is as follows:

[0171]

[0172] in represents the learnable weights, and is a learnable bias. r after forward update t ,

[0173] Then the forward unit and backward unit corresponding to the temporal position are used to generate the state representation of the output layer

[0174]

[0175] Two layers of Bi-LSTM networks are stacked to deeply memorize and encode sequence features. The hidden layer of the second Bi-LSTM is also set to 128.

[0176] In the third stage, the subject will focus their attention at certain moments in a trial and relax at other times. Focusing on the sequence characteristics of the subject's focus period and ignoring the relaxation period can reduce the number of parameters and improve classification accuracy. We use an extended self-attention mechanism to assign weights to each EEG signal sample by exploring the intrinsic importance of each sample. The structure of self-attention is as follows Figure 4 As shown in the upper part. The self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation. First, the output layer is mapped to the attention vector

[0177]

[0178] in represents the learnable weights, is a learnable bias. The attention vector is randomly initialized and learned jointly during training. The softmax activation function ensures that the sum of all calculated weights is 1 to obtain the importance weight of each time step

[0179]

[0180] When copying and expanding the importance weights into Then, the weighted sum of all EEG timestamps is calculated as the timestamp focus expressed as ⊙ represents the Hardman product.

[0181] f Ta =∑ t α i ⊙v i (12)

[0182] Step 5) Spatial feature extraction and emotion classification

[0183] In the last part of the proposed CFMN, we first use the convolution kernel size k d is 3, step length s d Conv1D with a value of 1 is used to extract the features of C leads. The temporal features between different channels are recombined into new spatiotemporal features.

[0184] f st =conv1d(f Ta ) (13)

[0185] conv1d() convolutional layer

[0186] The Flatten operation converts a multidimensional vector into a one-dimensional output

[0187] O=flatten(f st ) (14)

[0188] flatten() expansion function

[0189] The fully connected layer then projects the high-dimensional spatiotemporal features into the emotion space to be classified. The softmax function calculates the conditional probability of each emotion category and selects the category corresponding to the maximum conditional probability as the predicted output category.

[0190] Then, the cross entropy error is evaluated on all labeled samples.

[0191] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. The EEG recognition system based on nonlinear cross-frequency modulation network is characterized by: It includes a preprocessing module, a frequency band decomposition and feature representation module, a cross-frequency coupling layer, a CNN1D-ALSTM model, a spatial decoding module, and an emotion classification module: The preprocessing module is used to receive the collected original EEG signals and perform preprocessing; The frequency band decomposition and feature representation module is used to perform frequency band division on the preprocessed EEG signal based on eMEMD, and nonlinearly extract the signal component of the target frequency band; The cross-frequency coupling layer is used to capture coupling information between different frequency bands and output cross-frequency coupling characteristics; The CNN1D-ALSTM model combines the temporal feature extraction capability of a one-dimensional convolutional neural network with the long sequence learning capability of a Bi-LSTM to capture the temporal features in the cross-frequency coupling features. The spatial decoding module is used to perform spatial decoding on the temporal features to obtain spatiotemporal features; The emotion classification module is used to project the spatiotemporal features into the emotion space to be classified; The frequency band decomposition and feature representation module specifically adopts the following method: S1. The remaining signal is set to ,initialization Equal to the input signal ; C represents the number of EEG signal channels, T represents the number of EEG signal sampling points, is a set of real numbers of C×T dimensions; S2. Generate a mask frequency sequence ,in , is the sampling frequency; is the mask frequency sequence The qth frequency value in , q is initialized to 1; （1） S3. Design mask signal set based on step strategy ,in is a self-defined constant, is the kth mask signal: （2） in (*) is the standard deviation, is the kth mask phase; S4. The first-order eigenmode function of the kth phase ; ; Represents the first-order eigenmode function with the kth phase operation, when season Increment by 1 and return to S3; Otherwise, calculate the qth order eigenmode function , ; S5. If , calculate the new residual signal as the original residual signal minus , Increment by 1 and return to S3 repeatedly; Otherwise, stop the loop; Thus, narrowband signals in four empirical frequency bands are obtained. , , , ; S6. In a single test, the length of The non-overlapping Hanning window of the narrowband signal in each experience frequency band is segmented. , , , The dimension becomes ;Will , , , Reshape into , and then complete the Conv1D convolution to get , , , , T f represents the number of features after convolution, ;k d Indicates the convolution kernel size, s d Indicates the step length; Narrowband signals in four empirical frequency bands , , , The corresponding time series are , , , ; The cross-frequency coupling layer is used to capture the coupling information between different frequency bands and output the cross-frequency coupling characteristics. Expressed as （4） in represents the Hadamard product, tanh(·) represents the tanh activation function, σ(·) represents the σ activation function, and the time series is activated by the σ activation function and , regularize the time series by the tanh function and ; The CNN1D-ALSTM model includes a Conv1D layer, a MaxPool layer, a Batch Norm layer, a first BiLSTM layer, a second BiLSTM layer, and a self-attention mechanism layer; The first stage consists of the Conv1D layer, the MaxPool layer, and the Batch Norm layer. The Conv1D layer acts as an automatic feature extractor with the help of the convolution kernel. The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization; The first BiLSTM layer and the second BiLSTM layer constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

2. The EEG recognition system based on nonlinear cross-frequency modulation network according to claim 1, characterized in that: The preprocessing module specifically performs the following preprocessing: Get the collected original EEG signal, the sampling frequency is , use blind source signal separation method to remove motion artifacts; downsampling is performed to unify the multi-channel sampling frequency, and a bandpass filter is used on the downsampled EEG signal to remove power frequency noise and irrelevant frequency components.

3. The EEG recognition system based on nonlinear cross-frequency modulation network according to claim 1, characterized in that: The spatial decoding module includes using a convolutional layer - Conv1D layer to extract temporal features of different channels, recombining the temporal features between different channels into new spatiotemporal features, and converting the spatiotemporal features from multi-dimensional features into one-dimensional output.

4. The EEG recognition method based on nonlinear cross-frequency modulation network is characterized by: The steps include: Step 1: Preprocess the collected raw EEG signals; Step 2: Perform frequency band division on the preprocessed EEG signal based on eMEMD, and extract the signal components of the target frequency band nonlinearly; Step 3: Capture the coupling information between different frequency bands and output the cross-frequency coupling characteristics; Step 4: Capturing the timing characteristics in the cross-frequency coupling characteristics; Step 5: spatially decode the temporal features to obtain spatiotemporal features; Step 6: Projecting the spatiotemporal features into the emotion space to be classified; The step 2 is specifically implemented as follows: S1. The remaining signal is set to ,initialization Equal to the input signal ; C represents the number of EEG signal channels, T represents the number of EEG signal sampling points, is a set of real numbers of C×T dimensions; S2. Generate a mask frequency sequence ,in , is the sampling frequency; is the mask frequency sequence The qth frequency value in , q is initialized to 1; （1） S3. Design mask signal set based on step strategy ,in is a self-defined constant, is the kth mask signal: （2） in (*) is the standard deviation, is the kth mask phase, ; S4. The first-order eigenmode function of the kth phase ; ; Represents the first-order eigenmode function with the kth phase operation, when season Increment by 1 and return to S3; Otherwise, calculate the qth order eigenmode function , ; S5. If , calculate the new residual signal as the original residual signal minus , Increment by 1 and return to S3 repeatedly; Otherwise, stop the loop; Thus, narrowband signals in four empirical frequency bands are obtained. , , , ; S6. In a single test, the length of The non-overlapping Hanning window of the narrowband signal in each experience frequency band is segmented. , , , The dimension becomes ;Will , , , Reshape into , and then complete the Conv1D convolution to get , , , , T f represents the number of features after convolution, ;k d Indicates the convolution kernel size, s d Indicates the step length; Narrowband signals in four empirical frequency bands , , , The corresponding time series are , , , ; Step 3: Output cross-frequency coupling characteristics Expressed as （4） in represents the Hadamard product, tanh(·) represents the tanh activation function, σ(·) represents the σ activation function, and the time series is activated by the σ activation function and , regularize the time series by the tanh function and ; In the step 4, a CNN1D-ALSTM model is used to capture the timing characteristics in the cross-frequency coupling characteristics; The CNN1D-ALSTM model includes a Conv1D layer, a MaxPool layer, a Batch Norm layer, a first BiLSTM layer, a second BiLSTM layer, and a self-attention mechanism layer; The first stage consists of the Conv1D layer, the MaxPool layer, and the Batch Norm layer. The Conv1D layer acts as an automatic feature extractor with the help of the convolution kernel. The MaxPool layer downsamples; and the Batch Norm layer is used for feature regularization; The first BiLSTM layer and the second BiLSTM layer constitute the second stage; the self-attention mechanism is used to assign importance to each recurrent encoding slice and aggregate this information to form the final representation, obtaining the temporal features in the cross-frequency coupling features.

5. The EEG recognition method based on nonlinear cross-frequency modulation network according to claim 4, characterized in that: In step 5, a convolutional layer (Conv1D layer) is used to extract temporal features of different channels, and the temporal features between different channels are recombined into new spatiotemporal features, and the spatiotemporal features are converted from multi-dimensional features to one-dimensional outputs; In step six, the conditional probability of each emotion category is calculated using the softmax function, and then the category corresponding to the maximum conditional probability is selected as the predicted output category.

Images