Electroencephalogram signal fine-grained emotion recognition method and system

Abstract

The present application relates to a kind of electroencephalogram fine-grained emotion recognition method, comprising: selecting subject electroencephalogram from FACED dataset, the frequency segmentation of preprocessed subject electroencephalogram is carried out;To the frequency segmentation of all electroencephalogram, using the edge center analysis method based on common fluctuation, construct common fluctuation edge time series ETS, common fluctuation edge functional connection matrix EFC;According to the high-order space-time characteristics of common fluctuation edge time series ETS, common fluctuation edge functional connection matrix EFC, construct CoEdge-STNet model, the CoEdge-STNet model is fine-grained emotion classification model;In different frequency band under subject, using CoEdge-STNet model carries out fine-grained emotion recognition;In different frequency band under cross-subject, using CoEdge-STNet model carries out fine-grained emotion recognition.The present application also relates to a kind of electroencephalogram fine-grained emotion recognition system.The present application can effectively identify fine-grained emotion, improve the accuracy of emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to a method and system for fine-grained emotion recognition of electroencephalogram signals. BACKGROUND

[0002] The importance of emotion to human physical and mental health cannot be ignored. With the continuous development of affective computing technology, people have begun to pay attention to more fine-grained emotion recognition, such as the recognition of specific emotion types such as joy, anger, fear, and sadness. Electroencephalogram (EEG) technology, as a non-invasive, fast, cost-effective and user-friendly emotion recognition tool, can objectively and accurately map the individual's internal emotional state by measuring the voltage fluctuations caused by the flow of ion current within the brain neurons.

[0003] However, current EEG emotion recognition only stays between a few simple emotions. Research on fine-grained emotion recognition can promote the development of affective computing, improve the accuracy and efficiency of fine-grained emotion recognition, and provide stronger support for the research and application of affective brain-computer interfaces.

[0004] Existing technology for emotion recognition based on electroencephalogram signals involves feature extraction in time domain and frequency domain on pre-processed electroencephalogram signals, followed by classification and recognition using machine learning or deep learning models. Feature extraction is a key factor in the entire pattern recognition process. However, electroencephalogram signals are high-dimensional and complex time series signals that change dynamically, and existing technology does not fully consider the dynamic interactions between brain regions. How to efficiently obtain high-dimensional nonlinear dynamic features of electroencephalogram signals and effectively recognize them is a difficult problem that needs to be solved in fine-grained emotion recognition of electroencephalogram signals.

[0005] The shortcomings of existing technology include at least:

[0006] (1) Limited types of emotion recognition

[0007] In the field of electroencephalogram emotion recognition, although there are commonly used datasets such as DEAP, DREAMER, and SEED series, these datasets still need to be improved in terms of the number of subjects, the types of emotions covered, and the diversity of scenarios, making it difficult to fully evaluate the stability and generalization ability of models in different real-world scenarios. Especially in the application of brain-computer interfaces, the requirements of real-time performance and generalization further increase the complexity and challenge of the research.

[0008] (2) Ignoring high-order interaction information between edges

[0009] Feature extraction is a key factor in the entire pattern recognition process. Features used for emotion recognition are mostly from time domain and frequency domain. Electroencephalogram signals are high-dimensional and complex time series signals that change dynamically. Existing technology often uses node-based functional connectivity, which may ignore high-order interaction information between edges and does not fully consider the dynamic interactions between brain regions.

[0010] (3) Ignoring information complementarity

[0011] The prior art generally focuses on classification recognition using extracted features, and ignores the time dependence contained in the original time sequence, which is crucial for revealing the dynamic change information of the electroencephalogram SUMMARY

[0012] Therefore, it is necessary to provide an electroencephalogram fine-grained emotion recognition method and system.

[0013] The application provides an electroencephalogram fine-grained emotion recognition method, which comprises the following steps: S1, selecting a subject electroencephalogram from a FACED data set, and performing frequency division and segmentation on the preprocessed subject electroencephalogram; S2, using a common fluctuation edge center analysis method based on all the segmented and frequency-divided subject electroencephalograms, constructing a common fluctuation edge time sequence ETS and a common fluctuation edge functional connection matrix EFC; S3, constructing a CoEdge-STNet model according to the high-order space-time characteristics of the common fluctuation edge time sequence ETS and the common fluctuation edge functional connection matrix EFC, wherein the CoEdge-STNet model is a fine-grained emotion classification model; S4, using the CoEdge-STNet model to perform fine-grained emotion recognition under different frequency bands of the subject; and S5, using the CoEdge-STNet model to perform fine-grained emotion recognition under different frequency bands of different subjects.

[0014] Preferably, the step of selecting a subject electroencephalogram from a FACED data set comprises: randomly selecting n subjects, or selecting fixed continuous n subjects according to subject numbers, wherein n is any natural number not more than the total number of subjects in the FACED data set.

[0015] Preferably, the step S1 comprises:

[0016] Each selected subject participates in 28 independent trials and collects 32-channel electroencephalogram signals with a time length of 30s; band-pass filtering is performed to obtain 1-4Hz Delta, 4-8Hz Theta, 8-14Hz Alpha, 14-30Hz Beta and 30-47Hz Gamma frequency band electroencephalogram signals; and the preprocessed electroencephalogram signals are segmented into non-overlapping windows with a length of 1s.

[0017] Preferably, the step S3 comprises:

[0018] The CoEdge-STNet model comprises two spatiotemporal collaborative convolution modules ST-Conv Black1, ST-Conv Black2 and a fully connected layer fc, wherein each spatiotemporal collaborative convolution module extracts spatiotemporal features through the combination of two time convolution layers and one graph convolution layer.

[0019] Preferably, the step S3 comprises:

[0020] Step S31, first spatiotemporal collaborative convolution module ST-Conv Black1: the input shape is (batch_size, num_nodes, num_features);

[0021] Step S32, the output of the first spatiotemporal collaborative convolution module ST-Conv Black1 is (batch_size, num_nodes, 64), wherein 64 represents the feature dimension of each node;

[0022] Step S33, second spatiotemporal collaborative convolution module ST-Conv Black2: the input shape is (batch_size, num_nodes, 64);

[0023] Step S34, the output of the second spatiotemporal collaborative convolution module ST-Conv Black2 is (32, 496, 128);

[0024] Step S35, flattening and Dropout: after passing through the two spatiotemporal collaborative convolution modules, a feature map with a shape of (batch_size, num_nodes, 128) is obtained, (batch_size, num_nodes, 128) is flattened into (batch_size, num_nodes*128), a Dropout layer is applied, and part of the neurons are randomly discarded to prevent overfitting;

[0025] Step S36, the data is processed through the fully connected layer fc, the flattened features are input into the fully connected layer, and the final output is obtained.

[0026] Preferably, the step S31 comprises:

[0027] a. First time convolution layer Convld-1: nn.Conv1d(in_channels, spatial_channels, kernel_size=3, padding=1) performs one-dimensional convolution, the convolution kernel size is 3, the step is default 1, and the padding is 1, which ensures that the length of the input and output is consistent;

[0028] Convolution operation: x = self.Convld-1(x), the input data of shape (batch_size, num_nodes, num_features) is transformed into (batch_size, num_nodes, 32) through convolution, and the number of features of each node is mapped to 32;

[0029] b. ReLU activation: x = F.relu(x), apply ReLU activation function to the output after convolution to enhance nonlinearity;

[0030] c. Transpose dimension: x = x.transpose(1, 2), convert the shape from (batch_size, num_nodes, 32) to (batch_size, 32, num_nodes) to prepare for graph convolution;

[0031] d. Graph convolution: x = self.spatial(x, efc);

[0032] e. ReLU activation: x = F.relu(x), apply ReLU activation function to the output after graph convolution again;

[0033] f. In order to send the data into the next time convolution layer Convld-2, transpose it again, transpose dimension: x = x.transpose(1, 2), transpose back to (batch_size, num_nodes, 32) again;

[0034] g. Second time convolution layer (Convld-2): the input shape is (batch_size, num_nodes, in_channels), where: in_channels is 32, and the output shape is (batch_size, num_nodes, TimeTrack_channels), TimeTrack_channels is 64.

[0035] Preferably, the step S33 comprises:

[0036] a. First time convolution layer Convld-1, the input shape is (batch_size, num_nodes, spatial_channels), and the convolution operation is performed along the node dimension, and the output shape is (batch_size, num_nodes, spatial_channels);

[0037] b. Apply ReLU activation function to the result after convolution;

[0038] c. Before applying graph convolution, transpose the data to make it conform to the input format of graph convolution;

[0039] d. Data enters the graph convolution layer GraphConvolution: the graph convolution layer propagates the features of each node and combines them with the features of the constructed functional connection matrix EFC of the co-fluctuation edge;

[0040] e. Again, apply the ReLU activation function to the result after graph convolution;

[0041] f. Transpose the data back to send it to the next temporal convolution layer Convld-2;

[0042] g. The second temporal convolution layer Convld-2 convolves along the node dimension, and the output shape is (batch_size, num_nodes, temporal_channels).

[0043] Preferably, the step S4 comprises:

[0044] Intra-subject, for all 28 trials representing 9 different emotion categories of each subject, three-fold cross-validation is performed, each fold including complete trials representing nine emotion categories, ensuring balanced emotion category and sample size allocation within each fold; The experimental results of each subject are reported as the average results of three-fold cross-validation.

[0045] Preferably, the step S5 comprises:

[0046] Cross-subject, for all 10 subjects, leave-one-out method is used for emotion category recognition, each time selecting all trial data of one subject as the validation set, and all trial data of the remaining subjects as the training set, the step is repeated 10 times, and the average result of 10 times is finally reported.

[0047] The application also provides an electroencephalogram fine-grained emotion recognition system, which comprises a frequency segmentation module, an analysis module, a model construction module, an intra-subject fine-grained emotion recognition module, and a cross-subject fine-grained emotion recognition module, wherein:

[0048] The frequency segmentation module is used to select subject electroencephalogram signals from the FACED data set, and to segment the preprocessed subject electroencephalogram signals by frequency;

[0049] The analysis module is used to construct the co-fluctuation edge time series ETS and the co-fluctuation edge functional connection matrix EFC using the edge center analysis method based on co-fluctuation for all segmented subject electroencephalogram signals;

[0050] The model construction module is configured to construct a CoEdge-STNet model according to high-order spatiotemporal characteristics of the Co-fluctuation edge time sequence ETS and the Co-fluctuation edge functional connection matrix EFC, and the CoEdge-STNet model is a fine-grained emotion classification model.

[0051] The intra-subject fine-grained emotion recognition module is configured to use the CoEdge-STNet model to perform fine-grained emotion recognition under different frequency bands of different subjects.

[0052] The cross-subject fine-grained emotion recognition module is configured to use the CoEdge-STNet model to perform fine-grained emotion recognition under different frequency bands of different subjects.

[0053] The present application can effectively recognize fine-grained emotions, can start from the edge center perspective, form a more rich feature space by combining the Co-fluctuation edge functional connection matrix (EFC) and the Co-fluctuation edge time sequence (ETS), provide a new idea for EEG emotion recognition feature extraction, the model can analyze the spatiotemporal characteristics of brain activity under various emotional states, provide a new perspective and method for brain function research, and help reveal the working principle and mechanism of the brain under complex emotional state. The beneficial effects of the present application include:

[0054] (1) Overcome the information loss caused by single feature

[0055] The present application combines the Co-fluctuation edge time sequence (ETS) constructed using the edge center perspective with the high-dimensional Co-fluctuation edge functional connection matrix (EFC) feature, realizes information complementation and dynamic analysis, and thus provides a more comprehensive and systematic perspective to reveal the mechanism of brain perception of complex emotions.

[0056] (2) Improve the accuracy and generalization of fine-grained emotion recognition

[0057] The present application can ensure that the model performs well in different populations and situations. In particular, it can improve the low recognition accuracy of the prior art.

[0058] (3) Promote the development of the field of affective computing

[0059] The present application can enable intelligent machines to understand human emotions and provide stronger support for the research and application of emotional brain-computer interfaces. In clinical practice, it can help treat some serious mental illnesses, such as refractory depression. In daily life, it can help ordinary people relieve anxiety and improve mood. In the game and entertainment industry, it can adjust the game difficulty or music recommendation according to the player's emotions, providing a more personalized experience. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1A flowchart of the electroencephalogram fine-grained emotion recognition method of the present application;

[0061] Figure 2 A technical route schematic diagram provided by the embodiment of the present application;

[0062] Figure 3 An intra-subject schematic diagram of the classification report of the FACED dataset-10 subjects-5 frequency bands-nine classification of fine-grained emotions provided by the embodiment of the present application;

[0063] Figure 4 A cross-subject schematic diagram of the classification report of the FACED dataset-10 subjects-5 frequency bands-nine classification of fine-grained emotions provided by the embodiment of the present application;

[0064] Figure 5 A hardware architecture diagram of the electroencephalogram fine-grained emotion recognition system of the present application. DETAILED DESCRIPTION

[0065] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Referring to Figure 1 , it is an operation flowchart of the preferred embodiment of the electroencephalogram fine-grained emotion recognition method of the present application. Please refer to Figure 2 :

[0067] Step S1, selecting subject electroencephalogram signals from the FACED (Finer-grained Affective Computing EEG Dataset) dataset, and performing frequency division and segmentation on the preprocessed subject electroencephalogram signals. Specifically:

[0068] The selected subject electroencephalogram signals from the FACED dataset can be randomly selected n subjects, or fixed continuous n subjects according to the subject number, and n is any natural number not more than the total number of subjects in the FACED dataset, such as n can be 10, 20 or 30.

[0069] In this embodiment, the first 10 subjects are selected from the FACED dataset, wherein each selected subject participated in 28 independent trials and collected 32-channel electroencephalogram signals with a duration of 30s; the band-pass filter is used to obtain the Delta, Theta, Alpha, Beta and Gamma frequency band electroencephalogram signals, which are 1-4Hz, 4-8Hz, 8-14Hz, 14-30Hz and 30-47Hz respectively; the preprocessed electroencephalogram signals are segmented into equal length segments with a length of 1s as a non-overlapping window. Among them, the 28 trials represent 9 different emotion categories, and the label of the emotion category is a digital label of 0-8, and the digital label 0-8 represents nine emotion categories to be recognized.

[0070] Step S2, for all subjects' EEG signals after frequency segmentation, use the edge center analysis method based on common fluctuation to construct the common fluctuation edge time series (ETS), the common fluctuation edge functional connection matrix (EFC). Specifically:

[0071] For all subjects' EEG signals after frequency segmentation, use the edge center analysis method to construct the common fluctuation edge time series ETS and the common fluctuation edge functional connection matrix EFC, obtain the edge time series matrix of each trial sample with a dimension of 496*250*30 (where: 496 represents the number of edges, 250 represents the sampling points, and 30 represents the number of segments), and the edge functional connection matrix of each trial sample with a dimension of 496*496*30 (where: 496 represents the number of edges, and 30 represents the number of segments).

[0072] Step S3, according to the high-order spatio-temporal characteristics of the common fluctuation edge time series (ETS) and the common fluctuation edge functional connection matrix (EFC), construct the CoEdge-STNet (Co-fluctuated Edge Spatio-Temporal Network) model. The CoEdge-STNet model is a fine-grained emotion classification model. Specifically:

[0073] The complete CoEdge-STNet model includes: 2 spatio-temporal collaborative convolution modules (ST-Conv Black) and a fully connected layer (fc). Wherein, each spatio-temporal collaborative convolution module extracts spatio-temporal features through the combination of two time convolution layers (Convld-1 and Convld-2) and a graph convolution layer (Graph Convolution).

[0074] The CoEdge-STNet model structure is:

[0075] • The first spatio-temporal collaborative convolution module (ST-Conv Black1): extracts spatio-temporal features through time convolution layer, graph convolution layer and activation function:

[0076] • Convld-1

[0077] • Graph Convolution

[0078] • Convld-2

[0079] • The second spatio-temporal collaborative convolution module (ST-Conv Black2): further extracts more complex spatio-temporal features. The specific structure is the same as ST-Conv Black1.

[0080] • Flatten and Dropout: Flatten the features and apply Dropout to prevent overfitting.

[0081] • Fully connected layer (fc): Map the extracted spatio-temporal features to the output class space.

[0082] The two ST-Conv Black above are processed in sequence in the network, and the output of ST-Conv Black1 is used as the input of ST-Conv Black2.

[0083] Temporal convolution layers are used to extract edge temporal sequence features; graph convolution layers are used to process the complexity of edge temporal sequences, while effectively capturing the spatial dependence contained in the functional connection matrix, thereby further enriching the feature representation.

[0084] The CoEdge-STNet model introduces ReLU activation functions after multiple convolution layers to improve expression ability and deep learning ability. The above nonlinear activation mechanism can significantly enhance the model's ability to recognize complex data patterns, enabling the model to learn higher-level features. After the features output by the temporal convolution layer are flattened, the final prediction result is generated through the fully connected layer. Dropout technology is applied before the fully connected layer. Dropout, as a regularization method, can randomly discard the output of a portion of neurons, effectively reducing overfitting and improving the model's generalization ability.

[0085] The CoEdge-STNet model is normalized before being fed into the model. The following are the parameter settings for the CoEdge-STNet model:

[0086] Parameter settings:

[0087] batch_size = 32, lr = 0.0001, weight_decay = 0.01, num_classes = 9

[0088] self.ST-Conv Black1 = ST-Conv Black(num_features, 32, 64, num_nodes)

[0089] self.ST-Conv Black2 = ST-Conv Black(64, 32, 128, num_nodes)

[0090] Model initialization settings:

[0091] model = NeuralNetwork(num_nodes, num_features, num_classes, dropout_rate = 0.5)

[0092] For the used FACED dataset, the shape of the input data is (32, 496, 250), where: 32 is the batch size, 496 is the number of co-fluctuation edge time series (num_nodes) constructed using edge center perspective, and 250 is the number of features for each co-fluctuation edge time series (num_features). The edge functional connection network size constructed by the co-fluctuation edge time series is 496*496.

[0093] Step S31, the first space-time collaborative convolution module ST-Conv Black1: the input shape is (batch_size, num_nodes, num_features), which is (32, 496, 250) in this embodiment, including the following processing steps:

[0094] a. The first time convolution layer (Convld-1): nn.Conv1d(in_channels, spatial_channels, kernel_size=3, padding=1) performs one-dimensional convolution, the convolution kernel size is 3, the step is default 1, and the padding is 1, which ensures that the length of the input and output is consistent.

[0095] Convolution operation: x = self.Convld-1(x), the input data of (batch_size, num_nodes, num_features) is converted to (batch_size, num_nodes, 32) through convolution. The number of features of each node is mapped to 32.

[0096] b. ReLU activation: x = F.relu(x), apply the ReLU activation function to the output after convolution to enhance nonlinearity.

[0097] c. Transpose dimension: x = x.transpose(1, 2), convert the shape from (batch_size, num_nodes, 32) to (batch_size, 32, num_nodes) to prepare for graph convolution.

[0098] d. Graph convolution (GraphConvolution): x = self.spatial(x, efc).

[0099] The first step of the graph convolution operation: support = torch.matmul(x, self.weight), this step is matrix multiplication, which transforms the input features through the weight matrix weight to obtain the linearly transformed features support.

[0100] Figure convolution operation second step: graph convolution spreads features through the adjacency matrix: output = torch.matmul(efc, support), that is, the matrix multiplication of the linearly transformed features support of the co-volatility edge function connection matrix (EFC).

[0101] Figure convolution operation third step: return output + self.bias, that is, add the bias term.

[0102] e. ReLU activation: x = F.relu(x), and then apply the ReLU activation function to the output of the graph convolution.

[0103] f. In order to send the data to the next time convolution layer t2, transpose it again, transpose the dimensions: x = x.transpose(1, 2), and transpose back to (batch_size, num_nodes, 32) again.

[0104] g. Second time convolution layer (Convld-2):

[0105] self.Convld-2 = nn.Conv1d(spatial_channels, TimeTrack_channels, kernel_size = 3, padding = 1), the second time convolution layer (Convld-2): the shape of the input is (batch_size, num_nodes, in_channels), where: in_channels is 32, the shape of the output is (batch_size, num_nodes, TimeTrack_channels), TimeTrack_channels is 64.

[0106] Step S32, the output of the first space-time collaborative convolution module ST-Conv Black1 is (batch_size, num_nodes, 64), where the feature dimension of each node is 64.

[0107] Step S33, the second space-time collaborative convolution module (ST-Conv Black2):

[0108] ST-Conv Black2 is the second space-time collaborative convolution module, the input shape is (batch_size, num_nodes, 64), including the following processing steps:

[0109] (Convld-1): nn.Conv1d(spatial_channels, Convld-2) channels, kernel_size=3, padding=1) spatial_channels = 32, TimeTrack_channels = 128.

[0110] a. First temporal convolutional layer (Convld-1): Convld-1 is a 1D convolutional layer with an input shape of (batch_size, num_nodes, spatial_channels), i.e., (32, 496, 32). The convolution operation is performed along the node dimension (i.e., convolving the features for each node), and the output shape is (batch_size, num_nodes, spatial_channels), i.e., (32, 496, 64).

[0111] b. Apply ReLU activation function to the convolved result: x.shape is still (32, 496, 64) because the ReLU activation function does not change the shape of the tensor.

[0112] c. Before applying the graph convolution, transpose the data to conform to the input format of the graph convolution: x.shape is transposed from (32, 496, 64) to (32, 64, 496).

[0113] d. Data enters the graph convolution layer (GraphConvolution):

[0114] The graph convolution layer propagates the features for each node and combines them with the constructed co-fluctuation edge functional connection matrix EFC. The output shape is still (32, 64, 496).

[0115] e. Apply ReLU activation function to the graph-convolved result again: x.shape is still (32, 64, 496).

[0116] f. Transpose the data back to send it to the next temporal convolutional layer: x.shape is transposed from (32, 64, 496) to (32, 496, 64), restoring the (batch_size, num_nodes, spatial_channels).

[0117] g. Second temporal convolutional layer (Convld-2):

[0118] Convld-2 is a 1D convolution layer, Convld-2 convolves along the node dimension, and the output shape is (batch_size, num_nodes, temporal_channels), i.e. (32, 496, 128).

[0119] Step S34, the output shape of the second spatio-temporal collaborative convolution module ST-Conv Black2 is (32, 496, 128).

[0120] Step S35, flattening and Dropout:

[0121] After passing through two spatio-temporal collaborative convolution modules, a feature map with a shape of (batch_size, num_nodes, 128) is obtained.

[0122] Next, flattening: x = x.view(x.size(0), -1), (batch_size, num_nodes, 128) is flattened into (batch_size, num_nodes*128), i.e. the num_nodes*128 dimensions are merged in order to pass to the fully connected layer. For the current embodiment, the shape after flattening is (batch_size, 496*128).

[0123] Dropout: x = self.dropout(x), apply the Dropout layer, randomly discard part of the neurons to prevent overfitting.

[0124] Step S36, fully connected layer (fc):

[0125] The data is processed by the fully connected layer fc, full connection: x = self.fc(x), the flattened features are input into the fully connected layer to obtain the final output. The shape of the fully connected layer is (batch_size, 9), i.e. each sample is classified into one of the 9 emotion categories. The weight matrix shape of the fc layer here is (num_nodes*128, num_classes), i.e. (496*128, 9). The flattened feature vector is multiplied by the weight matrix and a bias is added to obtain a 9-dimensional output vector, each dimension corresponding to the score of a class.

[0126] Step S4, in different frequency bands, the CoEdge-STNet model is used for fine-grained emotion recognition. That is, the present embodiment performs within-subject emotion recognition in different frequency bands (a total of 5). Each frequency band needs to be performed, and finally, 5 fine-grained emotion recognition results are reported for each frequency band. Please refer to Figure 3 , in particular:

[0127] In this embodiment, for each participant, three-fold cross-validation was performed on all 28 trials representing 9 different emotion categories. Each fold included complete trials representing all nine emotion categories, ensuring a balance between emotion categories and sample size within each fold. The experimental results for each participant were reported as the average of the three-fold cross-validation results.

[0128] Step S5: Fine-grained emotion recognition is performed using the CoEdge-STNet model across different frequency bands for each participant. That is, in this embodiment, cross-participant emotion recognition is performed in five different frequency bands. This process is required for each frequency band, and finally, one fine-grained emotion recognition result is reported for each of the five frequency bands. Please refer to the following documentation: Figure 4 Specifically:

[0129] In this embodiment, across all 10 participants, the leave-one-out method is used for emotion category identification. Each time, all trial data of one participant is selected as the validation set, and all trial data of the remaining participants are selected as the training set. This step is repeated 10 times, and finally the average result of the 10 times is reported.

[0130] See Figure 5 The diagram shown is a hardware architecture diagram of the fine-grained emotion recognition system 10 based on EEG signals of the present invention. Please refer to it as well. Figure 2 The system includes: a frequency segmentation module 101, an analysis module 102, a model building module 103, an intra-subject fine-grained emotion recognition module 104, and a cross-subject fine-grained emotion recognition module 105. Among them:

[0131] The frequency division and segmentation module 101 is used to select subject EEG signals from the FACED dataset and perform frequency division and segmentation on the preprocessed subject EEG signals. Specifically:

[0132] The selection of subject EEG signals from the FACED dataset can be done by randomly selecting n subjects or by selecting a fixed number of consecutive subjects according to their subject numbers. n is any natural number not exceeding the total number of subjects in the FACED dataset, such as 10, 20, or 30.

[0133] In this embodiment, the first 10 participants were selected from the FACED dataset. Each participant participated in 28 independent trials, and 32 channels of EEG signals were collected for a duration of 30 seconds. Bandpass filtering was used to obtain Delta, Theta, Alpha, Beta, and Gamma frequency bands of EEG signals, corresponding to 1–4 Hz, 4–8 Hz, 8–14 Hz, 14–30 Hz, and 30–47 Hz, respectively. The preprocessed EEG signals were segmented into equal-length segments with a 1-second non-overlapping window. The 28 trials represented 9 different emotion categories, labeled with numeric tags from 0 to 8, representing the nine emotion categories to be identified.

[0134] The analysis module 102 is used to construct the converging edge time series (ETS) and the converging edge functional connectivity matrix (EFC) using the converging edge-center analysis method based on converging edge-center analysis on all subject EEG signals after frequency segmentation. Specifically:

[0135] The analysis module 102 uses the edge-center analysis method to construct the co-wave edge time series (ETS) and the co-wave edge functional connectivity matrix (EFC) for all the subject EEG signals after frequency division and segmentation. This results in an edge time series matrix with a dimension of 496*250*30 for each trial sample (where 496 represents the number of edges, 250 represents the number of sampling points, and 30 represents the number of segments) and an edge functional connectivity matrix with a dimension of 496*496*30 for each trial sample (where 496 represents the number of edges and 30 represents the number of segments).

[0136] The model building module 103 is used to construct the CoEdge-STNet model based on the high-order spatiotemporal characteristics of the co-wave edge time series (ETS) and the co-wave edge functional connectivity matrix (EFC). The CoEdge-STNet model is a fine-grained sentiment classification model. Specifically:

[0137] The complete CoEdge-STNet model consists of two spatiotemporal co-convolutional modules (ST-Conv Black) and a fully connected layer (fc). Each spatiotemporal co-convolutional module extracts spatiotemporal features through a combination of two temporal convolutional layers (Convld-1 and Convld-2) and a graph convolutional layer.

[0138] The CoEdge-STNet model structure is as follows:

[0139] • First spatiotemporal co-convolutional module (ST-Conv Black1): Extracts spatiotemporal features through temporal convolutional layers, graph convolutional layers, and activation functions.

[0140] Convld-1

[0141] ·Graph Convolution

[0142] Convld-2

[0143] • Second spatiotemporal co-convolutional module (ST-Conv Black2): Further extracts more complex spatiotemporal features. Its specific structure is the same as ST-Conv Black1.

[0144] • Flattening and Dropout: Flatten the features and apply Dropout to prevent overfitting.

[0145] • Fully connected layer (fc): Maps the extracted spatiotemporal features to the output class space.

[0146] The two ST-Conv Blacks are processed sequentially in the network, with the output of ST-Conv Black1 serving as the input of ST-Conv Black2.

[0147] Temporal convolutional layers are used to extract edge time series features; graph convolutional layers are used to handle the complexity of edge time series and effectively capture the spatial dependencies contained in the functional connectivity matrix, thereby further enriching the feature representation.

[0148] The CoEdge-STNet model introduces ReLU activation functions after multiple convolutional layers to enhance expressive power and deep learning capabilities. This non-linear activation mechanism significantly improves the model's ability to recognize complex data patterns, enabling it to learn higher-level features. The features output from the temporal convolutional layers are flattened and then used to generate the final prediction results through fully connected layers. Dropout is applied before the fully connected layers. As a regularization technique, Dropout randomly discards a portion of the neuron's output, effectively reducing overfitting and improving the model's generalization ability.

[0149] Before feeding the data into the model, the co-wave edge time series ETS is normalized. The following are the parameter settings for the CoEdge-STNet model:

[0150] Parameter settings:

[0151] batch_size=32, lr=0.0001, weight_decay=0.01, num_classes=9

[0152] self.ST-Conv Black1=ST-Conv Black(num_features,32,64,num_nodes)

[0153] self.ST-Conv Black2=ST-Conv Black(64,32,128,num_nodes)

[0154] Model initialization settings:

[0155] model=NeuralNetwork(num_nodes,num_features,num_classes,dropout_rate=0.5)

[0156] For the FACED dataset used, the shape of the input data is (32, 496, 250), where 32 is the batch size, 496 is the number of co-wave edge time series constructed using the edge-center perspective (num_nodes), and 250 is the number of features (num_features) for each co-wave edge time series. The size of the edge-feature network constructed from the co-wave edge time series is 496*496.

[0157] First, the first spatiotemporal co-convolutional module ST-Conv Black1 has an input shape of (batch_size, num_nodes, num_features), which in this embodiment is (32, 496, 250), and includes:

[0158] a. First-time convolutional layer (Convld-1): nn.Conv1d(in_channels,spatial_channels,kernel_size=3,padding=1) performs a one-dimensional convolution with a kernel size of 3, a stride of 1 by default, and padding of 1 to ensure that the lengths of the input and output are consistent.

[0159] Convolution operation: x = self.Convld-1(x), which transforms the input data (batch_size, num_nodes, num_features) into (batch_size, num_nodes, 32) through convolution. The number of features for each node is mapped to 32.

[0160] b. ReLU activation: x = F.relu(x), applies the ReLU activation function to the output after convolution to enhance nonlinearity.

[0161] c. Transpose dimension: x = x.transpose(1,2), which transforms the shape from (batch_size, num_nodes, 32) to (batch_size, 32, num_nodes), preparing for graph convolution.

[0162] d. Graph Convolution: x = self.spatial(x, efc).

[0163] The first step in graph convolution is: support = torch.matmul(x, self.weight). This is a matrix multiplication, which transforms the input features through the weight matrix weight to obtain the linearly transformed features support.

[0164] The second step of graph convolution operation: Graph convolution propagates features through the adjacency matrix: output = torch.matmul(efc, support), which is to perform matrix multiplication on the feature support after linear transformation of the concentric edge functional connectivity matrix (EFC).

[0165] The third step of graph convolution operation: return output + self.bias, that is, add the bias term.

[0166] e.ReLU activation: x = F.relu(x), then apply the ReLU activation function to the output after graph convolution.

[0167] f. In order to feed the data into the next temporal convolutional layer t2, transpose it again with the transpose dimension: x = x.transpose(1,2), and then transpose it back to (batch_size, num_nodes, 32).

[0168] g. Second temporal convolutional layer (Convld-2):

[0169] self.Convld-2 = nn.Conv1d(spatial_channels,TimeTrack_channels,kernel_size=3,padding=1), the second temporal convolutional layer (Convld-2): the input shape is (batch_size,num_nodes,in_channels), where in_channels is 32, and the output shape is (batch_size,num_nodes,TimeTrack_channels), where TimeTrack_channels is 64.

[0170] Secondly, the output of the first spatiotemporal co-convolutional module ST-Conv Black1 is (batch_size, num_nodes, 64), where the feature dimension of each node is 64.

[0171] Next, the second spatiotemporal co-convolution module (ST-Conv Black2):

[0172] ST-Conv Black2 is the second spatiotemporal co-convolutional module, with an input shape of (batch_size, num_nodes, 64), including:

[0173] (Convld-2): nn.Conv1d(spatial_channels, Convld-_channels, kernel_size=3, padding=1) spatial_channels=32, TimeTrack_channels=128.

[0174] a. First-time convolutional layer (Convld-1): Convld-1 is a 1D convolutional layer with an input shape of (batch_size, num_nodes, spatial_channels), i.e. (32, 496, 32). This convolution operation performs convolution along the node dimension (i.e., convolution on the features of each node), and its output shape is (batch_size, num_nodes, spatial_channels), i.e. (32, 496, 64).

[0175] b. Apply the ReLU activation function to the result after convolution: x.shape remains (32,496,64) because the ReLU activation function does not change the shape of the tensor.

[0176] c. Before applying graph convolution, transpose the data to conform to the input format of graph convolution: x.shape is transposed from (32,496,64) to (32,64,496).

[0177] d. Data enters the graph convolution layer:

[0178] The graph convolutional layer propagates the features of each node and combines them with the features of the constructed concentric wave edge functional connectivity matrix EFC. The shape of the output is still (32, 64, 496).

[0179] e. Apply the ReLU activation function again to the result of the graph convolution: x.shape remains (32,64,496).

[0180] f. Transpose the data back so that it can be fed into the next temporal convolutional layer. To do this, x.shape is transposed from (32,64,496) to (32,496,64) and restored to (batch_size,num_nodes,spatial_channels).

[0181] g. Second temporal convolutional layer (Convld-2):

[0182] Convld-2 is a 1D convolutional layer. Convld-2 performs convolution along the node dimension, and the output shape is (batch_size, num_nodes, temporal_channels), which is (32, 496, 128).

[0183] Subsequently, the output shape of the second spatiotemporal co-convolution module ST-Conv Black2 is (32,496,128).

[0184] Then, flatten and dropout:

[0185] After passing through two spatiotemporal co-convolutional modules, a feature map with shape (batch_size, num_nodes, 128) is obtained.

[0186] Flattening: x = x.view(x.size(0), -1), flattens (batch_size, num_nodes, 128) into (batch_size, num_nodes * 128), that is, merging the num_nodes * 128 dimension so as to pass it to the fully connected layer. In the current embodiment, the flattened shape is (batch_size, 496 * 128).

[0187] Dropout: x = self.dropout(x) applies a Dropout layer to randomly drop some neurons to prevent overfitting.

[0188] Finally, the fully connected layer (fc):

[0189] The data is processed through a fully connected layer (fc). The fully connected layer is defined as: `x = self.fc(x)`. The flattened features are input into the fully connected layer to obtain the final output. The shape of the fully connected layer is (batch_size, 9), meaning each sample is classified into one of nine sentiment categories. The weight matrix of the fc layer has a shape of (num_nodes*128, num_classes), which is (496*128, 9). Multiplying the flattened feature vector by the weight matrix and adding a bias results in a 9-dimensional output vector, with each dimension corresponding to the score of a category.

[0190] The in-subject fine-grained emotion recognition module 104 is used to perform fine-grained emotion recognition using the CoEdge-STNet model at different frequency bands within the subject. That is, in this embodiment, in-subject emotion recognition is performed at different frequency bands (a total of 5), and this is required for each frequency band. Finally, one fine-grained emotion recognition result is reported for each of the 5 frequency bands. Please refer to [further details needed]. Figure 3Specifically:

[0191] In this embodiment, for each participant, three-fold cross-validation was performed on all 28 trials representing 9 different emotion categories. Each fold included complete trials representing all nine emotion categories, ensuring a balance between emotion categories and sample size within each fold. The experimental results for each participant were reported as the average of the three-fold cross-validation results.

[0192] The cross-subject fine-grained emotion recognition module 105 is used to perform fine-grained emotion recognition using the CoEdge-STNet model across different frequency bands for different subjects. That is, in this embodiment, cross-subject emotion recognition is performed separately in different frequency bands (a total of 5), and this process is required for each frequency band. Finally, one fine-grained emotion recognition result is reported for each of the 5 frequency bands. Please refer to [further details needed]. Figure 4 Specifically:

[0193] In this embodiment, across all 10 participants, the leave-one-out method is used for emotion category identification. Each time, all trial data of one participant is selected as the validation set, and all trial data of the remaining participants are selected as the training set. This step is repeated 10 times, and finally the average result of the 10 times is reported.

[0194] This invention combines the co-wave edge time series (ETS) constructed using the edge-center perspective with the features of the high-dimensional co-wave edge functional connectivity matrix (EFC), achieving information complementarity and dynamic analysis. This enables the CoEdge-STNet model to capture complex emotional states that are difficult to discover through time-domain or frequency-domain features alone, thus providing a more comprehensive and systematic perspective to reveal the intrinsic connections of emotional states.

[0195] Converging edge functional connectivity matrix (EFC) and converging edge time series (ETS) have different feature representations, and combining the two can form a richer feature space. Converging edge time series (ETS) can reflect the dynamic changes of EEG signals, which helps to understand the dynamic process of brain activity in emotional states; combined with converging edge functional connectivity matrix (EFC), the spatiotemporal characteristics of brain activity in emotional states can be analyzed in greater depth.

[0196] This invention can advance the field of affective computing, enabling intelligent machines to understand human emotions and providing stronger support for the research and application of affective brain-computer interfaces. Clinically, it can help treat some serious mental illnesses, such as treatment-resistant depression. In daily life, it can help ordinary people alleviate anxiety and improve mood. Furthermore, affective brain-computer interfaces also have enormous commercial value, revolutionizing the gaming and entertainment industries by adjusting game difficulty or music recommendations based on the player's emotions, providing a more personalized experience.

[0197] Although the present invention has been described with reference to the present preferred embodiments, those skilled in the art should understand that the above preferred embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for fine-grained emotion recognition based on electroencephalogram (EEG) signals, characterized in that, The method includes the following steps: S1, Select the subject's EEG signal from the FACED dataset and perform frequency segmentation on the preprocessed subject's EEG signal; S2. For all the EEG signals of the subjects after frequency division and segmentation, the edge-center analysis method based on the co-wave is used to construct the co-wave edge time series ETS and the co-wave edge functional connectivity matrix EFC. S3. Based on the high-order spatiotemporal characteristics of the co-wave edge time series ETS and the co-wave edge functional connectivity matrix EFC, a CoEdge-STNet model is constructed. The CoEdge-STNet model is a fine-grained emotion classification model. S4. Fine-grained emotion recognition was performed using the CoEdge-STNet model at different frequency bands within the subjects. S5, using the CoEdge-STNet model for fine-grained emotion recognition across different frequency bands of participants; where: Step S3 includes: The CoEdge-STNet model includes two spatiotemporal co-convolutional modules, ST-Conv Black1 and ST-Conv Black2, and a fully connected layer fc. Each spatiotemporal co-convolutional module extracts spatiotemporal features through a combination of two temporal convolutional layers and a graph convolutional layer.

2. The method as described in claim 1, characterized in that, The selection of subject EEG signals from the FACED dataset includes: randomly selecting n subjects, or selecting a fixed number of consecutive n subjects according to their subject numbers, where n is any natural number not exceeding the total number of subjects in the FACED dataset.

3. The method as described in claim 2, characterized in that, Step S1 includes: For each selected subject, 28 independent trials were conducted, and 32 channels of EEG signals were collected for a duration of 30 seconds. EEG signals in the Delta, Theta, Alpha, Beta, and Gamma bands of 1–4 Hz, 4–8 Hz, 8–14 Hz, 14–30 Hz, and 30–47 Hz were obtained through bandpass filtering. The preprocessed EEG signals were segmented into equal-length segments with a length of 1 second as a non-overlapping window.

4. The method as described in claim 3, characterized in that, Step S3 includes: Step S31, First spatiotemporal co-convolutional module ST-Conv Black1: Input shape is (batch_size, num_nodes, num_features); Step S32, the output of the first spatiotemporal co-convolutional module ST-Conv Black1 is (batch_size, num_nodes, 64), where 64 represents the feature dimension of each node; Step S33, the second spatiotemporal co-convolutional module ST-Conv Black2: input shape is (batch_size, num_nodes, 64); Step S34, the output of the second spatiotemporal co-convolution module ST-Conv Black2 is (32,496,128); Step S35, Flattening and Dropout: After passing through two spatiotemporal co-convolutional modules, a feature map with shape (batch_size, num_nodes, 128) is obtained. The (batch_size, num_nodes, 128) is flattened into (batch_size, num_nodes*128), and a Dropout layer is applied to randomly drop some neurons to prevent overfitting. In step S36, the data is processed through a fully connected layer fc, and the flattened features are input into the fully connected layer to obtain the final output.

5. The method as described in claim 4, characterized in that, Step S31 includes: a. First-time convolutional layer Convld-1: nn.Conv1d(in_channels,spatial_channels,kernel_size=3,padding=1) performs a one-dimensional convolution with a kernel size of 3, a default stride of 1, and padding of 1 to ensure that the lengths of the input and output are consistent. Convolution operation: x = self.Convld-1(x), which transforms the input data (batch_size, num_nodes, num_features) into (batch_size, num_nodes, 32) through convolution, and the number of features of each node is mapped to 32; b. ReLU activation: x = F.relu(x), applies the ReLU activation function to the output after convolution to enhance nonlinearity; c. Transpose dimension: x = x.transpose(1,2), which transforms the shape from (batch_size, num_nodes, 32) to (batch_size, 32, num_nodes), preparing for graph convolution; d. Graph Convolution: x = self.spatial(x, efc); e.ReLU activation: x = F.relu(x), then apply the ReLU activation function to the output after graph convolution; f. In order to feed the data into the next temporal convolutional layer Convld-2, transpose it again with the transpose dimension: x = x.transpose(1,2), and then transpose it back to (batch_size, num_nodes, 32); g. Second temporal convolutional layer (Convld-2): The input shape is (batch_size, num_nodes, in_channels), where in_channels is 32, and the output shape is (batch_size, num_nodes, TimeTrack_channels), where TimeTrack_channels is 64.

6. The method as described in claim 5, characterized in that, Step S33 includes: a. The first convolutional layer Convld-1 has an input shape of (batch_size, num_nodes, spatial_channels). This convolution operation performs convolution along the node dimension, and its output shape is (batch_size, num_nodes, spatial_channels). b. Apply the ReLU activation function to the convolution result; c. Before applying graph convolution, transpose the data to conform to the input format of graph convolution; d. Data enters the graph convolutional layer: The graph convolutional layer propagates the features of each node and combines them with the features of the constructed concentric edge functional connectivity matrix EFC; e. Apply the ReLU activation function again to the result of the graph convolution; f. Transpose the data back so that it can be fed into the next temporal convolutional layer, Convld-2; g. The second temporal convolutional layer, Convld-2, performs convolutions along the node dimension, and the output shape is (batch_size, num_nodes, temporal_channels).

7. The method as described in claim 6, characterized in that, Step S4 includes: Within the participants, three-fold cross-validation was performed on all 28 trials representing 9 different emotion categories for each participant. Each fold included complete trials representing all nine emotion categories to ensure a balance between the distribution of emotion categories and the number of samples within each fold. The experimental results for each participant were reported as the average of the three-fold cross-validation results.

8. The method as described in claim 7, characterized in that, Step S5 includes: Across all 10 participants, a leave-one-out method was used for emotion category identification. Each time, all trial data of one participant were selected as the validation set, and all trial data of the other participants were selected as the training set. This process was repeated 10 times, and the average result of the 10 trials was reported.

9. A fine-grained emotion recognition system for EEG signals utilizing the fine-grained emotion recognition method for EEG signals according to claim 1, characterized in that, The system includes a frequency segmentation module, an analysis module, a model building module, an intra-subject fine-grained emotion recognition module, and a cross-subject fine-grained emotion recognition module, among which: The frequency division and segmentation module is used to select the subject's EEG signal from the FACED dataset and perform frequency division and segmentation on the preprocessed subject's EEG signal; The analysis module is used to construct the co-wave edge time series (ETS) and the co-wave edge functional connectivity matrix (EFC) for all subject EEG signals after frequency segmentation and segmentation using the edge-center analysis method based on co-wave. The model building module is used to construct the CoEdge-STNet model based on the high-order spatiotemporal characteristics of the co-wave edge time series ETS and the co-wave edge functional connectivity matrix EFC. The CoEdge-STNet model is a fine-grained sentiment classification model. The in-subject fine-grained emotion recognition module is used to perform fine-grained emotion recognition using the CoEdge-STNet model at different frequency bands within the subject. The cross-subject fine-grained emotion recognition module is used to perform fine-grained emotion recognition using the CoEdge-STNet model across different frequency bands of subjects.

Images