Emotion state recognition method and device based on sparse dynamic graph convolutional neural network

Abstract

This application discloses a method and apparatus for emotion state recognition based on a sparse dynamic graph convolutional neural network, belonging to the field of electroencephalogram (EEG) signal processing technology. The method includes: acquiring the user's EEG data to be recognized; inputting the differential entropy features of multiple frequency bands corresponding to the EEG data to an emotion state recognition model, the emotion state recognition model including a feature extractor, a data augmentation module, and a downstream classifier, the feature extractor being constructed based on a sparse dynamic graph convolutional neural network; extracting features from the differential entropy features through the feature extractor and applying sparsity to obtain the first node features of the EEG data to be recognized; enhancing the first node features through the data augmentation module to generate second node features; classifying the second node features through the downstream classifier to obtain and output the user's emotion state. This method can quickly identify emotion states, extract sparse but effective information, and eliminate reliance on domain knowledge.

Description

Technical Field

[0001] This application belongs to the field of EEG signal processing technology, and in particular relates to a method and device for emotion state recognition based on sparse dynamic graph convolutional neural network. Background Technology

[0002] Electroencephalogram (EEG) is a non-invasive tool for recording brain neural activity. Due to its high temporal resolution and non-invasiveness, it is widely used in research on mental illnesses such as depression and anxiety related to mood states. By analyzing EEG signals from patients at rest or during tasks, patterns of brain dysfunction related to mood states can be revealed, such as abnormal activity in specific frequency bands like alpha and beta waves. EEG data is characterized by its multi-channel, multi-timepoint nature, high dimensionality, and unstructured nature.

[0003] Currently, EEG signals can be analyzed using feature engineering and traditional machine learning methods. However, feature engineering-based analysis of EEG signals using Fourier transform and Hilbert-Huang transform is time-consuming and relies on domain knowledge. Traditional machine learning methods such as support vector machines (SVM) and random forests have the drawback of not being able to fully exploit the spatiotemporal characteristics of EEG signals. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes an emotion state recognition method and apparatus based on a sparse dynamic graph convolutional neural network, which can quickly identify emotion states, extract sparse and effective information, and eliminate the dependence on domain knowledge.

[0005] In a first aspect, this application provides a method for emotion state recognition based on a sparse dynamic graph convolutional neural network, the method comprising:

[0006] Acquire the user's EEG data to be identified;

[0007] The differential entropy features of multiple frequency bands corresponding to the EEG data to be identified are input into the emotion state recognition model. The emotion state recognition model includes a feature extractor, a data augmentation module, and a downstream classifier. The feature extractor is constructed based on a sparse dynamic graph convolutional neural network.

[0008] The differential entropy features are extracted by a feature extractor and sparsity is applied to obtain the first node features of the EEG data to be identified.

[0009] The data augmentation module enhances the features of the first node to generate the features of the second node.

[0010] The downstream classifier classifies the features of the second node to obtain and output the user's emotional state;

[0011] The emotion state recognition model is based on multiple training samples, which include the sample differential entropy features of the sample EEG data and the emotion state labels corresponding to the sample EEG data.

[0012] According to one embodiment of this application, the feature extractor includes a graph filtering layer, a graph convolutional layer, a sparsity block, and an attention weight calculation module connected in sequence, and the input channel of the feature extractor corresponds one-to-one with the electrode nodes of the EEG electrodes of the EEG data to be identified;

[0013] The step of extracting features from the differential entropy features using a feature extractor and applying sparsity to obtain the first node features of the EEG data to be identified includes:

[0014] The differential entropy features are converted from the time domain to the spectral domain and the features of the graph are transformed through the graph filtering layer to obtain an electroencephalogram (EEG).

[0015] Through the graph convolutional layer and the sparse adjacency matrix, the node features of the electrode nodes corresponding to the EEG data to be identified are extracted from the EEG atlas, and the node features are nonlinearly mapped to obtain a feature map.

[0016] The feature map is sparsified using the sparsification block according to the threshold sparsification attention weight to obtain sparsified features.

[0017] The attention weight calculation module generates a dynamic adjacency matrix for the sparse features to obtain the first node features corresponding to the electrode nodes.

[0018] According to one embodiment of this application, convolution is performed in the feature extractor using the graph Laplacian operator:

[0019]

[0020] in, It is a filter function; It is the order of the Chebyshev polynomial. ; These are the coefficients of the Chebyshev polynomial; It is a diagonal matrix; Is The k-th order Chebyshev polynomial to be evaluated at point k is... yes The largest element in the diagonal entries, yes Identifier matrix.

[0021] According to one embodiment of this application, the data augmentation module includes an attention partitioning module and a feature mixing module connected in sequence. The step of enhancing the first node features through the data augmentation module to generate second node features includes:

[0022] The attention partitioning module performs attention partitioning on the features of the first node to obtain output features;

[0023] The output features are mixed using the feature mixing module to generate second node features.

[0024] According to one embodiment of this application, the attention partitioning module includes an encoder and a decoder, the decoder including a first layer of attention and a second layer of attention;

[0025] The step of performing attention partitioning on the features of the first node through the attention partitioning module to obtain output features includes:

[0026] The encoder captures long-distance dependencies of the first node features and groups them to obtain multiple node subsets.

[0027] Global features and attention weights are extracted from the multiple node subsets using the first layer of attention.

[0028] The output features are generated through the second layer of attention, based on the attention weights and the interaction between the global and local features.

[0029] According to one embodiment of this application, the downstream classifier includes a convolutional block, a Transformer encoder, and a classification module connected in sequence. The classification module includes a first-level classifier and a second-level classifier connected in sequence.

[0030] The step of classifying the second node features through the downstream classifier to obtain and output the user's emotional state includes:

[0031] Low-level local features are extracted from the second node module using the convolutional block;

[0032] The global features are obtained by capturing global dependencies from the low-level local features using the Transformer encoder.

[0033] The global features are classified by the first-level classifier. If the first recognition result is determined to be a normal state, the first recognition result is output as the emotional state.

[0034] If the first recognition result is determined to be an unconventional state, the global features are classified by the secondary classifier to obtain the user's second recognition result, which is then output as the emotional state.

[0035] According to one embodiment of this application, acquiring the user's EEG data to be identified includes:

[0036] Acquire the user's electroencephalogram (EEG) signals;

[0037] The EEG signal was subjected to bandpass filtering.

[0038] Independent component analysis is performed on the filtered EEG signal to obtain the EEG data to be identified.

[0039] Secondly, this application provides an emotion state recognition device based on a sparse dynamic graph convolutional neural network, the device comprising:

[0040] The acquisition module is used to acquire the user's EEG data to be identified;

[0041] The first processing module is used to input the differential entropy features of multiple frequency bands corresponding to the EEG data to be identified into the emotion state recognition model. The emotion state recognition model includes a feature extractor, a data augmentation module, and a downstream classifier. The feature extractor is constructed based on a sparse dynamic graph convolutional neural network.

[0042] The second processing module is used to extract features from the differential entropy features using a feature extractor and apply sparsity to obtain the first node features of the EEG data to be identified.

[0043] The third processing module is used to enhance the features of the first node through the data augmentation module to generate the features of the second node;

[0044] The fourth processing module is used to classify the features of the second node through the downstream classifier, obtain the user's emotional state, and output it.

[0045] The emotion state recognition model is based on multiple training samples, which include the sample differential entropy features of the sample EEG data and the emotion state labels corresponding to the sample EEG data.

[0046] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the emotion state recognition method based on a sparse dynamic graph convolutional neural network as described in the first aspect above.

[0047] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the emotion state recognition method based on a sparse dynamic graph convolutional neural network as described in the first aspect above.

[0048] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the emotion state recognition method based on sparse dynamic graph convolutional neural network as described in the first aspect.

[0049] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the emotion state recognition method based on a sparse dynamic graph convolutional neural network as described in the first aspect above.

[0050] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0051] This application provides a method and apparatus for emotion state recognition based on sparse dynamic graph convolutional neural networks, which has the following advantages over existing technologies:

[0052] (1) By introducing a sparse dynamic graph convolutional neural network, a correlation graph between EEG channels is dynamically constructed to mine high-dimensional sparse features. The graph structure is dynamically adjusted during training, and the optimal channel relationship can be constructed according to the signal characteristics of each sample. This solves the problem of dependence on fixed graph structure in traditional graph convolutional networks, making feature extraction more adaptive. By combining dynamic feature extraction and reasonable data augmentation techniques, the emotion state recognition model has higher accuracy and stability in multi-classification tasks. It can quickly identify emotion states, extract sparse and effective information, get rid of dependence on domain knowledge, and the obtained emotion state can be used as a basis for judging health status. Based on the emotion state, suggestions for relaxation and stress relief are given to users.

[0053] (2) It can dynamically adapt to the characteristics of EEG data of different users, extract features with discriminative ability, provide high-quality input for the classification of emotional state, and also make up for the small sample problem through data augmentation, generate richer sample distribution, alleviate class imbalance, and improve the robustness of downstream classifiers. Through sparse dynamic graph convolutional neural network and data augmentation technology that relies on attention mechanism, it can efficiently extract emotional state related features from EEG data, while alleviating the problem of insufficient data and providing technical support for accurate classification. Attached Figure Description

[0054] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0055] Figure 1 This is one of the flowcharts illustrating the emotion state recognition method based on a sparse dynamic graph convolutional neural network provided in this application embodiment;

[0056] Figure 2 This is the second flowchart of the emotion state recognition method based on sparse dynamic graph convolutional neural network provided in the embodiments of this application;

[0057] Figure 3 This is the third flowchart of the emotion state recognition method based on sparse dynamic graph convolutional neural network provided in the embodiments of this application;

[0058] Figure 4 This is a schematic diagram of the structure of the emotion state recognition system based on sparse dynamic graph convolutional neural network provided in the embodiments of this application;

[0059] Figure 5 This is a schematic diagram of the structure of the emotion state recognition device based on a sparse dynamic graph convolutional neural network provided in the embodiments of this application;

[0060] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0062] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0063] The following description, in conjunction with the accompanying drawings, details the emotion state recognition method, device, electronic device, and readable storage medium based on sparse dynamic graph convolutional neural networks provided in this application, through specific embodiments and application scenarios.

[0064] Among them, the emotion state recognition method based on sparse dynamic graph convolutional neural network can be applied to the terminal, specifically executed by the hardware or software in the terminal.

[0065] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0066] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0067] The emotion state recognition method based on sparse dynamic graph convolutional neural network provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the emotion state recognition method based on sparse dynamic graph convolutional neural network. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The following uses an electronic device as the execution subject to illustrate the emotion state recognition method based on sparse dynamic graph convolutional neural network provided in this application embodiment.

[0068] like Figure 1 As shown, this emotion state recognition method based on sparse dynamic graph convolutional neural networks includes:

[0069] Step 110: Obtain the user's EEG data to be identified;

[0070] Step 120: Input the differential entropy features of multiple frequency bands corresponding to the EEG data to be identified into the emotion state recognition model. The emotion state recognition model includes a feature extractor, a data augmentation module, and a downstream classifier. The feature extractor is constructed based on a sparse dynamic graph convolutional neural network.

[0071] Step 130: Extract features from the differential entropy features using a feature extractor and apply sparsity to obtain the first node features of the EEG data to be identified;

[0072] Step 140: Enhance the features of the first node using the data augmentation module to generate the features of the second node;

[0073] Step 150: Classify the second node features using the downstream classifier to obtain and output the user's emotional state;

[0074] The emotion state recognition model is based on multiple training samples, which include the sample differential entropy features of the sample EEG data and the emotion state labels corresponding to the sample EEG data.

[0075] Differential entropy (DE) features are used to describe the complexity and dynamic changes of EEG data.

[0076] In practice, electroencephalography (EEG) data of the user is acquired using head-mounted devices or electrode arrays. This EEG data can be collected from multiple channels to ensure sufficient spatial resolution.

[0077] Frequency band analysis methods such as Fourier transform or wavelet transform are used to decompose the EEG data to be identified into frequency bands, extracting signal features of frequency bands such as delta wave, theta wave, alpha wave, and beta wave, in order to calculate the differential entropy features of each frequency band.

[0078] The differential entropy features of multiple frequency bands are used as input data into the emotion state recognition model.

[0079] The feature extractor is built on a sparse dynamic graph convolutional neural network (Sparse DGCNN). Sparse DGCNN is suitable for processing temporal data with graph structures, such as electroencephalogram (EEG) data.

[0080] Graph Convolutional Neural Networks (GCNs) are deep learning models for processing non-Euclidean data (such as graphs and networks), excelling at modeling complex relationships and neighborhood information. Sparse DGCNN is an optimization of traditional GCNs, with its key feature being the dynamic construction of adjacency relationships. It can dynamically adjust the graph structure based on the input data during training, making it more adaptable to high-dimensional sparse data.

[0081] Traditional EEG feature extraction methods rely on manual design, require extensive domain knowledge, and struggle to extract high-dimensional spatiotemporal features. Static Graph Convolutional Networks (GCNs) depend on fixed graph structures, making them ill-suited to the dynamic and personalized channel correlations in EEG data.

[0082] The input differential entropy features are processed through multiple convolutional layers in Sparse DGCNN. Sparse DGCNN extracts deep features for emotion state recognition from the EEG signals; these features are the first node features. The sparsity design of Sparse DGCNN effectively reduces computational complexity and adapts to the dynamic changes of time-series data.

[0083] The data augmentation module can enhance the features of the first node through noise injection, data transformation, etc., to improve the robustness and generalization ability of the emotion state recognition model and further enhance the representativeness of the features.

[0084] Downstream classifiers can use deep learning classification algorithms such as Support Vector Machines (SVM) to classify the second-node features obtained after data augmentation, mapping the second-node features to specific emotional states. Emotional states can be the types of emotions that affect the user's health, as well as their corresponding levels, such as level 1 happiness, level 3 sadness, and level 2 anger. The output emotional states can be displayed through visualization tools or used as input for further operations, such as user interaction systems or biofeedback applications.

[0085] During the training of the emotion state recognition model, the collected training samples include the sample differential entropy features of the sample EEG data and the corresponding emotion state labels. The emotion state labels can be labeled by user self-report, expert evaluation or other emotion recognition methods.

[0086] These training samples are used to train the various modules of the emotion state recognition model, including the feature extractor, data augmentation module, and downstream classifier. During training, the model parameters can be adjusted using optimization algorithms such as backpropagation and gradient descent to minimize the error between the model's prediction and the actual emotion state.

[0087] The emotion state recognition method based on sparse dynamic graph convolutional neural networks provided in this application introduces a sparse dynamic graph convolutional neural network to dynamically construct a correlation graph between EEG channels, mine high-dimensional sparse features, and dynamically adjust the graph structure during training. This allows for the construction of optimal channel relationships based on the signal characteristics of each sample, solving the problem of dependence on fixed graph structures in traditional graph convolutional networks. This makes feature extraction more adaptive. By combining dynamic feature extraction and appropriate data augmentation techniques, the emotion state recognition model achieves higher accuracy and stability in multi-classification tasks, can quickly identify emotion states, extract sparse but effective information, and is free from dependence on domain knowledge. The obtained emotion states can be used as a basis for judging health status, and suggestions for relaxation and stress relief can be given to users based on their emotion states.

[0088] It should be noted that EEG signals are subject to significant interference and may contain substantial noise. Furthermore, the effective information is sparse, making it difficult to directly model using traditional classification methods. The EEG dataset has a small number of samples, and the distribution of different emotional states is uneven, which affects the generalization ability of the model.

[0089] This application uploads the collected EEG signals of users to a cloud server, uses EEG signal detection to preprocess the EEG signals of respondents, and inputs the obtained valid EEG data to be identified into an emotion state recognition model. After feature extraction by a feature extractor built on Sparse DGCNN, it can cope with the sparsity of effective information. After data augmentation by a data augmentation module, the downstream classifier detects and classifies the emotion state. In the downstream classifier, the first-level classifier determines whether the EEG data to be identified is a normal state. The second-level node features judged as abnormal states are further input into the second-level classifier for evaluation of the emotion state.

[0090] In some embodiments, such as Figure 2 As shown, the user's EEG data to be identified is obtained, including:

[0091] Acquire the user's electroencephalogram (EEG) signals;

[0092] The EEG signal was subjected to bandpass filtering.

[0093] Independent component analysis is performed on the filtered EEG signal to obtain the EEG data to be identified.

[0094] In actual implementation, the acquired user's EEG signal is bandpass filtered to remove high-frequency noise and power line noise.

[0095] Independent component analysis (ICA) separates independent components from the filtered EEG signal, removes baseline-related components, and restores pure EEG activity to obtain the EEG data to be identified. The independent components may include baseline-related components such as eye movements and muscle activity.

[0096] In practice, filtering the EEG signals can improve the signal quality of the EEG data to be identified, avoid interference from noise and spurious signals, and ensure the validity of the data.

[0097] like Figure 3 As shown, the emotion state recognition model includes a feature extractor, a data augmentation module, and a downstream classifier connected in sequence.

[0098] In some embodiments, the feature extractor includes a graph filtering layer, a graph convolutional layer, a sparsity block, and an attention weight calculation module connected in sequence, and the input channel of the feature extractor corresponds one-to-one with the electrode node of the EEG electrode of the EEG data to be identified;

[0099] The step of extracting features from the differential entropy features using a feature extractor and applying sparsity to obtain the first node features of the EEG data to be identified includes:

[0100] The differential entropy features are converted from the time domain to the spectral domain and the features of the graph are transformed through the graph filtering layer to obtain an electroencephalogram (EEG).

[0101] Through the graph convolutional layer and the sparse adjacency matrix, the node features of the electrode nodes corresponding to the EEG data to be identified are extracted from the EEG atlas, and the node features are nonlinearly mapped to obtain a feature map.

[0102] The feature map is sparsified using the sparsification block according to the threshold sparsification attention weight to obtain sparsified features.

[0103] The attention weight calculation module generates a dynamic adjacency matrix for the sparse features to obtain the first node features corresponding to the electrode nodes.

[0104] Understandably, Sparse DGCNN is used as the feature extractor of the detection model. A sparse adjacency matrix is ​​constructed, and the dynamic relationship between the brain regions corresponding to the captured nodes (channels) is captured by graph convolution. The high-dimensional features of the EEG channels corresponding to each system node are extracted and used as the first node feature output.

[0105] In actual implementation, the graph filtering layer transforms the differential entropy features from the time domain to the spectral graph domain, while using polynomial approximation obtained by Chebyshev polynomial approximation to improve the efficiency of convolution operations:

[0106]

[0107] in, It is a filter function; It is the order of the Chebyshev polynomial. ; These are the coefficients of the Chebyshev polynomial; It is a diagonal matrix; Is The k-th order Chebyshev polynomial to be evaluated at point k is... yes The largest element in the diagonal entries, yes Identifier matrix.

[0108] Perform a linear transformation on each node feature of the graph structure composed of EEG features:

[0109]

[0110] in, It is a trainable weight matrix. Representing node characteristics, These are the transformed features.

[0111] Furthermore, for node pairs compute nodes With nodes Attention scores between :

[0112]

[0113] in, This represents a learnable attention vector. express transpose, It is a non-linear activation function. and These are the transformed node features. This indicates a vector concatenation operation.

[0114] Furthermore, Softmax is used to score attention. Normalization is performed to obtain node pairs. attention weights :

[0115]

[0116] in, This represents the attention score between node i and node j. This represents the attention score between node i and node k. Let i represent the set of neighboring nodes of node i, that is, the set of nodes that are connected to node i by an edge.

[0117] Furthermore, a sparsity threshold is set for the calculated attention weight matrix. Perform sparsity operations, retaining weights greater than 100%. From the edges, we get:

[0118]

[0119] in, For node pairs The attention weights after sparsification; For node pairs Attention weights; This is the sparsification threshold.

[0120] The sparsified attention weight matrix The electroencephalogram (EEG) is obtained by using a sparse adjacency matrix.

[0121] Furthermore, the graph convolutional layer uses a sparse adjacency matrix to apply graph convolution operations to the graph structure composed of EEG features, extracting node features and capturing patterns in the graph structure. The node features are then non-linearly mapped using the ReLU activation function to obtain a feature map.

[0122] The optimal adjacency matrix is ​​learned simultaneously during training. Use the following loss function:

[0123]

[0124] in, and These represent the actual label vector and the predicted label vector of the training data, respectively. It is a real label Compared to predicted labels The average cross-entropy, It is the regularization weight. It is the L2 norm. This represents all model parameters. The network parameters are iteratively updated using the standard backpropagation method. and the best adjacency matrix .

[0125] The sparsification block sparsifies the attention weights based on a threshold while retaining important edges; the attention weight calculation module is used to generate a dynamic adjacency matrix.

[0126] In this embodiment, Sparse DGCNN dynamically selects important edges through sparsification, reducing the influence of irrelevant nodes and lowering computational complexity, while retaining key information, enabling the model to perform excellently on high-dimensional EEG data. Dynamic feature extraction significantly improves the collaborative efficiency of upstream and downstream modules, providing the classifier with more discriminative input features.

[0127] In some embodiments, the data augmentation module includes an attention partitioning module and a feature mixing module connected in sequence. The step of enhancing the first node features through the data augmentation module to generate second node features includes:

[0128] The attention partitioning module performs attention partitioning on the features of the first node to obtain output features;

[0129] The output features are mixed using the feature mixing module to generate second node features.

[0130] In some embodiments, the attention partitioning module includes an encoder and a decoder, the decoder including a first layer of attention and a second layer of attention;

[0131] The step of performing attention partitioning on the features of the first node through the attention partitioning module to obtain output features includes:

[0132] The encoder captures long-distance dependencies of the first node features and groups them to obtain multiple node subsets.

[0133] Global features and attention weights are extracted from the multiple node subsets using the first layer of attention.

[0134] The output features are generated through the second layer of attention, based on the attention weights and the interaction between the global and local features.

[0135] It should be noted that the modified Point MixSwap is used to build the data augmentation module. After the data augmentation module receives the extracted first node features, it groups the data based on the attention mechanism, divides the EEG channels of the input first node features into multiple semantically consistent subsets, and performs weighted mixing or swapping operations between the partitions on the grouped features to generate new second node features.

[0136] EEG data on emotional states suffers from small sample sizes and significant class imbalance, making it difficult for traditional data augmentation methods (such as time slicing and frequency transformation) to generate sufficiently diverse samples. Augmentation methods often lack consideration for the importance of features within the EEG data, easily introducing noise and leading to insufficient model generalization. Using a modified PointMixSwap for data augmentation alleviates the small sample size problem, enriches the training data distribution, and helps prevent overfitting. Simultaneously, the attention mechanism helps PointMixSwap focus on important feature points in the EEG signal, avoiding the introduction of irrelevant or noisy features during augmentation, thus making the augmented data more reliable.

[0137] Dynamic graph structures and attention enhancement mechanisms address the high noise and sparsity issues of EEG data, improving the system's robustness. Furthermore, sparsity and modular design reduce computational overhead, making it suitable for real-time emotion state recognition scenarios.

[0138] The data augmentation module includes an attention partitioning module and a feature fusion module.

[0139] The attention partitioning module consists of an encoder and a decoder. The encoder is composed of multiple self-attention layers, which are used to capture long-distance dependencies and improve the first node features extracted by the feature extractor by grouping them into several subsets and progressively enhancing them. The encoder uses multi-layer attention modules to capture long-distance dependencies and improve the features of a given feature extractor. The decoder consists of two attention modules. The first layer of attention (attn1) is used to extract global features and attention weights, and the second layer of attention (attn2) is used to generate the final output features based on the interaction between global and local features.

[0140] The feature fusion module swaps the output features of specific points using given attention weights, performs weighted fusion or swapping operations on the grouped features, generates enhanced second node features, and enhances the robustness of the model.

[0141] The encoder consists of two layers of attention.

[0142] The first layer of attention (attn1) extracts global features and attention weights, and the second layer of attention (attn2) generates the final output features based on the interaction between global and local features.

[0143] The intention partitioning module is used to partition the first node features Divide into multiple subsets based on points.

[0144] N first node features can be used as input feature points for Point MixSwap.

[0145] N first node features can be used as input feature points for Point MixSwap.

[0146] In the attention partitioning module, an attention mechanism is used to calculate weights based on the features of each point and the global context, and an importance score is calculated for each feature point. :

[0147]

[0148] in, It is the i-th feature point. These are the features input into the partitioning module; These are characteristics of points; , It is a learnable weight matrix; The scaling numerator is typically equal to the feature dimension F.

[0149] Furthermore, based on importance scores Divide the N first-node features into R subsets:

[0150]

[0151] The feature blending module performs weighted blending or swapping operations on the grouped features to generate enhanced features, including the following steps:

[0152] For the partitioned subset Feature weighting can be performed within the same subset:

[0153]

[0154] in, Features are the corresponding subset features from a sample. For the corresponding subset features from another sample, As a new feature after mixing, It's the mixing ratio.

[0155] Furthermore, subset features among different samples Feature exchange between them:

[0156]

[0157] Furthermore, the enhanced features of all subsets are recombine into the complete output features:

[0158]

[0159] The convolutional block contains four convolutional layers, which are used to extract low-level local features of the input data while preserving relative position information. By reducing the spatial dimension of the input, the computational cost of subsequent Transformer operations is reduced. The stride of each convolutional layer is 2, so feature map downscaling can be completed without using pooling layers.

[0160] In this embodiment, an emotion state recognition model is constructed by organically combining a sparse dynamic graph convolutional neural network and Point MixSwap. The emotion state recognition model consists of a feature extractor, a data augmentation module, and a downstream classifier. Sparse DGCNN is used as the feature extractor for the emotion state recognition model. After feature extraction and selection, the feature extractor is connected to the data augmentation module, which is composed of a modified Point MixSwap. The data augmentation module mixes and swaps node features to generate an enhanced feature set, which is finally connected to the downstream classifier. This model can be used for accurate identification of depression levels.

[0161] In some embodiments, the downstream classifier includes a convolutional block, a Transformer encoder, and a classification module connected in sequence. The classification module includes a first-level classifier and a second-level classifier connected in sequence.

[0162] The step of classifying the second node features through the downstream classifier to obtain and output the user's emotional state includes:

[0163] Low-level local features are extracted from the second node module using the convolutional block;

[0164] The global features are obtained by capturing global dependencies from the low-level local features using the Transformer encoder.

[0165] The global features are classified by the first-level classifier. If the first recognition result is determined to be a normal state, the first recognition result is output as the emotional state.

[0166] If the first recognition result is determined to be an unconventional state, the global features are classified by the secondary classifier to obtain the user's second recognition result, which is then output as the emotional state.

[0167] An improved LeViT model is used as the downstream classifier, and a hierarchical classifier is integrated into LeViT. The downstream classifier includes a convolutional block, a Transformer encoder, and a classification module, which is built based on the hierarchical classifier.

[0168] The enhanced second node features are used as input to the downstream classifier. The convolutional block serves as the starting point for the downstream classifier, containing four convolutional layers to extract low-level local features of the second node features while preserving relative positional information. By reducing the spatial dimension of the input, the computational cost for subsequent Transformer operations is reduced. Each convolution is followed by a batch normalization, which can be merged with the preceding convolutions for inference.

[0169] The Transformer encoder consists of multiple stacked encoder modules, which are composed of alternating multi-head attention mechanisms and multi-layer perceptron blocks. The Transformer encoder takes the low-level local features extracted by the convolutional layers as input and captures global dependencies through self-attention mechanisms and multi-head attention computation, thereby enhancing the ability to model the relationships and functional connections between channels.

[0170] The classification module includes a first-level classifier and a second-level classifier, which consists of a fully connected layer, an activation function, a sigmoid function, a softmax function, and a loss function, respectively handling routine state detection and state classification tasks.

[0171] The first-level classifier is responsible for classifying the input global features into two categories: normal and non-normal states. This task is a binary classification task and can use the Sigmoid activation function.

[0172] If the first-level classifier determines the state to be abnormal, the second-level classifier further performs fine-grained three-class classification (mild, moderate, and severe), i.e., classifying the emotional state. This task is a multi-classification task, using the Softmax activation function. The global features extracted from the Transformer encoder are input into the classification module to identify the category of emotional state (mild, moderate, and severe), thus completing the emotional state recognition.

[0173] Compared to standard ViT, LeViT optimizes computation and adjusts the attention mechanism in this part to reduce computational complexity. It includes the following steps:

[0174] The normalization layer normalizes each token. The calculation mechanism of the normalization layer is as follows:

[0175] Suppose a batch of data Where N represents the batch size and D represents the feature dimension for each sample. ( ) and each feature dimension ( The formula for calculating LayerNorm is as follows:

[0176]

[0177] in, Is the input data in the th... The first sample and the first Values ​​in each feature dimension; and They are the first The mean and variance of a sample across all feature dimensions are calculated using the following formula:

[0178]

[0179]

[0180] Furthermore, and They correspond to the first Learnable scaling factors and offsets for each feature dimension allow the model to recover the necessary expressive power after normalization; It is a small constant used to prevent the denominator from being zero;

[0181] Multi-head attention mechanisms can capture long-distance dependencies between image patches; self-attention considers information from all other patches in the image when processing a patch, identifying visual relevance and patterns.

[0182] The calculation method for the self-attention mechanism is as follows:

[0183] Given an element in a sequence, we need to compute the relationships between that element and all other elements. Specifically, for the element in the sequence... There are elements, which we represent as Represent all other elements as and .

[0184] Then, the self-attention value can be calculated using the following formula:

[0185]

[0186] in, Represents matrix multiplication. Indicates transpose. It is the dimension of the key vector. It is the square root of the diagonal element.

[0187] The multilayer perceptron module consists of fully connected layers, a GELU activation function, and a Dropout layer. The fully connected layers learn complex relationships between input data through linear transformations, generating new feature representations. The GELU activation function introduces non-linear transformations, enhancing the model's ability to learn complex patterns while maintaining training stability. The formula for calculating the GELU function is:

[0188]

[0189] in, It is the cumulative distribution function of the standard normal distribution. It is the error function. The GELU function has a small slope in the negative region and is close to linear in the positive region. This characteristic makes it have both the zero cutoff property of the ReLU activation function (avoiding gradient vanishing) and the smooth gradient property of the sigmoid function.

[0190] like Figure 4 As shown, the system includes: an EEG signal collection terminal, a cloud server, a display module, and a suggestion module;

[0191] The EEG signal collection terminal is used to collect and store EEG signals. Users can upload EEG signals here via mobile phone or computer through the client or webpage.

[0192] The cloud server is equipped with an emotion state recognition model. The cloud server can perform EEG signal detection preprocessing on the collected EEG signals and call the emotion state recognition model, etc.

[0193] The display module is used to show emotional states;

[0194] The suggestion module is mainly used to provide users with corresponding suggestions on relaxation, stress relief, and attention diversion for users in their current emotional state.

[0195] When a user uses this system to detect their emotional state, the user uploads their EEG signals from the EEG signal collection terminal. After the data is uploaded to the cloud server, it is first preprocessed by the EEG signal detection algorithm. After preprocessing, it is input into the emotional state recognition model. After feature extraction and data augmentation, a first-level classifier determines whether the sample is in a normal state. Global features that are judged as abnormal states are further input into a second-level classifier to identify the emotional state. The stress level is displayed on the display module, and the suggestion module provides the user with corresponding stress relief suggestions. The obtained emotional state can be used as a basis for judging health status, providing an effective auxiliary support for the detection of symptoms corresponding to emotional states, and is expected to have a positive impact on the fields of mental health and medicine.

[0196] Users can wirelessly upload EEG data from the EEG signal collection point to a cloud server using a webpage or client application. The raw EEG data of the respondents is then preprocessed, and the preprocessed data is input into the emotion state recognition model.

[0197] The feature extractor, based on Sparse DGCNN, extracts features from the graph structure constructed from multi-channel EEG data, and the obtained node features are input into the data augmentation module.

[0198] After receiving the input features, the data augmentation module groups the data based on the attention mechanism, dividing the input features into several subsets. It then performs weighted mixing or swapping operations on the grouped features to generate second node features, which can be directly used in the downstream classifier.

[0199] In the embodiments of this application, it is possible to dynamically adapt to the characteristics of EEG data of different users, extract features with discriminative ability, provide high-quality input for the classification of emotional states, and also make up for the small sample problem through data augmentation, generate a richer sample distribution, alleviate class imbalance, and improve the robustness of downstream classifiers. Through sparse dynamic graph convolutional neural networks and data augmentation technology that relies on attention mechanisms, it is possible to efficiently extract emotional state-related features from EEG data, while alleviating the problem of insufficient data and providing technical support for accurate classification.

[0200] The emotion state recognition method based on sparse dynamic graph convolutional neural networks provided in this application can be executed by an emotion state recognition device based on sparse dynamic graph convolutional neural networks. This application example illustrates the emotion state recognition device based on sparse dynamic graph convolutional neural networks provided in this application by using an emotion state recognition device based on sparse dynamic graph convolutional neural networks to execute the emotion state recognition method based on sparse dynamic graph convolutional neural networks.

[0201] This application also provides an emotion state recognition device based on a sparse dynamic graph convolutional neural network.

[0202] like Figure 5 As shown, the emotion state recognition device based on a sparse dynamic graph convolutional neural network includes:

[0203] The acquisition module 510 is used to acquire the user's EEG data to be identified;

[0204] The first processing module 520 is used to input the differential entropy features of multiple frequency bands corresponding to the EEG data to be identified into the emotion state recognition model. The emotion state recognition model includes a feature extractor, a data augmentation module and a downstream classifier. The feature extractor is constructed based on a sparse dynamic graph convolutional neural network.

[0205] The second processing module 530 is used to extract features from the differential entropy features through a feature extractor and apply sparsity to obtain the first node features of the EEG data to be identified.

[0206] The third processing module 540 is used to enhance the first node features through the data enhancement module to generate second node features;

[0207] The fourth processing module 550 is used to classify the features of the second node through the downstream classifier, obtain the user's emotional state, and output it.

[0208] The emotion state recognition model is based on multiple training samples, which include the sample differential entropy features of the sample EEG data and the emotion state labels corresponding to the sample EEG data.

[0209] The emotion state recognition device based on a sparse dynamic graph convolutional neural network provided in this application introduces a sparse dynamic graph convolutional neural network to dynamically construct a correlation graph between EEG channels, mine high-dimensional sparse features, and dynamically adjust the graph structure during training. It can construct the optimal channel relationship according to the signal characteristics of each sample, solving the problem of dependence on fixed graph structure in traditional graph convolutional networks, making feature extraction more adaptive. By combining dynamic feature extraction and reasonable data augmentation techniques, the emotion state recognition model has higher accuracy and stability in multi-classification tasks, can quickly identify emotion states, can extract sparse and effective information, get rid of dependence on domain knowledge, and the obtained emotion state can be used as a basis for judging health status. Based on the emotion state, suggestions for relaxation and stress relief can be given to users.

[0210] The emotion state recognition device based on sparse dynamic graph convolutional neural network in this application embodiment can be an electronic device or a component in an electronic device, such as an integrated circuit or chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM or self-service machine, etc. The embodiments of this application do not specifically limit it.

[0211] The emotion state recognition device based on sparse dynamic graph convolutional neural network in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0212] The emotion state recognition device based on sparse dynamic graph convolutional neural network provided in this application embodiment can realize the various processes implemented in the emotion state recognition method embodiment based on sparse dynamic graph convolutional neural network as described above. To avoid repetition, it will not be described again here.

[0213] In some embodiments, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, it implements the various processes of the above-described embodiment of the emotion state recognition method based on sparse dynamic graph convolutional neural network and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0214] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0215] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of the emotion state recognition method based on sparse dynamic graph convolutional neural network and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0216] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0217] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described emotion state recognition method based on a sparse dynamic graph convolutional neural network.

[0218] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0219] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the emotion state recognition method based on sparse dynamic graph convolutional neural network, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0220] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0221] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0222] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the emotion state recognition method based on sparse dynamic graph convolutional neural network of the various embodiments of this application.

[0223] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0224] In the description of this application, "multiple" means two or more.

[0225] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0226] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0227] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for emotion state recognition based on sparse dynamic graph convolutional neural networks, characterized in that, include: Acquire the user's EEG data to be identified; The differential entropy features of multiple frequency bands corresponding to the EEG data to be identified are input into the emotion state recognition model. The emotion state recognition model includes a feature extractor, a data augmentation module, and a downstream classifier. The feature extractor is constructed based on a sparse dynamic graph convolutional neural network. The differential entropy features are extracted by a feature extractor and sparsity is applied to obtain the first node features of the EEG data to be identified. The data augmentation module enhances the features of the first node to generate the features of the second node. The downstream classifier classifies the features of the second node to obtain and output the user's emotional state; The emotion state recognition model is based on multiple training samples, which include the sample differential entropy features of the sample EEG data and the emotion state labels corresponding to the sample EEG data. The feature extractor includes a graph filtering layer, an attention weight calculation module, a sparsification block, and a graph convolution layer connected in sequence. The input channels of the feature extractor correspond one-to-one with the electrode nodes of the EEG electrodes of the EEG data to be identified. The step of extracting features from the differential entropy features using a feature extractor and applying sparsity to obtain the first node features of the EEG data to be identified includes: The differential entropy features are converted from the time domain to the spectral domain and the features of the graph are transformed through the graph filtering layer to obtain an electroencephalogram (EEG). The attention weight calculation module extracts the node features of the electrode nodes corresponding to the EEG data to be identified from the EEG atlas, and calculates the attention weights based on the node features to obtain the attention weight matrix. The attention weight matrix is ​​sparsified using the sparsification block according to the threshold sparsification attention weights to obtain a sparse adjacency matrix. The graph convolutional layer utilizes the sparse adjacency matrix to perform graph convolution operations on the EEG atlas and performs nonlinear mapping on the node features to obtain a feature map, thereby obtaining the first node features corresponding to the electrode nodes.

2. The emotion state recognition method based on sparse dynamic graph convolutional neural network according to claim 1, characterized in that, In the feature extractor, convolution is performed using the graph Laplacian operator: ； in, It is a filter function; It is the order of the Chebyshev polynomial. ; These are the coefficients of the Chebyshev polynomial; It is a diagonal matrix; Is The k-th order Chebyshev polynomial to be evaluated at point k is... yes The largest element in the diagonal entries, yes Identifier matrix.

3. The emotion state recognition method based on sparse dynamic graph convolutional neural network according to claim 1, characterized in that, The data augmentation module includes an attention partitioning module and a feature mixing module connected in sequence. The step of enhancing the features of the first node through the data augmentation module to generate the features of the second node includes: The attention partitioning module performs attention partitioning on the features of the first node to obtain output features; The output features are mixed using the feature mixing module to generate second node features.

4. The emotion state recognition method based on sparse dynamic graph convolutional neural network according to claim 3, characterized in that, The attention partitioning module includes an encoder and a decoder, and the decoder includes a first layer of attention and a second layer of attention. The step of performing attention partitioning on the features of the first node through the attention partitioning module to obtain output features includes: The encoder captures long-distance dependencies of the first node features and groups them to obtain multiple node subsets. Global features and attention weights are extracted from the multiple node subsets using the first layer of attention. The output features are generated through the second layer of attention, based on the attention weights and the interaction between the global and local features.

5. The emotion state recognition method based on sparse dynamic graph convolutional neural network according to claim 1, characterized in that, The downstream classifier includes a convolutional block, a Transformer encoder, and a classification module connected in sequence. The classification module includes a first-level classifier and a second-level classifier connected in sequence. The step of classifying the second node features through the downstream classifier to obtain and output the user's emotional state includes: Low-level local features are extracted from the features of the second node using the convolutional block; The global features are obtained by capturing global dependencies from the low-level local features using the Transformer encoder. The global features are classified by the first-level classifier. If the first recognition result is determined to be a normal state, the first recognition result is output as the emotional state. If the first recognition result is determined to be an unconventional state, the global features are classified by the secondary classifier to obtain the user's second recognition result, which is then output as the emotional state.

6. The emotion state recognition method based on sparse dynamic graph convolutional neural network according to claim 1, characterized in that, The acquisition of the user's EEG data to be identified includes: Acquire the user's electroencephalogram (EEG) signals; The EEG signal was subjected to bandpass filtering. Independent component analysis is performed on the filtered EEG signal to obtain the EEG data to be identified.

7. An emotion state recognition device based on a sparse dynamic graph convolutional neural network, characterized in that, include: The acquisition module is used to acquire the user's EEG data to be identified; The first processing module is used to input the differential entropy features of multiple frequency bands corresponding to the EEG data to be identified into the emotion state recognition model. The emotion state recognition model includes a feature extractor, a data augmentation module, and a downstream classifier. The feature extractor is constructed based on a sparse dynamic graph convolutional neural network. The second processing module is used to extract features from the differential entropy features using a feature extractor and apply sparsity to obtain the first node features of the EEG data to be identified. The third processing module is used to enhance the features of the first node through the data augmentation module to generate the features of the second node; The fourth processing module is used to classify the features of the second node through the downstream classifier, obtain the user's emotional state, and output it. The emotion state recognition model is based on multiple training samples, which include the sample differential entropy features of the sample EEG data and the emotion state labels corresponding to the sample EEG data. The feature extractor includes a graph filtering layer, an attention weight calculation module, a sparsification block, and a graph convolution layer connected in sequence. The input channels of the feature extractor correspond one-to-one with the electrode nodes of the EEG electrodes of the EEG data to be identified. The step of extracting features from the differential entropy features using a feature extractor and applying sparsity to obtain the first node features of the EEG data to be identified includes: The differential entropy features are converted from the time domain to the spectral domain and the features of the graph are transformed through the graph filtering layer to obtain an electroencephalogram (EEG). The attention weight calculation module extracts the node features of the electrode nodes corresponding to the EEG data to be identified from the EEG atlas, and calculates the attention weights based on the node features to obtain the attention weight matrix. The attention weight matrix is ​​sparsified using the sparsification block according to the threshold sparsification attention weights to obtain a sparse adjacency matrix. The graph convolutional layer utilizes the sparse adjacency matrix to perform graph convolution operations on the EEG atlas and performs nonlinear mapping on the node features to obtain a feature map, thereby obtaining the first node features corresponding to the electrode nodes.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the emotion state recognition method based on sparse dynamic graph convolutional neural network as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the emotion state recognition method based on a sparse dynamic graph convolutional neural network as described in any one of claims 1-6.

Images