Emotional state recognition method and device based on sparse dynamic graph convolutional neural network

Abstract

The invention discloses an emotional state recognition method and device based on a sparse dynamic graph convolutional neural network, and belongs to the technical field of electroencephalogram signal processing. The method comprises the steps of obtaining to-be-recognized electroencephalogram data of a user; differential entropy features of multiple frequency bands corresponding to the electroencephalogram data to be recognized are input into an emotional state recognition model, the emotional state recognition model comprises a feature extractor, a data enhancement module and a downstream classifier, and the feature extractor is constructed based on a sparse dynamic graph convolutional neural network; performing feature extraction on the differential entropy feature through a feature extractor and applying sparsity to obtain a first node feature of the to-be-recognized electroencephalogram data; enhancing the first node feature through a data enhancement module to generate a second node feature; and classifying the second node features through a downstream classifier to obtain an emotional state of the user and outputting the emotional state. The method can quickly identify the emotional state, can extract sparse effective information, and gets rid of dependence on domain knowledge.

Description

Technical Field

[0001] This application belongs to the technical field of electroencephalogram signal processing, and particularly relates to a method and device for emotion state recognition based on a sparse dynamic graph convolutional neural network. Background Art

[0002] Electroencephalogram (EEG) is a non-invasive tool for recording brain nerve activities. Due to its high temporal resolution and non-invasiveness, it is widely used in the research of mental diseases related to emotion states, such as depression and anxiety. By analyzing the EEG signals of patients in the resting or task state, the brain dysfunction patterns related to emotion states can be revealed, such as abnormal activities in specific frequency bands such as alpha waves and beta waves. EEG data has the characteristics of multiple channels and multiple time points, with high dimensions and being unstructured.

[0003] Currently, EEG signals can be analyzed based on analysis methods of feature engineering, traditional machine learning methods, etc. When analyzing EEG signals based on feature engineering through Fourier transform, Hilbert-Huang transform, etc., it is time-consuming and relies on domain knowledge; when analyzing EEG signals through traditional machine learning methods such as support vector machine (SVM) and random forest, there are defects in that the spatio-temporal characteristics of EEG cannot be fully mined. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems existing in the prior art. For this purpose, this application proposes a method and device for emotion state recognition based on a sparse dynamic graph convolutional neural network, which can quickly recognize emotion states, can extract sparse and effective information, and gets rid of the dependence on domain knowledge.

[0005] In a first aspect, this application provides a method for emotion state recognition based on a sparse dynamic graph convolutional neural network, and the method includes: Obtain the EEG data to be recognized of a user; Input the differential entropy features of multiple frequency bands corresponding to the EEG data to be recognized into an emotion state recognition model, where the emotion state recognition model includes a feature extractor, a data enhancement module, and a downstream classifier, and the feature extractor is constructed based on a sparse dynamic graph convolutional neural network; Extract features from the differential entropy features through the feature extractor and impose sparsity to obtain the first node features of the EEG data to be recognized; Enhance the first node features through the data enhancement module to generate second node features; Classify the second node features through the downstream classifier to obtain and output the emotion state of the user; The emotional state recognition model is obtained based on multiple training samples, where the training samples include the sample differential entropy features of the sample EEG data and the emotional state labels corresponding to the sample EEG data.

[0006] According to an embodiment of the present application, the feature extractor includes a graph filtering layer, a graph convolutional layer, a sparsification block, and an attention weight calculation module connected in sequence. The input channels of the feature extractor correspond one-to-one with the electrode nodes of the EEG electrodes of the EEG data to be recognized. The feature extraction of the differential entropy features by the feature extractor and the imposition of sparsity to obtain the first node features of the EEG data to be recognized include: Through the graph filtering layer, the differential entropy features are converted from the time domain to the spectrogram domain and the features of the graph are transformed to obtain an EEG spectrogram. Through the graph convolutional layer, using a sparse adjacency matrix, the node features of the electrode nodes corresponding to the EEG data to be recognized are extracted from the EEG spectrogram, and a non-linear mapping is performed on the node features to obtain a feature map. Through the sparsification block, according to the threshold sparsification attention weights, the feature map is sparsified to obtain sparsified features. Through the attention weight calculation module, a dynamic adjacency matrix is generated for the sparsified features to obtain the first node features corresponding to the electrode nodes.

[0007] According to an embodiment of the present application, in the feature extractor, convolution is performed through a graph Laplacian operator:

[0008] where is a filtering function; is the order of the Chebyshev polynomial, ; is the coefficient of the Chebyshev polynomial; is a diagonal matrix; is at the k-th order Chebyshev polynomial evaluated at, is the largest element in the diagonal entries, is the identity matrix.

[0009] According to an embodiment of the present application, the data augmentation module includes an attention partitioning module and a feature mixing module connected in sequence. The augmentation of the first node features by the data augmentation module to generate second node features includes: Through the attention partitioning module, the first node features are subjected to attention partitioning to obtain output features. Through the feature mixing module, perform feature mixing on the output features to generate second node features.

[0010] According to an embodiment of the present application, the attention partitioning module includes an encoder and a decoder, and the decoder includes a first-layer attention and a second-layer attention; The performing attention partitioning on the first node features through the attention partitioning module to obtain output features includes: Capturing long-range dependencies and grouping the first node features through the encoder to obtain multiple node subsets; Extracting global features and attention weights for the multiple node subsets through the first-layer attention; Generating the output features through the second-layer attention based on the attention weights according to the interaction between the global features and the local features.

[0011] According to an embodiment of the present application, the downstream classifier includes a convolutional block, a Transformer encoder, and a classification module connected in sequence, and the classification module includes a first-level classifier and a second-level classifier connected in sequence. The classifying the second node features through the downstream classifier to obtain and output the emotional state of the user includes: Extracting low-level local features for the second node module through the convolutional block; Capturing global dependencies for the low-level local features through the Transformer encoder to obtain global features; Classifying the global features through the first-level classifier, and in the case where it is determined that the generated first recognition result is a normal state, outputting the first recognition result as the emotional state; In the case where it is determined that the first recognition result is an abnormal state, classifying the global features through the second-level classifier to obtain a second recognition result of the user and outputting it as the emotional state.

[0012] According to an embodiment of the present application, the obtaining the electroencephalogram data to be recognized of the user includes: Obtaining the electroencephalogram signal of the user; Performing band-pass filtering on the electroencephalogram signal; Performing independent component analysis on the filtered electroencephalogram signal to obtain the electroencephalogram data to be recognized.

[0013] In a second aspect, the present application provides an emotional state recognition device based on a sparse dynamic graph convolutional neural network, and the device includes: An acquisition module for acquiring the electroencephalogram data to be recognized of the user; The first processing module is configured to input the differential entropy features of multiple frequency bands corresponding to the electroencephalogram data to be recognized into an emotion state recognition model, where the emotion state recognition model includes a feature extractor, a data enhancement module, and a downstream classifier, and the feature extractor is constructed based on a sparse dynamic graph convolutional neural network; The second processing module is configured to perform feature extraction on the differential entropy features through the feature extractor and impose sparsity to obtain the first node features of the electroencephalogram data to be recognized; The third processing module is configured to enhance the first node features through the data enhancement module to generate second node features; The fourth processing module is configured to classify the second node features through the downstream classifier to obtain the emotion state of the user and output it; The emotion state recognition model is obtained based on multiple training samples, where the training samples include sample differential entropy features of sample electroencephalogram data and emotion state labels corresponding to the sample electroencephalogram data.

[0014] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the emotion state recognition method based on a sparse dynamic graph convolutional neural network as described in the first aspect above.

[0015] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the emotion state recognition method based on a sparse dynamic graph convolutional neural network as described in the first aspect above.

[0016] In a fifth aspect, the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the emotion state recognition method based on a sparse dynamic graph convolutional neural network as described in the first aspect.

[0017] In a sixth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the emotion state recognition method based on a sparse dynamic graph convolutional neural network as described in the first aspect above.

[0018] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application.

[0019] The emotion state recognition method and device based on a sparse dynamic graph convolutional neural network provided by the present application have the following beneficial effects compared with the prior art: (1) By introducing a sparse dynamic graph convolutional neural network, a correlation graph between EEG channels is dynamically constructed to mine high-dimensional sparse features. During the training process, the graph structure is dynamically adjusted, enabling the construction of an optimal channel relationship according to the signal characteristics of each sample, solving the problem of the traditional graph convolutional network's dependence on a fixed graph structure, making feature extraction more adaptable. By combining dynamic feature extraction and reasonable data augmentation techniques, the emotion state recognition model has higher accuracy and stability in multi-classification tasks, can quickly recognize the emotion state, can extract sparse and effective information, gets rid of the dependence on domain knowledge, and the obtained emotion state can be used as a basis for judging the health state. According to the emotion state, suggestions for relaxation and stress relief are given to the user.

[0020] (2) It can dynamically adapt to the EEG data characteristics of different users, extract discriminative features, provide high-quality inputs for the classification of emotion states, and can also make up for the small sample problem through data augmentation, generate a more diverse sample distribution, alleviate class imbalance, and improve the robustness of the downstream classifier. Through the sparse dynamic graph convolutional neural network and data augmentation techniques relying on the attention mechanism, it is possible to efficiently extract emotion state-related features from EEG data while alleviating the problem of insufficient data, providing technical support for accurate classification. Brief Description of the Drawings

[0021] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, where: Figure 1 is one of the schematic flowcharts of the emotion state recognition method based on a sparse dynamic graph convolutional neural network provided by an embodiment of the present application; Figure 2 is another schematic flowchart of the emotion state recognition method based on a sparse dynamic graph convolutional neural network provided by an embodiment of the present application; Figure 3 is yet another schematic flowchart of the emotion state recognition method based on a sparse dynamic graph convolutional neural network provided by an embodiment of the present application; Figure 4 is the schematic structural diagram of the emotion state recognition system based on a sparse dynamic graph convolutional neural network provided by an embodiment of the present application; Figure 5 is the schematic structural diagram of the emotion state recognition device based on a sparse dynamic graph convolutional neural network provided by an embodiment of the present application; Figure 6 is the schematic structural diagram of the electronic device provided by an embodiment of the present application. Detailed Embodiments

[0022] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0023] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally means an "or" relationship between the associated objects before and after.

[0024] Next, in conjunction with the accompanying drawings, through specific embodiments and their application scenarios, the emotion state recognition method based on a sparse dynamic graph convolutional neural network, the emotion state recognition device based on a sparse dynamic graph convolutional neural network, an electronic device, and a readable storage medium provided by the embodiments of the present application will be described in detail.

[0025] Among them, the emotion state recognition method based on a sparse dynamic graph convolutional neural network can be applied to a terminal, and specifically can be executed by hardware or software in the terminal.

[0026] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad).

[0027] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.

[0028] The emotional state recognition method based on a sparse dynamic graph convolutional neural network provided by the embodiments of this application. The execution subject of this emotional state recognition method based on a sparse dynamic graph convolutional neural network can be an electronic device or a functional module or entity in the electronic device that can implement this emotional state recognition method based on a sparse dynamic graph convolutional neural network. The electronic devices mentioned in the embodiments of this application include, but are not limited to, mobile phones, tablet computers, computers, cameras, and wearable devices, etc. Here, taking the electronic device as the execution subject as an example, the emotional state recognition method based on a sparse dynamic graph convolutional neural network provided by the embodiments of this application will be described.

[0029] As Figure 1 shown, this emotional state recognition method based on a sparse dynamic graph convolutional neural network includes: Step 110, obtain the brain electrical data to be recognized of the user; Step 120, input the differential entropy features of multiple frequency bands corresponding to the brain electrical data to be recognized into an emotional state recognition model. The emotional state recognition model includes a feature extractor, a data enhancement module, and a downstream classifier. The feature extractor is constructed based on a sparse dynamic graph convolutional neural network; Step 130, perform feature extraction on the differential entropy features through the feature extractor and impose sparsity to obtain the first node features of the brain electrical data to be recognized; Step 140, enhance the first node features through the data enhancement module to generate second node features; Step 150, classify the second node features through the downstream classifier to obtain the emotional state of the user and output it; The emotional state recognition model is obtained based on multiple training samples. The training samples include the sample differential entropy features of the sample brain electrical data and the emotional state labels corresponding to the sample brain electrical data.

[0030] Differential entropy (DE) features are used to describe the complexity and dynamic changes of brain electrical data.

[0031] In actual execution, through an electroencephalogram (EEG) acquisition device such as a head-mounted device or an electrode array, the brain electrical signal data of the user is obtained. These brain electrical signal data can be collected from multiple channels to ensure sufficient spatial resolution.

[0032] Perform frequency band decomposition on the brain electrical data to be recognized through frequency band analysis methods such as Fourier transform or wavelet transform, and extract the signal features of frequency bands such as δ wave, θ wave, α wave, and β wave to calculate the differential entropy features of each frequency band.

[0033] Take the differential entropy features of multiple frequency bands as input data and input them into the emotional state recognition model. The feature extractor is constructed based on the Sparse Dynamic Graph Convolutional Neural Network (Sparse DGCNN). Sparse DGCNN is suitable for processing time-series data with graph structures, such as EEG data.

[0034] The Graph Convolutional Neural Network (GCN) is a deep learning model for processing non-Euclidean data (such as graphs, networks, etc.), which is good at modeling complex relationships and neighborhood information. Sparse DGCNN is an optimization of the traditional GCN. Its significant feature is the dynamic construction of adjacency relationships, which can dynamically adjust the graph structure according to the input data during the training process to make it more suitable for high-dimensional sparse data.

[0035] Traditional EEG feature extraction methods rely on manual design, require a lot of domain knowledge, and are difficult to extract high-dimensional spatio-temporal features. The dependence of the static Graph Convolutional Network (GCN) on the fixed graph structure makes it difficult to adapt to the dynamic and personalized channel correlations in EEG data.

[0036] The input differential entropy features will be processed through multiple convolutional layers in Sparse DGCNN. Sparse DGCNN will extract deep features for emotion state recognition from EEG signals, and these features are the first node features. The sparse design of Sparse DGCNN can effectively reduce the computational complexity and adapt to the dynamic changes of time-series data.

[0037] The data augmentation module can enhance the first node features through noise injection, data transformation, etc. to improve the robustness and generalization ability of the emotion state recognition model and further enhance the representativeness of the features.

[0038] The downstream classifier can classify the second node features obtained after data augmentation through classification algorithms of deep learning such as Support Vector Machine (SVM), and map the second node features to specific emotion states. The emotion states can be the types of emotions that affect the user's health and the corresponding levels. For example, happy level 1, sad level 3, and angry level 2, etc. The output emotion states can be displayed through visualization tools or used as the input for further operations, such as user interaction systems or biofeedback applications.

[0039] During the training process of the emotion state recognition model, the collected training samples include the sample differential entropy features of the sample EEG data and the corresponding emotion state labels. The emotion state labels can be calibrated through self-report by users, expert evaluation, or other emotion recognition methods.

[0040] Using these training samples, each module of the emotional state recognition model is trained, including the feature extractor, data augmentation module, and downstream classifier. During the training process, optimization algorithms such as backpropagation and gradient descent can also be used to adjust the model parameters to minimize the error between the model prediction and the actual emotional state.

[0041] According to the emotional state recognition method based on the sparse dynamic graph convolutional neural network provided by the embodiments of the present application, by introducing the sparse dynamic graph convolutional neural network, the correlation graph between EEG channels is dynamically constructed to mine high-dimensional sparse features, and the graph structure is dynamically adjusted during the training process, which can construct the optimal channel relationship according to the signal characteristics of each sample, solve the problem of the dependence of the traditional graph convolutional network on the fixed graph structure, make the feature extraction more adaptable, and through the combination of dynamic feature extraction and reasonable data augmentation technology, the emotional state recognition model has higher accuracy and stability in multi-classification tasks, can quickly recognize the emotional state, can extract sparse and effective information, gets rid of the dependence on domain knowledge, and the obtained emotional state can be used as the basis for judging the health state. According to the emotional state, suggestions for relaxation and stress relief are given to the user.

[0042] It should be noted that the EEG signals are greatly affected by interference, may have significant noise, and at the same time the effective information is sparse, making it difficult to directly use traditional classification methods for modeling. The samples in the EEG dataset are few, and the distribution of different emotional states is uneven, which affects the generalization ability of the model.

[0043] In this application, the collected EEG signals of the user are uploaded to the cloud server, and the EEG signals of the interviewee are preprocessed using EEG signal detection. The obtained effective EEG data to be recognized is input into the emotional state recognition model. After feature extraction by the feature extractor constructed based on Sparse DGCNN, it can handle the sparsity of effective information. After data augmentation by the data augmentation module, the downstream classifier detects and classifies the emotional state. In the downstream classifier, the first-level classifier judges whether the EEG data to be recognized is in a normal state, and the second-node features judged to be in an abnormal state are continuously input into the second-level classifier to evaluate the emotional state.

[0044] In some embodiments, as Figure 2 shown, obtaining the EEG data to be recognized of the user includes: Obtaining the EEG signals of the user; Performing band-pass filtering on the EEG signals; Performing independent component analysis on the filtered EEG signals to obtain the EEG data to be recognized.

[0045] In actual execution, band-pass filtering is performed on the obtained EEG signals of the user to remove high-frequency noise and power line noise.

[0046] Through independent component analysis, independent components are separated from the filtered EEG signals, components related to the baseline are removed, and pure EEG activities are restored to obtain the EEG data to be recognized. Among them, the independent components may include components related to baseline drift such as eye movements and muscle activities.

[0047] In actual implementation, filtering the EEG signals can improve the signal quality of the EEG data to be recognized, avoid the interference of noise and pseudo signals, and ensure the validity of the data.

[0048] As Figure 3 shown, the emotion state recognition model includes a feature extractor, a data augmentation module, and a downstream classifier connected in sequence.

[0049] In some embodiments, the feature extractor includes a graph filtering layer, a graph convolutional layer, a sparsification block, and an attention weight calculation module connected in sequence. The input channels of the feature extractor correspond one-to-one with the electrode nodes of the EEG electrodes of the EEG data to be recognized; The feature extraction of the differential entropy feature by the feature extractor and the application of sparsity to obtain the first node feature of the EEG data to be recognized include: Through the graph filtering layer, the differential entropy feature is converted from the time domain to the spectrogram domain and the features of the graph are transformed to obtain an electroencephalogram spectrum; Through the graph convolutional layer, through a sparse adjacency matrix, the node features corresponding to the electrode nodes of the EEG data to be recognized are extracted from the electroencephalogram spectrum, and the node features are non-linearly mapped to obtain a feature map; Through the sparsification block, the feature map is sparsified according to the threshold sparsification attention weight to obtain a sparsified feature; Through the attention weight calculation module, a dynamic adjacency matrix is generated for the sparsified feature to obtain the first node feature corresponding to the electrode node.

[0050] It can be understood that Sparse DGCNN is used as the feature extractor of the detection model to construct a sparse adjacency matrix, and the dynamic relationship between brain regions corresponding to nodes (channels) is captured through graph convolution to extract high-dimensional features of the EEG channels corresponding to each system node as the first node feature output.

[0051] In actual implementation, the graph filtering layer converts the differential entropy feature from the time domain to the spectrogram domain, and at the same time uses polynomial approximation obtained by approximating with Chebyshev polynomials to improve the efficiency of convolution operations:

[0052] Among them, is a filtering function; is the order of the Chebyshev polynomial, ; are the coefficients of the Chebyshev polynomial; is a diagonal matrix; is at the k-th order Chebyshev polynomial evaluated at, is the largest element in the diagonal entries, is the identity matrix.

[0053] Perform a linear transformation on each node feature of the graph structure composed of EEG features:

[0054] where, is a trainable weight matrix, represents the node feature, is the transformed feature.

[0055] Furthermore, for the node pair , calculate the attention score between node and node :

[0056] where, represents a learnable attention vector, represents the transpose of, is a non-linear activation function, and are the transformed node features, represents the vector concatenation operation.

[0057] Furthermore, use Softmax to normalize the attention score to obtain the attention weight of the node pair :

[0058] where, represents the attention score between node i and node j, represents the attention score between node i and node k, represents the set of neighbor nodes of node i, that is, the set of nodes connected to node i by edges.

[0059] Furthermore, a sparsity threshold is set for the calculated attention weight matrix to perform a sparsification operation, retaining the edges with weights greater than to obtain:

[0060] where is the sparsified attention weight for the node pair ; is the attention weight for the node pair ; is the sparsity threshold.

[0061] Taking the sparsified attention weight matrix as the sparse adjacency matrix to obtain the electroencephalogram spectrum.

[0062] Furthermore, the graph convolutional layer uses the sparse adjacency matrix to apply graph convolutional operations to the graph structure composed of EEG features, extract node features and capture patterns in the graph structure, and the node features are non-linearly mapped through the ReLu activation function to obtain the feature map.

[0063] During the training process, the optimal adjacency matrix is learned simultaneously, using the following loss function:

[0064] where and represent the actual label vector and the predicted label vector of the training data respectively, is the true label relative to the predicted label average cross-entropy, is the regularization weight, is the L2 norm, represents all model parameters. Using the standard backpropagation method to iteratively update the network parameters and the optimal adjacency matrix .

[0065] The sparsification block sparsifies the attention weights according to the threshold, retaining the important edges; the attention weight calculation module is used to generate the dynamic adjacency matrix.

[0066] In this embodiment, Sparse DGCNN dynamically selects important edges through sparsification processing, reduces the influence of irrelevant nodes, reduces the computational complexity, and at the same time retains the key information, enabling the model to perform excellently in high-dimensional EEG data. The dynamic feature extraction significantly improves the cooperation efficiency of the upstream and downstream modules and provides more discriminative input features for the classifier.

[0067] In some embodiments, the data augmentation module includes an attention partitioning module and a feature mixing module connected in sequence. Augmenting the first node feature through the data augmentation module to generate a second node feature includes: Performing attention partitioning on the first node feature through the attention partitioning module to obtain an output feature; Performing feature mixing on the output feature through the feature mixing module to generate a second node feature.

[0068] In some embodiments, the attention partitioning module includes an encoder and a decoder, and the decoder includes a first layer of attention and a second layer of attention; Performing attention partitioning on the first node feature through the attention partitioning module to obtain an output feature includes: Capturing long-range dependencies and grouping the first node feature through the encoder to obtain a plurality of node subsets; Extracting global features and attention weights for the plurality of node subsets through the first layer of attention; Generating the output feature through the second layer of attention based on the attention weights according to the interaction between the global features and the local features.

[0069] It should be noted that, using the modified Point MixSwap to construct the data augmentation module, after the data augmentation module receives the extracted first node feature, the data is grouped based on the attention mechanism, and the EEG channels of the input first node feature are divided into a plurality of subsets with consistent semantics, and a weighted mixing or swapping operation between intervals is performed on the grouped features to generate a new second node feature.

[0070] The amount of EEG data samples in the emotional state is small, and the problem of class imbalance is significant. Traditional data augmentation (such as time slicing, frequency transformation) cannot generate samples with strong enough diversity. The augmentation method lacks consideration of the importance of internal features of EEG data and is prone to introducing noise, resulting in insufficient generalization ability of the model. Using the modified Point MixSwap to augment the data alleviates the small sample problem, enriches the training data distribution, and helps prevent model overfitting. At the same time, the attention mechanism helps PointMixSwap focus on important feature points in the EEG signal, avoiding introducing irrelevant or noisy features during the augmentation process, and making the augmented data more credible.

[0071] The dynamic graph structure and the attention enhancement mechanism solve the problems of high noise and sparsity of EEG data, and improve the robustness of the system. In addition, the sparsification and modular design reduce the computational overhead and are suitable for the scenario of real-time emotional state recognition.

[0072] The data augmentation module includes an attention partitioning module and a feature mixing module.

[0073] The attention partitioning module consists of an encoder and a decoder. The encoder is composed of multiple self-attention layers, which are used to capture long-range dependencies, group the first node features extracted by the improved feature extractor, divide the first node features into several subsets and gradually enhance them. The encoder uses a multi-layer attention module to capture long-range dependencies and improve the features of a given feature extractor; the decoder consists of two layers of attention modules, uses the first layer of attention (attn1) to extract global features and attention weights, and uses the second layer of attention (attn2) to generate the final output features based on the interaction between global features and local features.

[0074] The feature mixing module uses the given attention weights to exchange the output features of specific points, performs weighted mixing or exchange operations on the grouped features, generates enhanced second node features, and enhances the robustness of the model.

[0075] The encoder consists of 2 layers of attention.

[0076] The global features and attention weights are extracted through the first layer of attention (attn1), and the final output features are generated using the second layer of attention (attn2) based on the interaction between global features and local features.

[0077] The attention partitioning module is used to divide the first node features into multiple subsets by points.

[0078] The N first node features can be used as the input feature points of Point MixSwap.

[0079] The N first node features can be used as the input feature points of Point MixSwap.

[0080] In the attention partitioning module, the attention mechanism is used to calculate weights based on the features of points and the global context, and calculate the importance scores for each feature point :

[0081] where is the i-th feature point, is the feature input into the partitioning module; is the feature of the point; 、 are learnable weight matrices; is the scaling factor, usually equal to the feature dimension F.

[0082] Furthermore, according to the importance scores , divide N first node features into R subsets:

[0083] The feature mixing module performs weighted mixing or swapping operations on the grouped features to generate enhanced features, including the following steps: For the divided subsets , feature weighted mixing can be performed within the same subset:

[0084] Among them, is the corresponding subset feature from one sample, is the corresponding subset feature from another sample, is the new feature after mixing, is the mixing ratio.

[0085] Furthermore, perform feature swapping between the subset features among different samples :

[0086] Furthermore, recombine the enhanced features of all subsets into complete output features:

[0087] The convolutional block contains four convolutional layers, which are used to extract low-level local features of the input data while retaining relative position information. By reducing the spatial dimension of the input, the computational amount for subsequent Transformer operations is reduced. The stride of each convolutional layer is 2, so that the feature map can be downscaled without using a pooling layer.

[0088] In this embodiment, an emotion state recognition model is constructed by organically combining a sparse dynamic graph convolutional neural network and Point MixSwap. The emotion state recognition model consists of a feature extractor, a data augmentation module, and a downstream classifier. Sparse DGCNN is used as the feature extractor of the emotion state recognition model. After feature extraction and feature selection, the feature extractor is connected to the data augmentation module composed of the modified Point MixSwap. The data augmentation module mixes and swaps the node features to generate an enhanced feature set, and finally connects to the downstream classifier. This model can be used for accurate recognition of the degree of depression.

[0089] In some embodiments, the downstream classifier includes a convolutional block, a Transformer encoder, and a classification module connected in sequence. The classification module includes a primary classifier and a secondary classifier connected in sequence. Classifying the second node features through the downstream classifier to obtain and output the emotional state of the user includes: Extracting low-level local features from the second node module through the convolutional block; Capturing global dependencies of the low-level local features through the Transformer encoder to obtain global features; Classifying the global features through the primary classifier. In the case where the generated first recognition result is a normal state, output the first recognition result as the emotional state; In the case where the first recognition result is an abnormal state, classify the global features through the secondary classifier to obtain the second recognition result of the user and output it as the emotional state.

[0090] Adopt an improved LeViT model as the downstream classifier, integrate the hierarchical classifier into LeViT. The downstream classifier includes: a convolutional block, a Transformer encoder, and a classification module. The classification module is constructed based on the hierarchical classifier; The enhanced second node features are used as the input of the downstream classifier. The convolutional block is the starting point of the downstream classifier, which includes four convolutional layers, used to extract the low-level local features of the second node features, while retaining the relative position information. By reducing the spatial dimension of the input, the computational amount for subsequent Transformer operations is reduced. Each convolution is followed by a batch normalization, and the batch normalization can be combined with the previous convolution for inference; The Transformer encoder is stacked by multiple encoder modules. The encoder module is composed of an alternating multi-head self-attention mechanism (Multi-Head Attention) and a multi-layer perceptron block (MLP Block). The Transformer encoder takes the low-level local features extracted by the convolutional layer as the input, and captures global dependencies through the self-attention mechanism and multi-head attention calculation, enhancing the modeling ability for inter-channel relationships and functional connections; The classification module includes a primary classifier and a secondary classifier, which are composed of fully connected layers, activation functions, sigmoid functions, softmax functions, and loss functions, and are respectively used to handle normal state detection and state classification tasks.

[0091] The primary classifier is responsible for classifying the input global features into two categories, including normal state and abnormal state. This task is a binary classification task and can use the Sigmoid activation function.

[0092] In the case where the first-level classifier determines an unconventional state, the second-level classifier further performs fine-grained three-class classification (mild, moderate, severe), that is, classifies the emotional state. This task is a multi-classification task and uses the Softmax activation function. The global features extracted from the Transformer encoder are input into the classification module to identify the category of the emotional state (mild, moderate, severe), and the identification of the emotional state is completed.

[0093] Compared with the standard ViT, LeViT has made computational optimizations and adjustments to the attention mechanism in this part to reduce the computational complexity. It includes the following steps: The normalization layer normalizes each token, and the calculation mechanism of the normalization layer is as follows: Assume a batch of data , where N represents the batch size and D represents the feature dimension. For each sample ( ) and each feature dimension ( ), the calculation formula of LayerNorm is as follows:

[0094] Among them, is the value of the input data on the th sample and the th feature dimension; and are the mean and variance of the th sample on all feature dimensions respectively, and the calculation formulas are as follows:

[0095]

[0096] Furthermore, and are the learnable scale factor and offset corresponding to the th feature dimension respectively. These two parameters allow the model to recover the necessary expressive ability after normalization; is a small constant used to prevent the denominator from being zero; The multi-head attention mechanism can capture the long-range dependencies between image patches; self-attention considers the information of all other patches in the image when processing a certain patch and identifies visual correlations and patterns; The calculation method of the Self-Attention mechanism is as follows: Given an element in a sequence, we need to calculate the relationship between this element and all other elements. Specifically, for the th element in the sequence, we represent it as , and represent all other elements as and .

[0097] Then, the self-attention value can be calculated using the following formula:

[0098] where represents matrix multiplication, represents transpose, is the dimension of the key vector, is the square root of the diagonal element.

[0099] The multi-layer perceptron block consists of a fully-connected layer, a GELU activation function, and a Dropout layer; the fully-connected layer learns the complex relationships between input data through linear transformation and generates new feature representations; the GELU activation function introduces non-linear transformation to enhance the model's learning ability for complex patterns while maintaining the stability of training. The calculation formula of the GELU function is:

[0100] where is the cumulative distribution function of the standard normal distribution, is the error function. The GELU function has a smaller slope in the negative value region and is close to linear in the positive value region. This property makes it have both the zero truncation property of the ReLU activation function (to avoid gradient vanishing) and the smooth gradient property of the sigmoid function.

[0101] As Figure 4 shown, the system includes: an EEG signal collection terminal, a cloud server, a display module, and a suggestion module; The EEG signal collection terminal is used to collect and save EEG signals, and users can upload the EEG signals here through a mobile phone or computer on the client or web page; The emotion state recognition model is deployed in the cloud server, and the cloud server can perform preprocessing of EEG signal detection and call the emotion state recognition model on the collected EEG signals; The display module is used to display the emotion state; The suggestion module is mainly used to provide corresponding suggestions for users in the current emotion state, such as relaxing the mind, relieving stress, and diverting attention.

[0102] When the user uses this system to detect the emotional state, the user uploads the electroencephalogram (EEG) signal from the EEG signal collection terminal. After the data is uploaded to the cloud server, it is first preprocessed by the EEG signal detection algorithm. After the preprocessing is completed, it is input into the emotional state recognition model. After feature extraction and data augmentation, the first-level classifier is used to determine whether the sample is in a normal state. The global features determined to be in an abnormal state are continuously input into the second-level classifier to identify the emotional state, and the stress level is displayed on the display module, and corresponding stress relief suggestions are provided to the user through the suggestion module. The obtained emotional state can be used as the basis for judging the health state, providing an effective auxiliary support means for the detection of symptoms corresponding to the emotional state, and is expected to have a positive impact on the field of mental health and the medical field.

[0103] The user can use the web page or client to upload the EEG data from the EEG signal collection terminal to the cloud server through wireless communication, preprocess the original EEG data of the interviewee, and input the preprocessed data into the emotional state recognition model. The feature extractor based on Sparse DGCNN extracts features from the graph structure constructed by multi-channel EEG data, and the obtained node features are input into the data augmentation module.

[0104] After receiving the input features, the data augmentation module groups the data based on the attention mechanism, divides the input features into several subsets, and performs weighted mixing or swapping operations on the grouped features to generate the second node features. The generated second node features can be directly used for the downstream classifier.

[0105] In the embodiments of the present application, it is possible to dynamically adapt to the EEG data characteristics of different users, extract discriminative features, provide high-quality inputs for the classification of emotional states, and also make up for the small sample problem through data augmentation, generate a richer sample distribution, alleviate class imbalance, and improve the robustness of the downstream classifier. Through the sparse dynamic graph convolutional neural network and the data augmentation technology relying on the attention mechanism, it is possible to efficiently extract features related to emotional states from EEG data, while alleviating the problem of insufficient data, providing technical support for accurate classification.

[0106] The emotional state recognition method based on the sparse dynamic graph convolutional neural network provided by the embodiments of the present application may be executed by an emotional state recognition device based on the sparse dynamic graph convolutional neural network. In the embodiments of the present application, taking the emotional state recognition device based on the sparse dynamic graph convolutional neural network executing the emotional state recognition method based on the sparse dynamic graph convolutional neural network as an example, the emotional state recognition device based on the sparse dynamic graph convolutional neural network provided by the embodiments of the present application is described.

[0107] The embodiments of the present application also provide an emotional state recognition device based on the sparse dynamic graph convolutional neural network.

[0108] As Figure 5 shown, the emotion state recognition device based on the sparse dynamic graph convolutional neural network includes: An acquisition module 510, configured to acquire the electroencephalogram data to be recognized of a user; A first processing module 520, configured to input the differential entropy features of multiple frequency bands corresponding to the electroencephalogram data to be recognized into an emotion state recognition model, where the emotion state recognition model includes a feature extractor, a data enhancement module, and a downstream classifier, and the feature extractor is constructed based on a sparse dynamic graph convolutional neural network; A second processing module 530, configured to perform feature extraction on the differential entropy features through the feature extractor and impose sparsity to obtain first node features of the electroencephalogram data to be recognized; A third processing module 540, configured to enhance the first node features through the data enhancement module to generate second node features; A fourth processing module 550, configured to classify the second node features through the downstream classifier to obtain and output the emotion state of the user; The emotion state recognition model is obtained based on multiple training samples, and the training samples include sample differential entropy features of sample electroencephalogram data and emotion state labels corresponding to the sample electroencephalogram data.

[0109] According to the emotion state recognition device based on the sparse dynamic graph convolutional neural network provided in the embodiments of the present application, by introducing a sparse dynamic graph convolutional neural network, dynamically constructing a correlation graph between EEG channels, mining high-dimensional sparse features, dynamically adjusting the graph structure during the training process, being able to construct an optimal channel relationship according to the signal characteristics of each sample, solving the problem of dependence of the traditional graph convolutional network on a fixed graph structure, making feature extraction more adaptable, and by combining dynamic feature extraction and reasonable data enhancement techniques, enabling the emotion state recognition model to have higher accuracy and stability in multi-classification tasks, being able to quickly recognize the emotion state, being able to extract sparse and effective information, getting rid of the dependence on domain knowledge, and the obtained emotion state can be used as a basis for judging the health state, and according to the emotion state, giving suggestions for the user to relax and relieve stress.

[0110] The emotion state recognition device based on the sparse dynamic graph convolutional neural network in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, an in-vehicle electronic device, a Mobile Internet Device (MID), an Augmented Reality (AR) / Virtual Reality (VR) device, a robot, a wearable device, an Ultra-Mobile Personal Computer (UMPC), a netbook, or a Personal Digital Assistant (PDA), etc. It can also be a server, a Network Attached Storage (NAS), a Personal Computer (PC), a Television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0111] The emotion state recognition device based on the sparse dynamic graph convolutional neural network in the embodiments of the present application can be a device with an operating system. The operating system can be the Android operating system, the iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0112] The emotion state recognition device based on the sparse dynamic graph convolutional neural network provided in the embodiments of the present application can implement each process implemented by the emotion state recognition method embodiment based on the sparse dynamic graph convolutional neural network in the above embodiments. To avoid repetition, it will not be elaborated here.

[0113] In some embodiments, as Figure 6 shown, the embodiments of the present application further provide an electronic device 600, including a processor 601, a memory 602, and a computer program stored on the memory 602 and executable on the processor 601. When the program is executed by the processor 601, it implements each process of the emotion state recognition method embodiment based on the sparse dynamic graph convolutional neural network described above, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0114] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0115] The embodiments of the present application further provide a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements each process of the above-mentioned embodiment of the emotion state recognition method based on a sparse dynamic graph convolutional neural network, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0116] Among them, the processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk or optical disc, etc.

[0117] The embodiments of the present application further provide a computer program product, including a computer program. When the computer program is executed by a processor, it implements the above-mentioned emotion state recognition method based on a sparse dynamic graph convolutional neural network.

[0118] Among them, the processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disk or optical disc, etc.

[0119] The embodiments of the present application further provide a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above-mentioned embodiment of the emotion state recognition method based on a sparse dynamic graph convolutional neural network, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0120] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-on-chip, system chip, chip system or system-on-a-chip, etc.

[0121] It should be noted that in this text, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, article or device comprising such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0122] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described example methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the emotion state recognition method based on a sparse dynamic graph convolutional neural network in various embodiments of the present application.

[0123] In the description of the present application, the "first feature", "second feature" may include one or more of such features.

[0124] In the description of the present application, the meaning of "a plurality" is two or more.

[0125] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them fall within the protection scope of the present application.

[0126] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0127] Although the embodiments of this application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of this application, and the scope of this application is defined by the claims and their equivalents.

Claims

1. A method for recognizing emotional states based on a sparse dynamic graph convolutional neural network, characterized in that, Including: Obtain the electroencephalogram (EEG) data to be recognized of the user; Input the differential entropy features of multiple frequency bands corresponding to the EEG data to be recognized into an emotion state recognition model, where the emotion state recognition model includes a feature extractor, a data augmentation module, and a downstream classifier, and the feature extractor is constructed based on a sparse dynamic graph convolutional neural network; Extract features from the differential entropy features through the feature extractor and impose sparsity to obtain the first node features of the EEG data to be recognized; Enhance the first node features through the data augmentation module to generate second node features; Classify the second node features through the downstream classifier to obtain and output the emotion state of the user; The emotion state recognition model is obtained based on multiple training samples, where the training samples include the sample differential entropy features of the sample EEG data and the emotion state labels corresponding to the sample EEG data.

2. The emotional state recognition method based on a sparse dynamic graph convolutional neural network according to claim 1, characterized in that, The feature extractor includes a graph filtering layer, a graph convolutional layer, a sparsification block, and an attention weight calculation module connected in sequence, and the input channels of the feature extractor correspond one-to-one with the electrode nodes of the EEG electrodes of the EEG data to be recognized; The step of extracting features from the differential entropy features through the feature extractor and imposing sparsity to obtain the first node features of the EEG data to be recognized includes: Through the graph filtering layer, convert the differential entropy features from the time domain to the spectrogram domain and transform the features of the graph to obtain an EEG spectrogram; Through the graph convolutional layer, extract the node features of the electrode nodes corresponding to the EEG data to be recognized in the EEG spectrogram through a sparse adjacency matrix and perform a non-linear mapping on the node features to obtain a feature map; Through the sparsification block, sparsify the attention weights according to a threshold and sparsify the feature map to obtain sparsified features; Through the attention weight calculation module, generate a dynamic adjacency matrix for the sparsified features to obtain the first node features corresponding to the electrode nodes.

3. The method for recognizing emotional states based on a sparse dynamic graph convolutional neural network according to claim 2, characterized in that In the feature extractor, perform convolution through a graph Laplacian operator: ； Among them, is a filtering function; is the order of the Chebyshev polynomial, ; are the coefficients of the Chebyshev polynomial; is a diagonal matrix; is at the k-th order Chebyshev polynomial evaluated at, is the largest element in the diagonal entries, is the identity matrix.

4. The emotional state recognition method based on a sparse dynamic graph convolutional neural network according to claim 1, wherein The data augmentation module includes an attention partitioning module and a feature mixing module connected in sequence, and the step of enhancing the first node features through the data augmentation module to generate second node features includes: Through the attention partitioning module, perform attention partitioning on the first node features to obtain output features; Through the feature mixing module, perform feature mixing on the output features to generate second node features.

5. The emotional state recognition method based on a sparse dynamic graph convolutional neural network according to claim 4, characterized in that The attention partitioning module includes an encoder and a decoder, and the decoder includes a first layer of attention and a second layer of attention; The step of performing attention partitioning on the first node features through the attention partitioning module to obtain output features includes: Through the encoder, capture long-range dependencies and group the first node features to obtain multiple node subsets; Extract global features and attention weights from the multiple node subsets through the first layer of attention; Through the second layer of attention, based on the attention weights, generate the output features according to the interaction between the global features and the local features.

6. The method for recognizing emotional states based on a sparse dynamic graph convolutional neural network according to claim 1, wherein The downstream classifier includes a convolutional block, a Transformer encoder, and a classification module connected in sequence. The classification module includes a primary classifier and a secondary classifier connected in sequence. The process of classifying the second node features through the downstream classifier to obtain and output the emotional state of the user includes: Extracting low-level local features from the second node module through the convolutional block; Capturing global dependencies of the low-level local features through the Transformer encoder to obtain global features; Classifying the global features through the primary classifier. When it is determined that the generated first recognition result is a normal state, the first recognition result is output as the emotional state; When it is determined that the first recognition result is an abnormal state, the global features are classified through the secondary classifier to obtain the second recognition result of the user and output it as the emotional state.

7. The method for recognizing emotional states based on a sparse dynamic graph convolutional neural network according to claim 1, characterized in that The process of obtaining the electroencephalogram data to be recognized of the user includes: Obtaining the electroencephalogram signal of the user; Performing band-pass filtering on the electroencephalogram signal; Performing independent component analysis on the filtered electroencephalogram signal to obtain the electroencephalogram data to be recognized.

8. An emotional state recognition device based on a sparse dynamic graph convolutional neural network, characterized in that It includes: An acquisition module for acquiring the electroencephalogram data to be recognized of the user; A first processing module for inputting the differential entropy features of multiple frequency bands corresponding to the electroencephalogram data to be recognized into an emotional state recognition model. The emotional state recognition model includes a feature extractor, a data enhancement module, and a downstream classifier. The feature extractor is constructed based on a sparse dynamic graph convolutional neural network; A second processing module for extracting features from the differential entropy features through the feature extractor and imposing sparsity to obtain the first node features of the electroencephalogram data to be recognized; A third processing module for enhancing the first node features through the data enhancement module to generate second node features; A fourth processing module for classifying the second node features through the downstream classifier to obtain and output the emotional state of the user; The emotional state recognition model is obtained based on multiple training samples. The training samples include the sample differential entropy features of the sample electroencephalogram data and the emotional state labels corresponding to the sample electroencephalogram data.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the emotional state recognition method based on a sparse dynamic graph convolutional neural network according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the emotional state recognition method based on a sparse dynamic graph convolutional neural network according to any one of claims 1-7.

Images