An electroencephalogram emotion recognition method based on identity mode authentication

Abstract

The present application provides an electroencephalogram emotion recognition method based on identity mode verification, which comprises the following steps: 1) collecting multi-channel electroencephalogram data and setting basic identity mode number; 2) extracting the spectral features of the electroencephalogram data, constructing the electroencephalogram channel mapping relationship and channel connection graph, and forming three-dimensional spectral features; 3) realizing identity mode verification based on tensor space measurement; 4) setting a distance threshold according to the channel connection graph, and then constructing a co-occurrence matrix of the channel; 5) extracting graph embedding features using an automatic encoder, with the input being the electroencephalogram spectral features and the output being the co-occurrence matrix; 6) designing an emotion classifier, and realizing emotion classification using shared Multi-Head Attention layers and special FeedForward layers. The present application combines tensor learning, graph embedding, attention mechanism and other methods, and proposes solutions to problems such as cross-subject time domain inadaptation, and can effectively improve the cross-subject time emotion recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electroencephalogram emotion classification and identity verification, and in particular to an electroencephalogram emotion recognition method based on identity mode verification. BACKGROUND

[0002] Emotion accompanies human life, is a complex psychological process, and plays a very important role in people's production and life. Emotion recognition is a multidisciplinary subject involving computer science, psychology, cognitive science, etc. Emotion recognition has wide applications, such as depression recognition, polygraph test, stress regulation and emotion regulation. When a person is in a certain emotional state, the body will produce some external and internal changes, such as expression, voice, posture and physiological activity, etc. Therefore, emotion can be analyzed based on these body changes.

[0003] Signal data used for emotion recognition can be divided into two categories: non-physiological signals and physiological signals. Typical non-physiological signals include facial expressions and voices, while physiological signals include electroencephalogram, electromyogram and electrocardiogram, etc. Among different types of physiological signals, the unique reaction of electroencephalogram to human emotional state makes it the best choice for emotion recognition. However, physiological signals have the problems of high interference and weak information by themselves, and also have the problem of "subject independence", that is, there is a large difference between different subjects.

[0004] Tensor feature uses the superiority of tensor data form over vector data form, retains the spatial structure information of features, mines the relative relationship between features, enhances the expression ability of tensor space, and greatly reduces the computational complexity. Graph neural network can mine the relationship between each sample point through unique node connection, and then aggregate the same type of nodes and separate different types of nodes. Meanwhile, tensor learning is introduced to distinguish identity mode and graph neural network to distinguish emotion classification to solve the time domain inadaptability of cross-subjects. SUMMARY

[0005] The problem to be solved by the present application is to provide an electroencephalogram emotion recognition method based on identity mode verification, which can solve the problem of small intra-class and inter-class difference of physiological signals, and can reduce the domain inadaptability of cross-subjects in the field of electroencephalogram emotion recognition, so as to improve the emotion classification accuracy under the condition of cross-subjects.

[0006] In order to solve the above technical problems, the present application provides an electroencephalogram emotion recognition method based on identity mode verification, comprising the following steps:

[0007] Step 1: Collecting multi-channel electroencephalogram data and setting the number of basic identity modes;

[0008] Step 2: Extract the spectral features of the electroencephalogram data, construct the electroencephalogram channel mapping relationship and channel connection graph according to the spatial position of the electrodes specified by the international standard 10 / 20 system, and form a three-dimensional spatial spectral feature through the channel mapping relationship;

[0009] Step 3: Identity pattern verification based on tensor space measurement, forming multiple identity pattern sets;

[0010] Step 4: According to the electroencephalogram channel connection graph, set the channel distance threshold, and then construct the co-occurrence matrix of the channel;

[0011] Step 5: Use automatic encoder to extract graph embedding features, the input is multi-channel electroencephalogram spectral features, and the output is the co-occurrence matrix constructed according to the electroencephalogram channel connection graph;

[0012] Step 6: Design an emotion classifier, use shared Multi-Head Attention layers and special FeedForward layers to realize emotion classification, shared means all identity pattern set networks are shared, and special means each identity pattern set network is unique;

[0013] In step 1, according to the multi-channel electroencephalogram data, set the basic number of identity pattern sets, emphasize the concept of "people are divided into groups", and the electroencephalogram of different subjects has certain similarity, which can be utilized to reduce the domain adaptation situation across subjects.

[0014] In step 2, forming a three-dimensional spatial spectral feature includes the following steps:

[0015] Step 2.1: Multi-channel electroencephalogram signal is denoted as Where C is the number of channels, and D is the time dimension of the collected signal. According to the Welch method, extract the spectral features of the 4-45Hz frequency band of the multi-channel electroencephalogram data, where the wave band is divided into theta (4-7Hz), alpha (8-13Hz), beta (14-29Hz), and gamma (30-45Hz);

[0016] Step 2.2: According to the spatial position of the electrodes specified by the international standard 10 / 20 system, construct the electroencephalogram channel mapping relationship, which is essentially a two-dimensional mapping matrix, which can map the C-dimensional channel into a P×Q distribution, P and Q are the length and width of the mapping matrix respectively; The electroencephalogram channel connection graph is the connection relationship between channels, which is essentially the connection between electrodes and the connection distance;

[0017] Step 2.3: According to the spectral features and mapping relationship, construct a three-dimensional spatial spectral feature.

[0018] In step 3, the algorithm for identity pattern verification based on tensor space measurement includes the following steps:

[0019] Step 3.1: Tensor feature. Tensor feature takes advantage of the superiority of tensor data form over vector data form, retains the spatial structure information of features, mines the relative relationship between features, enhances the expression ability of tensor space, greatly reduces the computational complexity, converts the extracted vector features into tensor features according to the lead mode and time-frequency mode, thereby retaining the spatial structure information of multi-channel electroencephalogram signals and other multi-modal characteristics.

[0020] Step 3.2: Tensor space learning. According to the rich projection direction selection provided by the tensor data, a series of optimal modal projection matrices b and w under the unit constraint I are learned by generalized tensor discriminant analysis. The optimization formula is as follows:

[0021]

[0022]

[0023] In the formula, ζ represents the convergence factor, and tr(.) represents the trace. The optimal projection direction of the multilinear tensor space is learned by generalized tensor discriminant analysis, thereby learning the stronger expression ability of the tensor space and further improving the discriminability and precision.

[0024] Step 3.3: Tensor network based on tensor contraction. In the multilinear tensor space under the optimal projection direction, the contracted tensor is obtained by using the alternating least squares, the tensor network is constructed by tensor contraction, thereby distributing the data to the corresponding identity class, so as to complete the identity mode classification, and the loss function is calculated by using cross entropy, and the loss of identity mode classification is denoted as L D . The multilinear expression ability of the tensor network is used to reduce the training parameters and training time, thereby providing higher real-time performance for industrial application.

[0025] In step 4, the construction of the co-occurrence matrix of the electroencephalogram channel includes the following steps:

[0026] Step 4.1: According to the electroencephalogram channel connection diagram, the channel distance threshold is set to 40% of the electrode sagittal line (the front and back connecting line from the nasal root to the occipital tuberosity), and when the connection distance between electrodes is less than the threshold, it is considered that the channel where the electrode is located has a co-occurrence relationship, otherwise there is no co-occurrence relationship;

[0027] Step 4.2: Calculate the co-occurrence value m ij of each channel with other channels, and the calculation formula is:

[0028]

[0029] In the formula, ξ ijdenotes the co-occurrence relationship between channel i and j, if there is a co-occurrence relationship, then ij =1, otherwise 0;

[0030] Step 4.3: Form the co-occurrence matrix M according to the co-occurrence values between all channels.

[0031] In step 5, the autoencoder is used to extract the graph embedding features, the input is the multi-channel electroencephalogram spectrum features, and the output is the co-occurrence matrix constructed according to the electroencephalogram channel connection graph; through the learning of the autoencoder, the task is to pass through the front layer network extract intermediate features from the multi-channel electroencephalogram spectrum features S and pass through the back layer network restore the intermediate features to the co-occurrence matrix M, and the target can be described as:

[0032]

[0033] where μ is the square of the number of channels, and the intermediate features are graph embedding features, which contain the information of point features (electroencephalogram spectrum features) and edge features (co-occurrence matrix); the restored co-occurrence matrix has a deviation, and the loss caused by the deviation is L M .

[0034] In step 6, the emotion classifier is designed, and the Multi-HeadAttention layer Attntion share shared by all identity mode set networks and the FeedForward layer FFd par specific to each identity mode set network are used to realize emotion classification, and the process is described as:

[0035]

[0036] The cross entropy is used to calculate the loss, and the emotion classification loss is L0, and the total loss is obtained as follows:

[0037] L=λL D +(1-λ)(L M +L0),

[0038] Where λ can be adjusted by weighting the losses of identity recognition and emotion classification to optimize the model and obtain more accurate emotion classification results.

[0039] The beneficial effects of the present application are that the tensor learning, graph embedding, attention mechanism and other methods are combined, solutions are proposed for cross-subject time domain inadaptation and other problems, and the cross-subject time emotion recognition accuracy can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a schematic diagram of the method flow of the present application;

[0041] Figure 2 is a schematic diagram of tensor space learning;

[0042] Figure 3 is a schematic diagram of graph embedding feature extraction;

[0043] Figure 4 is a schematic diagram of an emotion classifier. DETAILED DESCRIPTION

[0044] The present application is further illustrated in conjunction with the accompanying drawings and specific embodiments, to highlight the advantages of the present application:

[0045] As Figure 1 shown, the present application provides an electroencephalogram emotion recognition method based on identity pattern verification, which specifically includes the following steps:

[0046] Step 1: Collecting multi-channel electroencephalogram data, setting the basic number of identity patterns;

[0047] Step 2: Extracting the spectral features of the electroencephalogram data, constructing the electroencephalogram channel mapping relationship and channel connection graph according to the spatial position of the electrodes specified by the international standard 10 / 20 system, and forming three-dimensional spectral features through the channel mapping relationship;

[0048] Step 3: Identity pattern verification based on tensor space measurement, forming multiple identity pattern sets;

[0049] Step 4: Setting the channel distance threshold according to the electroencephalogram channel connection graph, and then constructing the co-occurrence matrix of the channels;

[0050] Step 5: Extracting graph embedding features using an autoencoder, with the input being multi-channel electroencephalogram spectral signals and the output being the co-occurrence matrix constructed according to the electroencephalogram channel connection graph;

[0051] Step 6: Designing an emotion classifier, using shared Multi-Head Attention layers and special FeedForward layers to achieve emotion classification, with sharing meaning that all identity pattern set networks are shared, and special meaning that each identity pattern set network is unique;

[0052] In step 1, according to the multi-channel electroencephalogram data, the basic number of identity pattern sets is set, emphasizing the concept of "people are divided into groups", and the electroencephalogram of different subjects has a certain similarity, which can be utilized to reduce the domain inadaptation situation across subjects.

[0053] In step 2, forming three-dimensional spectral features includes the following steps:

[0054] Step 2.1: Multi-channel electroencephalogram signal is recorded as where C is the number of channels, and D is the time dimension of the collected signal. According to the Welch method, the spectral features of the multi-channel electroencephalogram data in the 4-45 Hz frequency band are extracted, and the wave band is divided into theta (4-7 Hz), alpha (8-13 Hz), beta (14-29 Hz), and gamma (30-45 Hz);

[0055] Step 2.2: According to the spatial position of the electrodes specified by the international standard 10 / 20 system, a brain electrical channel mapping relationship is constructed, which is essentially a two-dimensional mapping matrix that can map C-dimensional channels into a P x Q distribution, where P and Q are the length and width of the mapping matrix, respectively. The electroencephalogram channel connection diagram is the connection relationship between channels, which is essentially the connection between electrodes and the connection distance.

[0056] Step 2.3: According to the spectral features and mapping relationship, a three-dimensional spatial spectral feature is constructed.

[0057] As shown in Figure 2 , in step 3, the algorithm for identity pattern verification based on tensor space measurement includes the following steps:

[0058] Step 3.1: Feature tensorization. Tensor features use the superiority of tensor data form over vector data form to retain spatial structure information of features and mine the relative relationship between features, thereby laying a solid data foundation for tensor-based machine learning algorithms. In addition, tensor features provide more abundant projection direction selection for tensor space, enhancing the expression ability of tensor space. Finally, tensor features can significantly reduce computational complexity compared to vector features, providing higher real-time performance for industrial applications. Tensor space learning. Therefore, the extracted vector features are converted into tensor features according to the lead mode and time-frequency mode, thereby retaining the spatial structure information of multi-channel electroencephalogram signals and other multi-modal characteristics.

[0059] Step 3.2: Tensor space learning. According to the abundant projection direction selection provided by tensor data, a series of optimal modal projection matrices b under unit constraint I are learned by generalized tensor discriminant analysis through inter-class dispersion S w and intra-class dispersion S The optimization formula is as follows:

[0060]

[0061]

[0062] where ζ represents the convergence factor, and tr(.) represents the trace. By learning the optimal projection direction of the multilinear tensor space through generalized tensor discriminant analysis, the tensor space learns a stronger expression ability, thereby improving the discriminability and precision.

[0063] Step 3.3: Tensor Network Based on Tensor Shrinking. In the multilinear tensor space under the optimal projection direction, shrinking tensors are obtained using alternating least squares. Tensor networks are constructed through tensor shrinking to assign data to corresponding identity classes, thereby completing identity pattern classification. The loss function is calculated using cross-entropy, and the loss for identity pattern classification is denoted as L. D By leveraging the multilinear representation capabilities of tensor networks, training parameters and training time can be reduced, providing higher real-time performance for industrial applications.

[0064] Step 4, constructing the co-occurrence matrix of EEG channels, includes the following steps:

[0065] Step 4.1: Based on the EEG channel connectivity map, set the channel distance threshold to 40% of the electrode sagittal line (the anterior-posterior connecting line from the root of the nose to the external occipital protuberance). When the connection distance between electrodes is less than this threshold, it is considered that there is a co-occurrence relationship in the channel where the electrode is located; otherwise, there is no co-occurrence relationship.

[0066] Step 4.2: Calculate the co-occurrence value m of each channel with other channels. ij The calculation formula is as follows:

[0067]

[0068] In the formula ξ ij Let ξ represent the co-occurrence relationship between channels i and j. If a co-occurrence relationship exists, then ξ... ij =1, otherwise 0;

[0069] Step 4.3: Form a co-occurrence matrix M based on the co-occurrence values ​​of all channels.

[0070] like Figure 3 As shown, in step 5, an autoencoder is used to extract graph embedding features. Its input is multi-channel EEG spectral features, and its output is a co-occurrence matrix constructed based on the EEG channel connectivity map. The autoencoder is then used for learning, with the task being to learn from the previous network layers. Extracting intermediate features from multi-channel EEG spectral features S Then through the back layer network intermediate features The reconstructed co-occurrence matrix is ​​M. Both the front-end and back-end networks consist of a combination of linear (Dense) layers and nonlinear (ReLU) layers. The network learning objective can be described as:

[0071]

[0072] In the formula, μ is the square of the number of channels, and the intermediate feature The graph embedding feature contains both the point feature (EEG frequency spectrum feature) and the edge feature (co-occurrence matrix); the recovered co-occurrence matrix has a deviation, and the loss caused by the deviation is L M .

[0073] As shown in Figure 4 , in step 6, the emotion classifier is designed, and the Multi-Head Attention layer Attention shared by all identity mode set networks and the Feed Forward layer FFd specific to each identity mode set network are used to implement emotion classification. share par The Multi-Head Attention layer first represents the graph embedding feature as Query, Key, and Value through a multi-head linear network, then extracts the features using a scaled self-attention mechanism, and then adds them to the initial features, and then inputs them to the Feed Forward layer through Layer Normalization; the Feed Forward layer first learns through a linear-nonlinear-linear network, and then undergoes the same Add&Normalization operation, and finally undergoes a linear layer to obtain the classification result. The process is described as follows:

[0074]

[0075] The cross-entropy is used to calculate the loss, and the emotion classification loss is denoted as L0, and the total loss is obtained as follows:

[0076] L = λL D + (1-λ)(L M + L0),

[0077] Where λ can be adjusted to weight the loss of identity recognition and emotion classification to optimize the model and obtain more accurate emotion classification results.​

Claims

1. An electroencephalogram emotion recognition method under identity-based mode verification, characterized in that, Includes the following steps: Step 1: Collect multi-channel EEG data and set the basic number of identity patterns; Step 2: Extract the spectral features of the EEG data, and construct the EEG channel mapping relationship and channel connection diagram according to the spatial location of the electrodes as specified by the international standard 10 / 20 system. The three-dimensional spatial spectral features are formed through the channel mapping relationship. Step 3: Perform identity pattern verification based on tensor space metrics to form multiple identity pattern sets; the algorithm for identity pattern verification based on tensor space metrics includes the following steps: Step 3.1: Feature Tensorization; Tensor features leverage the advantages of tensor data over vector data to preserve the spatial structure information of features, explore the relative relationships between features, enhance the expressive power of tensor space, and significantly reduce computational complexity. The extracted vector features are converted into tensor features according to lead modality and time-frequency modality, thereby preserving the spatial structure information and other multimodal characteristics of multichannel EEG signals. Step 3.2: Tensor space learning; learning a generalized tensor discriminant analysis by inter-class scatter and intra-class scatter learning a series of optimal modal projection matrices under unit constraint I The optimization formula is as follows: ； ； In the formula denotes a convergence factor, denotes a trace; the optimal projection direction of the multilinear tensor space is learned through the generalized tensor discriminant analysis, so that the stronger expression ability of the tensor space is learned, and the discrimination and precision are improved. Step 3.3: Tensor network based on tensor contraction; in the multi-linear tensor space under the optimal projection direction, the contraction tensor is obtained by using the alternating least squares, the tensor network is constructed by tensor contraction, and the data is distributed to the corresponding identity class, so as to complete the identity mode classification, the loss function is calculated by using cross entropy, and the loss of identity mode classification is denoted as ; the multi-linear expression ability of the tensor network is used to reduce the training parameters and training time, and higher real-time performance is provided for industrial application; Step 4: Based on the EEG channel connectivity map, set the channel distance threshold, and then construct the channel co-occurrence matrix; Step 5: Use an autoencoder to extract graph embedding features. Its input is multi-channel EEG spectral features, and its output is a co-occurrence matrix constructed based on the EEG channel connectivity map. Step 6: Design an emotion classifier that uses a shared Multi-Head Attention layer and a special Feed Forward layer to classify emotions. The shared layer means that all identity pattern sets share the same network, while the special layer means that each identity pattern set is unique to the network.

2. The electroencephalogram emotion recognition method based on identity mode authentication according to claim 1, characterized in that, In step 1, a basic set of identity patterns is set based on multi-channel EEG data, emphasizing the concept of "birds of a feather flock together". Different subjects have certain similarities in their EEG data, which can be utilized to reduce domain maladaptation when crossing subject boundaries.

3. The electroencephalogram emotion recognition method based on identity mode authentication according to claim 1, characterized in that, In step 2, forming three-dimensional spatial spectral features includes the following steps: Step 2.1: Multi-channel electroencephalogram signal is recorded as Wherein C is the number of channels, D is the time dimension of the collected signal; the spectral features of the multi-channel electroencephalogram data in the 4~45Hz frequency band are extracted according to the Welch method, including theta, alpha, beta and gamma wave bands, wherein the theta band is 4~7Hz, the alpha band is 8~13Hz, the beta band is 14~29Hz, and the gamma band is 30~45Hz; Step 2.2: Based on the spatial location of the electrodes as specified in the international standard 10 / 20 system, construct the EEG channel mapping relationship. Its essence is a two-dimensional mapping matrix, which can map the C-dimensional channels into a P×Q distribution, where P and Q are the length and width of the mapping matrix, respectively. The EEG channel connection diagram shows the connection relationship between each channel, which is essentially the connection status and connection distance between electrodes. Step 2.3: Construct three-dimensional spatial spectral features based on spectral characteristics and mapping relationships.

4. The electroencephalogram emotion recognition method based on identity mode authentication according to claim 1, characterized in that, In step 4, constructing the co-occurrence matrix of EEG channels includes the following steps: Step 4.1: Based on the EEG channel connectivity map, set the channel distance threshold to 40% of the electrode sagittal line, where the electrode sagittal line is the anterior-posterior connecting line from the root of the nose to the external occipital protuberance; when the connection distance between electrodes is less than this threshold, it is considered that there is a co-occurrence relationship in the channel where the electrode is located; otherwise, there is no co-occurrence relationship. Step 4.2: Calculate co-occurrence values of each channel with other channels The formula is: ； wherein denotes the co-occurrence relation of channels i and j, and is 1 if there is a co-occurrence relation, otherwise 0; , otherwise 0; Step 4.3: Form a co-occurrence matrix M based on the co-occurrence values ​​of all channels.

5. The electroencephalogram emotion recognition method based on identity mode authentication according to claim 1, characterized in that, In the step 5, the graph embedding features are extracted by using an autoencoder, the input of which is the multi-channel electroencephalogram spectrum features, and the output is the co-occurrence matrix constructed according to the electroencephalogram channel connection graph; learning is performed through the autoencoder, the task of which is to reconstruct the co-occurrence matrix M through the front-layer network The multi-channel electroencephalogram spectrum features Extract intermediate features , and then through the rear-layer network The intermediate features are restored to the co-occurrence matrix M, and the target can be described as: ​ ； In the formula is the square of the number of channels, the intermediate feature is the graph embedding feature, which contains both point features and edge features, where the point features are electroencephalogram spectrum features, and the edge features are co-occurrence matrices; the recovered co-occurrence matrix has a deviation, and the loss caused by the deviation is .

6. The electroencephalogram emotion recognition method based on identity mode authentication according to claim 1, characterized in that, In the step 6, the emotion classifier is designed, adopting the Multi-Head Attention layer shared by all the identity mode set networks and the Feed Forward layer specific to each identity mode set network The implementation of the emotion classification is described as follows: ； The loss is calculated by cross-entropy, and the emotion classification loss is denoted as The overall loss is obtained as follows: ； wherein The loss weighting of identity recognition and emotion classification can be adjusted to optimize the model to obtain more accurate emotion classification results.

Images