Emotion recognition method based on multi-band Riemannian manifold space learning

Abstract

The invention discloses an emotion recognition method based on multi-band Riemannian manifold space learning, and belongs to the technical field of biological signal recognition, and the method comprises the steps: carrying out the self-adaptive frequency band division of an original signal through variational mode decomposition, restraining the mode aliasing, and extracting a multi-band sub-signal; further mapping the sub-signals to a Riemannian manifold space by using a multivariate phase space reconstruction + Riemannian manifold regularization Fisher discriminant analysis module, representing the geometrical characteristics of the signals through a covariance matrix, enhancing the characteristic separability by combining tangent space projection and discriminant analysis, and fusing the characteristic vectors by using an attention mechanism to obtain a multi-element phase space reconstruction and Riemannian manifold regularization Fisher discriminant analysis model; and finally, inputting the fused feature vectors into a classification model constructed by a graph convolutional neural network, and carrying out classification processing. According to the method, the problems of insufficient utilization of frequency band information and weak manifold feature discrimination in traditional emotion recognition are effectively solved, and the recognition precision and robustness are remarkably improved in scenes such as physiological signals.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of biological signal recognition, and particularly relates to an emotion recognition method based on multi-band Riemannian manifold space learning. BACKGROUND

[0002] With the development of artificial intelligence, human-computer interaction pursues higher intelligence and humanization, and emotion recognition becomes the key to realizing natural and efficient interaction, and has wide application prospects in many fields such as medical treatment and education, such as assisting medical diagnosis and helping teaching strategy adjustment.

[0003] At present, emotion recognition relies on information such as images, voices and physiological signals. Image recognition is easily affected by light, shielding and individual expression differences; the accuracy of voice recognition is reduced in noisy environments, and the voice content may not match the true emotion. In contrast, physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG) and galvanic skin response (GSR) are the research focus because they can directly reflect the internal state of the human body and are less controlled by subjective consciousness. Among them, EEG signals contain rich emotional information, but have the characteristics of nonlinearity, non-stationarity and large individual differences, and traditional methods based on feature extraction and classifiers are difficult to mine the inherent characteristics.

[0004] In traditional physiological signal emotion recognition technology, time domain, frequency domain and frequency domain features are often extracted, and support vector machines, artificial neural networks and other classifiers are used. However, these methods have limitations, single or combined features cannot fully represent the signal, classic classifiers have poor performance when processing high-dimensional and nonlinear data, and it is difficult to cope with individual differences, resulting in unsatisfactory recognition accuracy and generalization ability.

[0005] In recent years, the theory of Riemannian manifold space learning has emerged, which can better describe the distribution of nonlinear data, but traditional methods based on a single frequency band ignore the cooperation of different frequency bands and cannot fully utilize multi-band information, limiting the performance improvement of emotion recognition.

[0006] Therefore, we propose an emotion recognition method based on multi-band Riemannian manifold space learning to solve the above problems. SUMMARY

[0007] The purpose of the present application is to solve the problem of ignoring the cooperation of different frequency bands and failing to fully utilize multi-band information in the prior art, and to propose an emotion recognition method based on multi-band Riemannian manifold space learning.

[0008] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0009] The emotion recognition method based on multi-band Riemannian manifold space learning comprises:

[0010] S01: Collecting initial EEG data, adding emotion labels to the initial EEG data, and performing preprocessing to obtain a first signal;

[0011] S02: establishing a variational model, dividing the first signal into frequency sub-bands, and extracting emotion-related frequency sub-band features, wherein the frequency sub-band division is performed based on the modal components and center frequencies of different frequency components;

[0012] S03: constructing a covariance matrix for the sub-bands through multivariate phase space reconstruction, and mapping the covariance matrix to a Riemannian manifold space;

[0013] S04: Mapping the covariance matrix to the Riemann tangent plane space to obtain a first vector, and using Riemann Fisher discriminant analysis to extract a discriminative feature of the Riemann tangent plane space to obtain a first feature;

[0014] S05: Establish a multi-head attention mechanism, calculate the attention coefficient of each attention head based on the first feature, and perform weighted summation of the first features using the attention coefficient to obtain the second feature;

[0015] S06: Propagate the second feature in the graph convolution layer to update the third feature; and apply a nonlinear activation function after the graph convolution operation;

[0016] S07: Map the third feature to the category space of emotion recognition, and apply the softmax function to convert the output into the probability distribution of each emotional state.

[0017] Preferably, the emotion labels include happy, sad and calm.

[0018] Preferably, the modal components are obtained by performing variational modal decomposition on the first signal.

[0019] Preferably, the variational model is established by performing variational modal decomposition on the first signal, extracting modal components of different frequencies, and selecting the number of modes and the quadratic penalty factor.

[0020] Preferably, after mapping the covariance matrix to the Riemannian manifold space in step S03, Riemannian geometry tools are used to calculate the Riemannian geodesic distance, tangent space and Riemannian geometry mean to mine the time-frequency-space structure information of the physiological signal.

[0021] Preferably, the covariance matrix is ​​constructed by using a multivariate phase space reconstruction technique for each sub-band, embedding the one-dimensional first signal into a multidimensional phase space vector through delayed coordinate embedding, and using N phase space vectors, each with a dimension of m×1.

[0022] Preferably, the first feature is obtained by projecting the first vector to a low-dimensional space through a projection matrix, which is solved by maximizing an objective function after calculating an intra-class scatter matrix and an inter-class scatter matrix and introducing a regularization term.

[0023] Preferably, the probability distribution is obtained by multiplying the output of the graph convolution layer with a learnable weight matrix, adding a bias vector to obtain an unnormalized score vector, and applying a Softmax function to the score vector.

[0024] In summary, the technical effects and advantages of the present application are: at the feature processing level, the present application divides the emotion-related frequency band by means of the variational model, combines the Riemannian manifold space with the Riemannian tangent plane Fisher discriminant analysis, effectively captures the nonlinear statistical characteristics of the electroencephalogram signal, breaks through the limitations of traditional Euclidean space methods for non-stationary signal processing, and significantly enhances the discriminant ability of different emotional state features; at the model construction level, the multi-head attention mechanism and the graph convolution layer are innovatively introduced, the former solves the problem of insufficient capture of complex dependency relationships by traditional feature splicing methods through multi-view adaptive feature weighted fusion, and the latter strengthens the spatial correlation understanding and complex pattern expression ability of the model for emotional features by modeling the spatial correlation of the electroencephalogram channels and nonlinear activation. Finally, through the full connection layer and the softmax function, accurate classification is realized, and each link cooperates to form a complete technical closed loop from data acquisition to classification and recognition, and the defects of the existing methods in the aspects of multi-band feature utilization, nonlinear processing and spatial correlation modeling are overcome. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The step flowchart in the present application is shown in the figure;

[0026] Figure 2 The system flowchart in the present application is shown in the figure;

[0027] Figure 3 The sub-band signal graph according to the variational model in the variational modal decomposition in the present application is shown in the figure. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments.

[0029] As shown in Figure 1 and Figure 2 , the emotion recognition method based on multi-band Riemannian manifold space learning includes:

[0030] S01: Collect initial electroencephalogram data, add emotion labels to the initial electroencephalogram data, and pre-process to obtain a first signal;

[0031] S02: Establishing a variational model, performing frequency band division on the first signal to obtain sub-bands, and extracting frequency band features related to emotions from the sub-bands, wherein the frequency band division is completed according to modal components and center frequencies of different frequency components;

[0032] S03: Reconstructing a covariance matrix through multivariate phase space for the sub-bands, and mapping the covariance matrix to a Riemannian manifold space;

[0033] S04: Mapping the covariance matrix to a Riemannian tangent plane space to obtain a first vector, and extracting a feature with discriminability of the Riemannian tangent plane space by using Riemann Fisher discriminant analysis to obtain a first feature;

[0034] S05: Establishing a multi-head attention mechanism, calculating an attention coefficient of each attention head through the first feature, and performing weighted summation on the first feature through the attention coefficient to obtain a second feature;

[0035] S06: Propagating and updating the second feature in a graph convolution layer to obtain a third feature, and applying a nonlinear activation function after the graph convolution operation;

[0036] S07: Mapping the third feature to a category space of emotion recognition, and applying a softmax function to convert the output into a probability distribution of each emotional state.

[0037] Specifically, the specific process of step S01 includes the following:

[0038] S11, collecting multi-channel electroencephalogram signals through an electroencephalogram device, collecting different electroencephalogram signals according to three emotional states (happy, sad, and calm) of a subject, adding corresponding label information, and then using a band-pass filter to retain the part of the electroencephalogram signal related to emotions (generally 0.5-40Hz), and the transfer function of the filter is expressed as:

[0039]

[0040] where a k and b k are the coefficients of the filter, and N and M are the orders of the filter, respectively.

[0041] After removing the artifacts in the electroencephalogram signal, it is normalized (using minimum-maximum normalization) to facilitate the use of the subsequent variational modal decomposition model:

[0042]

[0043] The specific process of step S02 is as follows:

[0044] S21, the filtered signal is decomposed into modal components of different frequencies by variational mode decomposition. The specific process is as follows: a suitable modal number (k) and a quadratic penalty factor (a) are selected to establish a variational model, which is represented as:

[0045]

[0046] S22, then the preprocessed EEG signal is divided into different frequency bands according to the modal components u k (t) and the center frequency ω κ , the modal components u k (t) can be obtained by the optimization process of the variational model, and the center frequency ω κ can be calculated by the following formula:

[0047]

[0048] As Figure 3 shown, the signal is divided into multiple sub-frequency bands, and the EEG frequency band features related to emotion are taken.

[0049] The specific implementation process of step S03 is as follows:

[0050] S31, the present application utilizes the multivariate phase space reconstruction technology for each sub-frequency band, and utilizes the delay coordinate embedding step to embed the one-dimensional EEG signal x(n) into a multi-dimensional phase space vector, which is represented as follows:

[0051] X(t)=[x(t),x(t-τ),x(t-2τ),…,x(t-(m-1)τ)] T

[0052] Wherein, m is the embedding dimension, which determines the dimension of the reconstructed phase space; τ is the delay time, which affects the independence between the reconstructed vectors.

[0053] S32, utilize N phase space vectors X1, X2, …, X n , each vector has a dimension of m x 1, and a covariance matrix C is constructed, which is defined as:

[0054]

[0055] Wherein, μ is the mean vector of all phase space vectors:

[0056]

[0057] S33, the constructed covariance matrix is mapped to the Riemannian manifold space, and Riemannian geometric tools including calculation of Riemannian geodesic distance, tangent space and Riemannian geometric mean are used to mine the time-frequency-space structure information of physiological signals.

[0058] The specific procedure of step S04 is as follows:

[0059] S41, data points on the Riemannian manifold are mapped into tangent space by using Riemann logarithm mapping, to obtain a vector V of tangent space i , which is convenient for optimization calculation in Euclidean space, and the formula is:

[0060]

[0061] Wherein, C0 is a mean point on the Riemannian manifold, C i is another point on the Riemannian manifold.

[0062] This step is to further express the tangent feature vector in the vector space, find the transformation matrix of enhancing class-related information and removing irrelevant information, and reconstruct the discriminant feature vector in the vector space.

[0063] S42, then calculate the intra-class scatter matrix S ω and the inter-class scatter matrix S b , and the formulas are respectively:

[0064]

[0065] Wherein: ω i represents the i-th sample set, N i is the number of samples in ω i .

[0066] S43, introduce a regularization term to improve the stability and generalization ability of the model, and the formula is:

[0067]

[0068] Wherein: λ is the regularization parameter, and I is the unit matrix.

[0069] S44, solve the projection matrix W * by maximizing the following objective function, and the formula is:

[0070]

[0071] S45, project the vector V in the tangent space to the low-dimensional space:

[0072]

[0073] Wherein: y i is the low-dimensional feature vector after projection.

[0074] The specific procedure of step S05 is as follows:

[0075] S51, define a multi-head attention mechanism, and then use the features y extracted in S04 i For each attention head h, calculate the attention coefficient:

[0076]

[0077] where: q (h) is the query matrix of the hth head, K (h) is the key matrix of the hth head, y i is the ith value matrix, and d is the feature dimension.

[0078] S52, according to the obtained attention coefficient weighted sum of the feature vector y i to obtain the fused feature z:

[0079]

[0080] The specific process of step S06 is as follows:

[0081] S61, construct a graph convolution layer, in which the node similarity is calculated by the fused feature z (without graph structure), which is expressed as:

[0082]

[0083] S62, sort each node according to the similarity, keep the top K neighbors, generate a binary adjacency matrix A[i,j], and then perform symmetric processing, preprocessing, and adding a self-loop to the adjacency matrix A:

[0084]

[0085] Then perform the normalization operation:

[0086]

[0087] where:

[0088]

[0089] where, is a diagonal matrix.

[0090] S63, propagate and update the features through the graph structure, thereby obtaining the new feature expression of each node, which is expressed as

[0091]

[0092] where: H ( n) is the input feature matrix, each row representing the feature of a node, H (n+1) is the output feature matrix, is the normalized adjacency matrix, W fc is a learnable weight matrix, and σ is a nonlinear activation function.

[0093] S64, apply a nonlinear activation function after the graph convolution operation to enhance the nonlinear representation ability of the model:

[0094]

[0095] where H (n+2) is the output of the graph convolution.

[0096] The specific process of step S07 is as follows:

[0097] S71, define an output layer of a fully connected layer, multiply the output H (n+2) of the graph convolution layer by a learnable weight matrix W fc , and add a bias vector b fc to obtain an unnormalized score vector o:

[0098] o = W fc H (n+2) + b fc

[0099] S72, apply a Softmax function to the score vector o to convert it into a probability distribution p i for each emotional state:

[0100]

[0101] where p i is the probability that the input sample belongs to the i-th emotional category, and C is the number of emotional categories (here C = 3, happy, sad, and calm).

[0102] The technical solutions in the embodiments of the present application have at least the following technical effects or advantages: the present application improves the accuracy and robustness of emotion recognition by using the multi-band Riemann manifold space learning and multi-modal feature fusion technology. In the feature processing layer, the S02 to S04 steps divide the emotion-related frequency band by using the variational model, combine the Riemann manifold space and the Riemann tangent plane Fisher discriminant analysis, effectively capture the nonlinear statistical characteristics of the electroencephalogram signal, break through the limitations of the traditional Euclidean space method for non-stationary signal processing, and significantly enhance the discriminant ability of different emotional state features; in the model construction layer, the S05 to S06 steps innovatively introduce the multi-head attention mechanism and the graph convolution layer, the former solves the problem of insufficient capture of complex dependency relationship by using the traditional feature splicing method, and the latter uses the electroencephalogram channel space correlation modeling and nonlinear activation to strengthen the spatial correlation understanding and complex pattern expression ability of the model for emotional features. Finally, precise classification is realized through the full connection layer and the softmax function, and each link cooperates to form a complete technical closed loop from data acquisition to classification and recognition, overcoming the defects of the existing methods in the aspects of multi-band feature utilization, nonlinear processing and spatial correlation modeling.

[0103] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes to the technical solutions and the inventive concept of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. An emotion recognition method based on multi-band Riemannian manifold space learning, characterized by: include: S01: Collecting initial EEG data, adding emotion labels to the initial EEG data, and performing preprocessing to obtain a first signal; S02: establishing a variational model, dividing the first signal into frequency sub-bands, and extracting emotion-related frequency sub-band features, wherein the frequency sub-band division is performed based on the modal components and center frequencies of different frequency components; S03: constructing a covariance matrix for the sub-bands through multivariate phase space reconstruction, and mapping the covariance matrix to a Riemannian manifold space; S04: Mapping the covariance matrix to the Riemann tangent plane space to obtain a first vector, and using Riemann Fisher discriminant analysis to extract a discriminative feature of the Riemann tangent plane space to obtain a first feature; S05: Establish a multi-head attention mechanism, calculate the attention coefficient of each attention head based on the first feature, and perform weighted summation of the first features using the attention coefficient to obtain the second feature; S06: Propagate the second feature in the graph convolution layer to update the third feature; And apply nonlinear activation function after graph convolution operation; S07: Map the third feature to the category space of emotion recognition, and apply the softmax function to convert the output into the probability distribution of each emotional state.

2. The emotion recognition method based on multi-band Riemannian manifold space learning according to claim 1, characterized in that: The emotion labels include happy, sad, and calm.

3. The emotion recognition method based on multi-band Riemannian manifold space learning according to claim 1, characterized in that: The modal components are obtained by performing variational modal decomposition on the first signal.

4. The emotion recognition method based on multi-band Riemannian manifold space learning according to claim 1, characterized in that: The variational model is established by performing variational modal decomposition on the first signal, extracting modal components of different frequencies, and selecting the number of modes and the quadratic penalty factor.

5. The emotion recognition method based on multi-band Riemannian manifold space learning according to claim 1, characterized in that: After mapping the covariance matrix to the Riemannian manifold space in step S03, Riemannian geometry tools are used to calculate the Riemannian geodesic distance, tangent space and Riemannian geometry mean to mine the time-frequency-space structure information of the physiological signal.

6. The emotion recognition method based on multi-band Riemannian manifold space learning according to claim 1, characterized in that: The covariance matrix is ​​constructed by using a multivariate phase space reconstruction technique for each sub-band, embedding the one-dimensional first signal into a multidimensional phase space vector through delayed coordinate embedding, and using N phase space vectors, each with a dimension of m×1.

7. The emotion recognition method based on multi-band Riemannian manifold space learning according to claim 1, characterized in that: The first feature is obtained by projecting the first vector into a low-dimensional space through a projection matrix, and the projection matrix is ​​obtained by calculating the intra-class scatter matrix and the inter-class scatter matrix and introducing a regularization term, and then solving the maximization objective function.

8. The emotion recognition method based on multi-band Riemannian manifold space learning according to claim 1, characterized in that: The probability distribution is obtained by multiplying the output of the graph convolution layer by a learnable weight matrix, adding a bias vector, and obtaining an unnormalized score vector, and applying the Softmax function to the score vector.

Images