A method, device and equipment suitable for multi-modal physiological signal emotion recognition
By using a unified regularization method based on subspace FCM and deep learning, a multi-label TSK fuzzy system is constructed, which solves the problems of modal differences and label correlation in multimodal physiological signal emotion recognition, and achieves efficient and accurate emotion recognition.
Patent Information
- Application Number
- CN202310252010.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-15
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2043-03-15
AI Technical Summary
Existing technologies fail to effectively consider the differences between different modal signals and the correlation between emotion labels in multimodal physiological signal emotion recognition, resulting in insufficient classification accuracy and gradient vanishing problems during training.
Subspace FCM is used to reduce the dimensionality of physiological signals. A multi-label TSK fuzzy system is constructed by combining fuzzy rules and a unified regularization method of deep learning. An efficient emotion recognition model is established by feature subspace clustering and membership function filtering.
It improves the accuracy of emotion recognition, reduces the risk of overfitting, enables correlation analysis of different emotion triggers, and improves the training efficiency and accuracy of the model.
Smart Images
Figure CN116401619B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of human-computer interaction, and more particularly, relates to a method, device and equipment for emotion recognition suitable for multi-modal physiological signals. BACKGROUND
[0002] Emotion plays an important role in human daily life behavior. It can affect people's beliefs and decisions, and also help build human cognitive processes. Emotion computing, as an important tool to help machines recognize, simulate and process human emotions, is widely used in emergency warning, rehabilitation, intelligent robots and other fields.
[0003] Researchers are used to using facial expressions, speech intonation or vocabulary to estimate people's state, which is called natural language processing (NLP). However, in the case of emotion computing, physiological signals such as electroencephalogram, blood oxygen and galvanic skin may perform better. For example, in high-risk construction sites, monitoring heart rate and blood oxygen with devices such as smart bracelets is usually more practical than taking facial expressions. In addition, in recent years, various devices for remotely monitoring physiological signals have emerged. In recent years, physiological signal datasets for emotion recognition have also been established, such as DEAP and DECAF, which collect multi-channel signals. More signals and channels guarantee more information, so many researchers have turned their attention to physiological signals.
[0004] There have been a lot of related work in the aspect of emotion classification with multi-sensor data: Liu et al. designed an alignment scheme for CNN based on tools such as canonical correlation analysis (CCA), aligning EEG with other peripheral nervous signals for emotion computing. Wen et al. adopted Multi-Variant Correlation method to extract features of physiological signals such as HR, GSR, and obtained good classification results using the method of random forest Kim et al. proposed a multi-modal framework for emotion recognition based on several symmetric frontal and temporal EEG electrodes and pulse signals, which fused and recognized the extracted electroencephalogram and heart rate features through convolutional long short-term memory neural network (convlstm). Yang combined the collected blood oxygen, galvanic skin, electromyographic signals with the behavior information of the subjects, and realized the binary classification of positive and negative emotions using deep neural networks based on attention mechanism.
[0005] However, these works still have some challenges that have not been solved: 1) Most of the works do not consider the difference of different modal signals, especially the importance in identifying different emotions. In fact, many physiological signals have high weight in identifying a certain emotion, but they may be completely redundant in another task. 2) Compared with other classification tasks, the labels of emotion computing tasks often have relevance, such as excitement often accompanied by anger or excitement. How to integrate this hidden connection into the identification task is also a problem to be solved.
[0006] Multi-label TSK is an improved TSK fuzzy system algorithm proposed in recent years to solve the problem of multi-label data. By introducing label correlation, the overall classification accuracy can be improved. This method can fully exploit the non-linear relationship between emotional labels and multi-modal physiological signals. However, too many types of feature inputs result in a sharp increase in the number of fuzzy rules, which causes problems such as training gradient disappearance. Therefore, the present application optimizes and improves the multi-label fuzzy system in the emotion recognition scene. SUMMARY
[0007] In view of the improvement needs of the prior art, the present application provides a method, device and equipment suitable for emotion recognition of multi-modal physiological signals, aiming to comprehensively analyze and extract the data characteristics of various physiological signals, and establish a fuzzy rule set, thereby realizing efficient and accurate emotion recognition.
[0008] To achieve the above purpose, in a first aspect, the present application provides a method for emotion recognition of multi-modal physiological signals, comprising the following steps:
[0009] S1, feature extraction is performed on the original physiological signal data to obtain an original feature space with a feature dimension D; d features are randomly selected from the original feature space, and the selection is repeated t times to obtain t feature subspaces; wherein d
[0010] S2, clustering analysis is performed on each feature subspace to obtain a clustering center and a corresponding membership function;
[0011] S3, the information entropy of all membership functions is calculated, and the feature subspaces corresponding to the first several membership functions with the smallest entropy values are selected;
[0012] S4, the clustering center and the membership function of the feature subspace selected in S3 are used to construct a fuzzy rule, and an emotion recognition model based on the fuzzy rule is established, wherein the emotion recognition model is used for emotion recognition of input physiological signals.
[0013] Further, the loss function for training the emotion recognition model is represented as:
[0014]
[0015] wherein W represents a parameter matrix of the fuzzy rule, and T represents a transpose; n = 1, 2, …, N, N is a sample quantity, k = 1, 2, …, K, K is a rule quantity, represents a fuzzy antecedent parameter of the n th sample in the k th rule; Y represents a real classification label, represents a square of a matrix F-norm, ||.||1 represents a 1-norm of a matrix, a and β represent first and second hyperparameters, L represents a label category number, w p and w q respectively represent fuzzy consequent parameters corresponding to the label p and the label q, r pq = 1 - c pq , c pq represents a Pearson correlation coefficient between the label p and the label q.
[0016] Further, a loss function for training the sentiment recognition model is represented as:
[0017]
[0018] wherein W represents a parameter matrix of the fuzzy rule, and T represents a transpose; n = 1, 2, …, N, N is a sample quantity, k = 1, 2, …, K, K is a rule quantity, represents a fuzzy antecedent parameter of the n th sample in the k th rule; Y represents a real classification label, represents a square of a matrix F-norm, ||.||1 represents a 1-norm of a matrix, a and β represent first and second hyperparameters, L represents a label category number, w p and w q respectively represent fuzzy consequent parameters corresponding to the label p and the label q, r pq = 1 - c pq , c pq represents a Pearson correlation coefficient between the label p and the label q; λ u represents a third hyperparameter.
[0019] Further, in the S3, an information entropy H(U) of the membership function is represented as:
[0020]
[0021] wherein u in represents a membership function of the n th sample to the i th cluster center, N is a sample quantity, and c is a cluster center quantity.
[0022] To achieve the above object, in a second aspect, the present application provides a device for emotion recognition of multi-modal physiological signals, comprising:
[0023] a feature subspace obtaining module, configured to extract features from the original physiological signal data to obtain an original feature space with a feature dimension D, randomly select d features from the original feature space, and repeat the selection t times to obtain t feature subspaces, wherein d < D;
[0024] a feature subspace clustering module, configured to perform clustering analysis on each feature subspace to obtain a clustering center and a corresponding membership function;
[0025] a feature subspace screening module, configured to calculate information entropy of all membership functions, and screen out feature subspaces corresponding to the first several membership functions with the smallest entropy values;
[0026] an emotion recognition module, configured to construct fuzzy rules using the clustering centers and membership functions of the screened feature subspaces, and establish an emotion recognition model based on the fuzzy rules, wherein the emotion recognition model is used to perform emotion recognition on input physiological signals.
[0027] To achieve the above object, in a third aspect, the present application provides an electronic device, comprising: a processor; a memory storing a computer executable program, wherein the program, when executed by the processor, causes the processor to execute the emotion recognition method for multi-modal physiological signals as described in the first aspect.
[0028] To achieve the above object, in a fourth aspect, the present application provides a computer readable storage medium storing a plurality of instructions, wherein the instructions are adapted to be loaded by a processor to execute the emotion recognition method for multi-modal physiological signals as described in the first aspect.
[0029] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:
[0030] (1) The present application uses subspace clustering analysis to process the pre-training set, reduces the dimension of the input sample to different dimensional subspaces, and establishes fuzzy rules and membership functions in different subspaces; through the dimension reduction processing of features, the problems such as gradient disappearance in the training process are prevented.
[0031] (2) The present application introduces the concept of uniform regularization in deep learning to accelerate training and reduce the risk of overfitting. BRIEF DESCRIPTION OF DRAWINGS
[0032] Figure 1This is one of the flowcharts of an emotion recognition method applicable to multimodal physiological signals proposed in this invention;
[0033] Figure 2 This is the second flowchart of an emotion recognition method applicable to multimodal physiological signals proposed in this invention;
[0034] Figure 3 The present invention presents a heatmap of the correlation between emotion tags. Detailed Implementation
[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0036] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0037] This invention uses a novel multi-label TSK fuzzy system as its main framework, effectively mining the mapping relationship between various physiological signals and different emotional labels. Compared with traditional recognition methods, it better considers the correlation between different emotional triggers. Simultaneously, this invention incorporates a newly proposed subspace FCM for feature dimensionality reduction, preventing problems such as gradient vanishing during training. Furthermore, the algorithm introduces the concept of unified regularization from deep learning, accelerating training and reducing the risk of overfitting. The algorithm includes the following steps:
[0038] 1. Perform feature extraction on the original data to construct a pre-training set.
[0039] 2. Subspace FCM is used to process the pre-training set and compute new training feature sets and membership functions. The purpose of subspace FCM is to reduce the dimensionality of the input samples to different dimensional subspaces and establish fuzzy rules and membership functions in different subspaces.
[0040] 3. Based on the membership function obtained in step 2, establish rule sets for different feature subspaces, and construct the objective function based on the correlation coefficient and logistic regression loss function.
[0041] 4. Apply a unified regularization method to make the training trigger strength of each rule as equal as possible.
[0042] 5. Repeat steps 3 and 4 until the overall model converges.
[0043] 6. New user test: directly use the trained general model to complete the prediction task without new user collecting any data.
[0044] The application will be described in more detail below with reference to specific examples.
[0045] Embodiment one
[0046] Referring to Figure 1 , in combination with Figure 2 and Figure 3 , the application provides a method for emotion recognition suitable for multi-modal physiological signals, comprising steps S1 to S4.
[0047] Step S1, feature extraction is performed on the original physiological signal data to obtain an original feature space with a feature dimension of D; d features are randomly selected from the original feature space, and the process is repeated t times to obtain t feature subspaces; wherein d < D.
[0048] In this embodiment, the original data of the physiological signal is first preprocessed, including time domain band pass filtering, removing artificial artifacts, etc., and feature extraction is performed. Assuming that the extracted feature dimension is D. Taking Table 1 as an example, for electroencephalogram signals, an international standard electrode implantation method is used, and 6 different frequency bands are extracted; for central nervous signals such as skin electricity and electromyography, their amplitude, mean value, and power spectral density function time domain feature information are extracted.
[0049] Table 1: Feature extraction example
[0050]
[0051]
[0052] The representation of the input data is given: X = [x1, x2,..., x N ] ∈ R m×N , where N is the number of samples and m is the number of features of the sample. Randomly select d features from the original feature space to form a subspace, repeat t times to obtain t feature subspaces.
[0053] Step S2, clustering analysis is performed on each feature subspace to obtain the clustering center and the corresponding membership function.
[0054] In this embodiment, for each feature subspace, FCM is used for clustering analysis to obtain the clustering center v i and the membership function u i of the nth sample to the clustering center v in ; wherein i = 1, 2,..., c, c is the number of clustering centers, n = 1, 2,..., N, and N is the number of samples.
[0055] Step S3, calculate the information entropy of all membership functions, and screen out the feature subspace corresponding to the top several membership functions with the minimum entropy value.
[0056] In this embodiment, according to the clustering effect of different feature subspaces, the suitable subspace is screened out by greedy optimization, and the optimization is judged according to the information entropy H(U) of the clustering membership function.
[0057]
[0058] wherein, u in represents the membership function of the nth sample to the ith cluster center, N is the sample quantity, and c is the number of cluster centers.
[0059] The greater the entropy value of clustering is, the less accurate the calculation of the cluster center is, that is, the greater the sample overlap is, so the feature subspace with small entropy is greedily selected.
[0060] Step S4, construct the fuzzy rule by using the cluster center and the membership function of the feature subspace screened out in step S3, and establish the emotion recognition model based on the fuzzy rule, wherein the emotion recognition model is used for performing emotion recognition on the input physiological signal.
[0061] In this embodiment, the cluster center and the membership function of different subspaces obtained by the above steps are used to construct the fuzzy rule, and the form of the fuzzy rule can be expressed as:
[0062]
[0063]
[0064] wherein is the fuzzy antecedent parameter of the jth feature in the kth rule, w k is the fuzzy consequent parameter in the kth rule, w k0 is the 0th correction term of the fuzzy consequent parameter in the kth rule, d is the dimension of the input vector, and K is the number of rules.
[0065] The cluster center v i and the membership function u in in step S2 can be derived as follows:
[0066]
[0067]
[0068]
[0069]
[0070] where, is the cluster center of the jth feature in the kth rule, h is the hyperparameter, and x nj is the jth feature in the nth sample.
[0071] The formula for predicting the sentiment type using fuzzy rules can be expressed as:
[0072]
[0073] Further let
[0074]
[0075]
[0076] The formula can be simplified as:
[0077]
[0078] where W represents the parameter matrix of fuzzy rules, T represents the transpose, and the objective function can be expressed as:
[0079]
[0080] where, Y represents the true classification label, is the square of the matrix F-norm, ||.||1 represents the 1-norm of the matrix, and β||W||1 is the regularization term.
[0081] Since there are multiple evaluation labels for sentiment classification, the above process is extended to multi-label, which is:
[0082]
[0083] In order to consider the correlation of multiple labels, the label correlation loss is introduced as a term in the objective function:
[0084]
[0085] r pq =1-c pq
[0086] where c pq represents the Pearson correlation coefficient between label p and label q.
[0087] Further, considering that in the training process of fuzzy learning, it is easy to appear that some rules are difficult to be activated and triggered, a uniform regularization method is used to optimize the fuzzy rule parameters, and by adding a uniform regularization term to the objective function, all rules are made to be at the same triggering level:
[0088]
[0089] Where, λ u Since the hyperparameters are adjustable, the final training loss function is obtained as follows:
[0090]
[0091] Finally, a mini-batch training method is adopted, in which N samples are selected from the total data N each time. S Substitute each element into the objective function and calculate. Thus, we obtain ▽J(W) t ), iteration parameter W t+1 ←W t -μ▽J(W t ).
[0092] Furthermore, the algorithm proposed in this invention significantly outperforms other traditional non-deep algorithms, and has lower model complexity and computational cost. As shown in Appendix Table 2, it outperforms other methods in three of the four main sentiment dimensions. Due to the introduction of multi-label learning, this invention can simultaneously observe the correlation of audience performance under different emotional triggers, as shown in Appendix Table 2. Figure 3 The emotional tag relationship heatmap shows that liking something is often accompanied by pleasant emotions, etc., so that users can further analyze and utilize it.
[0093] Table 2 Comparison with existing methods
[0094]
[0095] Example 2
[0096] An emotion recognition device applicable to multimodal physiological signals, comprising:
[0097] The feature subspace acquisition module is used to extract features from the raw physiological signal data to obtain an original feature space with a feature dimension of D; d features are randomly selected from the original feature space and repeated t times to obtain t feature subspaces; where d <D;
[0098] The feature subspace clustering module is used to perform cluster analysis on each feature subspace to obtain the cluster centers and corresponding membership functions;
[0099] The feature subspace filtering module is used to calculate the information entropy of all membership functions and filter out the feature subspaces corresponding to the top few membership functions with the smallest entropy values.
[0100] The emotion recognition module is configured to construct fuzzy rules by using the clustering centers and membership functions of the filtered feature subspaces, and to establish an emotion recognition model based on the fuzzy rules, the emotion recognition model being configured to perform emotion recognition on an input physiological signal.
[0101] The related technical solutions are the same as those of Embodiment One, and thus are not described herein again.
[0102] Embodiment Three
[0103] An electronic device includes a processor, and a memory storing computer executable programs, the programs, when executed by the processor, causing the processor to perform the emotion recognition method for multi-modal physiological signals as described in Embodiment One.
[0104] The related technical solutions are the same as those of Embodiment One, and thus are not described herein again.
[0105] Embodiment Four
[0106] A computer readable storage medium stores a plurality of instructions, the instructions being adapted to be loaded by a processor to execute the emotion recognition method for multi-modal physiological signals as described in Embodiment One.
[0107] The related technical solutions are the same as those of Embodiment One, and thus are not described herein again.
[0108] Those skilled in the art will easily understand that the above description is only the preferred embodiments of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A method for emotion recognition suitable for multi-modal physiological signals, characterized in that, The method comprises the following steps: S1, feature extraction is performed on original physiological signal data to obtain an original feature space with a feature dimension of D ; randomly select d features from the original feature space, and repeat t times to obtain t feature subspaces; wherein, d D ; S2, performing clustering analysis on each feature subspace to obtain a clustering center and a corresponding membership function; S3, calculating information entropy of all membership functions, and screening feature subspaces corresponding to a plurality of membership functions with minimum entropy values; S4, constructing fuzzy rules by using the clustering center and the membership function of the feature subspace screened in S3, and establishing a sentiment recognition model based on the fuzzy rules, wherein the sentiment recognition model is used for performing sentiment recognition on an input physiological signal; The emotion recognition model of the fuzzy rule is: wherein, W a parameter matrix representing the fuzzy rule, T denotes the transpose, the objective function of which can be represented as: wherein, , Y denotes the true classification label, denotes the squared matrix F-norm, denotes the 1-norm of a matrix, is a regularization term; is determined by the fuzzy antecedent parameters in the fuzzy rule.
2. The method for emotion recognition suitable for multi-modal physiological signals according to claim 1, characterized in that, A loss function for training the sentiment recognition model is represented as: in, W The parameter matrix representing the fuzzy rule. T Indicates transpose; , , , N For the sample size, , , K For the number of rules, Indicates the first k Rule number 1 n Fuzzy antecedent parameters for each sample; Y Indicates the actual category label, Denotes the square of the F-norm of a matrix. Denotes the 1 norm of a matrix. Indicates the first and second hyperparameters. L Indicates the number of label categories. and Representing tags respectively p With tags q The corresponding fuzzy consequent parameters, , Indicates label p With tags q The Pearson correlation coefficient between them.
3. The method for emotion recognition suitable for multi-modal physiological signals according to claim 1, characterized in that, A loss function for training the sentiment recognition model is represented as: in, W The parameter matrix representing the fuzzy rule. T Indicates transpose; , , , N For the sample size, , , K For the number of rules, Indicates the first k Rule number 1 n Fuzzy antecedent parameters for each sample; Y Indicates the actual category label, Denotes the square of the F-norm of a matrix. Denotes the 1 norm of a matrix. Indicates the first and second hyperparameters. L Indicates the number of label categories. and Representing tags respectively p With tags q The corresponding fuzzy consequent parameters, , Indicates label p With tags q The Pearson correlation coefficient between them; This represents the third hyperparameter.
4. The method for emotion recognition suitable for multi-modal physiological signals according to claim 1, characterized in that, In the S3, the information entropy of the membership function is represented as: wherein, denotes the membership function of the n th sample to the i th cluster center, N is the number of samples, c is the number of cluster centers.
5. An emotion recognition device suitable for multi-modal physiological signals, characterized in that, The method comprises the following steps: The feature subspace obtaining module is configured to perform feature extraction on the original physiological signal data to obtain an original feature space with a feature dimension of D ; randomly select d features from the original feature space, and repeat the selection t times to obtain t feature subspaces; and d D A feature subspace clustering module is configured to perform clustering analysis on each feature subspace to obtain a clustering center and a corresponding membership function; A feature subspace screening module is configured to calculate information entropy of all membership functions, and screen feature subspaces corresponding to a plurality of membership functions with minimum entropy values; A sentiment recognition module is configured to construct fuzzy rules by using the clustering center and the membership function of the feature subspace screened, and establish a sentiment recognition model based on the fuzzy rules, wherein the sentiment recognition model is used for performing sentiment recognition on an input physiological signal; The fuzzy rule sentiment recognition model is: wherein, W a parameter matrix representing the fuzzy rule, T denotes the transpose, the objective function of which can be represented as: wherein, , Y denotes the true classification label, denotes the squared matrix F-norm, denotes the 1-norm of a matrix, is a regularization term; is determined by the fuzzy antecedent parameters in the fuzzy rule.
6. An electronic device, comprising: The method comprises the following steps: A processor; A memory storing a computer executable program, wherein the program, when executed by the processor, causes the processor to perform the sentiment recognition method for multi-modal physiological signals according to any one of claims 1-4.
7. A computer readable storage medium characterized in that, The computer readable storage medium stores a plurality of instructions, which are adapted to be loaded by a processor to execute the sentiment recognition method for multi-modal physiological signals according to any one of claims 1-4.