Emotion recognition method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an emotion recognition method, device, equipment and medium, on one hand, emotion recognition is carried out based on electroencephalogram data, and the accuracy of emotion recognition is improved due to the fact that an electroencephalogram has high sensitivity to emotion; on the other hand, emotion recognition is carried out on the basis of an emotion recognition model which comprises a shallow feature extractor, a class decoupler, a domain decoupler and a prototype reasoning layer and is trained on the basis of a pairwise learning mechanism, target domain data and emotion tags are not depended on, and therefore the recognition accuracy is further improved; and the label noise resistance of the model and the robustness of the model are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, device, equipment and medium. Background Art

[0002] As an important physiological signal, emotion is an indispensable tool in human communication, and many studies have shown that there is an inseparable relationship between emotion and mental illness. Human emotions can be identified through language, facial expressions, body movements and physiological signals. Electroencephalography (EEG) is an electrophysiological signal collected directly from the scalp. It records the microvolt voltage fluctuations during brain activity and is more sensitive to emotional changes. It can detect immediate reactions to emotional stimuli and is an important feature for revealing emotional states. Studies have also shown that EEG signals perform better than other physiological signals in emotional assessment.

[0003] In recent years, researchers have developed many EEG-based emotion recognition models. However, due to individual differences in EEG signals, the distribution of EEG signals from different subjects also varies, which does not meet the assumption that the data of different subjects are independent and identically distributed during modeling. As a result, the performance of the model is seriously degraded when it is used on new subjects.

[0004] To solve this problem, researchers introduced transfer learning methods, which have achieved good results in solving the problem of individual differences in EEG signals. However, transfer learning methods require target domain data to participate in the training process during model training, which requires retraining every time the model is used, greatly increasing the cost of using the model. In addition, this type of model relies on high-precision data labels and is less resistant to label noise. Summary of the invention

[0005] In view of the above, it is necessary to provide an emotion recognition method, device, equipment and medium, aiming to solve the problems of inaccurate emotion recognition and the high cost and weak noise resistance of the adopted emotion recognition model.

[0006] An emotion recognition method, the emotion recognition method comprising:

[0007] In response to an emotion recognition instruction for a target user, obtaining the electroencephalogram data of the target user as data to be processed, and obtaining an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer;

[0008] Extracting target shallow features of the data to be processed by using the shallow feature extractor;

[0009] Extracting target class features from the target shallow features using the class decoupler, and extracting target domain features from the target shallow features using the domain decoupler;

[0010] The target class features and the target domain features are input into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

[0011] According to a preferred embodiment of the present invention, before acquiring the emotion recognition model pre-trained based on the paired learning mechanism, the method further includes:

[0012] Constructing an initial model; wherein the initial model also includes a domain discriminator for determining the domain to which the input feature belongs and a class discriminator for determining the class to which the input feature belongs; after processing the class feature extracted by the class decoupler based on the gradient reversal layer, the processed feature is input to the domain discriminator; after processing the domain feature extracted by the domain decoupler based on the gradient reversal layer, the processed feature is input to the class discriminator; the class feature extracted by the class decoupler is input to the class discriminator, and the domain feature extracted by the domain decoupler is input to the domain discriminator;

[0013] Constructing a class discriminator loss function, a domain discriminator loss function and a paired learning loss function, and constructing a model loss function according to the class discriminator loss function, the domain discriminator loss function and the paired learning loss function;

[0014] The initial model is trained based on the model loss function to obtain the emotion recognition model.

[0015] According to a preferred embodiment of the present invention, constructing a class discriminator loss function, a domain discriminator loss function and a pairwise learning loss function comprises:

[0016] The class discriminator loss function is expressed as follows:

[0017]

[0018] Among them, L cls (θ c ) represents the class discriminator parameter θ c The corresponding class discriminator loss; express Binary cross entropy loss between ; Represents the true class label of the i-th sample; Represents the class feature of the i-th sample; express Binary cross entropy loss between ; Represents the domain features of the i-th sample; It represents the feature obtained after the domain feature of the i-th sample is processed by the gradient reversal layer.

[0019] According to a preferred embodiment of the present invention, the construction of the class discriminator loss function, the domain discriminator loss function and the paired learning loss function further includes:

[0020] The domain discriminator loss function is expressed as follows:

[0021]

[0022] Among them, L dom (θ d ) represents the domain discriminator parameter θ d The corresponding domain discriminator loss; express Binary cross entropy loss between ; represents the true domain label of the i-th sample;

[0023] express Binary cross entropy loss between ; Represents the features of the class features of the i-th sample after being processed by the gradient reversal layer.

[0024] According to a preferred embodiment of the present invention, the construction of the class discriminator loss function, the domain discriminator loss function and the paired learning loss function further includes:

[0025] The pairwise learning loss function is expressed as follows:

[0026]

[0027] Among them, L pairwise (θ) represents the pairwise learning loss corresponding to the pairwise learning parameter θ; N represents the total number of samples; L bce (r ij ,g(x i ,x j ; θ)) represents r ij 、g(x i ,x j ; The binary cross entropy loss between θ; g(x i ,x j ; θ) represents the i-th sample x i With the jth sample x j is the similarity measure based on θ; β represents the weight parameter; R represents the soft regularization parameter;

[0028] When the true class label of the i-th sample is the same as the true class label of the j-th sample, r ij =1; when the true class label of the i-th sample is different from the true class label of the j-th sample, rij =0;

[0029] in, l i Represents the output of the prototype inference layer and the sample x i The corresponding eigenvector; l j Represents the output of the prototype inference layer and the sample x j The corresponding eigenvector; l i ·l j Indicates l i With l j The dot product operation between g(x i ,x j ; The value range of θ) is [0,1];

[0030] Where R = || P T PI|| F ; P is a matrix, each row of P represents the domain prototype of the test users belonging to the same source domain in the training sample set; I represents the unit matrix.

[0031] According to a preferred embodiment of the present invention, the inputting the target class feature and the target domain feature into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user includes:

[0032] Obtain the pre-calculated domain prototype of each domain and multiple class prototypes under each domain prototype;

[0033] Calculating the similarity between the target domain feature and each domain prototype based on a bilinear transformation method, and obtaining the domain prototype with the highest similarity to the target domain feature as the target domain prototype;

[0034] Calculating the similarity between the target class feature and each class prototype under the target domain prototype, and obtaining the class prototype with the highest similarity to the target class feature as the target class prototype;

[0035] The predicted emotion type is determined according to the target class prototype.

[0036] According to a preferred embodiment of the present invention, before obtaining the pre-calculated domain prototype of each domain and the multiple class prototypes under each domain prototype, the method further includes:

[0037] The domain prototype μ of the mth test user in the training sample set d It is expressed as follows:

[0038]

[0039] in,

[0040] Among them, Q mrepresents the set of domain features of all samples corresponding to the mth test user in the training sample set; |Q m | represents the total number of all samples corresponding to the mth test user in the training sample set;

[0041] Domain prototype μ d The class prototype μ corresponding to the preset emotion type c It is expressed as follows:

[0042]

[0043] in,

[0044] Among them, W m represents the set of samples classified as the preset emotion type in all samples corresponding to the mth test user in the training sample set; |W m | represents the total number of samples classified as the preset emotion type in all samples corresponding to the mth test user in the training sample set.

[0045] An emotion recognition device, the emotion recognition device comprising:

[0046] An acquisition unit, configured to, in response to an emotion recognition instruction for a target user, acquire the electroencephalogram data of the target user as data to be processed, and acquire an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer;

[0047] An extraction unit, used for extracting target shallow features of the data to be processed by using the shallow feature extractor;

[0048] The extraction unit is further used to extract the target class features in the target shallow features by using the class decoupler, and to extract the target domain features in the target shallow features by using the domain decoupler;

[0049] A processing unit is used to input the target class feature and the target domain feature into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

[0050] A computer device, comprising:

[0051] a memory storing at least one instruction; and

[0052] A processor executes the instructions stored in the memory to implement the emotion recognition method.

[0053] A computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the emotion recognition method.

[0054] It can be seen from the above technical solutions that, on the one hand, emotion recognition is performed based on EEG data, which improves the accuracy of emotion recognition because EEG is highly sensitive to emotions; on the other hand, emotion recognition is performed based on an emotion recognition model that includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer and is trained based on a paired learning mechanism. It does not rely on target domain data and emotion labels, thereby further improving the accuracy of recognition and enhancing the model's ability to resist label noise and the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of a preferred embodiment of the emotion recognition method of the present invention.

[0056] Figure 2 This is the architecture diagram of the shallow feature extractor, class decoupler, domain decoupler, class discriminator, domain discriminator and prototype reasoning layer in the emotion recognition method of the present invention.

[0057] Figure 3 It is a functional module diagram of a preferred embodiment of the emotion recognition device of the present invention.

[0058] Figure 4 It is a schematic diagram of the structure of a computer device of a preferred embodiment of the emotion recognition method of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] like Figure 1 FIG. 1 is a flowchart of a preferred embodiment of the emotion recognition method of the present invention. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0061] The emotion recognition method is applied to one or more computer devices, which are devices that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and whose hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0062] The computer device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0063] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0064] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0065] Among them, Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0067] The network where the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0068] S10, in response to an emotion recognition instruction for a target user, obtaining the electroencephalography (EEG) data of the target user as data to be processed, and obtaining an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer.

[0069] In this embodiment, the emotion recognition instruction can be triggered by relevant personnel according to actual needs, or it can be automatically triggered when it is detected that electroencephalogram data is uploaded to a designated interface.

[0070] The designated interface may be a user interaction interface.

[0071] In this embodiment, before obtaining the emotion recognition model pre-trained based on the paired learning mechanism, the method further includes:

[0072] Constructing an initial model; wherein the initial model also includes a domain discriminator for determining the domain to which the input feature belongs and a class discriminator for determining the class to which the input feature belongs; after processing the class feature extracted by the class decoupler based on the gradient reversal layer, the processed feature is input to the domain discriminator; after processing the domain feature extracted by the domain decoupler based on the gradient reversal layer, the processed feature is input to the class discriminator; the class feature extracted by the class decoupler is input to the class discriminator, and the domain feature extracted by the domain decoupler is input to the domain discriminator;

[0073] Constructing a class discriminator loss function, a domain discriminator loss function and a paired learning loss function, and constructing a model loss function according to the class discriminator loss function, the domain discriminator loss function and the paired learning loss function;

[0074] The initial model is trained based on the model loss function to obtain the emotion recognition model.

[0075] The architecture diagram of the shallow feature extractor, the class decoupler, the domain decoupler, the class discriminator, the domain discriminator and the prototype reasoning layer can be found in Figure 2 .

[0076] In the above embodiment, the EEG feature is regarded as the superposition of two deep features, namely, the superposition of domain-invariant class features and class-invariant domain features. The semantic information contained in the class feature expresses which class the corresponding sample belongs to, and the semantic information contained in the domain feature expresses which domain the corresponding sample belongs to. The distribution difference of EEG signals between different test users can be regarded as the difference in domain features of corresponding samples of different test users, which causes the deviation in the distribution of class features.

[0077] In order to better decouple the two types of features, the above embodiment introduces a domain discriminator and a class discriminator. Among them, the domain discriminator is used to determine which domain the input feature belongs to, and the class discriminator is used to determine which class the input feature belongs to. After receiving the domain feature as input, the domain discriminator can accurately determine the domain to which the domain feature belongs. At the same time, on the contrary, after receiving the domain feature as input, the class discriminator cannot use the domain feature to determine the class to which the sample belongs, that is, the domain feature has been completely decoupled from the class feature, and only contains information related to the domain, not information related to the class. Similarly, the class feature only contains class-related information, not domain-related information. In order to achieve the above purpose, before the class feature enters the domain discriminator and before the domain feature enters the class discriminator, it first passes through the gradient reversal layer (GRL), and adversarial training is achieved in this way. Specifically, the binary cross entropy loss function can be used to optimize the discriminator. The output of the discriminator will first be converted into a probability value through a sigmoid layer, and then compared with the true label, thereby converting the multi-classification problem into multiple independent binary classification problems.

[0078] Specifically, the class discriminator loss function is expressed as follows:

[0079]

[0080] Among them, L cls (θ c ) represents the class discriminator parameter θ c The corresponding class discriminator loss; express Binary cross entropy loss between ; Represents the true class label of the i-th sample; Indicates the i-th to the characteristics.

[0082] Specifically, the domain discriminator loss function is expressed as follows:

[0083]

[0084] Among them, L dom (θ d ) represents the domain discriminator parameter θ d The corresponding domain discriminator loss; express Binary cross entropy loss between ; represents the true domain label of the i-th sample; express Binary cross entropy loss between ; Represents the features of the class features of the i-th sample after being processed by the gradient reversal layer.

[0085] In addition, existing emotion recognition models usually adopt a point learning model, which uses similarity as the predicted sample label and then inputs it into the loss function together with the actual label of the sample for supervised training. This point-by-point learning strategy only focuses on the relationship between a single sample feature and each prototype, without considering the relationship between samples. For example, samples of the same category should be clustered together, and samples of different categories should be far away from each other. Therefore, this learning strategy has high requirements for the accuracy of sample labels. Incorrect labels will mislead the learning of the model and cause the model accuracy to decrease.

[0086] To address this problem, in order to increase the model's resistance to label noise, this embodiment uses a paired learning strategy to replace point-by-point learning. Different from point-by-point learning, the paired learning mechanism considers the relationship between sample pairs and captures the relative relationship between samples through paired calculations. Specifically, the paired learning loss function is expressed as follows:

[0087]

[0088] Among them, L pairwise (θ) represents the pairwise learning loss corresponding to the pairwise learning parameter θ; N represents the total number of samples; L bce (r ij ,g(x i ,x j ; θ)) represents r ij 、g(x i ,x j ; The binary cross entropy loss between θ; g(x i ,x j ; θ) represents the i-th sample x i With the jth sample x j is the similarity measure based on θ; β represents the weight parameter; R represents the soft regularization parameter;

[0089] When the true class label of the i-th sample is the same as the true class label of the j-th sample, r ij =1; when the true class label of the i-th sample is different from the true class label of the j-th sample, r ij =0;

[0090] in, l i Represents the output of the prototype inference layer and the sample x i The corresponding eigenvector; l j Represents the output of the prototype inference layer and the sample x j The corresponding eigenvector; l i ·l j Indicates l i With lj The dot product operation between g(x i ,x j ; The value range of θ) is [0,1];

[0091] Where R = || P T PI|| F ; P is a matrix, each row of P represents the domain prototype of the test users belonging to the same source domain in the training sample set; I represents the unit matrix.

[0092] Among them, r ij The value of is determined according to the sample label, which can improve the stability of the model during training and the generalization ability of the model.

[0093] The pairwise learning loss function is equivalent to a clustering loss. In a metric space, the pairwise learning loss function achieves sample classification by bringing similar samples closer together and separating samples of different classes. Because the pairwise learning loss function takes into account the relationship between samples, it has a stronger ability to resist label noise than the point learning strategy that only considers one data point.

[0094] The purpose of adding soft regularization is to prevent the model from overfitting.

[0095] S11, using the shallow feature extractor to extract target shallow features of the data to be processed.

[0096] In this embodiment, the shallow feature extractor may be a multilayer perceptron (MLP).

[0097] S12, using the class decoupler to extract target class features from the target shallow features, and using the domain decoupler to extract target domain features from the target shallow features.

[0098] In this embodiment, the class decoupler and the domain decoupler may also be multi-layer perceptrons.

[0099] S13, inputting the target class features and the target domain features into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

[0100] In this embodiment, the target class feature and the target domain feature are input into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user, which includes:

[0101] Obtain the pre-calculated domain prototype of each domain and multiple class prototypes under each domain prototype;

[0102] Calculating the similarity between the target domain feature and each domain prototype based on a bilinear transformation method, and obtaining the domain prototype with the highest similarity to the target domain feature as the target domain prototype;

[0103] Calculating the similarity between the target class feature and each class prototype under the target domain prototype, and obtaining the class prototype with the highest similarity to the target class feature as the target class prototype;

[0104] The predicted emotion type is determined according to the target class prototype.

[0105] For example, when the emotion represented by the target class prototype is positive, the predicted emotion type is positive.

[0106] Specifically, before obtaining the pre-calculated domain prototype of each domain and the multiple class prototypes under each domain prototype, the method further includes:

[0107] The domain prototype μ of the mth test user in the training sample set d It is expressed as follows:

[0108]

[0109] in,

[0110] Among them, Q m represents the set of domain features of all samples corresponding to the mth test user in the training sample set; |Q m | represents the total number of all samples corresponding to the mth test user in the training sample set;

[0111] Domain prototype μ d The class prototype μ corresponding to the preset emotion type c It is expressed as follows:

[0112]

[0113] in,

[0114] Among them, W m represents the set of samples classified as the preset emotion type in all samples corresponding to the mth test user in the training sample set; |W m | represents the total number of samples classified as the preset emotion type in all samples corresponding to the mth test user in the training sample set.

[0115] Correspondingly, the formula of the bilinear transformation method can be expressed as follows:

[0116]

[0117] in, express With μ d The similarity between them is calculated based on the bilinear transformation method; Represents the target domain feature; S∈R d×d , is a trainable randomly initialized bilinear transformation matrix that is not affected by positive definiteness or symmetry. The model updates the weights of the bilinear matrix through back propagation.

[0118] Furthermore, after determining the target domain prototype with the highest similarity to the target domain features, various prototypes μ of the domain are selected. c (c=1,2,…,n), compare the similarity between the sample class features and the prototypes. When comparing the class features with the class prototypes, the cosine similarity is used:

[0119]

[0120] Among them, l i represents the similarity between the target domain feature and the i-th domain prototype, d cos (·) indicates cosine similarity calculation.

[0121] Among them, for samples in the same domain, the domain labels are the same.

[0122] It can be seen that for domain features, assuming that there is a prototype representation in each domain, the domain prototype can be obtained through prototype reasoning. The domain prototype represents the representation characteristics of the corresponding domain, and the domain characteristics of the samples in the domain are distributed around the domain prototype. For each domain, the domain prototype can be regarded as the "center of mass" of all domain features. Similarly, for each category in a domain, the corresponding class prototype can be obtained through prototype reasoning. The class prototype also represents the representation characteristics of each category in the entire domain, that is, the "center of mass" of the class features of each category. The prototypes of the two types of features can be calculated as the average value of the corresponding sample features.

[0123] Moreover, during the training, the prototype of each test user is calculated based on the features corresponding to all its samples, and each prototype will be updated during the training process to better represent the features. However, when the model is actually used for emotion recognition, each prototype is fixed.

[0124] In this embodiment, the individual differences of EEG signals are regarded as feature shifts caused by the superposition of domain features and class features. Therefore, the semantic structure of EEG is encoded based on the class decoupler and the domain decoupler, and after obtaining more fine-grained features after feature decoupling, the domain prototype and class prototype of each subject user in the source domain are obtained through the prototype inference layer, which highly summarizes the overall characteristics of the data of a single subject user in the source domain. By comparing the similarity between the sample domain features of the target domain and the prototypes of each domain in the source domain, the distribution that best fits the sample is determined to reduce the impact of individual differences in EEG signals. During the training process of the model, the target domain data is invisible to the model. The model uses a paired learning mechanism to replace the traditional point-by-point learning strategy, regards emotion recognition as a paired learning problem, considers the relationship between sample pairs, and greatly reduces the reliance on high-precision emotion labels, thereby enhancing the model's ability to resist label noise and model robustness.

[0125] A large number of experiments have shown that the emotion recognition method used in this embodiment reduces the model's dependence on target domain data, and without using target domain data, the model accuracy still exceeds the classic deep transfer learning method.

[0126] Specifically, during the experiment, the embodiment was evaluated on the generally recognized public databases SEED and SEED-IV. Three emotion categories were identified on the SEED dataset: negative, neutral, and positive. Four emotions were identified on the SEED-IV dataset: happy, sad, calm, and fear.

[0127] In order to fully evaluate the model performance and make a sufficient comparison with existing methods, this embodiment uses two different cross-validation methods for verification.

[0128] (1) Leave-one-out cross-validation across subjects in a single session.

[0129] This method is the most widely used verification method in EEG-based emotion recognition tasks. Under this method, this embodiment uses the data of a single session of a subject user in the dataset as the target domain, and the data of the single sessions of the remaining subject users as the source domain. Repeat the training and verification so that each subject user is regarded as a target once, and the final result is the average of the performance of all subject users. In order to be consistent with other existing studies, this embodiment only uses the first session in the cross-subject single-session cross-validation. The experimental results are shown in Tables 1 and 2 below:

[0130] Table 1. Leave-one-out cross-validation results for SEED dataset across subjects in a single session

[0131]

[0132] Table 2 Single-session leave-one-out cross-validation results of SEED-IV dataset

[0133]

[0134] Table 1 shows the results of the cross-subject leave-one-out test on the SEED dataset (three-category emotion recognition) for the model of this embodiment, and the results are presented in the form of mean / standard deviation. Table 2 shows the results of the cross-subject leave-one-out test on the SEED-IV dataset (four-category emotion recognition) for the model of this embodiment. The recognition performance of the proposed model on the SEED dataset is 82.88%±05.23%, and the recognition performance on the SEED-IV dataset is 65.15%±10.34%.

[0135] Obviously, on the SEED dataset, compared with traditional machine learning methods and non-deep transfer learning methods, this embodiment has significant performance advantages and also exceeds a series of non-deep transfer learning methods (such as TCA, MMD, KPCA, CORAL, GFK, SA). Compared with the best non-deep transfer learning method (CORAL), this embodiment still improves by 11.40%.

[0136] In addition, it is worth noting that other deep transfer learning methods use target domain data during training, while the target domain data is invisible during the training process of the model in this embodiment. However, this embodiment still obtains similar results to other deep transfer learning methods that use target domain data in this case, and even significantly exceeds some deep transfer learning methods, with a performance improvement of 7.46% compared to DDC and 5.23% compared to MS-MDA.

[0137] Similar results were found on the SEED-IV dataset, and the performance of the model in this embodiment exceeded many deep transfer learning methods that used target domain data.

[0138] (2) Leave-one-out cross-validation across subjects and sessions.

[0139] In order to be closer to the actual application scenario, this embodiment also evaluates the effect of the model on unknown objects and unknown sessions. Similar to the above method, all the session data of one test user in the data set is used as the target domain, and all the session data of the remaining test users is used as the source domain. Repeated training and verification are performed on each data set, so that each test user in the data set is regarded as a target domain once, and the final result is the average of all test users. For specific verification results, please see Tables 3 and 4 below.

[0140] Table 3. Leave-one-out cross-validation results for the SEED dataset across subjects and sessions

[0141]

[0142] Table 4. Leave-one-out cross-validation results for SEED-IV dataset across subjects and sessions

[0143]

[0144] In order to be closer to the actual application scenario of the emotional brain-computer interface, a cross-subject and cross-session leave-one-out cross-validation was also performed to verify the performance of the model for different dates (sessions) and different subject user data. As shown in Tables 3 and 4, the recognition performance of the emotion recognition method proposed in this embodiment on the SEED dataset is 79.34% ± 06.34%, and the recognition performance on the SEED-IV dataset is 63.16% ± 09.03%. Compared with the cross-subject single-session leave-one-out cross-validation, this embodiment not only considers the differences between subjects, but also takes the differences between different sessions into consideration. For EEG-based emotion recognition tasks, this evaluation scheme poses the greatest challenge to the effectiveness of the model. Compared with existing studies, the emotion recognition method proposed in this embodiment still surpasses the classic deep transfer learning method without using target domain data, and has a smaller standard deviation. These results show that the model proposed in this embodiment can still have good performance without using target domain data, and can effectively solve the challenges brought by the differences between subjects and sessions in the EEG-based emotion recognition task, and has good effectiveness and generalization.

[0145] In order to evaluate the performance of the model under label noise and verify the anti-label noise ability and robustness brought by the introduction of paired learning, a certain number of labels in the source domain data are randomly replaced with different labels to simulate the situation where the data labels have label noise in the actual application scenario. Specifically, a cross-subject single-session leave-one-out cross-validation is performed in the SEED dataset, and η% of the labels in the source domain data are replaced with random labels different from the original labels. The model is then trained using the source domain data, and the performance of the model is verified on the target domain data. Since the target domain data is invisible to the model during the model training process, and the target domain data will also be used to evaluate the anti-label noise performance of the subsequent model, we only contaminate the source domain labels. During the verification process, we tested the model performance in five cases where η% values ​​were 0%, 5%, 10%, 20%, and 30%, respectively, and the model accuracy and standard deviation were 82.88%±5.32%, 81.46%±5.54%, 80.32%±6.39%, 79.79%±6.09%, and 79.01%±7.46%, respectively.

[0146] The results show that when the label noise rate η% gradually increases from 5% to 30%, the model performance decreases slightly, and the downward trend is relatively gentle, with a decrease rate of 3.87%. These results show that the model we proposed has good robustness in the face of label noise, and the model performance will not drop dramatically or significantly due to the presence of label noise.

[0147] It can be seen from the above technical solutions that, on the one hand, emotion recognition is performed based on EEG data, which improves the accuracy of emotion recognition because EEG is highly sensitive to emotions; on the other hand, emotion recognition is performed based on an emotion recognition model that includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer and is trained based on a paired learning mechanism. It does not rely on target domain data and emotion labels, thereby further improving the accuracy of recognition and enhancing the model's ability to resist label noise and the robustness of the model.

[0148] like Figure 3 , which is a functional module diagram of a preferred embodiment of the emotion recognition device of the present invention. The emotion recognition device 11 includes an acquisition unit 110, an extraction unit 111, and a processing unit 112. The module / unit referred to in the present invention refers to a series of computer program segments that can be executed by a processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0149] The acquisition unit 110 is used to respond to the emotion recognition instruction of the target user, acquire the electroencephalogram data of the target user as the data to be processed, and acquire the emotion recognition model pre-trained based on the paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer;

[0150] The extraction unit 111 is used to extract the target shallow features of the data to be processed by using the shallow feature extractor;

[0151] The extraction unit 111 is further configured to extract target class features from the target shallow features using the class decoupler, and to extract target domain features from the target shallow features using the domain decoupler;

[0152] The processing unit 112 is used to input the target class feature and the target domain feature into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

[0153] It can be seen from the above technical solutions that, on the one hand, emotion recognition is performed based on EEG data, which improves the accuracy of emotion recognition because EEG is highly sensitive to emotions; on the other hand, emotion recognition is performed based on an emotion recognition model that includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer and is trained based on a paired learning mechanism. It does not rely on target domain data and emotion labels, thereby further improving the accuracy of recognition and enhancing the model's ability to resist label noise and the robustness of the model.

[0154] like Figure 4 The figure is a schematic diagram of the structure of a computer device of a preferred embodiment of the emotion recognition method of the present invention.

[0155] The computer device 1 may include a memory 12 , a processor 13 , and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13 , such as an emotion recognition program.

[0156] Those skilled in the art will appreciate that the schematic diagram is merely an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 may be a bus-type structure or a star-type structure. The computer device 1 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the computer device 1 may also include input and output devices, network access devices, etc.

[0157] It should be noted that the computer device 1 is only an example, and other existing or future electronic products that are suitable for the present invention should also be included in the protection scope of the present invention and included here by reference.

[0158] The memory 12 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 12 may be an internal storage unit of the computer device 1, such as a mobile hard disk of the computer device 1. In other embodiments, the memory 12 may also be an external storage device of the computer device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 1. Further, the memory 12 may also include both an internal storage unit of the computer device 1 and an external storage device. The memory 12 may be used not only to store application software and various types of data installed in the computer device 1, such as the code of the emotion recognition program, etc., but also to temporarily store data that has been output or is to be output.

[0159] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the computer device 1, and uses various interfaces and lines to connect the various components of the entire computer device 1, and executes or executes programs or modules stored in the memory 12 (for example, executing emotion recognition programs, etc.), and calls data stored in the memory 12 to execute various functions of the computer device 1 and process data.

[0160] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned various emotion recognition method embodiments, for example Figure 1 Steps shown.

[0161] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, an extraction unit 111, and a processing unit 112.

[0162] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of the emotion recognition method described in each embodiment of the present invention.

[0163] If the module / unit integrated in the computer device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0164] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0165] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0166] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0167] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by only one straight line, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0168] Although not shown, the computer device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 13 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The computer device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0169] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the computer device 1 and other computer devices.

[0170] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the computer device 1 and to display a visual user interface.

[0171] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0172] It can be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0173] Combination Figure 1 , the memory 12 in the computer device 1 stores a plurality of instructions to implement an emotion recognition method, and the processor 13 can execute the plurality of instructions to implement:

[0174] In response to an emotion recognition instruction for a target user, obtaining the electroencephalogram data of the target user as data to be processed, and obtaining an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer;

[0175] Extracting target shallow features of the data to be processed by using the shallow feature extractor;

[0176] Extracting target class features from the target shallow features using the class decoupler, and extracting target domain features from the target shallow features using the domain decoupler;

[0177] The target class features and the target domain features are input into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

[0178] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0179] It should be noted that the data involved in this case were all obtained legally.

[0180] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0181] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0182] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0184] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0185] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0186] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the present invention can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. An emotion recognition method, characterized in that: The emotion recognition method comprises: In response to an emotion recognition instruction for a target user, obtaining the electroencephalogram data of the target user as data to be processed, and obtaining an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer; Extracting target shallow features of the data to be processed by using the shallow feature extractor; Extracting target class features from the target shallow features using the class decoupler, and extracting target domain features from the target shallow features using the domain decoupler; The target class features and the target domain features are input into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

2. The emotion recognition method according to claim 1, characterized in that: Before acquiring the emotion recognition model pre-trained based on the paired learning mechanism, the method further includes: Constructing an initial model; wherein the initial model also includes a domain discriminator for determining the domain to which the input feature belongs and a class discriminator for determining the class to which the input feature belongs; after processing the class feature extracted by the class decoupler based on the gradient reversal layer, the processed feature is input to the domain discriminator; after processing the domain feature extracted by the domain decoupler based on the gradient reversal layer, the processed feature is input to the class discriminator; the class feature extracted by the class decoupler is input to the class discriminator, and the domain feature extracted by the domain decoupler is input to the domain discriminator; Constructing a class discriminator loss function, a domain discriminator loss function and a paired learning loss function, and constructing a model loss function according to the class discriminator loss function, the domain discriminator loss function and the paired learning loss function; The initial model is trained based on the model loss function to obtain the emotion recognition model.

3. The emotion recognition method according to claim 2, characterized in that: The construction of the class discriminator loss function, the domain discriminator loss function and the paired learning loss function includes: The class discriminator loss function is expressed as follows: Among them, L cls (θ c ) represents the class discriminator parameter θ c The corresponding class discriminator loss; express Binary cross entropy loss between ; Represents the true class label of the i-th sample; Represents the class feature of the i-th sample; express Binary cross entropy loss between ; Represents the domain features of the i-th sample; It represents the feature obtained after the domain feature of the i-th sample is processed by the gradient reversal layer.

4. The emotion recognition method according to claim 3, characterized in that: The constructing of the class discriminator loss function, the domain discriminator loss function and the paired learning loss function also includes: The domain discriminator loss function is expressed as follows: Among them, L dom (θ d ) represents the domain discriminator parameter θ d The corresponding domain discriminator loss; express Binary cross entropy loss between ; represents the true domain label of the i-th sample; express Binary cross entropy loss between ; Represents the features of the class features of the i-th sample after being processed by the gradient reversal layer.

5. The emotion recognition method according to claim 4, characterized in that: The constructing of the class discriminator loss function, the domain discriminator loss function and the paired learning loss function also includes: The pairwise learning loss function is expressed as follows: Among them, L pairwise (θ) represents the pairwise learning loss corresponding to the pairwise learning parameter θ; N represents the total number of samples; L bce (r ij ,g(x i ,x j ; θ)) represents r ij 、g(x i ,x j ; The binary cross entropy loss between θ; g(x i ,x j ; θ) represents the i-th sample x i With the jth sample x j is the similarity measure based on θ; β represents the weight parameter; R represents the soft regularization parameter; Among them, when the true class label of the i-th sample is the same as the true class label of the j-th sample, r ij =1; when the true class label of the i-th sample is different from the true class label of the j-th sample, r ij =0; in, l i Represents the output of the prototype inference layer and the sample x i The corresponding eigenvector; l j Represents the output of the prototype inference layer and the sample x j The corresponding eigenvector; l i ·l j Indicates l i With l j The dot product operation between g(x i ,x j ; The value range of θ) is [0,1]; Where R = || P T PI|| F ; P is a matrix, each row of P represents the domain prototype of the test users belonging to the same source domain in the training sample set; I represents the unit matrix.

6. The emotion recognition method according to claim 5, characterized in that: The inputting the target class feature and the target domain feature into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user includes: Obtain the pre-calculated domain prototype of each domain and multiple class prototypes under each domain prototype; Calculating the similarity between the target domain feature and each domain prototype based on a bilinear transformation method, and obtaining the domain prototype with the highest similarity to the target domain feature as the target domain prototype; Calculating the similarity between the target class feature and each class prototype under the target domain prototype, and obtaining the class prototype with the highest similarity to the target class feature as the target class prototype; The predicted emotion type is determined according to the target class prototype.

7. The emotion recognition method according to claim 6, characterized in that: Before obtaining the pre-calculated domain prototype of each domain and the multiple class prototypes under each domain prototype, the method further includes: The domain prototype μ of the mth test user in the training sample set d It is expressed as follows: in, Among them, Q m represents the set of domain features of all samples corresponding to the mth test user in the training sample set; |Q m | represents the total number of all samples corresponding to the mth test user in the training sample set; Domain prototype μ d The class prototype μ corresponding to the preset emotion type c It is expressed as follows: in, Among them, W m represents the set of samples classified as the preset emotion type in all samples corresponding to the mth test user in the training sample set; |W m | represents the total number of samples classified as the preset emotion type in all samples corresponding to the mth test user in the training sample set.

8. An emotion recognition device, characterized in that: The emotion recognition device comprises: An acquisition unit, configured to, in response to an emotion recognition instruction for a target user, acquire the electroencephalogram data of the target user as data to be processed, and acquire an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer; An extraction unit, used for extracting target shallow features of the data to be processed by using the shallow feature extractor; The extraction unit is further used to extract the target class features in the target shallow features by using the class decoupler, and to extract the target domain features in the target shallow features by using the domain decoupler; A processing unit is used to input the target class features and the target domain features into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

9. A computer device, characterized in that: The computer device comprises: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the emotion recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the emotion recognition method as described in any one of claims 1 to 7.

Images