Emotion recognition method, device, equipment and medium

Abstract

The present invention relates to the field of artificial intelligence technology and provides an emotion recognition method, apparatus, device and medium. On the one hand, emotion recognition is performed based on electroencephalogram (EEG) data. Since EEG is highly sensitive to emotions, the accuracy of emotion recognition is improved. On the other hand, emotion recognition is performed based on an emotion recognition model that includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer and is trained based on a paired learning mechanism. The method does not rely on target domain data and emotion labels, thereby further improving the accuracy of recognition and enhancing the model's ability to resist label noise and the robustness of the model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, device, equipment and medium. Background Art

[0002] Emotion, as an important physiological signal, is an indispensable tool for human communication, and many studies have shown an inseparable relationship between emotions and mental illness. Human emotions can be identified through language, facial expressions, body movements, and physiological signals. Electroencephalography (EEG) is an electrophysiological signal collected directly from the scalp. It records microvolt-level voltage fluctuations during brain activity and is more sensitive to emotional changes. It can detect immediate reactions to emotional stimuli and is an important feature for revealing emotional states. Studies have also shown that EEG signals perform better than other physiological signals in emotional assessment.

[0003] In recent years, researchers have developed many EEG-based emotion recognition models. However, due to individual differences in EEG signals, the distribution of EEG signals from different subjects also varies, which does not meet the assumption that the data of different subject users are independent and identically distributed during modeling. As a result, the performance of the model is seriously degraded when it is used on new subjects.

[0004] To address this issue, researchers have introduced transfer learning methods, which have achieved significant success in addressing individual variability in EEG signals. However, transfer learning methods require target domain data for model training, requiring retraining each time the model is used, significantly increasing its cost. Furthermore, these models rely on highly accurate data labels and are less resistant to label noise. Summary of the Invention

[0005] In view of the above, it is necessary to provide an emotion recognition method, device, equipment and medium to solve the problems of inaccurate emotion recognition and the high cost and weak noise resistance of the emotion recognition model adopted.

[0006] An emotion recognition method, comprising:

[0007] In response to an emotion recognition instruction for a target user, obtaining electroencephalogram data of the target user as data to be processed, and obtaining an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer;

[0008] Extracting target shallow features of the data to be processed using the shallow feature extractor;

[0009] Extracting target class features from the target shallow features using the class decoupler, and extracting target domain features from the target shallow features using the domain decoupler;

[0010] The target class features and the target domain features are input into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

[0011] According to a preferred embodiment of the present invention, before obtaining the emotion recognition model pre-trained based on the paired learning mechanism, the method further includes:

[0012] Constructing an initial model; wherein the initial model also includes a domain discriminator for determining the domain to which the input feature belongs and a class discriminator for determining the class to which the input feature belongs; processing the class features extracted by the class decoupler based on the gradient reversal layer, and inputting the processed features into the domain discriminator; processing the domain features extracted by the domain decoupler based on the gradient reversal layer, and inputting the processed features into the class discriminator; inputting the class features extracted by the class decoupler into the class discriminator, and inputting the domain features extracted by the domain decoupler into the domain discriminator;

[0013] Constructing a class discriminator loss function, a domain discriminator loss function, and a paired learning loss function, and constructing a model loss function based on the class discriminator loss function, the domain discriminator loss function, and the paired learning loss function;

[0014] The initial model is trained based on the model loss function to obtain the emotion recognition model.

[0015] According to a preferred embodiment of the present invention, constructing the class discriminator loss function, the domain discriminator loss function and the pairwise learning loss function includes:

[0016] The class discriminator loss function is expressed as follows:

[0017]

[0018] Among them, L cls (θ c ) represents the class discriminator parameter θ c The corresponding class discriminator loss; express Binary cross entropy loss between ; Represents the true class label of the i-th sample; Represents the class feature of the i-th sample; express Binary cross entropy loss between ; represents the domain feature of the i-th sample; It represents the feature obtained by processing the domain feature of the i-th sample through the gradient reversal layer.

[0019] According to a preferred embodiment of the present invention, constructing the class discriminator loss function, the domain discriminator loss function and the pairwise learning loss function further includes:

[0020] The domain discriminator loss function is expressed as follows:

[0021]

[0022] Among them, L dom (θ d ) represents the domain discriminator parameter θ d The corresponding domain discriminator loss; express Binary cross entropy loss between ; represents the true domain label of the i-th sample;

[0023] express Binary cross entropy loss between ; It represents the feature obtained after the class feature of the i-th sample is processed by the gradient reversal layer.

[0024] According to a preferred embodiment of the present invention, constructing the class discriminator loss function, the domain discriminator loss function and the pairwise learning loss function further includes:

[0025] The pairwise learning loss function is expressed as follows:

[0026]

[0027] Among them, L pairwise (θ) represents the pairwise learning loss corresponding to the pairwise learning parameter θ; N represents the total number of samples; L bce (r ij ,g(x i ,x j ;θ)) represents r ij 、g(x i ,x j ; The binary cross entropy loss between θ); g(x i ,x j ; θ) represents the i-th sample x i With the jth sample x j is the similarity measure based on θ; β represents the weight parameter; R represents the soft regularization parameter;

[0028] Among them, when the true class label of the i-th sample is the same as the true class label of the j-th sample, r ij =1; when the true class label of the i-th sample is different from the true class label of the j-th sample, rij =0;

[0029] in, l i Represents the output of the prototype inference layer and the sample x i The corresponding eigenvector; l j Represents the output of the prototype inference layer and the sample x j The corresponding eigenvector; l i ·l j Indicates l i With l j The dot product operation between g(x i ,x j ; The value range of θ) is [0,1];

[0030] Where R = || P T PI|| F ; P is a matrix, each row of P represents the domain prototype of the test users belonging to the same source domain in the training sample set; I represents the identity matrix.

[0031] According to a preferred embodiment of the present invention, inputting the target class features and the target domain features into the prototype inference layer for processing to obtain the predicted emotion type of the target user includes:

[0032] Get the pre-calculated domain prototype of each domain and multiple class prototypes under each domain prototype;

[0033] Calculating the similarity between the target domain feature and each domain prototype based on a bilinear transformation method, and obtaining the domain prototype with the highest similarity to the target domain feature as the target domain prototype;

[0034] Calculating the similarity between the target class feature and each class prototype under the target domain prototype, and obtaining the class prototype with the highest similarity to the target class feature as the target class prototype;

[0035] The predicted emotion type is determined according to the target class prototype.

[0036] According to a preferred embodiment of the present invention, before obtaining the pre-calculated domain prototype of each domain and the multiple class prototypes under each domain prototype, the method further includes:

[0037] The domain prototype μ of the mth test user in the training sample set d It is expressed as follows:

[0038]

[0039] in,

[0040] Among them, Q mrepresents the set of domain features of all samples corresponding to the mth test user in the training sample set; |Q m | represents the total number of all samples corresponding to the mth test user in the training sample set;

[0041] Domain prototype μ d The class prototype μ corresponding to the preset emotion type c It is expressed as follows:

[0042]

[0043] in,

[0044] Among them, W m represents the set of samples classified as the preset emotion type among all samples corresponding to the mth test user in the training sample set; |W m | represents the total number of samples classified as the preset emotion type in all samples corresponding to the mth test user in the training sample set.

[0045] An emotion recognition device, comprising:

[0046] an acquisition unit, configured to, in response to an emotion recognition instruction for a target user, acquire the target user's electroencephalogram (EEG) data as data to be processed, and acquire an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype inference layer;

[0047] an extraction unit, configured to extract target shallow features of the data to be processed using the shallow feature extractor;

[0048] The extraction unit is further configured to extract target class features from the target shallow features using the class decoupler, and to extract target domain features from the target shallow features using the domain decoupler;

[0049] A processing unit is used to input the target class features and the target domain features into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

[0050] A computer device, comprising:

[0051] a memory storing at least one instruction; and

[0052] A processor executes instructions stored in the memory to implement the emotion recognition method.

[0053] A computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the emotion recognition method.

[0054] It can be seen from the above technical solutions that, on the one hand, emotion recognition is performed based on EEG data, which improves the accuracy of emotion recognition because EEG is highly sensitive to emotions; on the other hand, emotion recognition is performed based on an emotion recognition model that includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype inference layer and is trained based on a paired learning mechanism. It does not rely on target domain data and emotion labels, thereby further improving the accuracy of recognition and enhancing the model's ability to resist label noise and the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of a preferred embodiment of the emotion recognition method of the present invention.

[0056] Figure 2 This is the architecture diagram of the shallow feature extractor, class decoupler, domain decoupler, class discriminator, domain discriminator and prototype reasoning layer in the emotion recognition method of the present invention.

[0057] Figure 3 It is a functional module diagram of a preferred embodiment of the emotion recognition device of the present invention.

[0058] Figure 4 It is a structural diagram of a computer device of a preferred embodiment of the emotion recognition method of the present invention. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the emotion recognition method of the present invention. The order of the steps in the flow chart can be changed and some steps can be omitted according to different requirements.

[0061] The emotion recognition method is applied to one or more computer devices, which are devices that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Their hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0062] The computer device may be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0063] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0064] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0065] Among them, Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0067] The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0068] S10, in response to an emotion recognition instruction for a target user, obtaining the target user's electroencephalography (EEG) data as data to be processed, and obtaining an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype inference layer.

[0069] In this embodiment, the emotion recognition instruction can be triggered by relevant personnel according to actual needs, or it can be automatically triggered when it is detected that electroencephalogram data is uploaded to a designated interface.

[0070] The designated interface may be a user interaction interface.

[0071] In this embodiment, before obtaining the emotion recognition model pre-trained based on the paired learning mechanism, the method further includes:

[0072] Constructing an initial model; wherein the initial model also includes a domain discriminator for determining the domain to which the input feature belongs and a class discriminator for determining the class to which the input feature belongs; processing the class features extracted by the class decoupler based on the gradient reversal layer, and inputting the processed features into the domain discriminator; processing the domain features extracted by the domain decoupler based on the gradient reversal layer, and inputting the processed features into the class discriminator; inputting the class features extracted by the class decoupler into the class discriminator, and inputting the domain features extracted by the domain decoupler into the domain discriminator;

[0073] Constructing a class discriminator loss function, a domain discriminator loss function, and a paired learning loss function, and constructing a model loss function based on the class discriminator loss function, the domain discriminator loss function, and the paired learning loss function;

[0074] The initial model is trained based on the model loss function to obtain the emotion recognition model.

[0075] The architecture diagram of the shallow feature extractor, class decoupler, domain decoupler, class discriminator, domain discriminator and prototype inference layer can be found in Figure 2 .

[0076] In the above embodiment, EEG features are regarded as the superposition of two deep features, namely the superposition of domain-invariant class features and class-invariant domain features. The semantic information contained in the class features expresses which class the corresponding sample belongs to, and the semantic information contained in the domain features expresses which domain the corresponding sample belongs to. The distribution differences of EEG signals between different test users can be regarded as differences in the domain features of the corresponding samples of different test users, which causes the shift in the distribution of class features.

[0077] In order to better decouple the two types of features, the above embodiment introduces a domain discriminator and a class discriminator. Among them, the domain discriminator is used to determine which domain the input feature belongs to, and the class discriminator is used to determine which class the input feature belongs to. After receiving the domain feature as input, the domain discriminator can accurately determine the domain to which the domain feature belongs. At the same time, on the contrary, after receiving the domain feature as input, the class discriminator cannot use the domain feature to determine the class to which the sample belongs, that is, the domain feature has been completely decoupled from the class feature, and only contains information related to the domain, not information related to the class. Similarly, the class feature only contains class-related information, not domain-related information. In order to achieve the above purpose, before the class feature enters the domain discriminator and before the domain feature enters the class discriminator, it first passes through the gradient reversal layer (GRL), and this is used to implement adversarial training. Specifically, the binary cross entropy loss function can be used to optimize the discriminator. The output of the discriminator will first be converted into a probability value through the sigmoid layer, and then compared with the true label, thereby converting the multi-classification problem into multiple independent binary classification problems.

[0078] Specifically, the class discriminator loss function is expressed as follows:

[0079]

[0080] Among them, L cls (θ c ) represents the class discriminator parameter θ c The corresponding class discriminator loss; express Binary cross entropy loss between ; Represents the true class label of the i-th sample; Indicates the i-th

[0081] to the characteristics.

[0082] Specifically, the domain discriminator loss function is expressed as follows:

[0083]

[0084] Among them, L dom (θ d ) represents the domain discriminator parameter θ d The corresponding domain discriminator loss; express Binary cross entropy loss between ; represents the true domain label of the i-th sample; express Binary cross entropy loss between ; It represents the feature obtained after the class feature of the i-th sample is processed by the gradient reversal layer.

[0085] Furthermore, existing emotion recognition models typically employ a point-by-point learning model, which uses similarity as the predicted sample label and then feeds it into a loss function along with the actual sample label for supervised training. This point-by-point learning strategy focuses solely on the relationship between individual sample features and prototypes, without considering the relationships between samples. For example, samples of the same category should cluster together, while samples of different categories should stay apart. Consequently, this learning strategy places high demands on the accuracy of sample labels; incorrect labels can mislead the model's learning and reduce its accuracy.

[0086] To address this issue, and to increase the model's resistance to label noise, this embodiment uses a paired learning strategy to replace point-by-point learning. Unlike point-by-point learning, the paired learning mechanism considers the relationship between sample pairs and captures the relative relationship between samples through paired calculations. Specifically, the paired learning loss function is expressed as follows:

[0087]

[0088] Among them, L pairwise (θ) represents the pairwise learning loss corresponding to the pairwise learning parameter θ; N represents the total number of samples; L bce (r ij ,g(x i ,x j ;θ)) represents r ij 、g(x i ,x j ; The binary cross entropy loss between θ); g(x i ,x j ; θ) represents the i-th sample x i With the jth sample x j is the similarity measure based on θ; β represents the weight parameter; R represents the soft regularization parameter;

[0089] Among them, when the true class label of the i-th sample is the same as the true class label of the j-th sample, r ij =1; when the true class label of the i-th sample is different from the true class label of the j-th sample, r ij =0;

[0090] in, l i Represents the output of the prototype inference layer and the sample x i The corresponding eigenvector; l j Represents the output of the prototype inference layer and the sample x j The corresponding eigenvector; l i·l j Indicates l i With l j The dot product operation between g(x i ,x j ; The value range of θ) is [0,1];

[0091] Where R = || P T PI|| F ; P is a matrix, each row of P represents the domain prototype of the test users belonging to the same source domain in the training sample set; I represents the identity matrix.

[0092] Among them, r ij The value of is determined according to the sample label, which can improve the stability of the model during training and the generalization ability of the model.

[0093] The pairwise learning loss function is equivalent to a clustering loss. In a metric space, the pairwise learning loss function classifies samples by bringing similar samples closer together and separating samples of different classes. Because the pairwise learning loss function takes into account the relationship between samples, it is more resistant to label noise than point learning strategies that only consider a single data point.

[0094] The purpose of adding soft regularization is to prevent the model from overfitting.

[0095] S11, using the shallow feature extractor to extract target shallow features of the data to be processed.

[0096] In this embodiment, the shallow feature extractor may be a multilayer perceptron (MLP).

[0097] S12: Utilize the class decoupler to extract target class features from the target shallow features, and utilize the domain decoupler to extract target domain features from the target shallow features.

[0098] In this embodiment, both the class decoupler and the domain decoupler may be multi-layer perceptrons.

[0099] S13: Input the target class features and the target domain features into the prototype inference layer for processing to obtain the predicted emotion type of the target user.

[0100] In this embodiment, the target class feature and the target domain feature are input into the prototype inference layer for processing to obtain the predicted emotion type of the target user, including:

[0101] Get the pre-calculated domain prototype of each domain and multiple class prototypes under each domain prototype;

[0102] Calculating the similarity between the target domain feature and each domain prototype based on a bilinear transformation method, and obtaining the domain prototype with the highest similarity to the target domain feature as the target domain prototype;

[0103] Calculating the similarity between the target class feature and each class prototype under the target domain prototype, and obtaining the class prototype with the highest similarity to the target class feature as the target class prototype;

[0104] The predicted emotion type is determined according to the target class prototype.

[0105] For example, when the emotion represented by the target class prototype is positive, the predicted emotion type is positive.

[0106] Specifically, before obtaining the pre-calculated domain prototype of each domain and the multiple class prototypes under each domain prototype, the method further includes:

[0107] The domain prototype μ of the mth test user in the training sample set d It is expressed as follows:

[0108]

[0109] in,

[0110] Among them, Q m represents the set of domain features of all samples corresponding to the mth test user in the training sample set; |Q m | represents the total number of all samples corresponding to the mth test user in the training sample set;

[0111] Domain prototype μ d The class prototype μ corresponding to the preset emotion type c It is expressed as follows:

[0112]

[0113] in,

[0114] Among them, W m represents the set of samples classified as the preset emotion type among all samples corresponding to the mth test user in the training sample set; |W m | represents the total number of samples classified as the preset emotion type in all samples corresponding to the mth test user in the training sample set.

[0115] Correspondingly, the formula of the bilinear transformation method can be expressed as follows:

[0116]

[0117] in, express With μ d The similarity between them is calculated based on the bilinear transformation method; Represents the target domain feature; S∈R d×d , is a trainable randomly initialized bilinear transformation matrix that is not affected by positive definiteness or symmetry. The model updates the weights of the bilinear matrix through back propagation.

[0118] Furthermore, after determining the target domain prototype with the highest similarity to the target domain features, various prototypes μ of the domain are selected. c (c=1,2,…,n), compare the similarity between the sample class features and the prototypes. When comparing the class features with the prototypes, the cosine similarity is used:

[0119]

[0120] Among them, l i Indicates the similarity between the target domain feature and the i-th domain prototype, d cos (·) indicates cosine similarity calculation.

[0121] Among them, for samples in the same domain, the domain labels are the same.

[0122] As can be seen, for domain features, assuming that each domain has a prototype representation, we can obtain the domain prototype through prototype inference. The domain prototype represents the representational characteristics of the corresponding domain, and the domain characteristics of samples in that domain are distributed around the domain prototype. For each domain, the domain prototype can be regarded as the "center of mass" of all domain features. Similarly, for each category in a domain, the corresponding class prototype can be obtained through prototype inference. The class prototype also represents the representational characteristics of each category in the entire domain, that is, the "center of mass" of each category's class characteristics. The prototype of two types of features can be calculated as the average of the corresponding sample features.

[0123] Furthermore, during training, each user's prototype is calculated based on the features corresponding to all samples. Each prototype is updated during training to better represent the features. However, when the model is actually used for emotion recognition, each prototype is fixed.

[0124] In this embodiment, the individual differences in EEG signals are regarded as feature offsets caused by the superposition of domain features and class features. Therefore, the semantic structure of EEG is encoded based on the class decoupler and the domain decoupler to obtain more fine-grained features after feature decoupling. Then, the domain prototype and class prototype of each subject user in the source domain are obtained through the prototype inference layer, which highly summarizes the overall characteristics of the data of a single subject user in the source domain. By comparing the similarity between the sample domain features of the target domain and the prototypes of each domain in the source domain, the distribution that best fits the sample is determined to reduce the impact of individual differences in EEG signals. During the training process of the model, the target domain data is invisible to the model. The model uses a paired learning mechanism to replace the traditional point-by-point learning strategy, regards emotion recognition as a paired learning problem, considers the relationship between sample pairs, greatly reduces the dependence on high-precision emotion labels, and thus enhances the model's ability to resist label noise and model robustness.

[0125] A large number of experiments have shown that the emotion recognition method adopted in this embodiment reduces the model's dependence on target domain data, and even without using target domain data, the model accuracy still exceeds that of the classic deep transfer learning method.

[0126] Specifically, during the experiment, this embodiment was evaluated on the widely recognized public databases SEED and SEED-IV. Three emotion categories were identified on the SEED dataset: negative, neutral, and positive. Four emotions were identified on the SEED-IV dataset: happy, sad, calm, and fear.

[0127] In order to fully evaluate the performance of the model and make a sufficient comparison with existing methods, this embodiment adopts two different cross-validation methods for verification.

[0128] (1) Leave-one-out cross-validation across subjects in a single session.

[0129] This method is the most widely used verification method in EEG-based emotion recognition tasks. Under this method, this embodiment uses the data of a single session of a test user in the dataset as the target domain, and the data of the single sessions of the remaining test users as the source domain. Repeat the training and verification so that each test user is regarded as a target once, and the final result is the average of the performance of all test users. To be consistent with other existing studies, this embodiment only uses the first session in the cross-test single session cross-validation. The experimental results are shown in Tables 1 and 2 below:

[0130] Table 1. Leave-one-out cross-validation results for the SEED dataset across subjects in a single session.

[0131]

[0132] Table 2 Single-session leave-one-out cross-validation results of SEED-IV dataset

[0133]

[0134] Table 1 shows the results of a leave-one-out-across-subject test on the SEED dataset (three-category emotion recognition), with the results presented as mean / standard deviation. Table 2 shows the results of a leave-one-out-across-subject test on the SEED-IV dataset (four-category emotion recognition). The proposed model achieved recognition performance of 82.88% ± 05.23% on the SEED dataset and 65.15% ± 10.34% on the SEED-IV dataset.

[0135] Clearly, on the SEED dataset, this embodiment demonstrates significant performance advantages over traditional machine learning methods and non-deep transfer learning methods, surpassing a range of non-deep transfer learning methods (e.g., TCA, MMD, KPCA, CORAL, GFK, and SA). Compared to the state-of-the-art non-deep transfer learning method (CORAL), this embodiment achieves an 11.40% improvement.

[0136] It's also worth noting that other deep transfer learning methods use target domain data during training, whereas this target domain data is invisible during the training of the model in this embodiment. However, this embodiment still achieves similar results to other deep transfer learning methods that use target domain data, and even significantly surpasses some of these methods, achieving a 7.46% performance improvement over DDC and a 5.23% improvement over MS-MDA.

[0137] Similar results were found on the SEED-IV dataset, where the model performance of this embodiment surpassed many deep transfer learning methods that used target domain data.

[0138] (2) Leave-one-out cross-validation across subjects and sessions.

[0139] To better reflect real-world application scenarios, this example also evaluated the model's performance for unknown subjects and sessions. Similar to the previous method, all session data from one test user in the dataset was used as the target domain, and all session data from the remaining test users was used as the source domain. Training and validation were repeated for each dataset, ensuring that each test user in the dataset was considered the target domain once. The final result was the average across all test users. See Tables 3 and 4 below for detailed validation results.

[0140] Table 3. Leave-one-out cross-validation results for the SEED dataset across subjects and sessions.

[0141]

[0142] Table 4. Leave-one-out cross-validation results for the SEED-IV dataset across subjects and sessions.

[0143]

[0144] In order to be closer to the actual application scenarios of emotional brain-computer interfaces, a cross-subject and cross-session leave-one-out cross-validation was also performed to verify the performance of the model for different dates (sessions) and different subject user data. As shown in Tables 3 and 4, the emotion recognition method proposed in this embodiment has a recognition performance of 79.34% ± 06.34% on the SEED dataset and a recognition performance of 63.16% ± 09.03% on the SEED-IV dataset. Compared with the cross-subject single-session leave-one-out cross-validation, this embodiment not only takes into account the differences between subjects, but also takes into account the differences between different sessions. For EEG-based emotion recognition tasks, this evaluation scheme poses the greatest challenge to the effectiveness of the model. Compared with existing studies, the emotion recognition method proposed in this embodiment still surpasses the classic deep transfer learning method without using target domain data, and has a smaller standard deviation. These results show that the model proposed in this embodiment can still achieve good performance without using target domain data. It can effectively address the challenges brought about by the differences between subjects and sessions in the EEG-based emotion recognition task and has good effectiveness and generalization.

[0145] To evaluate the model's performance under label noise and verify its robustness and robustness to label noise brought about by pairwise learning, a certain number of labels in the source domain data are randomly replaced with different labels to simulate label noise in real-world scenarios. Specifically, a cross-subject, single-session, leave-one-out cross-validation is performed on the SEED dataset. η% of the labels in the source domain data are replaced with random labels different from the original labels. The model is then trained on the source domain data and its performance is verified on the target domain data. Since the target domain data is invisible to the model during training and will be used to evaluate the model's robustness to label noise, only the source domain labels are contaminated. During validation, the model performance is tested with η% values ​​of 0%, 5%, 10%, 20%, and 30%, achieving model accuracy and standard deviation of 82.88% ± 5.32%, 81.46% ± 5.54%, 80.32% ± 6.39%, 79.79% ± 6.09%, and 79.01% ± 7.46%, respectively.

[0146] The results show that as the label noise rate η% increases from 5% to 30%, the model performance decreases slightly, and the downward trend is relatively gentle, with a rate of decrease of 3.87%. These results show that our proposed model is robust to label noise and that the model performance does not experience a sharp or significant drop due to the presence of label noise.

[0147] It can be seen from the above technical solutions that, on the one hand, emotion recognition is performed based on EEG data, which improves the accuracy of emotion recognition because EEG is highly sensitive to emotions; on the other hand, emotion recognition is performed based on an emotion recognition model that includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype inference layer and is trained based on a paired learning mechanism. It does not rely on target domain data and emotion labels, thereby further improving the accuracy of recognition and enhancing the model's ability to resist label noise and the robustness of the model.

[0148] like Figure 3 , which is a functional block diagram of a preferred embodiment of an emotion recognition device according to the present invention. The emotion recognition device 11 includes an acquisition unit 110, an extraction unit 111, and a processing unit 112. As used herein, a module / unit refers to a series of computer program segments that can be executed by a processor and perform fixed functions, and is stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0149] The acquisition unit 110 is configured to, in response to an emotion recognition instruction for a target user, acquire the target user's electroencephalogram data as data to be processed, and acquire an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype inference layer;

[0150] The extraction unit 111 is configured to extract target shallow features of the data to be processed using the shallow feature extractor;

[0151] The extraction unit 111 is further configured to extract target class features from the target shallow features using the class decoupler, and to extract target domain features from the target shallow features using the domain decoupler;

[0152] The processing unit 112 is configured to input the target class features and the target domain features into the prototype inference layer for processing to obtain the predicted emotion type of the target user.

[0153] It can be seen from the above technical solutions that, on the one hand, emotion recognition is performed based on EEG data, which improves the accuracy of emotion recognition because EEG is highly sensitive to emotions; on the other hand, emotion recognition is performed based on an emotion recognition model that includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype inference layer and is trained based on a paired learning mechanism. It does not rely on target domain data and emotion labels, thereby further improving the accuracy of recognition and enhancing the model's ability to resist label noise and the robustness of the model.

[0154] like Figure 4 , which is a structural diagram of a computer device of a preferred embodiment of the emotion recognition method of the present invention.

[0155] The computer device 1 may include a memory 12 , a processor 13 , and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13 , such as an emotion recognition program.

[0156] Those skilled in the art will understand that the schematic diagram is merely an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 may have either a bus structure or a star structure. The computer device 1 may also include more or less other hardware or software than shown in the figure, or a different arrangement of components. For example, the computer device 1 may also include input and output devices, network access devices, etc.

[0157] It should be noted that the computer device 1 is only an example. Other existing or future electronic products that are suitable for the present invention should also be included in the scope of protection of the present invention and included here by reference.

[0158] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a mobile hard disk of the computer device 1. In other embodiments, the memory 12 can also be an external storage device of the computer device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 1. Furthermore, the memory 12 can also include both an internal storage unit of the computer device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the computer device 1, such as the code of the emotion recognition program, etc., but can also be used to temporarily store data that has been output or is to be output.

[0159] In some embodiments, the processor 13 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the computer device 1, connecting the various components of the entire computer device 1 using various interfaces and lines. It executes or executes programs or modules stored in the memory 12 (such as executing an emotion recognition program) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0160] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned various emotion recognition method embodiments, such as Figure 1 Steps shown.

[0161] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, an extraction unit 111, and a processing unit 112.

[0162] The above-mentioned integrated unit implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, computer device, or network device, etc.) or a processor to execute the portion of the emotion recognition method described in various embodiments of the present invention.

[0163] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments.

[0164] The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory, etc.

[0165] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0166] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0167] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The figure shows that only one straight line is used, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13.

[0168] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 via a power management device, thereby implementing functions such as charging management, discharging management, and power consumption management through the power management device. The power supply may also include one or more DC or AC power supplies, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be detailed here.

[0169] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the computer device 1 and other computer devices.

[0170] Optionally, the computer device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a display screen or a display unit, and is used to display information processed in the computer device 1 and to display a visual user interface.

[0171] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0172] It will be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation on the computer device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0173] Combine Figure 1 The memory 12 in the computer device 1 stores a plurality of instructions to implement an emotion recognition method, and the processor 13 can execute the plurality of instructions to implement:

[0174] In response to an emotion recognition instruction for a target user, obtaining electroencephalogram data of the target user as data to be processed, and obtaining an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer;

[0175] Extracting target shallow features of the data to be processed using the shallow feature extractor;

[0176] Extracting target class features from the target shallow features using the class decoupler, and extracting target domain features from the target shallow features using the domain decoupler;

[0177] The target class features and the target domain features are input into the prototype reasoning layer for processing to obtain the predicted emotion type of the target user.

[0178] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0179] It should be noted that the data involved in this case were all obtained legally.

[0180] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods.

[0181] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0182] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0183] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0184] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0185] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0186] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the present invention may also be implemented by a single unit or device through software or hardware. Terms such as first and second are used to indicate names and do not imply any particular order.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for emotion recognition, characterized in that: The emotion recognition method comprises: In response to an emotion recognition instruction for a target user, obtaining electroencephalogram data of the target user as data to be processed, and obtaining an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype reasoning layer; Extracting target shallow features of the data to be processed using the shallow feature extractor; Extracting target class features from the target shallow features using the class decoupler, and extracting target domain features from the target shallow features using the domain decoupler; Inputting the target class features and the target domain features into the prototype inference layer for processing to obtain the predicted emotion type of the target user; Wherein, before obtaining the emotion recognition model pre-trained based on the paired learning mechanism, an initial model is constructed; wherein, the initial model also includes a domain discriminator for determining the domain to which the input feature belongs and a class discriminator for determining the class to which the input feature belongs; after processing the class features extracted by the class decoupler based on the gradient reversal layer, the processed features are input into the domain discriminator; after processing the domain features extracted by the domain decoupler based on the gradient reversal layer, the processed features are input into the class discriminator; the class features extracted by the class decoupler are input into the class discriminator, and the domain features extracted by the domain decoupler are input into the domain discriminator; a class discriminator loss function, a domain discriminator loss function and a paired learning loss function are constructed, and a model loss function is constructed based on the class discriminator loss function, the domain discriminator loss function and the paired learning loss function; the initial model is trained based on the model loss function to obtain the emotion recognition model.

2. The emotion recognition method according to claim 1, wherein: The construction of the class discriminator loss function, the domain discriminator loss function and the pairwise learning loss function includes: The class discriminator loss function is expressed as follows: ； in, Represents the class discriminator parameters The corresponding class discriminator loss; express 、 Binary cross entropy loss between ; Represents the true class label of the i-th sample; Represents the class feature of the i-th sample; express 、 Binary cross entropy loss between ; represents the domain feature of the i-th sample; It represents the feature obtained by processing the domain feature of the i-th sample through the gradient reversal layer.

3. The emotion recognition method according to claim 2, wherein: The constructing of the class discriminator loss function, the domain discriminator loss function and the paired learning loss function further includes: The domain discriminator loss function is expressed as follows: ； in, Represents the domain discriminator parameters The corresponding domain discriminator loss; express Binary cross entropy loss between ; represents the true domain label of the i-th sample; express Binary cross entropy loss between ; It represents the feature obtained after the class feature of the i-th sample is processed by the gradient reversal layer.

4. The emotion recognition method according to claim 3, wherein: The constructing of the class discriminator loss function, the domain discriminator loss function and the paired learning loss function further includes: The pairwise learning loss function is expressed as follows: ； in, Represents pairwise learning parameters The corresponding pairwise learning loss; N represents the total number of samples; express Binary cross entropy loss between ; represents the i-th sample With the jth sample Based on Similarity measure; β represents the weight parameter; R represents the soft regularization parameter; Among them, when the true class label of the i-th sample is the same as the true class label of the j-th sample, ; When the true class label of the i-th sample is different from the true class label of the j-th sample, ; in, ; Represents the output of the prototype reasoning layer and the sample The corresponding eigenvector; Represents the output of the prototype reasoning layer and the sample The corresponding eigenvector; express and Dot product operation between them; The value range is ; in, ; P is a matrix, each row of P represents the domain prototype of the test users belonging to the same source domain in the training sample set; I represents the identity matrix.

5. The emotion recognition method according to claim 4, wherein: Inputting the target class features and the target domain features into the prototype inference layer for processing to obtain the predicted emotion type of the target user includes: Get the pre-calculated domain prototype of each domain and multiple class prototypes under each domain prototype; Calculating the similarity between the target domain feature and each domain prototype based on a bilinear transformation method, and obtaining the domain prototype with the highest similarity to the target domain feature as the target domain prototype; Calculating the similarity between the target class feature and each class prototype under the target domain prototype, and obtaining the class prototype with the highest similarity to the target class feature as the target class prototype; The predicted emotion type is determined according to the target class prototype.

6. The emotion recognition method according to claim 5, wherein: Before obtaining the pre-calculated domain prototype of each domain and the multiple class prototypes under each domain prototype, the method further includes: The domain prototype of the mth test user in the training sample set It is expressed as follows: ； in, ; in, represents the set of domain features of all samples corresponding to the mth test user in the training sample set; represents the total number of all samples corresponding to the mth test user in the training sample set; Domain Prototype The class prototype corresponding to the preset emotion type It is expressed as follows: ； in, ; in, represents the set of samples classified as the preset emotion type among all samples corresponding to the mth test user in the training sample set; It represents the total number of samples classified as the preset emotion type in all samples corresponding to the mth test user in the training sample set.

7. An emotion recognition device, characterized in that: The emotion recognition device comprises: an acquisition unit, configured to, in response to an emotion recognition instruction for a target user, acquire the target user's electroencephalogram (EEG) data as data to be processed, and acquire an emotion recognition model pre-trained based on a paired learning mechanism; wherein the emotion recognition model includes a shallow feature extractor, a class decoupler, a domain decoupler, and a prototype inference layer; an extraction unit, configured to extract target shallow features of the data to be processed using the shallow feature extractor; The extraction unit is further configured to extract target class features from the target shallow features using the class decoupler, and to extract target domain features from the target shallow features using the domain decoupler; a processing unit, configured to input the target class features and the target domain features into the prototype inference layer for processing to obtain a predicted emotion type of the target user; Wherein, before obtaining the emotion recognition model pre-trained based on the paired learning mechanism, an initial model is constructed; wherein, the initial model also includes a domain discriminator for determining the domain to which the input feature belongs and a class discriminator for determining the class to which the input feature belongs; after processing the class features extracted by the class decoupler based on the gradient reversal layer, the processed features are input into the domain discriminator; after processing the domain features extracted by the domain decoupler based on the gradient reversal layer, the processed features are input into the class discriminator; the class features extracted by the class decoupler are input into the class discriminator, and the domain features extracted by the domain decoupler are input into the domain discriminator; a class discriminator loss function, a domain discriminator loss function and a paired learning loss function are constructed, and a model loss function is constructed based on the class discriminator loss function, the domain discriminator loss function and the paired learning loss function; the initial model is trained based on the model loss function to obtain the emotion recognition model.

8. A computer device, characterized in that: The computer device comprises: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the emotion recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the emotion recognition method according to any one of claims 1 to 6.

Images