Emotion recognition method and device based on electroencephalogram data, equipment and medium
By initializing and optimizing the target encoder, and utilizing the discriminator and loss calculation function, the problem of recognition difficulties caused by individual differences in EEG data was solved, and efficient emotion recognition was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-05-19
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, emotion recognition methods based on EEG data are difficult to accurately identify emotions without manual labeling due to individual differences and non-stationarity, thus limiting their application in emotion recognition scenarios.
By acquiring the source encoder and classifier trained based on the first type of EEG data and emotion classification results, the target encoder is initialized, and the parameters of the target encoder are optimized using a discriminator and a loss calculation function until the difference loss is minimized, thereby achieving emotion recognition on the second type of EEG data.
Without additional labeling, emotion recognition of another EEG dataset was achieved based on EEG data with known emotion classification results, thus improving recognition efficiency.
Smart Images
Figure CN116610978B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of digital medical technology, and in particular to a method, device, equipment and medium for emotion recognition based on electroencephalogram (EEG) data. Background Technology
[0002] Currently, with the rise of artificial intelligence technology, human emotional states can be identified through facial expressions and gestures. However, these methods often fail to provide accurate results. For example, a sad expression can encompass emotions such as frustration and anger; directly detecting emotions through facial expressions cannot yield entirely accurate results. In the current medical field, identifying patients' emotions can support auxiliary diagnosis and health management.
[0003] Since EEG data consists of spontaneous electrical signals generated by human brain activity and is obtained non-invasively, research on the identification and classification of EEG data can be applied to emotion recognition, command transfer, and other purposes. However, because EEG data is non-stationary and exhibits individual variability, studying the correlation between EEG data and emotion requires emotion reference labeling and calibration of EEG data from different subjects. These operations are time-consuming and labor-intensive, limiting its application in emotion recognition and hindering effective emotion analysis. Therefore, how to accurately identify emotions from EEG data without artificial labeling and promote its application in emotion recognition scenarios has become an urgent problem to be solved. Summary of the Invention
[0004] In view of this, embodiments of this application provide a method, apparatus, device, and medium for emotion recognition based on electroencephalogram (EEG) data, in order to solve the problem of how to accurately recognize emotions from EEG data without artificial labeling.
[0005] In a first aspect, embodiments of this application provide an emotion recognition method based on electroencephalogram (EEG) data, the emotion recognition method comprising:
[0006] Obtain the source encoder and classifier trained based on the first type of EEG data and the corresponding emotion classification results, and initialize the target encoder using the parameters of the source encoder to obtain the initial target encoder;
[0007] The source encoder is used to encode the first type of EEG data and the second type of EEG data to be identified respectively to obtain a first set of encoding results, and the initial target encoder is used to encode the first type of EEG data and the second type of EEG data respectively to obtain a second set of encoding results;
[0008] A preset discriminator is used to determine the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and a discrimination result for the corresponding encoding result is obtained. For any discrimination result, the discrimination result is compared with the encoder used by the corresponding encoding result to obtain a comparison result.
[0009] Based on all comparison results, a preset loss calculation function is used to calculate the difference loss between the source encoder and the initial target encoder in terms of encoding, wherein the difference loss is minimized when all comparison results are the same;
[0010] When the difference loss is not minimized, an adjustment parameter is generated based on the difference loss and the parameters of the initial target encoder to adjust the parameters in the initial target encoder. The adjusted target encoder is used as the initial target encoder. The process of using the source encoder to encode the first type of EEG data and the second type of EEG data to be identified is returned to be executed until the difference loss is minimized. The target encoder corresponding to the minimum difference loss is determined as the optimized target encoder.
[0011] After encoding the second type of EEG data using the optimized target encoder, the encoding result is input into the classifier, and the output classification result is the emotion recognition result of the second type of EEG data.
[0012] Secondly, embodiments of this application provide an emotion recognition device based on electroencephalogram (EEG) data, the emotion recognition device comprising:
[0013] An initialization module is used to obtain a source encoder and a classifier trained based on the first type of EEG data and the corresponding emotion classification results, and to initialize the target encoder using the parameters of the source encoder to obtain an initial target encoder.
[0014] The encoding module is used to encode the first type of EEG data and the second type of EEG data to be identified using the source encoder to obtain a first set of encoding results, and to encode the first type of EEG data and the second type of EEG data using the initial target encoder to obtain a second set of encoding results.
[0015] The discrimination module is used to use a preset discriminator to determine the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, to obtain the discrimination result of the corresponding encoding result, and for any discrimination result, to compare the discrimination result with the encoder used by the corresponding encoding result to obtain a comparison result;
[0016] The loss calculation module is used to calculate the difference loss between the source encoder and the initial target encoder in encoding based on all comparison results and using a preset loss calculation function, wherein the difference loss is minimized when all comparison results are the same;
[0017] The parameter optimization module is used to generate adjustment parameters based on the difference loss and the parameters of the initial target encoder when the difference loss is not minimized, so as to adjust the parameters in the initial target encoder, use the adjusted target encoder as the initial target encoder, return to execute the step of encoding the first type of EEG data and the second type of EEG data to be identified using the source encoder respectively, until the difference loss is minimized, and determine the target encoder corresponding to the minimum difference loss as the optimized target encoder.
[0018] The emotion recognition module is used to encode the second type of EEG data using the optimized target encoder, input the encoding result into the classifier, and output the classification result as the emotion recognition result of the second type of EEG data.
[0019] Thirdly, embodiments of this application provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the emotion recognition method as described in the first aspect.
[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the emotion recognition method as described in the first aspect.
[0021] The beneficial effects of this application embodiment compared with the prior art are as follows: This application obtains a source encoder and a classifier trained based on the first type of EEG data and the corresponding emotion classification result. The parameters of the source encoder are used to initialize the target encoder to obtain an initial target encoder. The first type of EEG data and the second type of EEG data are respectively input into the source encoder and the initial target encoder for encoding to obtain a first encoding result. A discriminator is used to discriminate the encoder to which the first encoding result belongs to, and a discrimination result is obtained. Based on the discrimination result, the parameters of the initial target encoder are optimized until the loss determined by the discrimination result is minimized to obtain an optimized target encoder. The optimized target encoder is used to encode the second type of EEG data to obtain a second encoding result. The second encoding result is input into the classifier, and the output classification result is the emotion recognition result of the corresponding second type of EEG data. Emotion recognition of another type of EEG data can be achieved based on EEG data with a known emotion classification result without the need for other labeling operations, thus improving recognition efficiency. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a schematic diagram of an application environment for an emotion recognition method based on electroencephalogram (EEG) data, as provided in Embodiment 1 of this application.
[0024] Figure 2 This is a flowchart illustrating an emotion recognition method based on electroencephalogram (EEG) data provided in Embodiment 2 of this application;
[0025] Figure 3 This is a schematic diagram of the structure of an emotion recognition device based on electroencephalogram (EEG) data provided in Embodiment 3 of this application;
[0026] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of this application. Detailed Implementation
[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0030] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0032] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0033] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0034] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0035] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0036] To illustrate the technical solution of this application, specific embodiments are described below.
[0037] The emotion recognition method based on electroencephalogram (EEG) data provided in Embodiment 1 of this application can be applied to, for example... Figure 1 In this application environment, the client and server communicate to form a cloud-edge joint system. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, server-side computer equipment, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.
[0038] See Figure 2 This is a flowchart illustrating an emotion recognition method based on electroencephalogram (EEG) data provided in Embodiment 2 of this application. The aforementioned emotion recognition method based on EEG data is applied to... Figure 1 The server-side component connects to a corresponding database to retrieve relevant EEG data. This computer can also connect to a client-side component, allowing the client to collect EEG data for the server. For example... Figure 2 As shown, this emotion recognition method based on EEG data may include the following steps:
[0039] Step S201: Obtain the source encoder and classifier trained based on the first type of EEG data and the corresponding emotion classification results, and initialize the target encoder using the parameters of the source encoder to obtain the initial target encoder.
[0040] In this application, the first type of EEG data can be the collected EEG signals of subject A, and the second type of EEG data can be the collected EEG signals of subject B, or the collected EEG signals of subject A under other physical conditions, such as non-health conditions corresponding to health conditions. The emotion classification result can refer to the result of manually labeling the EEG data according to the cognitive needs of emotions. For example, for a certain segment of EEG signal, if the subject exhibits emotional agitation, the emotion classification result corresponding to that EEG signal can be labeled as emotional agitation.
[0041] Based on the first type of EEG data and the corresponding emotion classification results, the source encoder and classifier can be trained. The source encoder and classifier that are trained when the training meets the iteration conditions or loss conditions are called the trained source encoder and classifier. With the model structure of the source encoder and classifier determined, the first type of EEG data and the corresponding emotion classification results are the training set. The specific training process will not be described in detail.
[0042] The target encoder and the source encoder are encoders that use the same model structure. The target encoder is used to recognize the second type of EEG data. The parameters in the source encoder are used to initialize the target encoder, so that the target encoder can quickly learn the training results of the source encoder. In other words, the target encoder has the ability to recognize the emotions in the first type of EEG data.
[0043] In one embodiment, after acquiring the source encoder and classifier trained based on the first type of EEG data and the corresponding emotion classification results, the method further includes:
[0044] Extract all encoding weight parameters from the source encoder;
[0045] The target encoder is initialized using the parameters of the source encoder to obtain the initial target encoder, which includes:
[0046] Use all the encoding weight parameters in the source encoder to assign values to the parameters at the corresponding positions in the target encoder, and determine the target encoder after the assignment as the initial target encoder.
[0047] Specifically, values are assigned to the encoding weight parameters in the target encoder, while other parameters can be learned during the training of the target encoder, reducing the complexity of initialization.
[0048] In step S202, the source encoder is used to encode the first type of EEG data and the second type of EEG data to be identified to obtain the first set of encoding results, and the initial target encoder is used to encode the first type of EEG data and the second type of EEG data to obtain the second set of encoding results.
[0049] Specifically, in this application, a source encoder is used to encode the first type of EEG data to obtain an encoding result, an initial target encoder is used to encode the first type of EEG data to obtain an encoding result, a source encoder is used to encode the second type of EEG data to be identified to obtain an encoding result, and an initial target encoder is used to encode the second type of EEG data to obtain an encoding result. In this case, after the two types of EEG data are encoded by two encoders, for each encoding result, the EEG data corresponding to the encoding result and the encoder used are recorded.
[0050] Step S203: Use a preset discriminator to determine the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and obtain the discrimination result of the corresponding encoding result. For any discrimination result, compare the discrimination result with the encoder used by the corresponding encoding result to obtain the comparison result.
[0051] If the discriminator is pre-trained, its judgment result on the encoder of the first encoded result will be true; if the discriminator is untrained, it needs to be trained before use, and the training set used for training is the content recorded after encoding with two encoders.
[0052] Optionally, before using a preset discriminator to determine the encoder to which all the encoded results in the first and second sets of encoded results belong, the following steps are also included:
[0053] Freeze the parameters of the source encoder and the initial target encoder, use the initial discriminator to distinguish the first encoding result and the second encoding result, and determine the encoder to which the first encoding result is identified and the encoder to which the second encoding result is identified.
[0054] The system checks whether the encoder to which the first encoding result is classified is consistent with the actual encoder used when the first encoding result is encoded, and also checks whether the encoder to which the second encoding result is classified is consistent with the actual encoder used when the second encoding result is encoded. If they are inconsistent, the parameters of the initial discriminator are adjusted to obtain the adjusted discriminator.
[0055] The adjusted discriminator is used as the initial discriminator. The parameters for freezing the source encoder and the initial target encoder are returned. The initial discriminator is used to distinguish the first encoding result and the second encoding result. The steps are: to determine the encoder to which the first encoding result is judged and the encoder to which the second encoding result is judged.
[0056] When it is detected that the encoder to which the first encoding result belongs is the same as the real encoder used when encoding the first encoding result, and when it is detected that the encoder to which the second encoding result belongs is the same as the real encoder used when encoding the second encoding result, a preset discriminator is obtained.
[0057] The discriminator is a self-trained discriminator, which is trained using data encoded with the encoder using type I and type II EEG data. This discriminator has good adaptability and can change according to the EEG data of different subjects, reducing its limitations.
[0058] By freezing the parameters of the source encoder and the initial target encoder, other conditions remain unchanged during the training of the discriminator, which can effectively ensure the rapid convergence of the discriminator during training and avoid the situation of failure to converge.
[0059] Step S204: Based on all the comparison results, use a preset loss calculation function to calculate the difference loss between the source encoder and the initial target encoder in encoding.
[0060] In this application, if all comparison results are the same, the difference loss is determined to be minimal. That is, when the discriminator can accurately distinguish the encoding results of the source encoder for the first type of EEG data and the second type of EEG data, as well as the encoding results of the target encoder for the first type of EEG data and the second type of EEG data, it is considered that the difference between the source encoder and the target encoder is small, and accurate encoding of the EEG data can be achieved.
[0061] The discrimination result is the discriminator's judgment on the encoding results of the source encoder and the initial target encoder. This discrimination result is used to characterize the difference in feature distribution extracted by the two during encoding. Based on this difference, the target encoder is trained (i.e. optimized) to form adversarial training, so that the difference becomes smaller and smaller, that is, the loss determined by the discrimination result is smaller.
[0062] Training the target encoder using a discriminator can eliminate inter-domain differences. Specifically, the data from subject A and subject B, along with the encoder, generate encoding vectors which are then fed to the discriminator for evaluation. This optimizes the target encoder, ensuring that the features obtained from the target encoder have a distribution that is roughly the same as those obtained from the source encoder.
[0063] Step S205: When the difference loss is not minimized, generate adjustment parameters based on the difference loss and the parameters of the initial target encoder to adjust the parameters in the initial target encoder. Use the adjusted target encoder as the initial target encoder and return to step S202 until the difference loss is minimized. Determine the target encoder corresponding to the minimum difference loss as the optimized target encoder.
[0064] In this application, the difference loss is not minimized, indicating that the target encoder has not yet reached its optimal state. Therefore, based on the difference loss and the original parameters, adjustment parameters can be generated. Based on the condition of minimizing the loss, the parameters of the target encoder can be optimized by gradient backpropagation or gradient descent.
[0065] After adjusting the parameters of the target encoder, return to steps S202 to S204 above until the difference loss is minimized. At this point, the target encoder is the optimized target encoder.
[0066] Optionally, the parameters in the initial target encoder are adjusted using adjustment parameters. The adjusted target encoder is then used as the initial target encoder. The process returns to the step of encoding the first type of EEG data and the second type of EEG data to be identified using the source encoder, respectively, until the difference loss is minimized. The target encoder corresponding to the minimum difference loss is determined as the optimized target encoder, including:
[0067] Freeze the parameters of the source encoder and discriminator;
[0068] Adjust the parameters in the initial target encoder using the adjustment parameters to obtain the adjusted target encoder, and use the adjusted target encoder as the initial target encoder.
[0069] Repeat the following steps until the calculated discrimination loss is minimized, and determine the target encoder corresponding to the minimum discrimination loss as the optimized target encoder:
[0070] The first set of encoding results is obtained by encoding the first type of EEG data and the second type of EEG data to be identified using the source encoder, and the second set of encoding results is obtained by encoding the first type of EEG data and the second type of EEG data using the initial target encoder.
[0071] The preset discriminator is used to determine the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and the discrimination result of the corresponding encoding result is obtained. For any discrimination result, the discrimination result is compared with the encoder used by the corresponding encoding result to obtain the comparison result.
[0072] Based on all the comparison results, the difference loss between the source encoder and the initial target encoder is calculated using a preset loss calculation function.
[0073] In the process of optimizing the parameters of the target encoder, it is necessary to freeze the source encoder and the discriminator.
[0074] Optionally, the source encoder and the target encoder are encoders with the same structure, and the encoder includes N sequentially connected feature extraction units consisting of one-dimensional convolution and a one-layer time series model;
[0075] The first set of encoding results is obtained by encoding the first type of EEG data and the second type of EEG data to be identified using the source encoder, respectively. The second set of encoding results is obtained by encoding the first type of EEG data and the second type of EEG data using the initial target encoder, respectively.
[0076] The first type of EEG data is input into the first feature extraction unit of the source encoder, and the extraction result is output. The step of inputting the extraction result of the previous feature extraction unit into the next feature extraction unit is executed in the order of all feature extraction units until all feature extraction units are executed. The extraction result of the last feature extraction unit is output as a coding result of the first group of coding results.
[0077] In one feature extraction unit, the convolutional layer convolves each EEG data or the extraction result output by the previous feature extraction unit, and outputs the convolution result to the time series model for encoding to obtain the extraction result representing the context relationship;
[0078] The first type of EEG data is input into the first feature extraction unit of the initial target encoder, and the extraction result is output. The step of inputting the extraction result of the previous feature extraction unit into the next feature extraction unit is executed in the order of all feature extraction units until all feature extraction units are executed. The extraction result of the last feature extraction unit is output as a coding result of the second group of coding results.
[0079] The second type of EEG data is input into the first feature extraction unit of the source encoder, and the extraction result is output. The step of inputting the extraction result of the previous feature extraction unit into the next feature extraction unit is executed in the order of all feature extraction units until all feature extraction units are executed. The extraction result of the last feature extraction unit is output as a coding result of the first group of coding results.
[0080] The second type of EEG data is input into the first feature extraction unit of the initial target encoder, and the extraction result is output. The step of inputting the extraction result of the previous feature extraction unit into the next feature extraction unit is executed in the order of all feature extraction units until all feature extraction units are executed. The extraction result of the last feature extraction unit is output as a coding result of the second group of coding results.
[0081] Among them, time series models can refer to models used to extract contextual relationship features. The architecture of such models can adopt long short-term memory models, bidirectional long short-term memory models, and other models that can effectively process time series data.
[0082] For example, the encoder used consists of multiple layers of one-dimensional convolution and multiple layers of time series models connected sequentially. The one-dimensional convolutional layers are responsible for deep feature extraction, and the time series models use long short-term memory, bidirectional long short-term memory (or a combination of both) to encode the parameters of temporal contextual relationships in order to accelerate the optimization process of the target encoder.
[0083] Optionally, based on the difference loss and the parameters of the initial target encoder, the following adjustment parameters are generated:
[0084] Calculate the descent gradient using the gradient descent method and the difference loss;
[0085] The parameters of the initial target encoder are reduced using a descent gradient, and the reduced parameters are used as adjustment parameters.
[0086] Specifically, the parameters of the target encoder are adjusted based on the difference loss to obtain the adjustment parameters, which makes the training process of the target encoder more conducive to convergence.
[0087] Step S206: After encoding the second type of EEG data using an optimized target encoder, the encoding result is input into a classifier, and the classification result is output as the emotion recognition result of the second type of EEG data.
[0088] In this application, for the second type of EEG data, a target encoder and a classifier can be used for classification and recognition. For different subjects, the differences in EEG data are mainly related to encoding, that is, they are highly correlated with feature extraction. The classifier is based on the encoding results. As long as there is no essential difference in the distribution of the encoding results of two different encoders, the classifier can perform classification.
[0089] The classifier can consist of multiple fully connected layers, which are used to decode the encoded vector and output the final classification result.
[0090] For example, a source encoder and classifier are trained using labeled EEG data from subject A. The trained source encoder is then used to initialize the target encoder, with its parameters frozen during adversarial training. The target encoder is initialized with the weights of the source encoder and repeatedly optimized during adversarial training. During training, the source encoder and target encoder receive data from A and B respectively and generate feature codes. The discriminator identifies whether these encoded features belong to the source encoder or the target encoder and minimizes the loss. This reduces the difference in feature distribution between the source and target encoders through adversarial training. The target encoder parameters are frozen when optimizing the discriminator, and the discriminator parameters are frozen when optimizing the target encoder. The trained classifier and target encoder are then used to infer the emotions corresponding to the EEG data from subject B.
[0091] When used in medical settings, the above-mentioned method is implemented on a server that stores reference data (i.e., type I EEG data). When it is necessary to perform emotion recognition on a patient, the patient's EEG data is collected through an EEG device and sent to the server through a client. The server executes the above-mentioned method to process the EEG data to obtain the emotion recognition result and sends the emotion recognition result to the client, so that doctors, patients, and others can observe the emotion recognition result.
[0092] This application embodiment obtains a source encoder and a classifier trained based on first-type EEG data and corresponding emotion classification results. The target encoder is initialized using the parameters of the source encoder to obtain an initial target encoder. First-type and second-type EEG data are respectively input into the source encoder and the initial target encoder for encoding to obtain a first encoding result. A discriminator is used to classify the encoder to which the first encoding result belongs to, obtaining a discrimination result. Based on the discrimination result, the parameters of the initial target encoder are optimized until the loss determined by the discrimination result is minimized, obtaining an optimized target encoder. The optimized target encoder is used to encode the second-type EEG data to obtain a second encoding result. The second encoding result is input into the classifier, and the output classification result is the emotion recognition result corresponding to the second-type EEG data. Emotion recognition of another EEG data can be achieved based on EEG data with a known emotion classification result without the need for other labeling operations, thus improving recognition efficiency.
[0093] Corresponding to the emotion recognition method based on EEG data in the above embodiments, Figure 3 This paper shows a structural block diagram of an emotion recognition device based on electroencephalogram (EEG) data according to Embodiment 3 of this application. The emotion recognition device is applied to... Figure 1 The server-side component connects to a corresponding database to retrieve relevant EEG data. This computer device can also connect to a client-side component, which can also collect EEG data for the server. For ease of explanation, only the parts relevant to the embodiments of this application are shown.
[0094] See Figure 3 The emotion recognition device includes:
[0095] Initialization module 31 is used to obtain the source encoder and classifier trained based on the first type of EEG data and the corresponding emotion classification results, and to initialize the target encoder using the parameters of the source encoder to obtain the initial target encoder;
[0096] Encoding module 32 is used to encode the first type of EEG data and the second type of EEG data to be identified using the source encoder to obtain the first set of encoding results, and to encode the first type of EEG data and the second type of EEG data using the initial target encoder to obtain the second set of encoding results.
[0097] The discrimination module 33 is used to use a preset discriminator to discriminate the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and to obtain the discrimination result of the corresponding encoding result. For any discrimination result, the discrimination result is compared with the encoder used by the corresponding encoding result to obtain the comparison result.
[0098] The loss calculation module 34 is used to calculate the difference loss between the source encoder and the initial target encoder in encoding based on all comparison results and using a preset loss calculation function, wherein the difference loss is minimized when all comparison results are the same.
[0099] The parameter optimization module 35 is used to generate adjustment parameters based on the difference loss and the parameters of the initial target encoder when the difference loss is not minimized, so as to adjust the parameters in the initial target encoder, use the adjusted target encoder as the initial target encoder, return to execute the step of encoding the first type of EEG data and the second type of EEG data to be identified using the source encoder respectively, until the difference loss is minimized, and determine the target encoder corresponding to the minimum difference loss as the optimized target encoder.
[0100] The emotion recognition module 36 is used to encode the second type of EEG data using an optimized target encoder, input the encoding result into a classifier, and output the classification result as the emotion recognition result of the second type of EEG data.
[0101] Optionally, the initialization module 31 includes:
[0102] The parameter extraction unit is used to extract all the encoding weight parameters in the source encoder after obtaining the source encoder and classifier trained based on the first type of EEG data and the corresponding emotion classification results.
[0103] The initialization assignment unit is used to assign values to the parameters at the corresponding positions in the target encoder using all the encoding weight parameters in the source encoder, and to determine the target encoder after assignment as the initial target encoder.
[0104] Optionally, the encoding discrimination module 32 includes:
[0105] The discriminator initial training unit is used to freeze the parameters of the source encoder and the initial target encoder before using the preset discriminator to discriminate the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and to use the initial discriminator to discriminate the first encoding results and the second encoding results to determine the encoder to which the first encoding results are discriminated and the encoder to which the second encoding results are discriminated.
[0106] The encoding result detection unit is used to detect whether the encoder to which the first encoding result is classified is consistent with the real encoder used when the first encoding result is encoded, and to detect whether the encoder to which the second encoding result is classified is consistent with the real encoder used when the second encoding result is encoded. If they are inconsistent, the parameters of the initial discriminator are adjusted to obtain the adjusted discriminator.
[0107] The discriminator optimization training unit is used to take the adjusted discriminator as the initial discriminator, return the parameters for freezing the source encoder and the initial target encoder, use the initial discriminator to discriminate the first encoding result and the second encoding result, and determine the encoder to which the first encoding result is discriminated and the encoder to which the second encoding result is discriminated.
[0108] The discriminator determination unit is used to obtain a preset discriminator when it detects that the encoder to which the first encoding result belongs is consistent with the real encoder used when the first encoding result is encoded, and when it detects that the encoder to which the second encoding result belongs is consistent with the real encoder used when the second encoding result is encoded, it obtains a preset discriminator.
[0109] Optionally, the parameter optimization module 35 includes:
[0110] The gradient calculation unit is used to calculate the descent gradient based on the gradient descent method and the difference loss.
[0111] The parameter adjustment unit is used to descent the parameters of the initial target encoder using the descent gradient, and then use the descent parameters as adjustment parameters.
[0112] Optionally, the parameter optimization module 35 includes:
[0113] Freeze the parameters of the source encoder and discriminator;
[0114] Adjust the parameters in the initial target encoder using the adjustment parameters to obtain the adjusted target encoder, and use the adjusted target encoder as the initial target encoder.
[0115] Repeat the following steps until the calculated discrimination loss is minimized, and determine the target encoder corresponding to the minimum discrimination loss as the optimized target encoder:
[0116] The first set of encoding results is obtained by encoding the first type of EEG data and the second type of EEG data to be identified using the source encoder, and the second set of encoding results is obtained by encoding the first type of EEG data and the second type of EEG data using the initial target encoder.
[0117] The preset discriminator is used to determine the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and the discrimination result of the corresponding encoding result is obtained. For any discrimination result, the discrimination result is compared with the encoder used by the corresponding encoding result to obtain the comparison result.
[0118] Based on all the comparison results, the difference loss between the source encoder and the initial target encoder is calculated using a preset loss calculation function.
[0119] Optionally, the source encoder and the target encoder are encoders with the same structure, and the encoder includes N sequentially connected feature extraction units consisting of one-dimensional convolution and a one-layer time series model;
[0120] Encoding module 32 includes:
[0121] The first encoding unit is used to input the first type of EEG data into the first feature extraction unit of the source encoder, output the extraction result, and execute the step of inputting the extraction result output by the previous feature extraction unit into the next feature extraction unit in the order of all feature extraction units until all feature extraction units have been executed, and output the extraction result of the last feature extraction unit as an encoding result of the first group of encoding results;
[0122] In one feature extraction unit, the convolutional layer convolves each EEG data or the extraction result output by the previous feature extraction unit, and outputs the convolution result to the time series model for encoding to obtain the extraction result representing the context relationship;
[0123] The second encoding unit is used to input the first type of EEG data into the first feature extraction unit of the initial target encoder, output the extraction result, and sequentially execute the step of inputting the extraction result output by the previous feature extraction unit into the next feature extraction unit in the order of all feature extraction units until all feature extraction units have been executed, and output the extraction result of the last feature extraction unit as an encoding result of the second set of encoding results;
[0124] The third encoding unit is used to input the second type of EEG data into the first feature extraction unit of the source encoder, output the extraction result, and execute the step of inputting the extraction result output by the previous feature extraction unit into the next feature extraction unit in the order of all feature extraction units until all feature extraction units have been executed. The extraction result of the last feature extraction unit is output as an encoding result of the first group of encoding results.
[0125] The fourth encoding unit is used to input the second type of EEG data into the first feature extraction unit of the initial target encoder, output the extraction result, and sequentially execute the step of inputting the extraction result output by the previous feature extraction unit into the next feature extraction unit in the order of all feature extraction units, until all feature extraction units have been executed, and output the extraction result of the last feature extraction unit as an encoding result of the second group of encoding results.
[0126] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.
[0127] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of this application. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executed by the processor, implements the steps in any of the above embodiments of the emotion recognition method based on EEG data.
[0128] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.
[0129] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0130] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.
[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.
[0132] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a computer device, it enables the computer device to execute the steps in the above method embodiments.
[0133] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0135] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0137] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. An emotion recognition method based on electroencephalogram (EEG) data, characterized in that, The emotion recognition method includes: Obtain the source encoder and classifier trained based on the first type of EEG data and the corresponding emotion classification results, and initialize the target encoder using the parameters of the source encoder to obtain the initial target encoder; The source encoder is used to encode the first type of EEG data and the second type of EEG data to be identified respectively to obtain a first set of encoding results, and the initial target encoder is used to encode the first type of EEG data and the second type of EEG data respectively to obtain a second set of encoding results; A preset discriminator is used to determine the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and a discrimination result for the corresponding encoding result is obtained. For any discrimination result, the discrimination result is compared with the encoder used by the corresponding encoding result to obtain a comparison result. The discrimination result represents the result of the preset discriminator in determining the encoder to which the encoding result belongs. Based on all comparison results, a preset loss calculation function is used to calculate the difference loss between the source encoder and the initial target encoder in terms of encoding, wherein the difference loss is minimized when all comparison results are the same; When the difference loss is not minimized, an adjustment parameter is generated based on the difference loss and the parameters of the initial target encoder to adjust the parameters in the initial target encoder. The adjusted target encoder is used as the initial target encoder. The process of using the source encoder to encode the first type of EEG data and the second type of EEG data to be identified is returned to be executed until the difference loss is minimized. The target encoder corresponding to the minimum difference loss is determined as the optimized target encoder. After encoding the second type of EEG data using the optimized target encoder, the encoding result is input into the classifier, and the classification result is the emotion recognition result of the second type of EEG data. Before using a preset discriminator to determine the encoder to which all the encoded results in the first group and the second group belong, the process further includes: Freeze the parameters of the source encoder and the initial target encoder, use the initial discriminator to discriminate the first encoding result and the second encoding result, and determine the encoder to which the first encoding result is discriminated and the encoder to which the second encoding result is discriminated; The system detects whether the encoder to which the first encoding result is classified is consistent with the actual encoder used when the first encoding result was encoded, and detects whether the encoder to which the second encoding result is classified is consistent with the actual encoder used when the second encoding result was encoded. If they are inconsistent, the parameters of the initial discriminator are adjusted to obtain the adjusted discriminator. The steps are as follows: using the adjusted discriminator as the initial discriminator, returning the parameters for freezing the source encoder and the initial target encoder, using the initial discriminator to discriminate the first encoding result and the second encoding result, and determining the encoder to which the first encoding result is discriminated and the encoder to which the second encoding result is discriminated; When it is detected that the encoder to which the first encoding result belongs is the same as the real encoder used when the first encoding result was encoded, and when it is detected that the encoder to which the second encoding result belongs is the same as the real encoder used when the second encoding result was encoded, the preset discriminator is obtained.
2. The emotion recognition method according to claim 1, characterized in that, After acquiring the source encoder and classifier trained based on the first type of EEG data and the corresponding emotion classification results, the process also includes: Extract all encoding weight parameters from the source encoder; The target encoder is initialized using the parameters of the source encoder to obtain the initial target encoder, which includes: The parameters at the corresponding positions in the target encoder are assigned values using all the encoding weight parameters in the source encoder, and the target encoder after the assignment is determined as the initial target encoder.
3. The emotion recognition method according to claim 1, characterized in that, The generation of adjustment parameters based on the difference loss and the parameters of the initial target encoder includes: Calculate the descent gradient based on the gradient descent method and the difference loss; The parameters of the initial target encoder are decreased using the descent gradient, and the decreased parameters are used as adjustment parameters.
4. The emotion recognition method according to claim 1, characterized in that, Adjusting the parameters in the initial target encoder using the adjustment parameters, using the adjusted target encoder as the initial target encoder, and returning to execute the step of encoding the first type of EEG data and the second type of EEG data to be identified using the source encoder respectively, until the difference loss is minimized, determining the target encoder corresponding to the minimum difference loss as the optimized target encoder includes: Freeze the parameters of the source encoder and the discriminator; The parameters in the initial target encoder are adjusted using the adjustment parameters to obtain the adjusted target encoder, and the adjusted target encoder is used as the initial target encoder. Repeat the following steps until the calculated discrimination loss is minimized, and determine the target encoder corresponding to the minimum discrimination loss as the optimized target encoder: The source encoder is used to encode the first type of EEG data and the second type of EEG data to be identified respectively to obtain a first set of encoding results, and the initial target encoder is used to encode the first type of EEG data and the second type of EEG data respectively to obtain a second set of encoding results; A preset discriminator is used to determine the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and a discrimination result for the corresponding encoding result is obtained. For any discrimination result, the discrimination result is compared with the encoder used by the corresponding encoding result to obtain a comparison result. Based on all the comparison results, a preset loss calculation function is used to calculate the difference loss between the source encoder and the initial target encoder in terms of encoding.
5. The emotion recognition method according to any one of claims 1 to 4, characterized in that, The source encoder and the target encoder are encoders with the same structure. The encoder includes N sequentially connected feature extraction units consisting of one-dimensional convolution and a one-layer time series model. The first set of encoding results is obtained by encoding the first type of EEG data and the second type of EEG data to be identified using the source encoder, and the second set of encoding results is obtained by encoding the first type of EEG data and the second type of EEG data using the initial target encoder, including: The first type of EEG data is input into the first feature extraction unit of the source encoder, and the extraction result is output. The step of inputting the extraction result output by the previous feature extraction unit into the next feature extraction unit is executed in the order of all feature extraction units until all feature extraction units are executed. The extraction result of the last feature extraction unit is output as an encoding result of the first group of encoding results. In one feature extraction unit, the convolutional layer convolves each EEG data or the extraction result output by the previous feature extraction unit, and outputs the convolution result to the time series model for encoding to obtain the extraction result representing the context relationship; The first type of EEG data is input into the first feature extraction unit of the initial target encoder, and the extraction result is output. The step of inputting the extraction result output by the previous feature extraction unit into the next feature extraction unit is executed in the order of all feature extraction units until all feature extraction units are executed. The extraction result of the last feature extraction unit is output as a coding result of the second group of coding results. The second type of EEG data is input into the first feature extraction unit of the source encoder, and the extraction result is output. The step of inputting the extraction result output by the previous feature extraction unit into the next feature extraction unit is executed in the order of all feature extraction units until all feature extraction units are executed. The extraction result of the last feature extraction unit is output as an encoding result of the first group of encoding results. The second type of EEG data is input into the first feature extraction unit of the initial target encoder, and the extraction result is output. The step of inputting the extraction result output by the previous feature extraction unit into the next feature extraction unit is executed in the order of all feature extraction units until all feature extraction units are executed. The extraction result of the last feature extraction unit is output as a coding result of the second group of coding results.
6. An emotion recognition device based on electroencephalogram (EEG) data, characterized in that, The emotion recognition device includes: An initialization module is used to obtain a source encoder and a classifier trained based on the first type of EEG data and the corresponding emotion classification results, and to initialize the target encoder using the parameters of the source encoder to obtain an initial target encoder. The encoding module is used to encode the first type of EEG data and the second type of EEG data to be identified using the source encoder to obtain a first set of encoding results, and to encode the first type of EEG data and the second type of EEG data using the initial target encoder to obtain a second set of encoding results. The discrimination module is used to use a preset discriminator to discriminate the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and to obtain the discrimination result of the corresponding encoding result. For any discrimination result, the discrimination result is compared with the encoder used by the corresponding encoding result to obtain a comparison result. The discrimination result represents the result of the preset discriminator in discriminating the encoder to which the encoding result belongs. The loss calculation module is used to calculate the difference loss between the source encoder and the initial target encoder in encoding based on all comparison results and using a preset loss calculation function, wherein the difference loss is minimized when all comparison results are the same; The parameter optimization module is used to generate adjustment parameters based on the difference loss and the parameters of the initial target encoder when the difference loss is not minimized, so as to adjust the parameters in the initial target encoder, use the adjusted target encoder as the initial target encoder, return to execute the step of encoding the first type of EEG data and the second type of EEG data to be identified using the source encoder respectively, until the difference loss is minimized, and determine the target encoder corresponding to the minimum difference loss as the optimized target encoder. The emotion recognition module is used to encode the second type of EEG data using the optimized target encoder, input the encoding result into the classifier, and output the classification result as the emotion recognition result of the second type of EEG data. The encoding discrimination module also includes: The discriminator initial training unit is used to freeze the parameters of the source encoder and the initial target encoder before using a preset discriminator to discriminate the encoder to which all the encoding results in the first group of encoding results and the second group of encoding results belong, and to use the initial discriminator to discriminate the first encoding results and the second encoding results to determine the encoder to which the first encoding results are discriminated and the encoder to which the second encoding results are discriminated. The encoding result detection unit is used to detect whether the encoder to which the first encoding result is identified is consistent with the real encoder used when the first encoding result is encoded, and to detect whether the encoder to which the second encoding result is identified is consistent with the real encoder used when the second encoding result is encoded. If they are inconsistent, the parameters of the initial discriminator are adjusted to obtain the adjusted discriminator. The discriminator optimization training unit is used to take the adjusted discriminator as the initial discriminator, return the parameters for freezing the source encoder and the initial target encoder, use the initial discriminator to discriminate the first encoding result and the second encoding result, and determine the encoder to which the first encoding result is discriminated and the encoder to which the second encoding result is discriminated. The discriminator determination unit is used to obtain the preset discriminator when it is detected that the encoder to which the first encoding result belongs is consistent with the real encoder used when the first encoding result is encoded, and when it is detected that the encoder to which the second encoding result belongs is consistent with the real encoder used when the second encoding result is encoded.
7. The emotion recognition device according to claim 6, characterized in that, The initialization module includes: The parameter extraction unit is used to extract all the encoding weight parameters in the source encoder after obtaining the source encoder and classifier trained based on the first type of EEG data and the corresponding emotion classification results. An initialization assignment unit is used to assign values to the parameters at the corresponding positions in the target encoder using all the encoding weight parameters in the source encoder, and to determine the target encoder after assignment as the initial target encoder.
8. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the emotion recognition method as described in any one of claims 1 to 5.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the emotion recognition method as described in any one of claims 1 to 5.