Emotion recognition method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of research and development design and artificial intelligence, is applied to an emotion recognition scene according to an electroencephalogram signal, and relates to an emotion recognition method, device and equipment and a storage medium thereof. Inputting the brain wave shape signal into a pre-trained waveform signal feature vector extraction model based on an improved wav2vec2.0 structure, and outputting a representation vector of the brain wave shape signal; and inputting the representation vector of the brain wave shape signal into a downstream pre-trained emotion classification and recognition model to obtain an emotion classification result output by the emotion classification and recognition model. According to the emotion recognition method, architecture reduction and improvement are carried out on an existing complex structure model, so that after improvement, the model structure becomes simple, the emotion recognition task requirement can still be met, and the emotion recognition efficiency is improved by applying the emotion recognition method to the psychology field or the customer emotion judgment field. Therefore, related personnel are assisted in performing emotion fluctuation recognition on the target object.

Description

Technical Field

[0001] The present application relates to the fields of research and development, design and artificial intelligence technology, and is applied to emotion recognition scenarios based on electroencephalogram signals, and in particular to an emotion recognition method, device, equipment and storage medium thereof. Background Art

[0002] Modern human-computer interaction technology has developed to the point where it can identify a person's emotional state based on physiological signals. Human emotional states can be identified through facial expressions, gestures, etc., but these identification methods often do not give accurate results. For example, a sad expression also includes emotions such as frustration and anger. If emotions are detected directly through facial expressions, it is impossible to accurately extract completely accurate results from them. In recent years, due to its huge application potential in emotion analysis, the use of EEG signals for emotion recognition has received widespread attention.

[0003] At present, the application of emotion recognition based on EEG signals has been affected by factors such as difficulty in data collection, lack of available data or small amount of available data sets. The previous single classifier or parallel multi-classifier model cannot combine the various subtasks of emotion recognition well, making the model interfered by a large number of redundant and irrelevant features and unable to effectively extract high-level semantics. How to design a highly available emotion classification model based on EEG signal data has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an emotion recognition method, device, equipment and storage medium thereof, so as to design a high-usability emotion classification model in combination with EEG signal data.

[0005] In order to solve the above technical problems, the embodiment of the present application provides an emotion recognition method, which adopts the following technical solutions:

[0006] An emotion recognition method comprises the following steps:

[0007] Obtaining an EEG waveform signal to be used for emotion recognition;

[0008] Input the EEG waveform signal into a pre-trained feature vector extraction model, and output a representation vector of the EEG waveform signal, wherein the feature vector extraction model includes a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure;

[0009] The representation vector of the EEG waveform signal is input into a downstream pre-trained emotion classification and recognition model to obtain an emotion classification result output by the emotion classification and recognition model.

[0010] Furthermore, before executing the step of inputting the EEG waveform signal into the pre-trained feature vector extraction model and outputting the representation vector of the EEG waveform signal, the method further includes:

[0011] Acquire a training data set, wherein the training data set includes a first type of training data and a second type of training data, the first type of training data is EEG waveform signal data without annotated emotion categories, and the second type of training data is EEG waveform signal data with annotated emotion categories;

[0012] The training data set is input into the waveform signal feature vector extraction model to be trained based on the improved wav2vec2.0 structure, and the waveform signal feature vector extraction model is trained by self-supervised learning to obtain the pre-trained feature vector extraction model.

[0013] Furthermore, the step of inputting the training data set into the waveform signal feature vector extraction model to be trained based on the improved wav2vec2.0 structure, and training the waveform signal feature vector extraction model by self-supervised learning to obtain the pre-trained feature vector extraction model specifically includes:

[0014] Inputting the first type of training data into the waveform signal feature vector extraction model to be trained based on the improved wav2vec2.0 structure to perform preliminary model training;

[0015] Obtain the corresponding characterization vector extraction results of all EEG waveform signal data of unlabeled emotion categories as the first characterization vector extraction result, and

[0016] Obtaining feature vector extraction parameters of a waveform signal feature vector extraction model that has been preliminarily trained;

[0017] Inputting the second type of training data into the initially trained waveform signal feature vector extraction model to extract the representation vector;

[0018] Obtaining the representation vector extraction results corresponding to all EEG waveform signal data of the labeled emotion categories as the second representation vector extraction result;

[0019] According to the second extraction result of the representation vector and the emotion category labeling results corresponding to all the EEG waveform signal data in the second type of training data, the feature vector extraction parameters are adjusted and optimized to obtain an adjusted and optimized waveform signal feature vector extraction model;

[0020] The adjusted and optimized waveform signal feature vector extraction model is used as the pre-trained feature vector extraction model.

[0021] Furthermore, the step of adjusting and optimizing the feature vector extraction parameters according to the second extraction result of the representation vector and the emotion category labeling results corresponding to all EEG waveform signal data in the second type of training data to obtain an adjusted and optimized waveform signal feature vector extraction model specifically includes:

[0022] Classify and sort the second extraction result of the representation vector based on the emotion category labeling results corresponding to all the EEG waveform signal data in the second type of training data;

[0023] Obtain the classification and arrangement results of the second extraction result of the representation vector on all emotion categories as the first classification and arrangement result;

[0024] According to the first classification and sorting results, identifying the important representation dimensions and non-important representation dimensions corresponding to all emotion categories respectively;

[0025] The feature vector extraction parameters are adjusted and optimized based on the important representation dimensions and the unimportant representation dimensions corresponding to all emotion categories, so as to obtain an adjusted and optimized waveform signal feature vector extraction model.

[0026] Furthermore, before executing the step of inputting the representation vector of the EEG waveform signal into a downstream pre-trained emotion classification and recognition model to obtain the emotion classification result output by the emotion classification and recognition model, the method further includes:

[0027] Using the adjusted and optimized waveform signal feature vector extraction model, extract the representation vector of the first type of training data again to obtain an updated first extraction result of the representation vector;

[0028] Inputting the updated first extraction result of the representation vector into a downstream emotion classification and recognition model to be trained to perform preliminary training of the emotion classification and recognition model;

[0029] Obtaining a classification result output by the downstream emotion classification and recognition model to be trained for the first extraction result of the updated representation vector as an actual classification result;

[0030] According to the first classification and arrangement result, the updated first extraction result of the representation vector is classified and arranged in combination with a preset feature distance algorithm to obtain a second classification and arrangement result;

[0031] Verifying the emotion classification and recognition model that has been preliminarily trained based on the second classification and sorting results and the actual classification results;

[0032] The verified emotion classification and recognition model is obtained as the pre-trained emotion classification and recognition model.

[0033] Furthermore, the step of classifying and arranging the updated first extraction result of the representation vector according to the first classification and arrangement result in combination with a preset feature distance algorithm to obtain a second classification and arrangement result specifically includes:

[0034] Step 701, identifying the corresponding representation vector extraction results of all EEG waveform signal data with marked emotion categories;

[0035] Step 702, identifying the classification domains of all EEG waveform signal data labeled with emotion categories according to the first classification and sorting results;

[0036] Step 703, identifying the extraction results of the representation vectors corresponding to all the EEG waveform signal data without the labeled emotion category from the updated first extraction results of the representation vectors;

[0037] Step 704, selecting any EEG waveform signal data from all EEG waveform signal data with marked emotion categories as the EEG waveform signal data with currently marked emotion categories;

[0038] Step 705, selecting any EEG waveform signal data from all EEG waveform signal data of unlabeled emotion categories as the EEG waveform signal data of the current unlabeled emotion category;

[0039] Step 706, using the Euclidean distance algorithm to calculate the characteristic distance value between the EEG waveform signal data of the current unlabeled emotion category and the EEG waveform signal data of the current labeled emotion category on the representation vector extraction result;

[0040] Step 707, if the characteristic distance value satisfies a preset characteristic distance threshold, the EEG waveform signal data that is not currently labeled with an emotion category is classified into the classification domain where the EEG waveform signal data that is currently labeled with an emotion category is located;

[0041] Step 708, repeatedly executing steps 704 to 707 until characteristic distance values ​​of all EEG waveform signal data of unlabeled emotion categories and all EEG waveform signal data of labeled emotion categories are calculated on the representation vector extraction results, and all EEG waveform signal data of unlabeled emotion categories are classified into classification domains, and then the repeated execution is stopped;

[0042] Step 709: Using the final classification domain division result as the second classification sorting result.

[0043] Furthermore, the step of verifying the emotion classification and recognition model that has been preliminarily trained based on the second classification and sorting results and the actual classification results specifically includes:

[0044] Calculating the consistency between the actual classification result and the second classification result;

[0045] If the consistency does not meet the preset consistency threshold, the emotion classification parameters of the emotion classification recognition model are updated and optimized, and the updated and optimized emotion classification recognition model is used to re-obtain the actual classification result, until the consistency between the re-obtained actual classification result and the second classification sorting result meets the preset consistency threshold, and the updating and optimization is stopped;

[0046] If the consistency meets a preset consistency threshold, the pre-training of the downstream emotion classification and recognition model to be trained is completed.

[0047] In order to solve the above technical problems, the embodiment of the present application also provides an emotion recognition device, which adopts the following technical solution:

[0048] An emotion recognition device, comprising:

[0049] An EEG waveform signal acquisition module is used to acquire EEG waveform signals to be used for emotion recognition;

[0050] A feature vector extraction module, used for inputting the EEG waveform signal into a pre-trained feature vector extraction model, and outputting a representation vector of the EEG waveform signal, wherein the feature vector extraction model includes a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure;

[0051] The emotion classification processing module is used to input the representation vector of the EEG waveform signal into the downstream pre-trained emotion classification recognition model to obtain the emotion classification result output by the emotion classification recognition model.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0053] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned emotion recognition method when executing the computer-readable instructions.

[0054] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0055] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the emotion recognition method described above.

[0056] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0057] The emotion recognition method described in the embodiment of the present application obtains the EEG waveform signal to be subjected to emotion recognition; inputs the EEG waveform signal into a waveform signal feature vector extraction model based on the improved wav2vec2.0 structure that has been pre-trained, and outputs a representation vector of the EEG waveform signal; inputs the representation vector of the EEG waveform signal into a downstream pre-trained emotion classification and recognition model to obtain the emotion classification result output by the emotion classification and recognition model. The emotion recognition method disclosed in the present application utilizes a waveform signal feature vector extraction model based on the improved wav2vec2.0 structure to reduce and improve the architecture of the existing complex structure model, so that after the improvement, the model structure becomes simple and can still meet the recognition task requirements. Moreover, the model training is performed using a self-supervised learning method, which reduces the workload of manual labeling, and the emotion recognition method is applied to the field of psychology or the field of customer emotion judgment to assist relevant personnel in identifying emotional fluctuations of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0060] Figure 2 is a flow chart of an embodiment of an emotion recognition method according to the present application;

[0061] Figure 3 is a flowchart of a specific embodiment of pre-training a feature vector extraction model in the emotion recognition method described in the present application;

[0062] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 302 is shown;

[0063] Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 406 is shown;

[0064] Figure 6 It is a flowchart of a specific embodiment of pre-training a downstream emotion classification and recognition model in the emotion recognition method described in the present application;

[0065] Figure 7 yes Figure 6A flowchart of a specific embodiment of step 604 is shown;

[0066] Figure 8 yes Figure 6 A flowchart of a specific embodiment of step 605 is shown;

[0067] Fig. 9 is a schematic structural diagram of an embodiment of an emotion recognition device according to the present application;

[0068] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0070] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0071] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0072] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0073] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0074] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0075] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0076] It should be noted that the emotion recognition method provided in the embodiment of the present application is generally executed by a server, and accordingly, the emotion recognition device is generally set in the server.

[0077] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0078] Continue to refer Figure 2 , shows a flow chart of an embodiment of an emotion recognition method according to the present application. The emotion recognition method comprises the following steps:

[0079] Step 201, obtaining an EEG waveform signal for emotion recognition.

[0080] In this embodiment, the target object's emotions are identified through EEG waveform signals. Specifically, the emotion recognition method can be applied to the field of psychology or the field of customer emotion judgment to assist relevant personnel in identifying the target object's emotion fluctuations.

[0081] Specifically, the emotion recognition method can be applied to the insurance business field. For example, during insurance customer negotiations or business conversations between agents and customers, the emotion recognition method can be used to assist agents or staff in identifying customer emotional fluctuations, so as to better assist insurance business work and improve work efficiency.

[0082] Step 202, input the EEG waveform signal into a pre-trained feature vector extraction model, and output a representation vector of the EEG waveform signal, wherein the feature vector extraction model includes a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure.

[0083] In this embodiment, the improved wav2vec2.0 structure has the number of transformer encoders reduced from 12 to 3 compared to the original wav2vec2.0 structure, and the three transformer encoders form a cascade structure. The output of each transformer encoder is used as the input of the multi-head attention mechanism to construct a three-layer classification dimension attention network.

[0084] Specifically, the feature vector extraction model can be used to extract feature vectors from the EEG waveform signals to be subjected to emotion recognition. When the final emotion classification result is output, three levels of classification results are output. The three levels can be freely set according to actual classification requirements. For example, at the first level, the emotions are classified into happiness, anger, fear and sadness according to their nature; at the second level, the emotions are classified into mood, passion and stress according to their state; at the third level, the emotions are classified into positive emotions and negative emotions according to their positivity. This achieves multi-level, i.e., multi-dimensional, emotion classification based on EEG waveform signals, and simultaneously identifies whether the target object's emotions are positive, whether they are stressful emotions, and the corresponding emotion description words.

[0085] Step 203: input the representation vector of the EEG waveform signal into a downstream pre-trained emotion classification and recognition model to obtain an emotion classification result output by the emotion classification and recognition model.

[0086] It should be understood that, since the EEG waveform signal is a series of continuous signals, the emotion classification result outputted in step 203 corresponds to different emotion categories at different time points, that is, there may be certain emotion fluctuations.

[0087] In this embodiment, the EEG waveform signal to be subjected to emotion recognition is obtained; the EEG waveform signal is input into a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure that has been pre-trained, and a representation vector of the EEG waveform signal is output; the representation vector of the EEG waveform signal is input into a downstream pre-trained emotion classification and recognition model, and an emotion classification result output by the emotion classification and recognition model is obtained. The emotion recognition method disclosed in the present application uses a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure to reduce and improve the architecture of the existing complex structure model, so that after the improvement, the model structure becomes simple and can still meet the recognition task requirements. Moreover, the model training is carried out in a self-supervised learning manner, which reduces the workload of manual labeling. The emotion recognition method is applied to the field of psychology or the field of customer emotion judgment to assist relevant personnel in identifying emotional fluctuations of the target object.

[0088] Continue to refer Figure 3 In some optional implementations, a step of pre-training a feature vector extraction model is also included before step 202. Figure 3 This is a flowchart of a specific embodiment of pre-training a feature vector extraction model in the emotion recognition method described in the present application, comprising the following steps:

[0089] Step 301, obtaining a training data set, wherein the training data set includes a first type of training data and a second type of training data, wherein the first type of training data is EEG waveform signal data without annotated emotion categories, and the second type of training data is EEG waveform signal data with annotated emotion categories;

[0090] Step 302, input the training data set into the waveform signal feature vector extraction model to be trained based on the improved wav2vec2.0 structure, and use self-supervised learning to train the waveform signal feature vector extraction model to obtain the pre-trained feature vector extraction model.

[0091] In this embodiment, since the collection of EEG waveform signal data itself has a certain degree of complexity, the labeled EEG waveform signal data in the actual data set is less. Therefore, in order to meet the needs of learning and training, the unlabeled EEG waveform signal data can be amplified or resampled to expand the data volume of the training data set.

[0092] By adopting a self-supervised learning method to learn and train the waveform signal feature vector extraction model, the pre-trained feature vector extraction model is obtained. There is no need for a large amount of labeling before training, and unlabeled EEG waveform signal data is directly used for learning and training, which reduces the human workload to a certain extent.

[0093] Furthermore, by using the waveform signal feature vector extraction model based on the improved wav2vec2.0 structure, compared with the existing wav2vec2.0 structure waveform signal feature vector extraction model, the number of transformer encoders is reduced from 12 to 3, which improves the learning and training speed to a certain extent. In addition, due to the small amount of EEG signal data, only the waveform signal feature vector extraction model of the improved wav2vec2.0 structure is required to realize emotion recognition. The existing complex structure model is architecturally reduced and improved, so that after the improvement, the model structure becomes simple and can still meet the recognition task requirements.

[0094] Continue to refer Figure 4 , Figure 4 yes Figure 3 The flowchart of a specific embodiment of step 302 shown includes the following steps:

[0095] Step 401, inputting the first type of training data into the waveform signal feature vector extraction model to be trained based on the improved wav2vec2.0 structure to perform preliminary model training;

[0096] Step 402, obtaining the characterization vector extraction results corresponding to all the EEG waveform signal data of unlabeled emotion categories as the first characterization vector extraction result, and

[0097] Step 403, obtaining feature vector extraction parameters of the waveform signal feature vector extraction model that has been preliminarily trained;

[0098] Step 404, inputting the second type of training data into the initially trained waveform signal feature vector extraction model to extract the representation vector;

[0099] Step 405, obtaining the characterization vector extraction results corresponding to all the EEG waveform signal data marked with emotion categories as the second characterization vector extraction result;

[0100] Step 406, adjusting and optimizing the feature vector extraction parameters according to the second extraction result of the representation vector and the emotion category labeling results corresponding to all the EEG waveform signal data in the second type of training data, to obtain an adjusted and optimized waveform signal feature vector extraction model;

[0101] Step 407: Using the adjusted and optimized waveform signal feature vector extraction model as the pre-trained feature vector extraction model.

[0102] Specifically, after preliminary training of the waveform signal feature vector extraction model based on the improved wav2vec2.0 structure using EEG waveform signal data of unlabeled emotion categories, the feature vector extraction parameters of the model after preliminary training are extracted. Then, a small amount of labeled EEG waveform signal data is used to perform parameter tuning. The entire feature vector extraction model learning and training process includes a process of preliminary training first and then adjusting and optimizing the feature vector extraction parameters, so that the feature vector extraction model completed by subsequent pre-training is adjusted in combination with the labeled EEG waveform signal data, thereby ensuring the scientificity and usability of the pre-trained feature vector extraction model.

[0103] Continue to refer Figure 5 , Figure 5 yes Figure 4 The flowchart of a specific embodiment of step 406 shown includes the following steps:

[0104] Step 501, classifying and sorting the second extraction result of the representation vector based on the emotion category labeling results corresponding to all the EEG waveform signal data in the second type of training data;

[0105] Step 502, obtaining the classification and arrangement results of the second extraction result of the representation vector on all emotion categories as the first classification and arrangement result;

[0106] Step 503: identifying important representation dimensions and non-important representation dimensions corresponding to all emotion categories according to the first classification and sorting results;

[0107] Step 504: Adjust and optimize the feature vector extraction parameters based on the important representation dimensions and the unimportant representation dimensions corresponding to all emotion categories, and obtain an adjusted and optimized waveform signal feature vector extraction model.

[0108] Specifically, when using a small amount of labeled EEG waveform signal data for parameter tuning, first, identify the emotion category labeling results corresponding to all EEG waveform signal data, classify and organize the second extraction results of the representation vector, and identify the important representation dimensions and unimportant representation dimensions corresponding to all emotion categories based on the classification and organization results. Finally, adjust and optimize the feature vector extraction parameters based on the important representation dimensions and unimportant representation dimensions corresponding to all emotion categories. This ensures that the pre-trained feature vector extraction model combines the important representation dimensions and unimportant representation dimensions corresponding to all emotion categories, thereby ensuring the availability of the model.

[0109] Continue to refer Figure 6 In some optional implementations, a step of pre-training a downstream emotion classification and recognition model is also included before step 203. Figure 6 This is a flowchart of a specific embodiment of pre-training a downstream emotion classification and recognition model in the emotion recognition method described in the present application, comprising the following steps:

[0110] Step 601, using the adjusted and optimized waveform signal feature vector extraction model, extracting the representation vector of the first type of training data again to obtain an updated first extraction result of the representation vector;

[0111] Step 602, inputting the updated first extraction result of the representation vector into a downstream emotion classification and recognition model to be trained, and performing preliminary training of the emotion classification and recognition model;

[0112] Step 603, obtaining the classification result output by the downstream emotion classification and recognition model to be trained for the first extraction result of the updated representation vector as the actual classification result;

[0113] Step 604, classifying and sorting the updated first extraction result of the representation vector according to the first classification and sorting result in combination with a preset feature distance algorithm to obtain a second classification and sorting result;

[0114] Step 605, verifying the emotion classification and recognition model that has been preliminarily trained based on the second classification and sorting results and the actual classification results;

[0115] Step 606: Obtain the verified emotion classification and recognition model as the pre-trained emotion classification and recognition model.

[0116] Specifically, on the one hand, the waveform signal feature vector extraction model after adjustment and optimization is used to extract the representation vector of the first type of training data again to obtain the updated first extraction result of the representation vector, and the first extraction result of the representation vector is used to perform preliminary training on the emotion classification and recognition model to obtain the actual classification result output by the model. On the other hand, according to the first classification and sorting result, the updated first extraction result of the representation vector is classified and sorted in combination with a preset feature distance algorithm to obtain a second classification and sorting result. Since the first classification and sorting result is a classification of the labeled EEG signal data, a verification method is subsequently used to verify the consistency between the actual classification result and the second classification and sorting result, so as to obtain the emotion classification and recognition model after successful verification as the pre-trained emotion classification and recognition model. The availability of the downstream emotion classification and recognition model is guaranteed.

[0117] Continue to refer Figure 7 , Figure 7 yes Figure 6 The flowchart of a specific embodiment of step 604 shown includes the following steps:

[0118] Step 701, identifying the corresponding representation vector extraction results of all EEG waveform signal data with marked emotion categories;

[0119] Step 702, identifying the classification domains of all EEG waveform signal data labeled with emotion categories according to the first classification and sorting results;

[0120] Step 703, identifying the extraction results of the representation vectors corresponding to all the EEG waveform signal data without the labeled emotion category from the updated first extraction results of the representation vectors;

[0121] Step 704, selecting any EEG waveform signal data from all EEG waveform signal data with marked emotion categories as the EEG waveform signal data with currently marked emotion categories;

[0122] Step 705, selecting any EEG waveform signal data from all EEG waveform signal data of unlabeled emotion categories as the EEG waveform signal data of the current unlabeled emotion category;

[0123] Step 706, using the Euclidean distance algorithm to calculate the characteristic distance value between the EEG waveform signal data of the current unlabeled emotion category and the EEG waveform signal data of the current labeled emotion category on the representation vector extraction result;

[0124] Step 707, if the characteristic distance value satisfies a preset characteristic distance threshold, the EEG waveform signal data that is not currently labeled with an emotion category is classified into the classification domain where the EEG waveform signal data that is currently labeled with an emotion category is located;

[0125] Step 708, repeatedly executing steps 704 to 707 until characteristic distance values ​​of all EEG waveform signal data of unlabeled emotion categories and all EEG waveform signal data of labeled emotion categories are calculated on the representation vector extraction results, and all EEG waveform signal data of unlabeled emotion categories are classified into classification domains, and then the repeated execution is stopped;

[0126] Step 709: Using the final classification domain division result as the second classification sorting result.

[0127] Specifically, a one-by-one comparison method was adopted, which fully considered that unlabeled EEG waveform signal data may hit multiple classification domains, ensuring the scientificity and comprehensiveness of the classification and organization, and ultimately ensuring the availability of the downstream emotion classification and recognition model.

[0128] Continue to refer Figure 8 , Figure 8 yes Figure 6 The flowchart of a specific embodiment of step 605 shown includes the following steps:

[0129] Step 801, calculating the consistency between the actual classification result and the second classification result;

[0130] Step 802: if the consistency does not meet the preset consistency threshold, the emotion classification parameters of the emotion classification recognition model are updated and optimized, and the updated and optimized emotion classification recognition model is used to re-obtain the actual classification result, until the consistency between the re-obtained actual classification result and the second classification result meets the preset consistency threshold, and the updating and optimization is stopped;

[0131] Step 803: If the consistency meets a preset consistency threshold, the pre-training of the downstream emotion classification and recognition model to be trained is completed.

[0132] In this embodiment, after calculating the consistency between the actual classification result and the second classification and sorting result, the emotion classification parameters of the emotion classification and recognition model can be continuously updated and optimized in combination with a preset consistency threshold. The consistency threshold can be a similarity value. Of course, the higher the consistency threshold, the closer the actual classification result is to the second classification and sorting result, and the more accurate the emotion recognition result of the emotion classification and recognition model is.

[0133] The present application obtains the EEG waveform signal to be subjected to emotion recognition; inputs the EEG waveform signal into a waveform signal feature vector extraction model based on the improved wav2vec2.0 structure that has been pre-trained, and outputs a representation vector of the EEG waveform signal; inputs the representation vector of the EEG waveform signal into a downstream pre-trained emotion classification and recognition model, and obtains the emotion classification result output by the emotion classification and recognition model. The emotion recognition method disclosed in the present application utilizes a waveform signal feature vector extraction model based on the improved wav2vec2.0 structure to reduce and improve the architecture of the existing complex structure model, so that after the improvement, the model structure becomes simple and can still meet the recognition task requirements. Moreover, the model training is performed using a self-supervised learning method, which reduces the workload of manual labeling, and the emotion recognition method is applied to the field of psychology or the field of customer emotion judgment to assist relevant personnel in identifying emotional fluctuations of the target object.

[0134] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0135] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0136] In an embodiment of the present application, the EEG waveform signal to be subjected to emotion recognition is obtained; the EEG waveform signal is input into a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure that has been pre-trained, and a representation vector of the EEG waveform signal is output; the representation vector of the EEG waveform signal is input into a downstream pre-trained emotion classification and recognition model, and an emotion classification result output by the emotion classification and recognition model is obtained. The emotion recognition method disclosed in the present application utilizes a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure to reduce and improve the architecture of the existing complex structure model, so that after the improvement, the model structure becomes simple and can still meet the recognition task requirements. Moreover, the model training is performed using a self-supervised learning method, which reduces the workload of manual labeling, and the emotion recognition method is applied to the field of psychology or the field of customer emotion judgment to assist relevant personnel in identifying emotional fluctuations of the target object.

[0137] Further references Fig. 9 , as a response to the above Figure 2 The present application provides an embodiment of an emotion recognition device, and the embodiment of the device is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0138] like Fig. 9 As shown, the emotion recognition device 900 described in this embodiment includes: an EEG waveform signal acquisition module 901, a feature vector extraction module 902 and an emotion classification processing module 903. Among them:

[0139] The EEG waveform signal acquisition module 901 is used to acquire the EEG waveform signal to be used for emotion recognition;

[0140] A feature vector extraction module 902 is used to input the EEG waveform signal into a pre-trained feature vector extraction model and output a representation vector of the EEG waveform signal, wherein the feature vector extraction model includes a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure;

[0141] The emotion classification processing module 903 is used to input the representation vector of the EEG waveform signal into the downstream pre-trained emotion classification recognition model to obtain the emotion classification result output by the emotion classification recognition model.

[0142] The present application obtains the EEG waveform signal to be subjected to emotion recognition; inputs the EEG waveform signal into a waveform signal feature vector extraction model based on the improved wav2vec2.0 structure that has been pre-trained, and outputs a representation vector of the EEG waveform signal; inputs the representation vector of the EEG waveform signal into a downstream pre-trained emotion classification and recognition model, and obtains the emotion classification result output by the emotion classification and recognition model. The emotion recognition method disclosed in the present application utilizes a waveform signal feature vector extraction model based on the improved wav2vec2.0 structure to reduce and improve the architecture of the existing complex structure model, so that after the improvement, the model structure becomes simple and can still meet the recognition task requirements. Moreover, the model training is performed using a self-supervised learning method, which reduces the workload of manual labeling, and the emotion recognition method is applied to the field of psychology or the field of customer emotion judgment to assist relevant personnel in identifying emotional fluctuations of the target object.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0144] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0145] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0146] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c which are interconnected and communicated via a system bus. It should be noted that: Fig.10 Only a computer device 10 having a component memory 10a, a processor 10b, and a network interface 10c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0147] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0148] The memory 10a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk equipped on the computer device 10, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10a can also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 10a is generally used to store the operating system and various application software installed on the computer device 10, such as a computer-readable instruction of an emotion recognition method, etc. In addition, the memory 10a can also be used to temporarily store various types of data that have been output or are to be output.

[0149] The processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run the computer-readable instructions or process data stored in the memory 10a, such as computer-readable instructions for running the emotion recognition method.

[0150] The network interface 10c may include a wireless network interface or a wired network interface. The network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0151] The computer device proposed in this embodiment belongs to the field of research and development design and artificial intelligence technology, and is used in the scenario of emotion recognition based on EEG signals. This application obtains the EEG waveform signal to be used for emotion recognition; inputs the EEG waveform signal into a waveform signal feature vector extraction model based on the improved wav2vec2.0 structure that has been pre-trained, and outputs a representation vector of the EEG waveform signal; inputs the representation vector of the EEG waveform signal into a downstream pre-trained emotion classification and recognition model to obtain the emotion classification result output by the emotion classification and recognition model. The emotion recognition method disclosed in this application uses a waveform signal feature vector extraction model based on the improved wav2vec2.0 structure to reduce and improve the architecture of the existing complex structure model, so that after the improvement, the model structure becomes simple and can still meet the recognition task requirements. Moreover, the model training is carried out in a self-supervised learning manner, which reduces the workload of manual labeling, and applies the emotion recognition method to the field of psychology or the field of customer emotion judgment to assist relevant personnel in identifying emotional fluctuations of the target object.

[0152] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the emotion recognition method as described above.

[0153] The computer-readable storage medium proposed in this embodiment belongs to the field of research and development design and artificial intelligence technology, and is applied to the scene of emotion recognition based on EEG signals. This application obtains the EEG waveform signal to be subjected to emotion recognition; inputs the EEG waveform signal into the pre-trained waveform signal feature vector extraction model based on the improved wav2vec2.0 structure, and outputs the representation vector of the EEG waveform signal; inputs the representation vector of the EEG waveform signal into the downstream pre-trained emotion classification and recognition model, and obtains the emotion classification result output by the emotion classification and recognition model. The emotion recognition method disclosed in this application uses the waveform signal feature vector extraction model based on the improved wav2vec2.0 structure to reduce and improve the architecture of the existing complex structure model, so that after the improvement, the model structure becomes simple, and can still meet the recognition task requirements, and the self-supervised learning method is used for model training, which reduces the workload of manual labeling, and applies the emotion recognition method to the field of psychology or customer emotion judgment to assist relevant personnel in identifying emotional fluctuations of the target object.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0155] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the scope of the patent of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to replace some of the technical features therein with equivalents. All equivalent structures made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, are similarly within the scope of patent protection of this application. The tools or components not of our company that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. An emotion recognition method, characterized in that: The steps include: Obtaining an EEG waveform signal to be used for emotion recognition; Input the EEG waveform signal into a pre-trained feature vector extraction model, and output a representation vector of the EEG waveform signal, wherein the feature vector extraction model includes a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure; The representation vector of the EEG waveform signal is input into a downstream pre-trained emotion classification and recognition model to obtain an emotion classification result output by the emotion classification and recognition model.

2. The emotion recognition method according to claim 1, characterized in that: Before executing the step of inputting the EEG waveform signal into a pre-trained feature vector extraction model and outputting a representation vector of the EEG waveform signal, the method further includes: Acquire a training data set, wherein the training data set includes a first type of training data and a second type of training data, the first type of training data is EEG waveform signal data without annotated emotion categories, and the second type of training data is EEG waveform signal data with annotated emotion categories; The training data set is input into the waveform signal feature vector extraction model to be trained based on the improved wav2vec2.0 structure, and the waveform signal feature vector extraction model is trained by self-supervised learning to obtain the pre-trained feature vector extraction model.

3. The emotion recognition method according to claim 2, characterized in that: The step of inputting the training data set into the waveform signal feature vector extraction model to be trained based on the improved wav2vec2.0 structure, and using a self-supervised learning method to train the waveform signal feature vector extraction model to obtain the pre-trained feature vector extraction model specifically includes: Inputting the first type of training data into the waveform signal feature vector extraction model to be trained based on the improved wav2vec2.0 structure to perform preliminary model training; Obtain the corresponding characterization vector extraction results of all EEG waveform signal data of unlabeled emotion categories as the first characterization vector extraction result, and Obtaining feature vector extraction parameters of a waveform signal feature vector extraction model that has been preliminarily trained; Inputting the second type of training data into the initially trained waveform signal feature vector extraction model to extract the representation vector; Obtaining the representation vector extraction results corresponding to all EEG waveform signal data of the labeled emotion categories as the second representation vector extraction result; According to the second extraction result of the representation vector and the emotion category labeling results corresponding to all the EEG waveform signal data in the second type of training data, the feature vector extraction parameters are adjusted and optimized to obtain an adjusted and optimized waveform signal feature vector extraction model; The adjusted and optimized waveform signal feature vector extraction model is used as the pre-trained feature vector extraction model.

4. The emotion recognition method according to claim 3, characterized in that: The step of adjusting and optimizing the feature vector extraction parameters according to the second extraction result of the representation vector and the emotion category labeling results corresponding to all the EEG waveform signal data in the second type of training data to obtain an adjusted and optimized waveform signal feature vector extraction model specifically includes: Classify and sort the second extraction result of the representation vector based on the emotion category labeling results corresponding to all the EEG waveform signal data in the second type of training data; Obtain the classification and arrangement results of the second extraction result of the representation vector on all emotion categories as the first classification and arrangement result; According to the first classification and sorting results, identifying the important representation dimensions and non-important representation dimensions corresponding to all emotion categories respectively; The feature vector extraction parameters are adjusted and optimized based on the important representation dimensions and the unimportant representation dimensions corresponding to all emotion categories, so as to obtain an adjusted and optimized waveform signal feature vector extraction model.

5. The emotion recognition method according to claim 4, characterized in that: Before executing the step of inputting the representation vector of the EEG waveform signal into a downstream pre-trained emotion classification and recognition model to obtain the emotion classification result output by the emotion classification and recognition model, the method further includes: Using the adjusted and optimized waveform signal feature vector extraction model, extract the representation vector of the first type of training data again to obtain an updated first extraction result of the representation vector; Inputting the updated first extraction result of the representation vector into a downstream emotion classification and recognition model to be trained to perform preliminary training of the emotion classification and recognition model; Obtaining a classification result output by the downstream emotion classification and recognition model to be trained for the first extraction result of the updated representation vector as an actual classification result; According to the first classification and arrangement result, the updated first extraction result of the representation vector is classified and arranged in combination with a preset feature distance algorithm to obtain a second classification and arrangement result; Verifying the emotion classification and recognition model that has been preliminarily trained based on the second classification and sorting results and the actual classification results; The verified emotion classification and recognition model is obtained as the pre-trained emotion classification and recognition model.

6. The emotion recognition method according to claim 5, characterized in that: The step of classifying and arranging the updated first extraction result of the representation vector according to the first classification and arrangement result in combination with a preset feature distance algorithm to obtain a second classification and arrangement result specifically includes: Step 701, identifying the corresponding representation vector extraction results of all EEG waveform signal data with marked emotion categories; Step 702, identifying the classification domains of all EEG waveform signal data labeled with emotion categories according to the first classification and sorting results; Step 703, identifying the extraction results of the representation vectors corresponding to all the EEG waveform signal data without the labeled emotion category from the updated first extraction results of the representation vectors; Step 704, selecting any EEG waveform signal data from all EEG waveform signal data with marked emotion categories as the EEG waveform signal data with currently marked emotion categories; Step 705, selecting any EEG waveform signal data from all EEG waveform signal data of unlabeled emotion categories as the EEG waveform signal data of the current unlabeled emotion category; Step 706, using the Euclidean distance algorithm to calculate the characteristic distance value between the EEG waveform signal data of the current unlabeled emotion category and the EEG waveform signal data of the current labeled emotion category on the representation vector extraction result; Step 707, if the characteristic distance value satisfies a preset characteristic distance threshold, the EEG waveform signal data that is not currently labeled with an emotion category is classified into the classification domain where the EEG waveform signal data that is currently labeled with an emotion category is located; Step 708, repeatedly executing steps 704 to 707 until characteristic distance values ​​of all EEG waveform signal data of unlabeled emotion categories and all EEG waveform signal data of labeled emotion categories are calculated on the representation vector extraction results, and all EEG waveform signal data of unlabeled emotion categories are classified into classification domains, and then the repeated execution is stopped; Step 709: Using the final classification domain division result as the second classification sorting result.

7. The emotion recognition method according to claim 5, characterized in that: The step of verifying the emotion classification and recognition model completed by the preliminary training based on the second classification and sorting result and the actual classification result specifically includes: Calculating the consistency between the actual classification result and the second classification result; If the consistency does not meet the preset consistency threshold, the emotion classification parameters of the emotion classification recognition model are updated and optimized, and the updated and optimized emotion classification recognition model is used to re-obtain the actual classification result, until the consistency between the re-obtained actual classification result and the second classification sorting result meets the preset consistency threshold, and the updating and optimization is stopped; If the consistency meets a preset consistency threshold, the pre-training of the downstream emotion classification and recognition model to be trained is completed.

8. An emotion recognition device, characterized in that: include: An EEG waveform signal acquisition module is used to acquire EEG waveform signals to be used for emotion recognition; A feature vector extraction module, used for inputting the EEG waveform signal into a pre-trained feature vector extraction model, and outputting a representation vector of the EEG waveform signal, wherein the feature vector extraction model includes a waveform signal feature vector extraction model based on an improved wav2vec2.0 structure; The emotion classification processing module is used to input the representation vector of the EEG waveform signal into the downstream pre-trained emotion classification recognition model to obtain the emotion classification result output by the emotion classification recognition model.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the emotion recognition method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the emotion recognition method according to any one of claims 1 to 7 are implemented.

Images