Emotion cognition recognition method, system and equipment based on electroencephalogram

By constructing an emotional cognitive consensus model of a multi-task learning framework, using the emotional and cognitive information in the EEG signal, the problem of insufficient recognition accuracy in the existing technology is solved, and a higher recognition accuracy rate is achieved.

CN118885877APending Publication Date: 2024-11-01HANGZHOU DIANZI UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411011995.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-01

Smart Images

  • Figure CN118885877A_ABST
    Figure CN118885877A_ABST
Patent Text Reader

Abstract

The invention discloses an electroencephalogram-based emotion cognition recognition method, system and device, and relates to the technical field of electroencephalogram recognizing.The method comprises the steps that to-be-recognized electroencephalogram signals of a subject are input into an emotion cognition consensus model, and a predicted emotion recognition result and a predicted cognition recognition result are obtained; the emotion cognition consensus model is obtained by training a target network under a multi-task learning framework, and mainly comprises the steps of taking each sample of a training data set as input of the target network, taking an error between a prediction result output by the target network and an actual result as a loss function, training and optimizing parameters of the target network, and obtaining an emotion cognition consensus model; and the emotion cognition consensus model obtained by final training can realize accurate recognition of emotion and cognition. The emotion cognition consensus model is trained by adopting a multi-task learning framework, and the relationship between the emotion recognition task and the cognition recognition task and the shared features can be learned and optimized together, so that the model can more accurately predict the emotion and cognition states of the subject.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of electroencephalogram (EEG) recognition, and in particular to a method, system and device for emotion recognition based on EEG. Background Art

[0002] Emotion and cognition are closely related rather than independent psychological processes. Emotion can affect cognitive processes, and cognition can also affect emotional experience. At the same time, the study of the interaction between emotion and cognition has important potential in clinical and applied fields. For example, it can help us better understand the connection between emotional disorders and cognitive disorders, thereby providing new ideas for treatment and intervention.

[0003] Electroencephalography (EEG) is a non-invasive neurophysiological technique with excellent temporal resolution. Compared with behavioral signals, it is difficult to disguise and therefore has strong reliability. In recent years, with the development of deep learning technology and the popularity of low-cost non-invasive EEG recording equipment, EEG-based emotion and cognitive recognition applications have been widely promoted and studied. Unfortunately, however, most of the existing emotion and cognitive recognition models are based on EEG for single-task recognition of cognition or emotion, which ignores the use of valuable shared information between the two, and there is no construction of emotion and cognition multi-task recognition models. Summary of the invention

[0004] The purpose of this application is to provide a method, system and device for emotion recognition based on electroencephalogram (EEG), which can improve the recognition accuracy of the emotion recognition consensus model by constructing an emotion recognition consensus model and training the emotion recognition consensus model using EEG signals containing emotions and cognition.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for emotion and cognitive recognition based on electroencephalogram, comprising:

[0007] Inputting the subject's EEG signal to be recognized into the emotion recognition consensus model to obtain the predicted emotion recognition result and the predicted cognitive recognition result;

[0008] The emotion recognition consensus model is trained under a multi-task learning framework, and the training method is as follows:

[0009] Constructing a training data set, each sample of the training data set includes an electroencephalogram signal to be recognized, and a corresponding actual emotion recognition result and an actual cognitive recognition result;

[0010] Construct a target network under a multi-task learning framework, the target network comprising a basic neural network, a multi-scale attention module, and a fully connected layer; wherein the basic neural network is used to perform a primary feature extraction on the model input data to obtain a first shared feature; the multi-scale attention module is used to perform a secondary feature extraction on the first shared feature to obtain a second shared feature; the fully connected layer is used to perform classification processing on the second shared feature to obtain a predicted emotion recognition result and a predicted cognitive recognition result; the model input data is obtained by preprocessing the electroencephalogram signal to be recognized and dividing it into a time window of fixed length;

[0011] Each sample of the training data set is used as the input of the target network, and the error between the predicted result and the actual result output by the target network is used as the loss function to train and optimize the parameters of the target network to obtain an emotion recognition consensus model; the predicted result includes the predicted emotion recognition result and the predicted cognitive recognition result, and the actual result includes the actual emotion recognition result and the actual cognitive recognition result.

[0012] In a second aspect, the present application provides a system for emotion and cognitive recognition based on electroencephalogram, comprising:

[0013] A data acquisition module, used for acquiring the electroencephalogram signal to be identified from the subject;

[0014] The emotion recognition consensus module is used to input the EEG signal to be recognized by the subject into the emotion recognition consensus model to obtain the predicted emotion recognition result and the predicted cognitive recognition result;

[0015] The emotion recognition consensus model is trained under a multi-task learning framework, and the training method is as follows:

[0016] Constructing a training data set, each sample of the training data set includes an electroencephalogram signal to be recognized, and a corresponding actual emotion recognition result and an actual cognitive recognition result;

[0017] Construct a target network under a multi-task learning framework, the target network comprising a basic neural network, a multi-scale attention module, and a fully connected layer; wherein the basic neural network is used to perform a primary feature extraction on the model input data to obtain a first shared feature; the multi-scale attention module is used to perform a secondary feature extraction on the first shared feature to obtain a second shared feature; the fully connected layer is used to perform classification processing on the second shared feature to obtain a predicted emotion recognition result and a predicted cognitive recognition result; the model input data is obtained by preprocessing the electroencephalogram signal to be recognized and dividing it into a time window of fixed length;

[0018] Each sample of the training data set is used as the input of the target network, and the error between the predicted result and the actual result output by the target network is used as the loss function to train and optimize the parameters of the target network to obtain an emotion recognition consensus model; the predicted result includes the predicted emotion recognition result and the predicted cognitive recognition result, and the actual result includes the actual emotion recognition result and the actual cognitive recognition result.

[0019] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for emotion and cognitive recognition based on electroencephalogram described above.

[0020] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0021] The present application provides a method, system and device for emotion and cognitive recognition based on electroencephalogram (EEG), the method comprising inputting an EEG signal to be recognized by a subject into an emotion cognitive consensus model to obtain a predicted emotion recognition result and a predicted cognitive recognition result; wherein the emotion cognitive consensus model is obtained by training under a multi-task learning framework, and the training method is as follows: constructing a training data set, each sample of the training data set comprises an EEG signal to be recognized, and a corresponding actual emotion recognition result and an actual cognitive recognition result; constructing a target network under the multi-task learning framework, the target network comprises a basic neural network, a multi-scale attention module and a fully connected layer; the basic neural network is used to extract features of the model input data once, and obtain the first A shared feature; a multi-scale attention module is used to perform secondary feature extraction on the first shared feature to obtain a second shared feature; a fully connected layer is used to classify the second shared feature to obtain a predicted emotion recognition result and a predicted cognitive recognition result; the model input data is obtained by preprocessing the electroencephalogram signal to be recognized and dividing it into a time window of fixed length; each sample of the training data set is used as the input of the target network, and the error between the predicted result and the actual result output by the target network is used as the loss function to train and optimize the parameters of the target network to obtain an emotion cognition consensus model; the predicted results include predicted emotion recognition results and predicted cognitive recognition results, and the actual results include actual emotion recognition results and actual cognitive recognition results.

[0022] This application builds an emotion recognition consensus model and uses a multi-task learning framework for training, so that the model can learn and optimize emotion recognition and cognitive recognition tasks together, thereby improving the accuracy of the model in emotion and cognitive recognition. Moreover, by using the error between the actual emotion and cognitive recognition results and the model output as the loss function, the model can be effectively guided to optimize during the training process, continuously improving the model's recognition ability to obtain more accurate emotion and cognitive recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 This is a diagram of an application environment of a method for emotion recognition based on electroencephalogram in one embodiment of the present application;

[0025] Figure 2 A schematic diagram of a method for emotion recognition based on electroencephalogram provided in one embodiment of the present application;

[0026] Figure 3 A flowchart of a method for training an emotion recognition consensus model provided in one embodiment of the present application;

[0027] Figure 4 A schematic diagram of an electroencephalogram signal acquisition method provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of an emotion cognition consensus model provided by another embodiment of the present application;

[0029] Figure 6 A schematic diagram of functional modules of a system for emotion and cognitive recognition based on electroencephalogram provided in another embodiment of the present application;

[0030] Figure 7 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0033] The method for emotion recognition based on EEG provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the electroencephalogram signal to be identified to the server 104. After the server 104 receives the electroencephalogram signal to be identified, for the electroencephalogram signal to be identified, the server 104 inputs the electroencephalogram signal to be identified by the subject into the emotion cognition consensus model to obtain the predicted emotion recognition result and the predicted cognitive recognition result. The server 104 can feedback the obtained predicted emotion recognition result and the predicted cognitive recognition result to the terminal 102. In addition, in some embodiments, the method of emotion cognition recognition based on electroencephalogram can also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 can directly perform emotion recognition and cognitive recognition on the electroencephalogram signal to be identified, or the server 104 can obtain the electroencephalogram signal to be identified from the data storage system, and perform emotion recognition and cognitive recognition on the electroencephalogram signal to be identified.

[0034] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, and IoT devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0035] In an exemplary embodiment, a method for emotion recognition based on electroencephalogram is provided. The method is executed by a computer device, which can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, and the main steps include:

[0036] Step 1: Input the subject's EEG signal to be recognized into the emotion recognition consensus model to obtain the predicted emotion recognition result and the predicted cognitive recognition result.

[0037] The emotion recognition consensus model is obtained by training under a multi-task learning framework. The training method includes the following steps 201 to 205: Figure 2 and Figure 3 As shown, where:

[0038] Step 201: construct a training data set, each sample of the training data set includes an electroencephalogram signal to be recognized, and a corresponding actual emotion recognition result and an actual cognitive recognition result.

[0039] Step 2011: Select three kinds of emotional stimulation pictures from the material library as stimuli used in the experiment, including positive emotional pictures, neutral emotional pictures and negative emotional pictures, and collect EEG signals at 32 designated electrode positions when the subjects watch the emotional stimulation pictures.

[0040] Step 2012: Figure 4 As shown in the figure, at the beginning of the experiment, a black background with a "+" fixation point in the middle was displayed on the screen for 3000ms; then, 144 ERP trials were conducted, each trial sequentially presenting a blank screen interval of 1200-1500ms, an emotional stimulus picture of 1800ms duration, a black background of about 400-500ms duration, and a standard stimulus (a square with a side length of 40mm) or a target stimulus (a circle with a diameter of 40mm) for 500ms; when the target stimulus circle appeared on the screen, the subjects were required to operate the key as quickly as possible and respond, and the standard stimulus and target stimulus appeared 108 times and 36 times, respectively, at a ratio of 3:1. At the end of the experiment, the black background with a "+" fixation point in the center of the screen was displayed again for 3000ms.

[0041] Step 2013: The actual emotion recognition results are obtained by observing the facial expressions of the subjects; the actual cognitive recognition results are obtained by having the subjects complete the Montreal Cognitive Assessment (MoCA), the Cognitive Ability Screening Inventory (CASI) and the Clinical Dementia Rating (CDR) collection.

[0042] Step 202: preprocess the EEG signal to be identified by the subject to obtain a preprocessed EEG signal; specifically, use a 0.5-40 Hz bandpass filter to remove power line interference and DC components in the EEG signal; then use the principal component analysis method to remove blink artifacts in the EEG signal, and set the time window length to 2.7 seconds. The EEG signal data size corresponding to each time window is (32, 675), where 32 represents the number of channels, and 675 represents the number of samples, which is equal to the sampling frequency of 250 Hz multiplied by the time window of 2.7 seconds, and finally obtain the model input data.

[0043] Step 203: Define a multi-task learning framework. In order to allow emotion and cognitive tasks to better share features with each other, we choose a hard parameter sharing architecture as our multi-task learning framework.

[0044] Step 204: Figure 5As shown, a target network under a multi-task learning framework is constructed, and the target network includes a basic neural network, a multi-scale attention module, and a fully connected layer, wherein the basic neural network is used to perform a primary feature extraction on the model input data to obtain a first shared feature; the multi-scale attention module is used to perform a secondary feature extraction on the first shared feature to obtain a second shared feature; the fully connected layer is used to classify the second shared feature to obtain a predicted emotion recognition result and a predicted cognitive recognition result.

[0045] Step 2041: Using EEGNet as the basic neural network of the emotion recognition consensus model; the basic neural network includes a temporal convolution layer, a deep convolution layer and a separable convolution layer; inputting the model input data into the temporal convolution layer for grouped convolution, extracting the temporal features of the model input data, and obtaining a first output result; inputting the first output result into the deep convolution layer for grouped convolution, extracting the spatial features of the first output result, and obtaining a second output result; inputting the second output result into the separable convolution layer for grouped convolution and point-by-point convolution, extracting the time domain features of the second output result, and performing weighted combination on the generated feature map in the spatial dimension to obtain a first shared feature.

[0046] Furthermore, the model input data is first upgraded to meet the input requirements of the model. The upgraded model input data is represented as X∈R 1×H×W ; Then use a convolution kernel of size (1, freq / / 2) to extract the temporal features of the model input data, and output a feature map with a channel number of F1, where frep represents the sampling frequency, and add additional pixels or values ​​around the model input data so that the output size is the same as the input size to ensure that the size of the feature map remains unchanged; The obtained feature map is sent to the BatchNormalization layer to improve the training speed, stability and performance of the neural network; The first output result is represented by X1, X1∈R 1×H×W ; The whole process is shown in formula (1):

[0047] X1=BN(X*f1)(1);

[0048] Where f1∈R 1×1 ×(freq / / 2)×F1 represents the convolution filter of the temporal convolution layer, and BN represents the related operations of the BatchNormalization layer.

[0049] Furthermore, the first output result X1 is used as the input of the deep convolution layer and the grouped convolution with a convolution kernel size of (channel, 1) is used to further extract the spatial features of the first output result, where channel is equal to the number of channels H, the number of groups is F1, the number of spatial filters in each group is D, and the number of final output feature maps is F1*D; the output features are then sent to the BatchNormalization layer, the Elu activation function layer, the average pooling layer and the Droupout layer in sequence, where the convolution kernel size of the average pooling layer is set to (1, 4) to achieve the purpose of downsampling; finally, the second output result X2 is obtained, and The whole process is shown in formula (1):

[0050] X2=Drop(Avg(Elu(BN(X1*f2))))(2);

[0051] in, Represents the convolutional filters of a depthwise convolutional layer.

[0052] Furthermore, the second output result X2 is used as the input of the separable convolution layer. First, a convolution kernel of size (1, 16) is used to perform group convolution on X2 to further extract the time domain features of the second output result, where the number of groups is F1*D, and the output size is made the same as the input size, and the number of output feature maps is also F1*D; then, a point-by-point convolution with a convolution kernel size of (1, 1) is performed, and the feature maps generated in the previous step are weighted combined in the spatial dimension, and the number of output feature maps is F2, where F2 = F1*D; then the output features are sequentially sent to the BatchNormalization layer, the Elu activation function layer, the average pooling layer and the Droupout layer, where the convolution kernel of the pooling layer is set to (1, 8), and the output result is the first shared feature, represented by X3, and at this time The whole process is shown in formula (3):

[0053] X3=D(Avg(Elu(BN(f4*(X2*f3))))) (3);

[0054] in, represents the convolutional filters of the depthwise convolutional layer, Represents a convolution filter for point-wise convolution.

[0055] Step 2042: The multi-scale attention module includes a multi-scale grouped convolution module and an efficient channel attention module; the first shared feature is input into the multi-scale grouped convolution module for grouped convolution to capture spatial information of different scales and generate a multi-scale feature map; the multi-scale feature map is input into the efficient channel attention module, corresponding weights are assigned in the channel dimension, and the second shared feature is obtained.

[0056] Furthermore, the feature map output by EEGNet is recorded as X'(X'∈R C'*H'*W' ), where C' = (F1*D), H' = 1, W' = (W / / 4), X' will be divided into S sub-feature maps as the input of the multi-scale grouped convolution module, denoted as [X0,X1,...,X S-1 ]; By slicing the EEG signal, the spatial information on each channel feature map can be extracted in parallel; then, different sizes of convolution kernels are applied to each sub-feature map through group convolution to capture spatial information of different scales, which is convenient for extracting more refined features; we set the number of groups to G, Finally, the feature maps of each scale are concatenated to generate a complete multi-scale feature map. The whole process is shown in formula (4) and formula (5):

[0057] F i =Conv(k i *k i ,G i )(X' i )i=0,1,2,…,S-1 (4)

[0058] F=Cat([F1,F2…,F S-1 ]) (5)

[0059] Among them, k i =2×(i+1)+1, represents the i-th sub-feature map, and F∈RC'*H'*W') represents the multi-scale feature map obtained by concatenating all sub-feature maps.

[0060] Furthermore, the obtained multi-scale feature map F i Send it to the efficient channel attention module to obtain the weight Z of the corresponding channel i Specifically, F i First, we pass the global average pooling layer to get F i The mean vector of the two-dimensional features (H'×W') on each channel has the shape Then, a one-dimensional convolution operation with a kernel size of K and a Sigmoid activation function are applied to the average vector, and finally a weight vector Z consisting of the weights of each channel is obtained. i and The determination of K value is shown in formula (6):

[0061]

[0062] Where C represents the number of channels, γ and b are set to 2 and 1, respectively, and |·| oddRepresents the nearest odd number; the calculation process of the entire efficient channel attention module is shown in formula (7):

[0063]

[0064] Among them, σ represents the related operations of the Sigmoid activation function, and Conv1D represents the one-dimensional convolution operation.

[0065] Further, the weighted EEG features are output; for Z i Apply the Softmax activation function to get att i ; By doing so, the interaction between local and global channel attention is achieved; the specific calculation process is shown in formula (8):

[0066]

[0067] Then, we will get the att i and the corresponding F i Multiply by channel dimension to get Y i , which represents the sub-feature map obtained after the input EEG signal passes through the MSEA module; we concatenate all sub-feature maps along the channel dimension to form a complete refined output feature map Out. The specific process is shown in formulas (9) and (10):

[0068] Y i =F i ⊙att i (9);

[0069] Out=Cat([Y0,Y1,...,Y i ]) (10);

[0070] Step 2043: The fully connected layer includes an input layer, three hidden layers and two output layers; wherein the input layer is used to receive the second shared features output by the multi-scale attention module; the hidden layer is used to perform dimensionality reduction processing on the second shared features, and process them into feature representations suitable for emotion and cognitive classification tasks respectively; the output layer is used to output predicted emotion recognition results and predicted cognitive recognition results respectively.

[0071] Furthermore, the second shared feature output by the multi-scale attention module is used as the input of the fully connected layer, and the feature is reduced in dimension through the fully connected layer and the final classification result is output; specifically, the weighted features of the multi-scale attention module are flattened as the input data received by the input layer, and then sent to the first hidden layer FC ts , which is responsible for reducing the dimension of the flattened shared features and further learning high-level features; the reduced-dimensional features are sent to the task-specific hidden layers FC corresponding to emotions and cognition.th And apply the ReLU activation function; then send it to the output layer FC of each task to And apply the ReLU activation function to get the final output vector. The whole process is shown in Equation 11:

[0072] O task =Softmax(FC to ReLU(FC th ReLU(FC ts (Out))))(11);

[0073] Among them, task∈{emotion, cognition}, n f Represents the number of features after flattening, n h1 Represents the number of nodes in the first hidden layer, n h2 The number of hidden layer nodes representing the task-specific layer, n h0 Represents the number of categories for each classification task.

[0074] Step 205: Use each sample of the training data set as the input of the target network, use the error between the predicted result output by the target network and the actual result as the loss function, train and optimize the parameters of the target network, and obtain the final emotion cognition consensus model; the predicted result includes the predicted emotion recognition result and the predicted cognitive recognition result, and the actual result includes the actual emotion recognition result and the actual cognitive recognition result; the emotion cognition consensus model is used to process the EEG signal to be recognized by the subject, and obtain the predicted emotion recognition result and the predicted cognitive recognition result.

[0075] Furthermore, before introducing the dynamic weighted average algorithm, we first define the emotion cross entropy loss function at round t and the cross entropy loss function of the cognitive task at round t, as shown in formula (12) and formula (13):

[0076] Loss emo (t) = crossentropy(T emo (t),O emo (t))(12);

[0077] Loss cog (t) = crossentropy(T cog (t),O cog (t))(13);

[0078] Among them, T emo (t), T cog (t) represent the actual emotion recognition result vector and the actual cognitive recognition result vector of the t-th training round, respectively.emo (t), O cog (t) represent the predicted emotion recognition result vector and the predicted cognitive recognition result vector of the tth training round respectively.

[0079] Then, the relative decline rate w of emotion and cognitive recognition tasks in round t is calculated respectively. emo (t), W cog (t), the loss weights λ of the emotion and cognitive tasks at round t are calculated by the relative decline rate emo (t), λ cog (t), the specific calculation process is shown in formula (14) and formula (15):

[0080]

[0081]

[0082] Among them, K represents the number of tasks, and in this application K=2; T is a hyperparameter that controls the sensitivity of each task weight. Experiments have shown that the accuracy of our classification tasks is highest when T=2, so T=2 is set in this application.

[0083] Finally, the error Loss between the predicted result output by the target network and the actual result in round t multi-task (t) is obtained by multiplying the loss value of each task in round t by the weight of round t and adding them together. The specific calculation is shown in formula (16):

[0084] Loss multi-task (t) = λ emo (t)×Loss emo (t)+λ cog (t)×Loss cog (t)(15).

[0085] By implementing the above steps 201 to 205, by constructing an emotion recognition consensus model and using a multi-task learning framework for training, the model can learn and optimize emotion recognition and cognitive recognition tasks together. Compared with the recognition model in the prior art, the present application improves the accuracy of the model in emotion and cognitive recognition, and the comparison results are shown in Table 1. Moreover, by using the error between the actual emotion and cognitive recognition results and the model output as the loss function, the model can be effectively guided to optimize during the training process, and the model recognition ability can be continuously improved to obtain more accurate emotion and cognitive recognition results.

[0086] Table 1 Comparison of prediction results between the emotion cognition consensus model and existing models

[0087]

[0088] The present application also provides an application scenario, which applies the above-mentioned method of emotion recognition based on EEG. Specifically: The emotion recognition method provided in this embodiment can be applied in the emotion recognition result display scenario. The emotion recognition result display scenario includes an emotion recognition link and a display link. The emotion recognition method provided in this embodiment is used in the emotion recognition link.

[0089] Based on the same inventive concept, the embodiment of the present application also provides an emotion and cognition recognition system for implementing the above-mentioned EEG-based emotion and cognition recognition method. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more emotion and cognition recognition system embodiments provided below can refer to the limitations of the emotion and cognition recognition method above, and will not be repeated here.

[0090] In an exemplary embodiment, Figure 6 As shown, a system for emotion and cognitive recognition based on electroencephalogram is provided, comprising:

[0091] The data acquisition module 301 is used to acquire the electroencephalogram signal to be identified from the subject;

[0092] The emotion recognition consensus module 302 is used to input the subject's electroencephalogram signal to be recognized into the emotion recognition consensus model to obtain a predicted emotion recognition result and a predicted cognitive recognition result;

[0093] The emotion recognition consensus model is trained under a multi-task learning framework, and the training method is as follows:

[0094] Constructing a training data set, each sample of the training data set includes an electroencephalogram signal to be recognized, and a corresponding actual emotion recognition result and an actual cognitive recognition result;

[0095] Construct a target network under a multi-task learning framework, the target network comprising a basic neural network, a multi-scale attention module, and a fully connected layer; wherein the basic neural network is used to perform a primary feature extraction on the model input data to obtain a first shared feature; the multi-scale attention module is used to perform a secondary feature extraction on the first shared feature to obtain a second shared feature; the fully connected layer is used to perform classification processing on the second shared feature to obtain a predicted emotion recognition result and a predicted cognitive recognition result; the model input data is obtained by preprocessing the electroencephalogram signal to be recognized and dividing it into a time window of fixed length;

[0096] Each sample of the training data set is used as the input of the target network, and the error between the predicted result and the actual result output by the target network is used as the loss function to train and optimize the parameters of the target network to obtain an emotion recognition consensus model; the predicted result includes the predicted emotion recognition result and the predicted cognitive recognition result, and the actual result includes the actual emotion recognition result and the actual cognitive recognition result.

[0097] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video tag processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for emotion and cognitive recognition based on electroencephalogram is implemented.

[0098] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0100] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for emotion recognition based on electroencephalogram, characterized in that: The method for emotion recognition based on electroencephalogram includes: Inputting the subject's EEG signal to be recognized into the emotion recognition consensus model to obtain the predicted emotion recognition result and the predicted cognitive recognition result; The emotion recognition consensus model is trained under a multi-task learning framework, and the training method is as follows: Constructing a training data set, each sample of the training data set includes an electroencephalogram signal to be recognized, and a corresponding actual emotion recognition result and an actual cognitive recognition result; Construct a target network under a multi-task learning framework, the target network comprising a basic neural network, a multi-scale attention module, and a fully connected layer; wherein the basic neural network is used to perform a feature extraction on the model input data once to obtain a first shared feature; the multi-scale attention module is used to perform a feature extraction on the first shared feature twice to obtain a second shared feature; the fully connected layer is used to classify the second shared feature to obtain a predicted emotion recognition result and a predicted cognitive recognition result; the model input data is data obtained after preprocessing the electroencephalogram signal to be recognized and dividing it into a time window of fixed length; Each sample of the training data set is used as the input of the target network, and the error between the predicted result and the actual result output by the target network is used as the loss function to train and optimize the parameters of the target network to obtain an emotion recognition consensus model; the predicted result includes the predicted emotion recognition result and the predicted cognitive recognition result, and the actual result includes the actual emotion recognition result and the actual cognitive recognition result.

2. The method for emotion recognition based on electroencephalogram according to claim 1, characterized in that: The basic neural network includes a temporal convolution layer, a depth convolution layer and a separable convolution layer; the feature extraction of the model input data is performed once to obtain the first shared feature, specifically including: Inputting the model input data into the time convolution layer for group convolution, extracting the time features of the model input data, and obtaining a first output result; Inputting the first output result into a deep convolutional layer for group convolution, extracting spatial features of the first output result, and obtaining a second output result; The second output result is input into a separable convolution layer for group convolution and point-by-point convolution, the time domain features of the second output result are extracted, and weighted combination is performed in the spatial dimension to obtain the first shared features.

3. The method for emotion recognition based on electroencephalogram according to claim 1, characterized in that: The multi-scale attention module includes a multi-scale grouped convolution module and an efficient channel attention module; performing secondary feature extraction on the first shared feature to obtain a second shared feature specifically includes: Inputting the first shared features into a multi-scale grouped convolution module for grouped convolution to capture spatial information of different scales and generate a multi-scale feature map; The multi-scale feature map is input into an efficient channel attention module, and a corresponding weight is assigned in the channel dimension to obtain a second shared feature.

4. The method for emotion recognition based on electroencephalogram according to claim 1, characterized in that: The fully connected layer includes an input layer, three hidden layers and two output layers; wherein the input layer is used to receive the second shared features output by the multi-scale attention module; the hidden layer is used to perform dimensionality reduction processing on the second shared features, and process them into feature representations suitable for emotion and cognitive classification tasks respectively; the output layer is used to output predicted emotion recognition results and predicted cognitive recognition results respectively.

5. The method for emotion and cognition recognition based on electroencephalogram according to claim 1, characterized in that: The loss function is: Loss multi-task (t)=λ emo (t)×Loss emo (t)+λ cog (t)×Loss cog (t); Among them, Loss multi-task (t) is the error between the predicted result and the actual result output by the target network in round t, λ emo (t) is the loss weight of the emotion task in round t, λ cog (t) is the cognitive task loss weight of round t, λ emo (t) and λ cog (t) is calculated by using a dynamic weighted average algorithm; Loss emo (t) is the emotional loss value of round t, Loss cog (t) is the cognitive loss value of round t, Loss emo (t) and Loss cog (t) is calculated by cross entropy.

6. The method for emotion recognition based on electroencephalogram according to claim 1, characterized in that: The EEG signal to be recognized by the subject is input into the emotion recognition consensus model to obtain the predicted emotion recognition result and the predicted cognitive recognition result, which specifically include: Preprocessing the electroencephalogram signal to be identified by the subject to obtain a preprocessed electroencephalogram signal; Divide the preprocessed EEG signal into time windows of fixed length to obtain model input data; The preprocessed electroencephalogram signal is input into the emotion recognition consensus model to obtain the predicted emotion recognition result and the predicted cognitive recognition result.

7. A system for emotion and cognitive recognition based on electroencephalogram, characterized in that: The system for emotion and cognitive recognition based on electroencephalogram includes: A data acquisition module, used for acquiring the electroencephalogram signal to be identified from the subject; The emotion recognition consensus module is used to input the EEG signal to be recognized by the subject into the emotion recognition consensus model to obtain the predicted emotion recognition result and the predicted cognitive recognition result; The emotion recognition consensus model is trained under a multi-task learning framework, and the training method is as follows: Constructing a training data set, each sample of the training data set includes an electroencephalogram signal to be recognized, and a corresponding actual emotion recognition result and an actual cognitive recognition result; Construct a target network under a multi-task learning framework, the target network comprising a basic neural network, a multi-scale attention module, and a fully connected layer; wherein the basic neural network is used to perform a primary feature extraction on the model input data to obtain a first shared feature; the multi-scale attention module is used to perform a secondary feature extraction on the first shared feature to obtain a second shared feature; the fully connected layer is used to perform classification processing on the second shared feature to obtain a predicted emotion recognition result and a predicted cognitive recognition result; the model input data is obtained by preprocessing the electroencephalogram signal to be recognized and dividing it into a time window of fixed length; Each sample of the training data set is used as the input of the target network, and the error between the predicted result and the actual result output by the target network is used as the loss function to train and optimize the parameters of the target network to obtain an emotion recognition consensus model; the predicted result includes the predicted emotion recognition result and the predicted cognitive recognition result, and the actual result includes the actual emotion recognition result and the actual cognitive recognition result.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for emotion and cognitive recognition based on electroencephalogram as described in any one of claims 1-6.

Citation Information

Cited By

  • Online brain grain recognition method for cross-session scene

    CN119622467A

  • Sleep environment management method, device and system, storage medium and program product

    CN121116075A