Emotion recognition model training method, computer equipment, storage medium and program product

Abstract

The invention discloses an emotion recognition model training method, computer equipment, a storage medium and a program product. The method comprises the steps of performing alignment processing on a continuous label data set and a corresponding stimulation image frame to obtain a preset continuous label data set; inputting the electroencephalogram sample data into an electroencephalogram encoder to extract electroencephalogram features; inputting a stimulation image frame corresponding to electroencephalogram sample data to the image encoder to extract image features; the sentiment classifier obtains a predicted sentiment category according to the electroencephalogram features, and first loss is obtained by combining a corresponding sample sentiment classification label; determining a second loss according to the electroencephalogram features and the image features; and performing parameter updating on the emotion recognition model at least according to the first loss and the second loss. According to the method and the device, cross-modal data fusion between the electroencephalogram data and the stimulation image frame is realized, more information can be extracted from multi-modal data, and the emotion recognition accuracy of an emotion recognition model is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an emotion recognition model training method, computer equipment, storage medium and program product. Background Art

[0002] With the development of deep learning technology, research on emotion recognition based on electroencephalogram (EEG) has made great progress. However, understanding the EEG-based emotion recognition task from the perspective of deep learning models is still in its early stages. Some studies have pointed out that mainstream datasets usually assign the same labels to a large number of samples collected under the same stimulus. These discrete labels cannot describe the emotional dynamics well, resulting in insufficient information obtained by the emotion recognition model and low emotion recognition accuracy. Summary of the invention

[0003] The embodiments of the present application provide an emotion recognition model training method, a computer device, a storage medium and a program product, which are used to solve at least one of the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a method for training an emotion recognition model, wherein the emotion recognition model includes an image encoder, an electroencephalogram encoder, and an emotion classifier; The method comprises: aligning a continuous label data set and a corresponding stimulus image frame to obtain a preset continuous label data set; the continuous label data set comprises electroencephalogram sample data and sample emotion classification labels; Inputting the EEG sample data into the EEG encoder to extract EEG features; inputting the stimulus image frame corresponding to the electroencephalogram sample data into the image encoder to extract image features; The emotion classifier obtains a predicted emotion category according to the EEG feature, and obtains a first loss in combination with a corresponding sample emotion classification label; determining a second loss according to the EEG feature and the image feature; Parameters of the emotion recognition model are updated at least according to the first loss and the second loss.

[0005] The present application obtains a preset continuous label data set by aligning the continuous label data set and the corresponding stimulus image frames, thereby realizing cross-modal data fusion between the electroencephalogram data and the stimulus image frames, so that the trained emotion recognition model can extract more information from the multimodal data, thereby helping to improve the accuracy of emotion recognition by the emotion recognition model.

[0006] In some embodiments, the emotion recognition model further includes an induction degree predictor, and the continuous label data set further includes a sample induction intensity label; The method further comprises: the induction degree predictor obtains predicted induction strength according to the electroencephalogram feature, and obtains a third loss in combination with the corresponding sample induction strength label; Updating parameters of the emotion recognition model at least according to the first loss and the second loss includes: The parameters of the emotion recognition model are updated according to the first loss to the third loss.

[0007] In some embodiments, the image encoder is obtained through pre-training; updating the parameters of the emotion recognition model includes: updating the parameters of the electroencephalogram encoder, the emotion classifier and the induction degree predictor.

[0008] In some embodiments, updating the parameters of the emotion recognition model according to the first loss to the third loss includes: The parameters of the emotion recognition model are updated according to the weighted values ​​of the first loss to the third loss; wherein the first loss is the emotion classification loss, the second loss is the feature similarity loss, and the third loss is the induction degree loss.

[0009] In some embodiments, the sentiment classification loss is configured with a first weight value, the induction degree loss is configured with a second weight value, and the weight value of the feature similarity loss decreases with the training process.

[0010] In some embodiments, before inputting the stimulation image frame corresponding to the electroencephalogram sample data into the image encoder, it also includes: adjusting the stimulation image frame to a preset size and configuring a preset number of channels.

[0011] In a second aspect, an embodiment of the present application provides a computer device, which is equipped with an emotion recognition model trained using the emotion recognition model training method described in any one of the present applications, and is used for emotion recognition.

[0012] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the emotion recognition model training method described in any one of the present applications.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program / instruction is stored, characterized in that when the computer program / instruction is executed by a processor, the steps of the emotion recognition model training method described in any one of the items of the present application are implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the emotion recognition model training method described in any one of the items of the present application are implemented.

[0015] The present application obtains a preset continuous label data set by aligning the continuous label data set and the corresponding stimulus image frames, thereby realizing cross-modal data fusion between the electroencephalogram data and the stimulus image frames, so that the trained emotion recognition model can extract more information from the multimodal data, thereby helping to improve the accuracy of emotion recognition by the emotion recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A flowchart of an embodiment of the emotion recognition model training method of the present application; Figure 2 A schematic diagram of the structure of an embodiment of the emotion recognition model of the present application; Figure 3 A schematic diagram of the structure of an embodiment of a computer device of the present application; Figure 4 A view of the multi-task results for some typical subjects in this application; Figure 5 In this application, during the training of the multi-task model, the loss function of different projects changes over time. Figure 6a This is a confusion matrix diagram for directly predicting classification results based on similarity in this application; Figure 6b This is the similarity ROC curve diagram in this application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0019] It should also be noted that, in this article, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0020] like Figure 1 The following is a method for training an emotion recognition model provided by an embodiment of the present application. Figure 2 The structure diagram of an embodiment of the emotion recognition model in the present application is shown, and the emotion recognition model includes: an image encoder, an electroencephalogram encoder and an emotion classifier; the emotion recognition model training method includes the following steps: S10, continuous label data set and corresponding stimulus image frames Perform alignment processing to obtain a preset continuous label data set; the continuous label data set includes electroencephalogram sample data and sample sentiment classification labels C tr .

[0021] Exemplarily, the continuous label data set can be obtained by referring to the following prior art (Tagging Continuous Labels for EEG-based Emotion Classification), and the specific implementation method is not limited in this application. The preset continuous label data set obtained after alignment includes EEG sample data, sample emotion classification labels and corresponding stimulus image frames. Among them, the stimulus image frame is the image frame in the sample video corresponding to the EEG sample data generated when the subject watches the sample video.

[0022] S20, inputting the EEG sample data into the EEG encoder to extract EEG features.

[0023] Exemplarily, an EEG encoder can be constructed by a one-block spectral-spatial-temporal adaptive graph convolutional neural network (for example, see the related technology: A Multi-domain Adaptive Graph Convolutional Network for EEG-based Emotion Recognition). This application does not limit the specific implementation method of constructing the EEG encoder.

[0024] S30, inputting the stimulation image frame corresponding to the electroencephalogram sample data into the image encoder to extract image features. The image encoder is obtained by pre-training. Exemplarily, the pre-trained image encoder can adopt the CLIP-ViT-B / 32 model, which is trained using the OpenCLIP open source code and the CLIP method proposed by OpenAI. It should be noted that the above is only an example and this application is not limited to this.

[0025] S40, the emotion classifier obtains a predicted emotion category according to the electroencephalogram feature, and obtains a first loss in combination with a corresponding sample emotion classification label. Exemplarily, the emotion classifier includes three fully connected layers, and the first loss is an emotion classification loss.

[0026] S50: Determine a second loss according to the EEG feature and the image feature. Exemplarily, the second loss is a feature similarity loss.

[0027] S60. Update parameters of the emotion recognition model at least according to the first loss and the second loss.

[0028] The present application obtains a preset continuous label data set by aligning the continuous label data set and the corresponding stimulus image frames, thereby realizing cross-modal data fusion between the electroencephalogram data and the stimulus image frames, so that the trained emotion recognition model can extract more information from the multimodal data, thereby helping to improve the accuracy of emotion recognition by the emotion recognition model.

[0029] In some embodiments, the emotion recognition model further includes an induction degree predictor, and the continuous label dataset further includes a sample induction intensity label. R tr ; The method also includes: the induction degree predictor obtains the predicted induction strength according to the electroencephalogram features, and obtains the third loss in combination with the corresponding sample induction strength label; the third loss is the induction degree loss; the induction degree predictor includes three fully connected layers.

[0030] The parameter updating of the emotion recognition model is performed at least according to the first loss and the second loss, including: the parameter updating of the emotion recognition model is performed according to the first loss to the third loss.

[0031] In some embodiments, the image encoder is obtained by pre-training. Accordingly, updating the parameters of the emotion recognition model includes: updating the parameters of the electroencephalogram encoder, the emotion classifier and the induction degree predictor.

[0032] In the process of realizing the present application, the inventor first discovered that the prior art relied on discrete emotion labels and could not accurately capture the dynamic changes of emotions. In addition, although EEG signals are significantly affected by video stimulation, the prior art fails to fully utilize the image information in the video. Therefore, the present application introduces a video feature extraction module into the model, which, combined with continuous emotion labels, enables the model to extract more information from multimodal data. In addition, the efficiency of feature extraction is improved in the present application by using pre-trained image encoders and EEG encoders.

[0033] In some embodiments, the parameters of the emotion recognition model are updated according to the first loss to the third loss, including: updating the parameters of the emotion recognition model according to the weighted values ​​of the first loss to the third loss; wherein the first loss is the emotion classification loss, the second loss is the feature similarity loss, and the third loss is the induction degree loss.

[0034] In this embodiment, through the weighted multi-task loss function, the emotion recognition model can achieve faster convergence during the training process and effectively avoid the overfitting problem.

[0035] In some embodiments, the sentiment classification loss is configured with a first weight value, the induction degree loss is configured with a second weight value, and the weight value of the feature similarity loss decreases with the training process. Exemplarily, the sentiment classification loss is configured with a first weight value that can take a value of 1, and the induction degree loss is configured with a second weight value that can take a value of 10.

[0036] In this embodiment, the weight value of the feature similarity loss is gradually reduced during the training process, which can promote the rapid convergence of the early EEG encoder and improve the training efficiency of the emotion recognition model.

[0037] In some embodiments, before inputting the stimulation image frame corresponding to the electroencephalogram sample data into the image encoder, the method further includes: adjusting the stimulation image frame to a preset size and configuring a preset number of channels. For example, the preset size may be 224*224, and the preset number of channels may be 3.

[0038] In some embodiments, the present application also provides a computer device, which is configured with an emotion recognition model trained using any emotion recognition model training method described in any one of the present applications, and is used for emotion recognition.

[0039] In some embodiments, the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the emotion recognition model training method described in any one of the present applications.

[0040] In some embodiments, the present application also provides a computer-readable storage medium having a computer program / instruction stored thereon, characterized in that when the computer program / instruction is executed by a processor, the steps of the emotion recognition model training method described in any one of the items of the present application are implemented.

[0041] In some embodiments, the present application also provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the emotion recognition model training method described in any one of the items of the present application are implemented.

[0042] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0043] Figure 3 is a schematic diagram of the hardware structure of a computer device for executing an emotion recognition model training method provided by another embodiment of the present application, such as Figure 3 As shown, the device includes: One or more processors 310 and memory 320, Figure 3 A processor 310 is taken as an example.

[0044] The device for executing the emotion recognition model training method may further include: an input device 330 and an output device 340 .

[0045] The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0046] The memory 320 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the emotion recognition model training method in the embodiment of the present application. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the emotion recognition model training method in the above method embodiment is implemented.

[0047] The memory 320 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the emotion recognition model training device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 320 may optionally include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the emotion recognition model training device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The input device 330 can receive input digital or character information and generate signals related to user settings and function control of the emotion recognition model training device. The output device 340 can include display devices such as display screens.

[0049] The one or more modules are stored in the memory 320, and when executed by the one or more processors 310, the emotion recognition model training method in any of the above method embodiments is executed.

[0050] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0051] In order to make the technical solution and effect of this application clearer, the research, implementation and experimental demonstration process of the invention are presented as follows: With the development of deep learning technology, the research on emotion recognition based on electroencephalogram (EEG) has made great progress. However, understanding the EEG-based emotion recognition task from the perspective of deep learning models is still in its early stages. Some studies have pointed out that mainstream datasets usually assign the same labels to a large number of samples collected under the same stimulus. These discrete labels cannot well describe the dynamics of emotion, resulting in insufficient information obtained by the model. Therefore, continuous labels are introduced in related technologies, proving the feasibility of learning these labels and their effectiveness in enhancing the model's understanding of emotion recognition tasks. However, these works either failed to demonstrate their beneficial effects or did not make full use of continuous labels. Therefore, in this application, a multi-task emotion recognition model is proposed, which combines two emotion recognition tasks. In addition, considering the advanced methods in other fields and the neglected information in the stimuli that contribute to the formation of EEG datasets, we use pre-trained models to extract image features of stimuli to assist the training process. Our proposed model makes full use of all available labels and stimuli, and achieves superior performance in both tasks compared with single-task models. In the proposed emotion classification task, our multi-task model achieves an accuracy of 86.08%, surpassing the performance of the single-task model, which achieves 84.60% and 83.13% accuracy with and without continuous labels, respectively. These findings not only highlight the performance gains brought by multi-task methods, but also advocate the integration of pre-trained models and advanced methods in related fields into the research of emotion brain-computer interfaces, aiming to enhance the understanding of the complex human brain.

[0052] 1. Introduction The field of affective brain-computer interface (aBCI) has been committed to promoting the understanding and regulation of emotions. In terms of the form of brain signals obtained by sensors, brain-computer interfaces can be divided into invasive and non-invasive types. In terms of the functions and uses achieved by brain-computer interfaces, brain-computer interfaces can be divided into two categories, namely motor brain-computer interfaces and affective brain-computer interfaces. In recent years, research on emotion recognition has progressed rapidly, leading to an increasing number of studies on training models using labeled EEG datasets. The most widely used EEG datasets include DREAMER, MAHNOB-HCI, DEAP, and SEED, all of which use videos as stimuli. In these datasets, EEG data is collected while subjects watch stimuli, and the data shares the same label with the stimuli, which is given by the subjects or annotators. However, such discrete labels cannot reflect the changing emotions of subjects while watching videos, provide limited information for the model, and prevent researchers from understanding the emotions expressed by EEG signals in detail.

[0053] Several studies have experimented with continuous label datasets. Most of these studies focused on features and models for continuous label prediction. Soleymani et al. created a continuous label dataset based on MAHNOB-HCI but failed to achieve significant performance in predicting EEG continuous labels. More recently, Ding et al. performed effective regression analysis of continuous emotion and arousal labels based on amplitude and phase features of EEG data. Thammasan et al. studied two other features, fractal dimension and power spectral density, while Hasanzadeh et al. explored different models. These studies have demonstrated the learnability of continuous labels, but whether continuous labels can provide richer information and whether the model can learn better than discrete labels is still unknown.

[0054] In order to evaluate the effectiveness of continuous labels, the related technology extends a dataset from the SEED dataset (the dataset can be obtained by referring to the following existing technology: TaggingContinuous Labels for EEG-based Emotion Classification), which includes discrete emotion labels and continuous emotion labels indicating the degree of emotion induction. The different values ​​of continuous labels indicate the significant differences in the intensity of emotions in the same video clip and between different subjects, which is exactly what discrete labels cannot capture. At the same time, they select highly induced samples based on continuous labels and use these samples to train the emotion classification model. Although the number of training samples is greatly reduced, the accuracy is getting higher and higher. This proves that the additional information provided by continuous labels has a positive effect on the model. They also designed a cross-dataset induced / non-induced classification task, highlighting the wider use of continuous labels. However, their method only considers the relative size of the degree of induction, ignoring the detailed information provided by the precise value of the continuous label.

[0055] Additionally, predicting continuous labels has limited effectiveness, suggesting that models face challenges in learning from them. Although many studies have found that stimulus content has a significant impact on EEG data, including key features of the stimulus may disrupt the training process. In the past few years, natural language processing and computer vision methods have experienced another explosion in development. Researchers in the field of BCI have also used these methods to successfully reconstruct images from EEG data. However, few attempts have been made to use these methods to assist in emotion classification tasks.

[0056] Considering the shortcomings of previous works and methods in other fields, we made the following contributions in this application: (1) designed a multi-task emotion recognition model supplemented by pre-trained stimulus image features; (2) demonstrated the superiority of the model in emotion classification tasks and emotion induction degree regression tasks, and proved the necessity of a single module; (3) proposed the concept that stimulation and progress of other modalities can guide aBCI research.

[0057] 2. Methods We consider both discrete and continuous labels in our model to directly show the improvement that continuous labels bring to the model, while we retain the value of labels to make full use of them. At the same time, we use leading pre-trained image encoders to extract features from stimuli to assist EEG feature extraction. This section introduces the detailed design of the model.

[0058] 2.1 Multi-task emotion recognition Previous studies have used the degree of emotion induction to filter the training set, which improves the classification accuracy, but ignores the information contained in the continuous labels. In this application, the degree of induction is also used as a prediction target to let the model distinguish between different levels of EEG signals, not just induction and non-induction. In this work, through the idea of ​​multi-task learning, these two tasks are integrated into one model to play to their strengths.

[0059] Multi-task learning is a method that enables tasks to share their model parameters in a hard or soft manner, which may benefit our method in the following aspects: (1) the parameters of similar tasks can be better learned through mutual assistance; (2) during the training of multi-task models, losses other than the target task can prevent the model from overfitting in the task and improve the generalization ability of the model. In addition, Li et al. explored several multi-task models and demonstrated their compatibility with aBCI tasks based on discrete annotated datasets and achieved better accuracy than single-task models.

[0060] Common multi-task learning methods can be divided into hard parameter sharing and soft parameter sharing. We adopt the former, that is, the input data of two tasks (for example, the sentiment classification task and the induction degree prediction task) are first passed through the same EEG encoder to generate general EEG features, and then passed through different downstream networks (for example, the sentiment classifier to be mentioned later). Dense c and induction degree predictor Dense i ) to further extract information related to each task. This structure is relatively simple and suitable for highly related tasks.

[0061] 2.2 EEG encoding based on pre-trained image encoder In addition to the deliberately collected labels, there is also potential information in the stimuli built by the dataset that can be utilized. Therefore, this application further integrates the information contained in the video material on the basis of the multi-task model to optimize the effect of the EEG encoder. In previous studies, we observed that when both the training set and the test set contain samples evoked by the same video material, the model tends to recognize the video material first and then the emotion type. This suggests that the EEG data may contain more information related to the video material itself, rather than just the emotions evoked by the video material in the subjects.

[0062] In fact, as early as 1999, researchers proposed the research direction of using data collected from the human brain to construct original visual images. The results of early work were limited by data quality and models, and the reconstruction effect was very limited. With the development of brain data technology and computer vision methods, research in this field is also constantly improving. In the study of Takagi et al., they used functional magnetic resonance imaging (fMRI) to collect brain data of subjects when they watched multiple pictures, and used visual cortical signals and the leading diffusion model in the field of computer vision to achieve high-resolution image reconstruction. Lan et al. also proposed image reconstruction based on EEG data. They mapped EEG signals to potential spaces of two particle sizes, and then applied the diffusion model to verify the feasibility of reconstructing images using EEG data. The above experiments show that human brain signals contain a lot of visual information.

[0063] Therefore, in this application, we try to add the corresponding video frame of the stimulus to each sample, and use the similarity between the image features of the video frame and the EEG features as one of the optimization targets. This measure not only helps the model converge quickly, but also makes up for the common problem of insufficient data in aBCI tasks and prevents overfitting.

[0064] Specifically, we capture the intermediate frame corresponding to the stimulus EEG sample in each window and apply the pre-trained model in the field of computer vision to extract image semantic features. The pre-trained image encoder used in this experiment adopts the CLIP-ViT-B / 32 model of LAION, which is trained using the OpenCLIP open source code and the CLIP method proposed by OpenAI. The reliability of this model has been rigorously verified. The ViT-B / 32 model used in this task is a model based on visual transformers with an input convolution kernel size of 32*32. After pre-training with the DataComp-1B dataset, the model can achieve a zero-point accuracy of 72.8% when tested on the ImageNet-1K dataset.

[0065] 2.3 Multi-task emotion recognition model The emotion recognition model in this application can be implemented as follows Figure 2A multi-task emotion recognition model is shown in FIG. The training process of the model is shown in Algorithm 1. The specific implementation steps are as follows: 1. Start a multi-task emotion recognition model consisting of trainable modules, including an EEG encoder Encoder eeg , Sentiment Classifier Dense c and induction degree predictor Dense i .

[0066] 2. Before training the model, each EEG sample must be compared with the frame image in the stimulus x f Alignment. Introducing a pre-trained image encoder Encoder img To extract image features: feat f = Encoder img ( x f ). Image encoder Encoder img In frozen state.

[0067] 3. During the training phase, the model receives raw input x eeg And extract EEG features: feat eeg = Encoder eeg ( x eeg ). Then predict the category and the degree of induction To update the parameters, back-propagation is applied according to a weighted loss function consisting of: (1) the cosine similarity loss between the EEG sample features and the image frame features L f ; (2) Cross entropy loss for sentiment classification L c ; (3) Mean square error loss of emotion induction degree prediction L i The former speeds up the training in the early stages, while the latter two focus on extracting emotion- and task-related information. The specific weights will be introduced in the experimental section.

[0068] 4. After the training, the trainable modules are frozen and the model can predict the emotion category and induction degree for testing.

[0069] 3. Experiment 3.1 Dataset We used the continuous labeling dataset in the related art (refer to the following prior art to obtain the continuous labeling dataset: Tagging Continuous Labels for EEG-based Emotion Classification), and recruited 8 subjects to watch video clips for emotion induction. Among them, there are 5 positive and 5 negative video clips of about 5 minutes, and the subjects are required to watch the clips twice, respectively, for EEG data collection and continuous labeling. The original EEG data was also processed, downsampled from 1000 Hz to 200Hz, bandpass filtered and denoised in the range of 0.5 Hz to 70 Hz, and differential entropy (DE) features of 15-second windows were extracted to obtain better performance. At the same time, the continuous labels were smoothed and limited to the range of [0, 5]. After preprocessing, each EEG feature sample has a corresponding positive / negative discrete label and an induction degree continuous label.

[0070] In order to incorporate stimulus information into features, it is necessary to extract the corresponding frame of each sample from the time window used to calculate the differential entropy feature. Before entering the image encoder, these frames will be resized to a uniform size of 224*224 and have 3 channels by the processor provided with the encoder to standardize the format of image input and improve encoding efficiency.

[0071] 3.2 Experimental Setup The experiment was conducted in a subject-dependent manner and a leave-one-out-cross-validation approach was used to group the samples based on the videos.

[0072] We construct the EEG encoder through a one-block spectral-spatial-temporal adaptive graph convolutional neural network (see related technology: A Multi-domain Adaptive Graph Convolutional Network for EEG-based Emotion Recognition), which is an enhancement of the GCN model used in previous work. The size of the output feature is 512, which is consistent with the feature dimension of the frame. Both the emotion classifier and the induction degree predictor consist of three fully connected layers.

[0073] Different loss functions have different weights. For sentiment classification and induction level prediction, the weights are 1 and 10 respectively to ensure the balance of training. The weight of feature similarity loss is set to ,in, ce It is the abbreviation of currentepoch, which means the current iteration round. te Total epoch is the abbreviation of total epoch, which means the total number of iterations. This weight is gradually reduced during the training process to promote fast convergence of the EEG encoder in the early stage and focus on downstream tasks in the later stage.

[0074] Table 1. Sentiment classification accuracy of the multi-task sentiment recognition model (mean ± std. %), where std represents standard deviation.

[0075] 3.3 Results We first evaluated the model performance on the sentiment classification task. Table 1 shows the accuracy of positive / negative sentiment classification using the multi-task sentiment recognition model and compares it to single-task models trained using the full (discrete) and filtered (continuous) training sets proposed in previous work.

[0076] From the results, we can see that the multi-task emotion recognition model achieves the highest accuracy in the emotion classification task. This result shows that the information provided by continuous labels has not been fully utilized in previous studies, while our multi-task model retains all samples instead of discarding those with less inductive samples, thus outperforming in terms of sample efficiency.

[0077] Table 2. Mean square error of the prediction of emotion induction degree (mean ± std.%) At the same time, the mean square error of the induction degree prediction task is shown in Table 2. The first method is to directly extract the average value of the induction degree from the training set of positive emotions and negative emotions respectively. The single-task method trains an induction degree prediction model for each emotion. The experimental results show that although the mean square error under the multi-task model is greatly reduced compared with the single-task model, it is still slightly higher than the mean method, which exposes the difficulty of continuous label prediction.

[0078] Figure 4 This is a view of the multi-task results of some typical subjects in this application. In order to understand whether the model has learned effective information, Figure 4 The following figure shows some of the prediction results of the multi-task model. Each column represents a subject and each row represents a stimulus. The red and blue curves are the true values, the green line is the predicted value, and the transparent green area under the green line is the correctly classified sample.

[0079] The overall trend of the predicted values ​​is consistent with the true values, so the model may have extracted information related to the degree of emotion induction. Since there is still a large difference between the predicted values ​​and the true values, the induction degree regression task seems to be more challenging than the classification task and deserves further optimization.

[0080] also, Figure 5 The trend of loss changes over time in different item loss functions during the training of the multi-task model is shown. In the early stages, the similarity loss decreases rapidly, the induction prediction loss decreases significantly, and the classification loss begins to decrease in the later stages. This phenomenon is consistent with the design of weighted losses. On the other hand, the model, especially the EEG encoder, may prefer to learn from image features first and then adjust the weights based on additional EEG information and emotional tasks. In addition, it is assumed that the similarity loss helps to distinguish subtle differences between similar emotional samples and enhance the effect of the induction prediction task.

[0081] 3.4 Ablation study However, the above results do not confirm whether these improvements are due to continuous labels, multi-task design, or image information from stimuli. To demonstrate the individual contribution of each module to improving information extraction from EEG data, we conducted further ablation studies. We conducted two additional experiments: removing continuous label information; and removing stimuli from the multi-task model by prohibiting the induced degree regression task or pre-trained image feature extractor. The results are shown in Table 3. All methods were trained and tested in the same way.

[0082] Table 3. Sentiment classification accuracy of each method in the ablation study (mean ± std.%). The middle two rows are two additional experiments, and the first and fourth rows are comparative experiments.

[0083] The results show that when the continuous label module or the stimulus feature module is disabled, the accuracy of emotion classification decreases, but is still higher than that of the single-task model. This result proves that the two additional modules in the multi-task model can significantly improve the model's understanding of EEG data from different perspectives to complete emotion-related tasks.

[0084] 4. Discussion This application introduces a pre-trained model from the field of computer vision into the task of emotion recognition based on EEG. It is worth exploring further to clarify the basic principles of why pre-trained image models are beneficial to the research of affective brain-computer interface (aBCI). On the one hand, image features can supplement the information from vision, so that the model can waste less time on understanding the visual-related information hidden in the EEG samples; on the other hand, image features may also contain information about the type of emotion implied by the video clip. Therefore, we conducted an experiment to discover the source of this information.

[0085] This time we do not connect EEG data and video frames, but focus on labels, i.e., the types of emotions expressed in natural language, to leverage CLIP. We directly use the natural language feature extraction module in the above pre-trained CLIP model to extract "positive" and "negative" label features, and calculate the similarity between each label feature and each frame feature, with the true label as the discrete label of the video clip.

[0086] The confusion matrix of the classification result is directly predicted based on the similarity. Figure 6a Although the prediction accuracy of negative emotions is as high as 99.34%, the prediction accuracy of positive emotions is only 24.96%. Figure 6b The similarity ROC curve is depicted in Figure 1, with an AUC value of up to 0.903, where the horizontal axis is the false positive rate and the vertical axis is the true positive rate. It can be considered that the pre-trained CLIP model has a certain ability to distinguish the potential emotional orientation in the video frame, but further model training is required for classification tasks. This phenomenon proves that models in other fields may play a positive role in aBCI tasks, but how to perfectly utilize these models remains to be explored.

[0087] 5. Conclusion EEG-based emotion recognition has made great progress in recent years. However, most works utilize discrete labels, which provide limited information to the model, while works on continuous labels do not fully utilize these labels. In this application, a multi-task emotion recognition model is proposed, which combines the discrete emotion classification task and the continuous induction degree prediction task, and incorporates the characteristics of the stimulus to facilitate the training process. The model learns entirely from continuous labels, improving the performance of both tasks, and we also reveal the potential of applying pre-trained encoders from other modalities. Although there is still much room for improvement in this method. But there is no doubt that the performance of the model can be improved by better utilizing continuous labels and stimulus information.

[0088] The computer device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0089] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0090] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0091] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0092] (5) Other electronic devices with data interaction functions.

[0093] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an emotion recognition model, characterized in that: The emotion recognition model includes an image encoder, an electroencephalogram encoder and an emotion classifier; The method comprises: Aligning the continuous label data set and the corresponding stimulus image frame to obtain a preset continuous label data set; the continuous label data set includes electroencephalogram sample data and sample emotion classification labels; Inputting the EEG sample data into the EEG encoder to extract EEG features; inputting the stimulus image frame corresponding to the electroencephalogram sample data into the image encoder to extract image features; The emotion classifier obtains a predicted emotion category according to the EEG feature, and obtains a first loss in combination with a corresponding sample emotion classification label; determining a second loss according to the EEG feature and the image feature; Parameters of the emotion recognition model are updated at least according to the first loss and the second loss.

2. The method according to claim 1, characterized in that The emotion recognition model further includes an induction degree predictor, and the continuous label data set further includes a sample induction intensity label; The method further comprises: the induction degree predictor obtains predicted induction strength according to the electroencephalogram feature, and obtains a third loss in combination with the corresponding sample induction strength label; Updating parameters of the emotion recognition model at least according to the first loss and the second loss includes: The parameters of the emotion recognition model are updated according to the first loss to the third loss.

3. The method according to claim 2, characterized in that The image encoder is obtained through pre-training; updating the parameters of the emotion recognition model includes: updating the parameters of the electroencephalogram encoder, the emotion classifier and the induction degree predictor.

4. The method according to claim 2 or 3, characterized in that: The updating of parameters of the emotion recognition model according to the first loss to the third loss includes: The parameters of the emotion recognition model are updated according to the weighted values ​​of the first loss to the third loss; wherein the first loss is the emotion classification loss, the second loss is the feature similarity loss, and the third loss is the induction degree loss.

5. The method according to claim 4, characterized in that The sentiment classification loss is configured with a first weight value, the induction degree loss is configured with a second weight value, and the weight value of the feature similarity loss decreases with the training process.

6. The method according to any one of claims 1 to 3, characterized in that Before inputting the stimulation image frame corresponding to the electroencephalogram sample data into the image encoder, the method further includes: adjusting the stimulation image frame to a preset size and configuring a preset number of channels.

7. A computer device, configured with an emotion recognition model trained using the method described in any one of claims 1 to 6, for performing emotion recognition.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images