A semi-supervised facial emotion recognition method based on brain-computer collaborative training

Abstract

The present invention discloses a semi-supervised facial emotion recognition method based on brain-computer collaborative training, which trains a cognitive model and a classification model on labeled images, uses the trained cognitive model and classification model to predict pseudo labels for unlabeled data, and uses the predicted consistent labels as pseudo labels for the images. The cognitive model and the classification model are updated and trained on the training set after the pseudo labels are amplified, and then the unlabeled images are re-predicted, and this process is repeated; the classification model with the best detection effect is used as the final classification model. The image to be tested is input into the final classification model, and the final classification model outputs the emotion type of the object in the image to be tested. The present invention obtains higher quality pseudo labels by alternately training the cognitive model and the classification model, and mutually verifying the prediction results, thereby preventing a single model from overfitting the pseudo-label data that predicts errors, thereby making more effective use of the unlabeled image data.

Description

Technical Field

[0001] The present invention belongs to the intersection of collaborative training, semi-supervised learning and emotion recognition, and specifically relates to a semi-supervised facial emotion recognition method based on brain-computer collaborative training. Background Art

[0002] Machine learning has achieved remarkable results in facial emotion recognition (FER) with the help of large labeled datasets, but manually labeling datasets is very expensive. Semi-supervised learning (SSL) is a machine learning method that aims to improve model performance by leveraging both labeled and unlabeled data. Semi-supervised learning is effective in many practical applications, including facial emotion recognition. Co-training is a well-known SSL paradigm, in which two redundant view classifiers are iteratively trained on different views, with each classifier's predictions augmenting the other's training set.

[0003] Because EEG signals contain both emotional cognition and visual information about the brain's response to visual imagery, EEG and visual images provide multi-perspective data suitable for collaborative training. Visual images reflect external visual information, while EEG signals reflect internal cognitive states. Brain-computer collaborative training complements traditional collaborative training, following a framework of iterative training and increasing training samples. It enables multi-perspective fusion, providing complementary information to improve performance. Visual images have been widely used in machine learning, using convolutional neural networks (CNNs) tailored for visual image classification. Visual images can be used to stimulate EEG signals, introducing new perspectives from cognitive models. EEG signals reflect neuronal activity and brain responses. Previous studies have shown that EEG signals contain both cognitive and visual information about visual stimuli. This provides the foundation for EEG signals to become a new perspective for collaborative training. Summary of the Invention

[0004] The purpose of this invention is to propose a semi-supervised facial emotion recognition method based on brain-computer collaborative training. EEG signals contain cognitive information about brain activity and visual information about the visual objects that triggered them. Similar to the goals of brain-computer interface (BCI) research, this invention provides a brain-computer collaborative training method that uses EEG signals and visual images as two perspectives for iterative learning.

[0005] In a first aspect, the present invention provides a semi-supervised facial emotion recognition method based on brain-computer collaborative training, the specific steps of which are as follows:

[0006] Step 1: Obtain labeled image data and EEG data; the labeled image data constitutes the training set. Unlabeled images are collected; these unlabeled images contain objects with different emotion categories but have not been labeled. EEG representations are extracted from the EEG data and classified according to the label category. The mean of all EEG representations corresponding to each category is taken as the EEG class representation for each category.

[0007] Step 2: Build a classification model to identify the emotion category of objects in the image; train the classification model using the training set; and build a cognitive model that maps images to EEG space. The cognitive model is trained using the training set and the EEG class representations corresponding to each sample.

[0008] Step 3: Input the unused portion of the unlabeled image into the classification model; use the output of the classification model as the first pseudo-label for the unlabeled image. Input the unlabeled image into the cognitive model to obtain an image representation; measure the similarity between the EEG representation and the image representation, and take the category corresponding to the EEG representation with the highest similarity as the second pseudo-label for the unlabeled image.

[0009] Step 4: Determine whether the first pseudo label and the second pseudo label of each unlabeled image are consistent; then convert the unlabeled image whose first pseudo label and the second pseudo label are consistent into a pseudo-labeled image; and expand the pseudo-labeled image into the training set.

[0010] Step 5: Use the training set to train the classification model and the cognitive model; and re-execute steps 3 and 4 until all unlabeled images have been used.

[0011] Step 6: Use the test set to test the classification model obtained in each round of iteration, and use the classification model with the best detection effect as the final classification model.

[0012] Step 7: Input the tested image into the final classification model; the final classification model outputs the emotion type of the object in the tested image.

[0013] Preferably, in step 1, the model for extracting EEG representations includes a temporal convolution block, a depthwise convolution block, and a separable convolution block, which are used to hierarchically process different input dimensions. The output of the separable convolution block is provided to a fully connected layer; the fully connected layer outputs the EEG representation.

[0014] Preferably, all convolutional layers in the temporal convolution block, the depthwise convolution block and the separable convolution block are batch normalized and activated by rectified linear units.

[0015] Preferably, the cognitive model and the classification model in step 2 both include a convolutional neural network consisting of three convolutional modules. Each convolutional module includes a convolution layer, a normalization layer, a nonlinear activation layer, and a maximum pooling layer. The output of the convolutional neural network in the cognitive model is connected to a fully connected layer projected to the EEG representation dimension, and the output of the convolutional neural network in the classification model is connected to a fully connected layer projected to the label class dimension.

[0016] Preferably, the cognitive model is trained using a triplet marginal loss function. During training, for each image representation v1 output by the cognitive model, two pairs of image representation / EEG representations (v1, e1) and (v1, e2) are given; the EEG representation e1 is a positive term of the same category as the image representation v1, and the EEG representation e2 is a negative term of a different category from the image representation v1. The triplet marginal loss function is expressed as follows:

[0017] max(d(v1,e1)-d(v1,e2)+margin,0)

[0018] Where d(·,·) is the Euclidean distance function and margin is the marginal parameter.

[0019] Preferably, the classification model described in step 5 uses cross entropy as a loss function for training.

[0020] Preferably, the image size of the image data in step 1 is 32×32; the number of channels of the EEG data is 62, and the number of records sampled for each channel is 500.

[0021] Preferably, the EEG data in step 1 is filtered to retain signals within a frequency range of 1 Hz to 75 Hz.

[0022] Preferably, there are seven types of emotions to be recognized, namely anger, disgust, fear, sadness, surprise, neutrality, and happiness.

[0023] In a second aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the aforementioned semi-supervised facial emotion recognition method based on brain-computer enhanced intelligence.

[0024] In a third aspect, the present invention provides a readable storage medium storing a computer program; when the computer program is executed by a processor, it is used to implement the aforementioned semi-supervised facial emotion recognition method based on brain-computer enhanced intelligence.

[0025] The beneficial effects of the present invention are as follows:

[0026] 1. The present invention obtains two sets of prediction results by alternately training the cognitive model and the classification model, and verifies each other to obtain higher-quality pseudo-labels, thereby reducing the pseudo-label data predicted incorrectly due to overfitting of a single model, thereby making fuller and more efficient use of unlabeled image data.

[0027] 2. The present invention provides a semi-supervised facial emotion recognition method based on brain-computer collaborative training, which uses cognitive models and classification models to expand the training set based on unlabeled images, and the cognitive model and classification model are iteratively updated using part of the unlabeled images each time, which helps to obtain the best classification model, thereby improving the performance of facial emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the present invention;

[0029] Figure 2 A framework diagram for collaborative iterative training of the classification model and cognitive model in the present invention;

[0030] Figure 3 This is a schematic diagram of cognitive model learning in step 3 of the present invention. DETAILED DESCRIPTION

[0031] like Figure 1 、 2 As shown in Figure 3, a semi-supervised facial emotion recognition method based on brain-computer collaborative training is described. The specific process is as follows:

[0032] Step 1: Obtain image data and EEG data

[0033] The present invention uses facial emotion images in the Chinese Facial Affective Image System (CFAPS) data set to induce EEG signals. In the CFAPS data set, the emotions expressed by facial images are divided into seven types (anger, disgust, fear, sadness, surprise, neutrality, and happiness). Before the experiment begins, a "+" sign will be displayed to prompt the subject to concentrate. The display time of each facial emotion image is 0.5 seconds. There is a 10-second black image as a buffer between different types of facial emotion images. After the acquisition of the EEG signal is completed, preprocessing will be performed to remove artifacts and improve the signal-to-noise ratio of the EEG signal. Preprocessing includes removing invalid fragments in the original data and removing artifacts caused by eye movements, blinking and muscle tremors. In addition, a Butterworth filter is used to filter out signal frequencies between 1-75Hz. In this embodiment, the EEG signal is mainly collected using the NeuroScan64-lead EEG signal cap.

[0034] Step 2: EEG representation learning

[0035] In the field of CNN applications in brain-computer interface (BCI) systems, EEGNet demonstrates an effective CNN design for EEG classification. Therefore, the EEGNet model is used to map EEG signals into EEG representations. It is a convolutional neural network consisting of temporal, depthwise, and separable convolution blocks, which can hierarchically process different input dimensions. All convolutional layers in the temporal, depthwise, and separable convolution blocks undergo batch normalization and rectified linear unit activation. The output of the separable convolution block is then fed into the final fully connected layer, whose size matches the EEG representation dimension. The EEG representations output by the fully connected layer are classified by label category, and the EEG representations within each category are averaged as the common representation for that category. A total of seven EEG class representations are obtained, one for each emotion category, summarizing the common cognitive patterns of that emotion.

[0036] Step 3: Build classification model and cognitive model

[0037] like Figure 2 As shown in the figure, the cognitive model and classification model are trained on labeled images to initialize the two models. The cognitive model and classification model are convolutional neural networks composed of three convolutional modules, each of which includes a convolutional layer, a normalization layer, a nonlinear activation layer, and a maximum pooling layer. The output of the convolutional neural network in the cognitive model is connected to a fully connected layer that projects to the EEG representation dimension, while the output of the convolutional neural network in the classification model is connected to a fully connected layer that projects to the label class dimension.

[0038] For the classification model, an image self-learning strategy is used to train the initial model using labeled images, and the trained model is used to predict unlabeled images as pseudo labels for visual images.

[0039] For the cognitive model, the category of the visual image is determined by measuring the similarity between the EEG class representation and the image representation. The cognitive model maps the visual image to the EEG space, and its only goal is to maximize the similarity between the corresponding data pairs (image representation and EEG class representation). In order to abstract the learning process from the task, the present invention uses triple marginal loss to train the cognitive model, with the aim of mapping the image to the EEG class representation space of the corresponding category, while keeping it away from the EEG class representation that does not correspond to the category. Therefore, for each image representation v1 output by the cognitive model, given two pairs of image representation / EEG class representation (v1, e1), and (v1, e2), the present invention regards v1 as an anchor item, e1 as a positive item of the same category as v1, and e2 as a negative item of a different category from v1. The loss function for training the cognitive model adopts the triple marginal loss function, which is expressed as:

[0040] max(d(v1,e1)-d(v1,e2)+margin,0)

[0041] Here, d(v, e) is the Euclidean distance function, which calculates the Euclidean distance between v and e. Margin is a predefined parameter that ensures sufficient distance between positive and negative terms. The loss is zero only when the distance (v1, e1) is less than the distance (v1, e2) and the difference is greater than the margin. The similarity between the visual representation of the unlabeled image and the representations of the seven EEG categories is measured using the trained cognitive model. The category corresponding to the EEG category representation with the highest similarity score is used as the pseudo-label for the visual image.

[0042] Step 4: Predict unlabeled images

[0043] Input some or all of the unused unlabeled images into the classification model; use the output of the classification model as the first pseudo-label for the unlabeled images. Input the unlabeled images into the cognitive model to obtain image representations; measure the similarity between the EEG representations and the image representations, and select the category corresponding to the EEG representation with the highest similarity as the second pseudo-label for the unlabeled images. Determine whether the first and second pseudo-labels of each unlabeled image are consistent; convert unlabeled images whose first and second pseudo-labels are consistent into pseudo-labeled images; and augment the pseudo-labeled images into the training set.

[0044] Step 5: Data enhancement for cognitive models and classification models

[0045] The training set after pseudo-label images is augmented to further train the cognitive model and classification model respectively, and step 4 is repeated after the training of the two models is completed until all unlabeled images are used.

[0046] Step 6: Image Classification

[0047] The classification model obtained in each round of iteration is tested using the test set, and the classification model with the best detection effect is used as the final classification model. The image under test is input into the final classification model, and the final classification model outputs the emotion type of the object in the image under test. Cross entropy is used as the loss function for training the final classification model, and the final classification model is trained using the training set obtained by expansion in step 5, and its effectiveness is verified in the test set. In order to perform quantitative evaluation, the experiment uses accuracy, precision, recall and F1 score indicators. Since the CFAPS dataset does not have a predefined labeled / unlabeled split, the present invention divides the dataset into four subsets: labeled data, unlabeled data, validation set and test set, with ratios of 4:4:1:1 and 6:2:1:1, respectively.

[0048] Table 1. Brain-computer collaborative training performance on the CFAPS dataset.

[0049]

[0050]

[0051] As shown in Table 1, the overall classification performance of the model first increases and then decreases with increasing iterations, demonstrating that collaborative training effectively improves model performance. Collaborative training enables the two models to complement each other. Iterative enhancements of the two models continuously improve pseudo-label quality, fully utilize unlabeled data, and enhance facial emotion recognition performance.

Claims

1. A semi-supervised facial emotion recognition method based on brain-computer collaborative training, characterized by: The specific steps are as follows: Step 1: Obtain labeled image data and EEG data; the labeled image data constitutes a training set; collect unlabeled images; extract EEG representations from the EEG data, and classify the EEG representations according to the label category, and take the average of all EEG representations corresponding to each category as the EEG class representation of each category; Step 2: Build a classification model to identify the emotion category of the object in the image; and use the training set to train the classification model; build a cognitive model that maps the image to the EEG space; and use the training set and the EEG class representation corresponding to each sample to train the cognitive model; Step 3: Input the unused portion of the unlabeled image into the classification model; use the output of the classification model as the first pseudo label of the unlabeled image; The unlabeled image is input into the cognitive model to obtain the image representation; the similarity between the EEG representation and the image representation is measured respectively, and the category corresponding to the EEG representation with the highest similarity is taken as the second pseudo label of the unlabeled image; Step 4: Determine whether the first pseudo label and the second pseudo label of each unlabeled image are consistent; then convert the unlabeled image whose first pseudo label and the second pseudo label are consistent into a pseudo-labeled image; Augment the training set with pseudo-labeled images; Step 5: Use the training set to train the classification model and cognitive model; And re-execute steps 3 and 4 until all unlabeled images are used; Step 6: Use the test set to test the classification model obtained in each round of iteration, and use the classification model with the best detection effect as the final classification model; Step 7: Input the tested image into the final classification model; the final classification model outputs the emotion type of the object in the tested image.

2. The method for semi-supervised facial emotion recognition based on brain-computer collaborative training according to claim 1, characterized in that: In the step 1, the model for extracting EEG representation includes a temporal convolution block, a depth convolution block and a separable convolution block, which are used to hierarchically process different input dimensions; the output of the separable convolution block is provided to the fully connected layer; and the fully connected layer outputs the EEG representation.

3. The method for semi-supervised facial emotion recognition based on brain-computer collaborative training according to claim 2, characterized in that: All convolutional layers in the described temporal convolution blocks, depthwise convolution blocks, and separable convolution blocks are batch normalized and activated with rectified linear units.

4. The method for semi-supervised facial emotion recognition based on brain-computer collaborative training according to claim 1, characterized in that: The cognitive model and classification model in step 2 both include a convolutional neural network consisting of three convolutional modules; each convolutional module includes a convolution layer, a normalization layer, a nonlinear activation layer, and a maximum pooling layer; the output of the convolutional neural network in the cognitive model is connected to a fully connected layer projected to the EEG representation dimension, and the output of the convolutional neural network in the classification model is connected to a fully connected layer projected to the label class dimension.

5. The method for semi-supervised facial emotion recognition based on brain-computer collaborative training according to claim 1, characterized in that: The cognitive model is trained using a triplet marginal loss function. During training, for each image representation v1 output by the cognitive model, two pairs of image representation / EEG representations (v1, e1) and (v1, e2) are given. The EEG representation e1 is a positive term of the same category as the image representation v1, and the EEG representation e2 is a negative term of a different category from the image representation v1. The triplet marginal loss function is expressed as follows: max(d(v1,e1)-d(v1,e2)+margin,0) Where d(·,·) is the Euclidean distance function and margin is the marginal parameter.

6. The method for semi-supervised facial emotion recognition based on brain-computer collaborative training according to claim 1, characterized in that: The classification model described in Step 5 uses cross entropy as the loss function for training.

7. The method for semi-supervised facial emotion recognition based on brain-computer collaborative training according to claim 1, characterized in that: The EEG data described in step 1 is filtered to retain signals in the frequency range of 1 Hz-75 Hz.

8. The method for semi-supervised facial emotion recognition based on brain-computer collaborative training according to claim 1, characterized in that: There are seven types of emotions recognized: anger, disgust, fear, sadness, surprise, neutral, and happiness.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The memory stores computer programs; the processor executes a semi-supervised facial emotion recognition method based on brain-computer enhanced intelligence as described in any one of claims 1-8.

10. A readable storage medium storing a computer program; characterized in that: When the computer program is executed by a processor, it is used to implement a semi-supervised facial emotion recognition method based on brain-computer enhanced intelligence as described in any one of claims 1 to 8.

Images