A sentiment recognition method and device based on ensemble learning and transfer learning

Abstract

The application discloses a kind of based on ensemble learning and transfer learning sentiment recognition method and device, belong to the field of brain-computer interface of sentiment computing.First based on deep transfer learning algorithm, using source domain labeled data to pre-train sentiment recognition model, then using the pre-trained sentiment recognition model to predict target domain unlabeled data, and select the first several high confidence prediction results as pseudo label, obtain target domain with pseudo label data.Then, by random nonlinear mapping to features, to increase feature diversity.Next, using source domain labeled data and target domain with pseudo label data to construct first data sample set, train a base classifier;Using target domain with pseudo label data and target domain remaining label data to construct second data sample set, train another base classifier.Iteration, finally sum all base classifier results to obtain the final output, to realize the identification of target domain unlabeled data.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of affective computing brain-computer interface, and more particularly, relates to a sentiment recognition method and device based on ensemble learning and transfer learning. BACKGROUND

[0002] Sentiment recognition is an important part of human-computer interaction systems. Sentiment brain-computer interface is a way of sentiment recognition, which analyzes the subject's emotion category corresponding to the current signal by collecting the subject's scalp electroencephalogram signal. In order to reduce the labeled data (referred to as the calibration data of the subject) required for training the sentiment recognition model, the model can be trained with the aid of other users' collected labeled data. The electroencephalogram signals between different users usually differ greatly, which leads to the assumption that the data between other users and the subject does not conform to the independent and identically distributed hypothesis. The model directly trained on other users performs poorly on new subjects, while transfer learning can solve this problem. Transfer learning aims to learn the knowledge of auxiliary data (source domain) and transfer it to target data (target domain), so as to train a high-performance model on the target domain with less or even zero labeled data.

[0003] Transfer learning methods have been widely used in sentiment brain-computer interface, which can be mainly divided into two categories:

[0004] 1) Traditional transfer learning algorithm: hand-crafted features are extracted from electroencephalogram signals, and traditional transfer learning algorithms are used for training. Scholars have proposed differential entropy (DE) - the average energy of electroencephalogram signals in a specific frequency band, which is widely used in the field of sentiment brain-computer interface; and have used transfer component analysis (TCA), kernel principal component analysis (KPCA), transductive parameter transfer (TPT) and other transfer learning methods on the sentiment brain-computer interface task. At the same time, they have also proposed an individual similarity transfer learning framework that measures individual differences using maximum mean discrepancy (MMD).

[0005] 2) Deep transfer learning algorithm: a transfer learning algorithm based on convolutional neural network, which directly extracts features from raw electroencephalogram signals and completes classification. Scholars have used deep transfer learning methods such as domain adaptation neural network (DANN), joint adaptation network (JAN), conditional domain adversarial network (CDAN) in the emotional brain-computer interface task, and tried to combine them with eye movement signal model to improve performance.

[0006] However, the existing emotion recognition algorithm based only on transfer learning has room for improvement in accuracy because it only maps features from the perspective of feature extraction to align different distributions, ignoring the impact of feature mapping on classification accuracy. SUMMARY

[0007] To overcome the defects of the prior art and improve the performance of the deep transfer learning algorithm proposed in the cross-subject task of emotional brain-computer interface, the present application provides an emotion recognition method and device based on ensemble learning and transfer learning. The present application is aimed at the unsupervised transfer learning scenario: a large amount of unlabeled electroencephalogram data has been collected from the target subject (target domain), and the data of other subjects (source domain) with labeled signals is needed to train the model to achieve the highest possible accuracy on the unlabeled data of the target domain.

[0008] To achieve the above-mentioned purpose, in a first aspect, the present application provides an emotion recognition method based on ensemble learning and transfer learning, comprising:

[0009] S1, pre-training an emotion recognition model using source domain labeled data, then using the pre-trained emotion recognition model to predict target domain unlabeled data, and selecting the top several high-confidence prediction results as pseudo-labels to obtain target domain pseudo-labeled data; the emotion recognition model includes a feature extractor and a classifier, the feature extractor is used to extract features from raw electroencephalogram signals, and the classifier is used to identify emotional categories;

[0010] S2, randomly and nonlinearly mapping the data features extracted in S1;

[0011] S3, using the source domain labeled data and the target domain pseudo-labeled data mapped by S2 to construct a first data sample set, and training a base classifier; if there is a sample belonging to the source domain labeled data and the prediction result is wrong, the weight of the sample is set to zero;

[0012] S4, constructing a second data sample set by using the target domain pseudo-labeled data mapped by S2 and the target domain remaining label data, and training another base classifier by constraining the model output to be consistent before and after random disturbance; wherein the label of the target domain remaining label data is calculated by the base classifier trained by S3;

[0013] S5, repeating S2 to S4 until the iteration stopping condition is reached, and summing all base classifier results to obtain the final output to realize the identification of the target domain unlabeled data.

[0014] Further, in S2, the process of random nonlinear mapping is represented as:

[0015] h k (z)=δ[ZS(z τ M k ,μ,σ)],

[0016] ZS[z τ M k ,μ,σ)=(z τ M k -μ)· / σ,

[0017] Wherein, h k (z) represents the mapped feature, δ represents any kind of nonlinear mapping activation function, Z=[z1, z2, …, z n ] represents a data matrix composed of all samples, n is the number of samples, z τ represents the transpose of z, M k represents a randomly generated matrix, μ and σ represent the mean and standard deviation of Z τ M k respectively, and the dot. represents element-by-element operation.

[0018] Further, in S3, in the kth cycle, the loss function of the 2k-1th base classifier is represented as:

[0019]

[0020] Wherein, wherein N T and N S represent the number of samples of the target domain pseudo-labeled data and the source domain labeled data respectively; l represents cross entropy loss, x i and y i represent the feature and label of the ith sample respectively; f 2k-1 represents the 2k-1th base classifier, and F 2k-2 represents the sum of all base classifiers trained before the current step.

[0021] Further, in the S3, when solving the loss function of the (2k-1)th base classifier, the loss of multi-classification is converted into the sum of a series of binary classification losses by using Taylor expansion.

[0022] Further, in the S4, in the second data sample set, a disturbance is added to the features of the target domain remaining label data, the disturbance is composed of randomly generated Gaussian noise; and another base classifier is trained by constraining the outputs of the model to be consistent before and after the disturbance is added.

[0023] To achieve the above object, in a second aspect, the present application provides a sentiment recognition device based on ensemble learning and transfer learning, comprising:

[0024] A data processing unit is configured to pre-train a sentiment recognition model by using source domain labeled data, and then predict target domain unlabeled data by using the pre-trained sentiment recognition model, and select the first several high-confidence prediction results as pseudo labels to obtain target domain pseudo-labeled data; the sentiment recognition model comprises a feature extractor and a classifier, the feature extractor is configured to extract features from original electroencephalogram signals, and the classifier is configured to identify sentiment categories;

[0025] A feature mapping unit is configured to perform random nonlinear mapping on the extracted data features;

[0026] A first training unit is configured to construct a first data sample set by using the mapped source domain labeled data and the target domain pseudo-labeled data, and train a base classifier; wherein if there is a sample belonging to the source domain labeled data and the prediction result is wrong, the weight of the sample is set to zero;

[0027] A second training unit is configured to construct a second data sample set by using the mapped target domain pseudo-labeled data and the target domain remaining label data, and train another base classifier by constraining the model outputs to be consistent before and after the sample is added with random disturbance; wherein the labels of the target domain remaining label data are calculated by the base classifier trained by the first training unit;

[0028] A sentiment recognition unit is configured to repeat the operations of the feature mapping unit, the first training unit and the second training unit until the iteration stopping condition is reached, and sum all the base classifier results to obtain the final output, so as to realize the recognition of the target domain unlabeled data.

[0029] To achieve the above object, in a third aspect, the present application provides an electronic device, comprising: a processor; a memory storing a computer executable program, the program, when executed by the processor, causes the processor to execute the sentiment recognition method based on ensemble learning and transfer learning as described in the first aspect.

[0030] In general, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0031] The first step of the present application is based on a deep transfer learning algorithm, which uses source domain labeled data to pre-train a sentiment recognition model, then uses the pre-trained sentiment recognition model to predict target domain unlabeled data, and selects the top several high-confidence prediction results as pseudo labels to obtain target domain data with pseudo labels. The second step is to increase the diversity of features by random nonlinear mapping of the features. The third step is to use the source domain labeled data and the target domain data with pseudo labels to construct a first data sample set and train a base classifier; because all data are labeled, this is a supervised transfer learning task. The fourth step is to use the target domain data with pseudo labels and the target domain data with remaining labels to construct a second data sample set and train another base classifier; since all data come from the target domain, the data meet the independent and identically distributed assumption, and it is a normal semi-supervised learning task. Repeat steps two to four to continuously train new base classifiers, and the weight of each step sample update will be used as the weight of the sample in the next iteration to calculate the loss. Finally, sum all the base classifier results to obtain the final output to realize the recognition of target domain unlabeled data. In this way, the present application combines ensemble learning with transfer learning on the basis of transfer learning, which can further improve the accuracy of sentiment recognition. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 One of the flowcharts of the sentiment recognition method based on ensemble learning and transfer learning provided by the embodiments of the present application;

[0033] Figure 2 The second flowchart of the sentiment recognition method based on ensemble learning and transfer learning provided by the embodiments of the present application;

[0034] Figure 3 The third flowchart of the sentiment recognition method based on ensemble learning and transfer learning provided by the embodiments of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0036] In the present application, the terms "first", "second", etc. (if any) in the present application and the drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0037] Ensemble learning is to train and fuse multiple models to improve the performance of the base model. Ensemble learning is mainly divided into Bagging and Boosting strategies. Bagging strategy is to select part of the data from the data set to train the model each time, repeat multiple times, and then integrate the multiple models obtained. Bagging strategy can improve the stability of the base model (reduce the variance of multiple experimental results). Boosting strategy is to train a base model with the initial data set, then adjust the weights of different samples in the data set according to the accuracy of the model classification, and the misclassified samples will get more weight and be paid more attention by the model. Repeat the above process multiple times, and finally integrate the multiple base models obtained. LogitBoost is an improvement of Boosting, which uses the gradient of the loss of the training base classifier as the basis for updating the sample weight, which can effectively solve the overfitting problem and improve the generalization performance of the model. Boosting can improve the accuracy of the base model. There is no algorithm idea of combining ensemble learning and transfer learning in sentiment recognition at present.

[0038] Based on this, the present application refines the existing deep transfer learning algorithm by using ensemble learning to obtain better performance, named "SS-TrBoosting", see the following specific embodiments.

[0039] Embodiment one

[0040] Referring to Figure 1 , in combination with Figure 2 and Figure 3 , the present application provides a sentiment recognition method based on ensemble learning and transfer learning, which comprises operations S1 to S5.

[0041] Operation S1, pre-training a sentiment recognition model using source domain labeled data, then using the pre-trained sentiment recognition model to predict the target domain unlabeled data, and selecting the top several high confidence prediction results as pseudo labels to obtain target domain pseudo labeled data; the sentiment recognition model comprises a feature extractor and a classifier, the feature extractor is used to extract features from the original electroencephalogram signal, and the classifier is used to identify the emotion category.

[0042] In this embodiment, sentiment recognition can be understood as, for example, a person watching a video, the brain signals of the person will be recorded, and the labels corresponding to the brain signals can be happy, sad, emotionless, etc.

[0043] As Figure 2As shown, the "Deep Unsupervised / Semi-supervised Transfer Learning Algorithm" section demonstrates an existing deep transfer learning algorithm, which consists of two parts: a feature extractor composed of a convolutional neural network and a classifier composed of a single-layer fully connected neural network. The features output by the feature extractor are used as input to the classifier and also as input to the algorithm of this invention. This classifier is used as the initial classifier for ensemble learning.

[0044] After pre-training, taking a single sample as an example, the feature extractor extracts z from the original EEG signal, and the model output is f(z) = [a1, a2, ..., a...]. c ], a c This refers to the probability of the c-th class being predicted. The final predicted value of the model is the class with the highest predicted probability, i.e., arg max(a i (i = 1, 2, ..., c) We record the predicted class of all samples after the pre-trained model and the probability of that class being predicted, as follows: Where c i This indicates the predicted category of the i-th sample. This indicates that the i-th sample is predicted as c. i The probability of each class is calculated, where n is the number of samples. These samples are then sorted, and the top 10% are used to assume their predicted class as their true class. This results in 10% of the target subject's data being labeled (hereinafter referred to as target domain pseudo-labeled data). The output of this step is the features of all samples and the pre-trained classifier f.

[0045] Operation S2 performs a random nonlinear mapping on the data features extracted in S1.

[0046] In this embodiment, as Figure 3 As shown, let the features and label of a sample in the source domain be (z... S y S The sample with pseudo-labels in the target domain is (z). T y T The unlabeled sample features in the target domain are z. U The three sets of samples are subjected to random nonlinear mappings, and the process of the k-th iteration is as follows:

[0047] h k (z)=δ[ZS(z τ M k ,μ,σ)],

[0048] ZS(z τ M k ,μ,σ)=(z τ M k -μ). / σ,

[0049] where h k (z) represents the mapped features, δ represents an arbitrary nonlinear mapping activation function, such as a sigmoid function; M k is a randomly generated matrix, μ and σ are the mean and standard deviation of Z τ M k , Z = [z1, z2, …, z n ] represents a data matrix composed of all samples, z τ represents the transpose of z, and the dot represents element-wise operation. Finally, we let the mapped features of a sample be x, i.e., x = h k (z).

[0050] Operation S3, a first data sample set is constructed using the source domain labeled data and the target domain pseudo-labeled data mapped in S2, and a base classifier is trained; wherein if there is a sample belonging to the source domain labeled data and the prediction result is wrong, the weight of the sample is set to zero.

[0051] As Figure 3 shown in the "supervised transfer learning" module, we select the source domain labeled data and the target domain pseudo-labeled data obtained in S2 to form a data set:

[0052]

[0053] where N T and N S represent the number of samples of the target domain pseudo-labeled data and the source domain labeled data, respectively, and x T and x S represent that the sample belongs to the target domain and the source domain, respectively.

[0054] Because all data are annotated, this is a supervised transfer learning task, and the LogitBoost algorithm is improved to train the base classifier and update the weight of each sample according to the training loss. In the kth cycle, we optimize the following loss to train the 2k-1th base classifier:

[0055]

[0056] where f 2k-1 represents the newly trained base classifier, l represents the cross-entropy loss, f 2k-2 represents the sum of all base classifiers trained before the current step, and if the current is the first cycle (k = 1), F is the initial classifier f output in S1; if the current is the kth cycle, i.e.

[0057]

[0058] After Taylor expansion of the above loss formula, subtracting and adding terms, and transforming the multi-class loss into the sum of a series of binary classification losses (a C-class problem is transformed into C binary classifications, each binary classification problem determining whether a sample belongs to the j-th class, j = 1, 2, ..., C), the formula is as follows:

[0059]

[0060] in, The weights represent the weights used to calculate the loss for the binary classification problem on the j-th class of the i-th sample.

[0061]

[0062]

[0063] p j (x i ) represents the i-th sample being F 2k-2 The model predicts the probability of a sample x belonging to the j-th class. i For source domain samples (i>N) T And when the model makes a prediction error, i.e., argmax[F 2k-2 (x i )]≠y i If we consider the source domain sample to be too different from the target domain, we will reset its weight to zero, that is, let After training using the above loss, the base classifier f is obtained. 2k-1 The overall integrated model at this point is represented as F. 2k-1 =F 2k-2 +f 2k-1 The updated sample weights are

[0064] Operation S4 involves constructing a second data sample set using the target domain data with pseudo-labels mapped by S2 and the remaining target domain data with labels. Another base classifier is trained by ensuring that the model output remains consistent before and after random perturbation of the samples. The labels of the remaining target domain data are calculated by the base classifier trained by S3.

[0065] like Figure 3 As shown in the "Semi-supervised learning" module, taking the k-th iteration as an example, the model F trained in S3 is first used. 2k-1 Calculate pseudo-labels for all unlabeled data in the target domain:

[0066]

[0067] We select the target domain data with pseudo-labels obtained in S2 (the pseudo-labels are calculated by the pre-trained deep transfer learning model in S1 and are not real labels) and the remaining target domain labeled data (using F...2k-1 The pseudo-label calculated by the model is used as the label of the sample to form a data set, and a disturbance is added to the features of the remaining label data of the target domain, and the disturbance is composed of randomly generated Gaussian noise, and the specific expression is as follows:

[0068]

[0069] Wherein, N T and N U respectively represent the number of pseudo-label data of the target domain and the number of remaining label data samples of the target domain. The disturbance ε i obeys Gaussian distribution with mean 0 and standard deviation ∑.

[0070] Since all the data come from the target domain, the data meet the independent and identically distributed assumption, and it is a normal semi-supervised learning task. For the remaining label data of the target domain, the present application adds a disturbance to it, uses the output of the sample before the disturbance as the pseudo-label of the sample after the disturbance, and trains a new base classifier according to the algorithm of LogitBoost in S3; for the pseudo-label data of the target domain, cross-entropy loss is used. Then, the loss is calculated according to the current data set in S3, and the difference from S3 is that the operation of setting the last to zero is removed. Finally, the trained 2kth base classifier f 2k is obtained, and the integrated model is updated to F 2k =F 2k-1 +f 2k , and the sample weight is calculated in the same way as in S3 (without the zero setting operation).

[0071] Operation S5, repeating S2 to S4 until the iteration stopping condition is reached, and summing all the base classifier results to obtain the final output, to realize the identification of the unlabeled data of the target domain.

[0072] In the present embodiment, the operations S2 to S4 are repeated K times, and the base classifiers are alternately generated in S3 and S4 during the circulation, so that the 2k-1th and 2kth base classifiers are respectively generated in the kth circulation. New base classifiers are continuously trained, and the weight of the sample updated at each step is used as the weight of the sample in the next iteration to calculate the loss; finally, all the base classifier results are summed to obtain the final output.

[0073] Embodiment two

[0074] A sentiment recognition device based on ensemble learning and transfer learning, comprising:

[0075] The data processing unit is configured to pre-train a sentiment recognition model by using source domain labeled data, and to predict target domain unlabeled data by using the pre-trained sentiment recognition model, and to select a plurality of high confidence prediction results as pseudo labels to obtain target domain pseudo labeled data.

[0076] The feature mapping unit is configured to perform random nonlinear mapping on the extracted data features.

[0077] The first training unit is configured to construct a first data sample set by using the mapped source domain labeled data and the target domain pseudo labeled data, and to train a base classifier.

[0078] The second training unit is configured to construct a second data sample set by using the mapped target domain pseudo labeled data and target domain remaining labeled data, and to train another base classifier by constraining the model output to be consistent before and after adding random disturbance.

[0079] The sentiment recognition unit is configured to repeat the operations of the feature mapping unit, the first training unit and the second training unit until an iteration stopping condition is reached, and to sum all base classifier results to obtain a final output to realize recognition of the target domain unlabeled data.

[0080] The related technical solutions are the same as those of Embodiment One, and will not be repeated here.

[0081] Embodiment Three

[0082] An electronic device includes a processor, and a memory storing a computer executable program, which, when executed by the processor, causes the processor to perform the sentiment recognition method based on integrated learning and transfer learning as described in Embodiment One.

[0083] The related technical solutions are the same as those of Embodiment One, and will not be repeated here.

[0084] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A sentiment recognition method based on ensemble learning and transfer learning, characterized in that, include: S1, the emotion recognition model is pre-trained using labeled data from the source domain, and then the pre-trained emotion recognition model is used to predict unlabeled data from the target domain. The top few high-confidence prediction results are selected as pseudo-labels to obtain pseudo-labeled data from the target domain. The emotion recognition model includes a feature extractor and a classifier. The feature extractor is used to extract features from the raw EEG signal, and the classifier is used to identify the emotion category. S2, random nonlinear mapping of the good data features extracted in S1; S3: Construct the first data sample set using the source domain labeled data and the target domain pseudo-labeled data after mapping S2, and train a base classifier; wherein, if there is a sample that belongs to the source domain labeled data and the prediction result is incorrect, then reset the weight of the sample to zero. S4: Construct a second data sample set using the target domain pseudo-labeled data mapped by S2 and the remaining label data of the target domain. Train another base classifier by ensuring that the model output is consistent before and after adding random perturbation to the samples. The labels of the remaining label data of the target domain are calculated by the base classifier trained by S3. S5. Repeat S2 to S4 until the iteration stopping condition is met, and sum the results of all base classifiers to obtain the final output, so as to realize the identification of unlabeled data in the target domain.

2. The emotion recognition method based on ensemble learning and transfer learning according to claim 1, characterized in that, In S2, the process of random nonlinear mapping is represented as follows: h k (z)=δ[ZS(z τ M k ,μ,σ)], ZS(z τ M k ,μ,σ)=(z τ M k -m). / s, Among them, h k (z) represents the mapped feature, δ represents any nonlinear mapping activation function, and Z = [z1, z2, ..., zn]. n ] represents the data matrix consisting of all samples, where n is the number of samples, and z is the number of samples. τ M represents the transpose of z. k Let Z be a randomly generated matrix, where μ and σ represent Z respectively. τ M k The mean and standard deviation are given by the dot (.), which indicates element-wise operations.

3. The emotion recognition method based on ensemble learning and transfer learning according to claim 1, characterized in that, In S3, during the k-th iteration, the loss function of the (2k-1)-th base classifier is expressed as: Wherein, N T and N S These represent the number of samples in the target domain with pseudo-labels and the number of samples in the source domain with labels, respectively; l represents the cross-entropy loss, and x represents the cross-entropy loss. i and y i f represents the feature and label of the i-th sample, respectively; 2k-1 F represents the 2k-1th base classifier. 2k-2 This represents the sum of all base classifiers that have been trained up to the current step.

4. The emotion recognition method based on ensemble learning and transfer learning according to claim 3, characterized in that, In S3, when solving the loss function of the 2k-1 base classifier, Taylor expansion is used to transform the multi-class loss into the sum of a series of binary classification losses.

5. The emotion recognition method based on ensemble learning and transfer learning according to claim 1, characterized in that, In step S4, in the second data sample set, a perturbation is added to the features of the remaining label data in the target domain. The perturbation consists of randomly generated Gaussian noise. Then, another base classifier is trained by ensuring that the output of the model is consistent between the two sets of data before and after the perturbation is added.

6. An emotion recognition device based on ensemble learning and transfer learning, characterized in that, include: The data processing unit is used to pre-train the emotion recognition model using labeled data from the source domain, and then use the pre-trained emotion recognition model to predict unlabeled data from the target domain, and select the top few high-confidence prediction results as pseudo-labels to obtain pseudo-labeled data from the target domain; the emotion recognition model includes a feature extractor and a classifier, the feature extractor is used to extract features from the raw EEG signal, and the classifier is used to identify the emotion category. The feature mapping unit is used to perform random nonlinear mapping on the extracted data features; The first training unit is used to construct a first data sample set using the mapped source domain labeled data and target domain pseudo-labeled data, and to train a base classifier; wherein, if there is a sample that belongs to the source domain labeled data and the prediction result is incorrect, the weight of the sample is reset to zero. The second training unit is used to construct a second data sample set using the mapped target domain pseudo-labeled data and the remaining target domain labeled data, and to train another base classifier by ensuring that the model output is consistent before and after adding random perturbation to the samples; wherein, the labels of the remaining target domain labeled data are calculated by the base classifier trained by the first training unit; The emotion recognition unit is used to repeat the operations of the feature mapping unit, the first training unit, and the second training unit until the iteration stopping condition is met, and to sum the results of all base classifiers to obtain the final output, so as to realize the recognition of unlabeled data in the target domain.

7. An electronic device, characterized in that, include: processor; A memory storing a computer-executable program, which, when executed by the processor, causes the processor to perform the emotion recognition method based on ensemble learning and transfer learning as described in any one of claims 1-5.

Images