A cross-domain sentiment classification method based on BERT, LSTM and width learning

Abstract

The present invention discloses a cross-domain emotion classification method based on BERT, LSTM and width learning, comprising the following steps: (1) obtaining product review data and preprocessing the review data; (2) using BERT to vectorize the text; (3) using LSTM to extract features from the data to obtain domain-invariant features and domain-specific feature vectors; (4) using the maximum mean difference (MMD) to measure the difference between domain-invariant features and distributions; (5) constructing a domain-invariant classifier based on DIF and a domain-specific classifier based on DSF; (6) co-training the above two classifiers and outputting classification results. The present invention combines the advantages of BERT, LSTM and width learning, so that the system can effectively obtain the features of the source domain, thereby improving the emotion classification effect on the target domain.

Description

Technical Field

[0001] The present invention relates to emotion classification technology, and in particular to a cross-domain emotion classification method based on BERT, LSTM and width learning. Background Art

[0002] According to the 2020 China Economic Annual Report data released by the National Bureau of Statistics, the national online retail sales in 2020 reached 11.76 trillion yuan, an increase of 10.9% over the previous year; among them, the online retail sales of physical goods reached 9.76 trillion yuan, an increase of 14.8%, accounting for 24.9% of the total retail sales of consumer goods, an increase of 4.2 percentage points over the previous year. Such a huge scale of online transactions has generated a huge amount of product review information on the Internet, a large part of which contains various emotions. Due to the huge amount of these review information, it is difficult to mine valuable information by simple manual processing alone; in addition, the distribution ratio of the scale of product review big data in various fields is very uneven; if all fields are modeled separately for sentiment classification of these review texts, it will take a lot of time and manpower. These are the basic problems that need to be solved urgently for cross-domain sentiment classification of product reviews.

[0003] Cross-domain text sentiment classification aims to utilize useful knowledge in the source domain (with sufficient labeled data) to promote sentiment classification in the target domain (with little or no labeled data). It can not only greatly reduce the dependence on labeled data, but also facilitate sentiment classification in the target domain that lacks labeled data. The types of emotions generally include seven types: joy, love, anger, sadness, fear, hatred, and desire; however, the existing technology mainly focuses on sentiment classification based on binary classification (positive, negative) or based on three-category classification (positive, negative, neutral). The sentiment classification method based on binary or three-category classification is difficult to fully express the complex inner world of human beings. It is not only difficult to effectively portray people's subtle emotional changes, but also difficult to comprehensively cover people's psychological state.

[0004] In addition, existing methods for text sentiment classification are generally oriented towards a single domain, and most of them use traditional machine learning models or deep learning models; these models can only classify the sentiment of text in a single domain. In addition, most existing cross-domain methods use deep transfer learning to solve sentiment classification tasks. Compared with traditional sentiment transfer methods, methods based on deep transfer learning can achieve end-to-end cross-domain sentiment transfer tasks and have better robustness and generalization capabilities. However, cross-domain sentiment classification methods based on deep transfer learning are all coarse-grained and cannot handle fine-grained sentiment classification tasks, that is, they cannot solve cross-domain text sentiment classification tasks. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a cross-domain emotion classification method based on BERT (Bidirectional Encoder Representation from Transformers), LSTM (Long Short-Term Memory) and width learning; the present invention combines the advantages of BERT, LSTM and width learning, so that the system can effectively obtain the features of the source domain, thereby improving the emotion classification effect on the target domain.

[0006] In order to achieve the above object, the present invention is implemented by the following technical scheme: a cross-domain emotion classification method based on BERT, LSTM and width learning, comprising the following steps:

[0007] (1) Obtain product review data and preprocess the review data;

[0008] (2) Use BERT to vectorize text;

[0009] (3) Use LSTM to extract features from the data and obtain vectors such as Domain Invariant Feature (DIF) and Domain Specific Feature (DSF);

[0010] (4) Use Maximum Mean Discrepancy (MMD) to measure domain-invariant features and differences between distributions;

[0011] (5) Construct a Domain Invariant Classifier (DIC) based on DIF and a Domain Specific Classifier (DSC) based on DSF;

[0012] (6) The above two classifiers are trained collaboratively and the classification results are output.

[0013] Furthermore, the step (2) is specifically as follows:

[0014] Use BERT to train source domain samples X s and the target domain labeled samples X tl The encoding is as follows:

[0015]

[0016] in and Respectively represent X s and X tl Using the vector of domain-invariant features generated by BERT, and Respectively represent X s and X tl Using the domain-specific feature vectors generated by BERT, BERT inv and BERT spec They respectively represent the encoding operation of DIF and DSF using BERT; Indicates that there is N s The source domain labeled dataset of samples, where represents the i-th sentence in the source domain, represents the one-hot encoding of the i-th labeled in the source domain; Indicates that there is N tl The target domain labeled dataset of samples is represents the i-th sentence in the target domain, represents the one-hot encoding of the i-th labeled in the target domain; and Represents the corresponding model parameters, and l represents the length of the text.

[0017] Furthermore, the step (3) is specifically as follows:

[0018] LSTM is used to extract domain-invariant features and domain-specific features:

[0019]

[0020] in and denote the domain-invariant features of the source and target, and denote and Use the vector generated by LSTM; and represent the domain-specific features of the source and target, and and Use the vector generated by LSTM; LSTM inv and LSTM spec They respectively represent the encoding operation of DIF and DSF using LSTM; and Represents the corresponding parameters, and q represents the dimension of the feature output.

[0021] Furthermore, the step (4) is specifically as follows:

[0022] Step 4-1: Use the maximum mean difference (MMD) to measure domain-invariant features and The differences between the distributions; the specific operations are as follows:

[0023] and The corresponding loss function between distributions is expressed as follows:

[0024]

[0025] By minimizing L sim It can reduce the difference in feature distribution between the source domain and the target domain, thereby achieving the extraction of domain-invariant features;

[0026] Step 4-2: Use negative maximum mean difference (MMD) to measure domain-specific features and The difference between the distributions, the corresponding loss function is expressed as follows:

[0027]

[0028] By minimizing L diff The feature distributions of the source domain and the target domain can be made as different as possible to effectively extract the unique features of the target domain.

[0029] Furthermore, the step (5) is specifically as follows:

[0030] Step 5-1: Construct a domain-invariant classifier DIC based on DIF

[0031] The features extracted by LSTM are nonlinearly mapped to generate multiple groups of enhanced nodes for width learning BL, that is, and Perform nonlinear mapping to generate n inv The i-th group of enhanced nodes is represented as follows:

[0032]

[0033] in represents a nonlinear activation function, and Represent fixed randomly generated weights and biases respectively;

[0034] Therefore, n inv The set of group enhancement nodes is represented as Then the estimated output of DIC is expressed as follows:

[0035]

[0036] in represents the output layer weight of DIC, A inv Represents all input features of DIC;

[0037] Step 5-2: Construct a domain-specific classifier DSC based on DSF

[0038] Since DSC only needs to classify samples in the target area, Perform nonlinear mapping operations to generate n spec The jth group of enhanced nodes is represented as follows:

[0039]

[0040] in and Represent fixed randomly generated weights and biases respectively;

[0041] Therefore, n spec The set of group enhancement nodes is represented as Then the estimated output of DSC is expressed as follows:

[0042]

[0043] in represents the output layer weight of DSC, A spec Represents all input characteristics of DSC;

[0044] Furthermore, the step (6) is specifically as follows:

[0045] Step 6-1: Update the parameters of feature representations DIF and DSF;

[0046] For DIF, the parameters need to be updated and Minimize the following loss function:

[0047]

[0048] Where α represents the weight of the loss term, and L(θ) represents the optimization parameter θ during training;

[0049] L c It is a cross entropy loss function, which is used to measure the difference between the real label and the predicted label in the source domain and the target domain. It is specifically expressed as follows:

[0050]

[0051] in express The i-th row of express The jth row of

[0052] Step 6-2: Update the parameters of classifiers DIC and DSC;

[0053] For DSF, the parameters need to be updated and Minimize the following loss function:

[0054]

[0055] Where γ represents the weight of the loss term, L t It only measures the difference between the actual label and the predicted label in the target field, which is specifically expressed as follows:

[0056]

[0057] Step 6-3: For DIC, you need to find a suitable make The actual output Y inv The difference between them is as small as possible; therefore, it is regarded as a least squares regularization problem, which is specifically expressed as follows:

[0058]

[0059] where Y inv Indicates the actual output, represents the estimated output, λ1 represents the regularization parameter;

[0060] Step 6-4: For DSC, you need to find a suitable make The actual output Y spec The difference between them is as small as possible; therefore, it is regarded as a least squares regularization problem, which is specifically expressed as follows:

[0061]

[0062] where Y spec Indicates the actual output, represents the estimated output, λ2 represents the regularization parameter;

[0063] Step 6-5: By ridge regression, we can get The optimal solution is as follows:

[0064]

[0065] Where λ1 represents the regularization parameter and I represents the identity matrix;

[0066] Step 6-6: By ridge regression, we can get The optimal solution is as follows:

[0067]

[0068] Where λ2 represents the regularization parameter and I represents the identity matrix;

[0069] Step 6-7: Since the training of the domain-invariant classifier and the domain-specific classifier is independent of each other, they map each target domain sample to the domain-invariant feature space and the domain-specific feature space respectively. Therefore, a collaborative training mechanism is introduced to train the model, that is, after each round of training, these two classifiers are used to predict the unlabeled data of the target domain, and p samples with the highest confidence are taken out and added to the training set to participate in the next round of training.

[0070] Compared with the prior art, the present invention has the following advantages and effects: [1] ,TextCNN [2] ,DANN [3] ,PBLM [4] ) found that the method of the present invention is used to process emotion classification. Since it comprehensively uses multiple technical means such as BERT, LSTM and width learning, this scheme integrates the advantages of the aforementioned multiple technologies, so that the system can effectively obtain the characteristics of the source field and effectively improve the emotion classification effect in the target field. The experimental results show that the accuracy of the method proposed in the present invention is better than that of the baseline model. The present invention can be effectively applied to public opinion monitoring, e-commerce, big search, information prediction, personalized recommendation, mental health prediction and other fields, with a wide range of application and good applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a flow chart of the cross-domain emotion classification method based on BERT, LSTM and width learning implemented by the present invention. DETAILED DESCRIPTION

[0072] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0073] Example

[0074] like Figure 1 As shown, the cross-domain emotion classification method based on BERT, LSTM and width learning in the embodiment of the present invention is implemented through the technical solution of the following specific steps:

[0075] Step 1: Obtain product review data (using a crawler system to collect data) and preprocess the review data (remove URLs, convert emoticons in short texts into corresponding emotional words, etc.; then, combine a small amount of manual annotation information and use machine annotation as the main method to complete the annotation and storage of the corpus).

[0076] Step 2: Use the language pre-training model BERT to vectorize the text;

[0077] Use the pre-trained model BERT to train the source domain sample X s and the target domain labeled samples X tl The encoding is as follows:

[0078]

[0079] in and Respectively represent X s and X tl Using the vector of domain-invariant features generated by BERT, and Respectively represent X s and X tl Using the domain-specific feature vectors generated by BERT, BERT inv and BERT spec Represents BERT respectively inv and BERT spec They respectively represent the encoding operation of DIF and DSF using BERT; Indicates that there is N s The source domain labeled dataset of samples, where represents the i-th sentence in the source domain, represents the one-hot encoding of the i-th labeled in the source domain; Indicates that there is N tl The target domain labeled dataset of samples is represents the i-th sentence in the target domain, represents the one-hot encoding of the i-th labeled in the target domain; and Represents the corresponding model parameters, and l represents the length of the text.

[0080] Step 3: Use the long short-term memory network LSTM to extract features from the data and obtain vectors such as domain-invariant features DIF and domain-specific features DSF;

[0081] LSTM is used to extract domain-invariant features and domain-specific features:

[0082]

[0083] in and denote the domain-invariant features of the source and target, and denote and Use the vector generated by LSTM; and represent the domain-specific features of the source and target, and and Use the vector generated by LSTM; LSTM inv and LSTM spec They respectively represent the encoding operation of DIF and DSF using LSTM; and Represents the corresponding parameters, and q represents the dimension of the feature output.

[0084] Step 4: Use the maximum mean difference (MMD) to measure domain invariant features and differences between distributions;

[0085] Step 4-1: Use the maximum mean difference (MMD) to measure domain-invariant features and The differences between the distributions; the specific operations are as follows:

[0086] and The corresponding loss function between distributions is expressed as follows:

[0087]

[0088] By minimizing L sim It can reduce the difference in feature distribution between the source domain and the target domain, thereby achieving the extraction of domain-invariant features;

[0089] Step 4-2: Use negative maximum mean difference (MMD) to measure domain-specific features and The difference between the distributions, the corresponding loss function is expressed as follows:

[0090]

[0091] By minimizing L diff The feature distributions of the source domain and the target domain can be made as different as possible to effectively extract the unique features of the target domain.

[0092] Step: 5: Construct a domain-invariant classifier DIC based on DIF and a domain-specific classifier DSC based on DSF;

[0093] Step 5-1: Construct a domain-invariant classifier DIC based on DIF

[0094] The features extracted by LSTM are nonlinearly mapped to generate multiple groups of enhanced nodes for width learning BL, that is, and Perform nonlinear mapping to generate n inv The i-th group of enhanced nodes is represented as follows:

[0095]

[0096] in represents a nonlinear activation function, and Represent fixed randomly generated weights and biases respectively;

[0097] Therefore, n inv The set of group enhancement nodes is represented as Then the estimated output of DIC is expressed as follows:

[0098]

[0099] in represents the output layer weight of DIC, A inv Represents all input features of DIC;

[0100] Step 5-2: Construct a domain-specific classifier DSC based on DSF

[0101] Since DSC only needs to classify samples in the target area, Perform nonlinear mapping operations to generate n spec The jth group of enhanced nodes is represented as follows:

[0102]

[0103] in and Represent fixed randomly generated weights and biases respectively;

[0104] Therefore, n spec The set of group enhancement nodes is represented as Then the estimated output of DSC is expressed as follows:

[0105]

[0106] in represents the output layer weight of DSC, A spec Represents all input characteristics of the DSC.

[0107] Step 6: Co-train the above two classifiers and output the classification results

[0108] Step 6-1: Update the parameters of feature representations DIF and DSF;

[0109] For DIF, the parameters need to be updated and Minimize the following loss function:

[0110]

[0111] Where α represents the weight of the loss term, and L(θ) represents the optimization parameter θ during training;

[0112] L c It is a categorical cross-entropy loss function that measures the difference between the true label and the predicted label in the source domain and the target domain. It is specifically expressed as follows:

[0113]

[0114] in express The i-th row of express The jth row of

[0115] Step 6-2: Update the parameters of classifiers DIC and DSC

[0116] For DSF, the parameters need to be updated and Minimize the following loss function:

[0117]

[0118] Where γ represents the weight of the loss term, L t It only measures the difference between the actual label and the predicted label in the target field, which is specifically expressed as follows:

[0119]

[0120] Step 6-3: For DIC, you need to find a suitable make The actual output Y inv The difference between them is as small as possible; therefore, it is regarded as a least squares regularization problem, which is specifically expressed as follows:

[0121]

[0122] where Y inv Indicates the actual output, represents the estimated output, λ1 represents the regularization parameter;

[0123] Step 6-4: For DSC, you need to find a suitable make The actual output Y spec The difference between them is as small as possible; therefore, it is regarded as a least squares regularization problem, which is specifically expressed as follows:

[0124]

[0125] where Y spec Indicates the actual output, represents the estimated output, λ2 represents the regularization parameter;

[0126] Step 6-5: By ridge regression, we can get The optimal solution is as follows:

[0127]

[0128] Where λ1 represents the regularization parameter and I represents the identity matrix;

[0129] Step 6-6: By ridge regression, we can get The optimal solution is as follows:

[0130]

[0131] Where λ2 represents the regularization parameter and I represents the identity matrix;

[0132] Step 6-7: Since the training of the domain-invariant classifier and the domain-specific classifier are independent of each other, they map each target domain sample to the domain-invariant feature space and the domain-specific feature space respectively; therefore, a collaborative training mechanism is introduced to train the model, that is, after each round of training, these two classifiers are used to predict the unlabeled data of the target domain, and p samples with the highest confidence are taken out and added to the training set to participate in the next round of training.

[0133] This embodiment collects data sets in four fields: movie reviews, hotels, clothing, and electronic products from Douban, Ctrip, Taobao, and JD.com. There are a total of 41,628 reviews (10,000 movie reviews, 10,009 hotels, 10,301 clothing, and 11,318 electronic products). There are seven types of emotions (happy, touched, angry, sad, fearful, disgusted, and surprised). The preprocessed text data is input into BERT for pre-training to generate word vectors; then, they are input into LSTM for feature extraction, and the output of LSTM is used as the feature node for width learning, and they are nonlinearly transformed to generate enhanced nodes; finally, ridge estimation is used to classify the text emotion type. Four baseline models (Bi-LSTM [1] 、TextCNN [2] DANN [3] PBLM [4] ) were experimentally compared, and the specific parameter settings are as follows: the number of groups of enhanced nodes n inv =n spec =20 and the number of nodes in each group is 50, the number of hidden layers of LSTM is 150, α=γ=0.5, λ=0.001, and p=5. The experimental results show that the method proposed in the present invention is better than the baseline model in terms of accuracy. The experimental results are shown in Table 1.

[0134] Table 1

[0135]

[0136] The above baseline model literatures are:

[0137] [1]Mike Schuster and Kuldip K.Paliwal. Bidirectional recurrent neural networks, IEEE Transactions on Signal Processing, 1997, 45(11): 2673-2681.

[0138] [2] Yoon Kim. Convolutional neural networks for sentence classification. Proceedings of the 2014Conference on Empirical Methods inNatural Language Processing, 2014, pp.1746-1751.

[0139] [3] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 2016, 17(1): 2030-2096.

[0140] [4]Yftah Ziser and Roi Reichart. Pivot based language modeling for improved neural domain adaptation. Proceedings of the 2018Conference of theNorth American Chapter of the Association for Computational Linguistics:HumanLanguage Technologies,2018,pp.1241-1251.

[0141] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A cross-domain sentiment classification method based on BERT, LSTM and width learning, characterized by The following steps are involved: (1) Obtain product review data and preprocess the review data; (2) Use BERT to vectorize text; (3) Use LSTM to extract features from the data and obtain domain-invariant features DIF and domain-specific feature vectors DSF; (4) Using the maximum mean difference (MMD) to measure domain-invariant features and differences between distributions; (5) Construct a domain-invariant classifier based on DIF and a domain-specific classifier based on DSF; (6) Co-train the above two classifiers and output the classification results; The step (2) is: Use BERT to train source domain samples X s and the target domain labeled samples X tl The encoding is as follows: in and Respectively represent X s and X tl Using the vector of domain-invariant features generated by BERT, and Respectively represent X s and X tl Using the domain-specific feature vectors generated by BERT, BERT inv and BERT spec They respectively represent the encoding operation of DIF and DSF using BERT; Indicates that there is N s The source domain labeled dataset of samples, where represents the i-th sentence in the source domain, represents the one-hot encoding of the i-th labeled in the source domain; Indicates that there is N tl The target domain labeled dataset of samples is represents the i-th sentence in the target domain, represents the one-hot encoding of the i-th labeled in the target domain; and represents the corresponding model parameters, l represents the text length; The step (3) is specifically: LSTM is used to extract domain-invariant features and domain-specific features: in and denote the domain-invariant features of the source and target, and denote and Use the vector generated by LSTM; and represent the domain-specific features of the source and target, and and Use the vector generated by LSTM; LSTM inv and LSTM spec They respectively represent the encoding operation of DIF and DSF using LSTM; and represents the corresponding parameters, and q represents the dimension of feature output; The step (4) is specifically as follows: Step 4-1: Use the maximum mean difference (MMD) to measure domain-invariant features and The differences between the distributions; the specific operations are as follows: and The corresponding loss function between distributions is expressed as follows: By minimizing L sim It can reduce the difference in feature distribution between the source domain and the target domain, thereby achieving the extraction of domain-invariant features; Step 4-2: Use negative maximum mean difference (MMD) to measure domain-specific features and The difference between the distributions, the corresponding loss function is expressed as follows: By minimizing L diff The feature distributions of the source domain and the target domain can be made as different as possible to effectively extract the unique features of the target domain; The step (5) is specifically as follows: Step 5-1: Construct a domain-invariant classifier DIC based on DIF The features extracted by LSTM are nonlinearly mapped to generate multiple groups of enhanced nodes for width learning BL, that is, and Perform nonlinear mapping to generate n inv The i-th group of enhanced nodes is represented as follows: in represents a nonlinear activation function, and Represent fixed randomly generated weights and biases respectively; Therefore, n inv The set of group enhancement nodes is represented as Then the estimated output of DIC is expressed as follows: in represents the output layer weight of DIC, A inv Represents all input features of DIC; Step 5-2: Construct a domain-specific classifier DSC based on DSF Since DSC only needs to classify samples in the target area, Perform nonlinear mapping operations to generate n spec The jth group of enhanced nodes is represented as follows: in and Represent fixed randomly generated weights and biases respectively; Therefore, n spec The set of group enhancement nodes is represented as Then the estimated output of DSC is expressed as follows: in represents the output layer weight of DSC, A spec Represents all input characteristics of DSC; The step (6) is specifically as follows: Step 6-1: Update the parameters of feature representations DIF and DSF; For DIF, the parameters need to be updated and Minimize the following loss function: Where α represents the weight of the loss term, and L(θ) represents the optimization parameter θ during training; L c It is a categorical cross-entropy loss function that measures the difference between the true label and the predicted label in the source domain and the target domain. It is specifically expressed as follows: in express The i-th row of express The jth row of Step 6-2: Update the parameters of classifiers DIC and DSC; For DSF, the parameters need to be updated and Minimize the following loss function: Where γ represents the weight of the loss term, L t It only measures the difference between the actual label and the predicted label in the target field, which is expressed as follows: Step 6-3: For DIC, the specific expression is as follows: where Y inv Indicates the actual output, represents the estimated output, λ1 represents the regularization parameter; Step 6-4: For DSC, the specific expression is as follows: where Y spec Indicates the actual output, represents the estimated output, λ2 represents the regularization parameter; Step 6-5: By ridge regression, we can get The optimal solution is as follows: Where λ1 represents the regularization parameter and I represents the identity matrix; Step 6-6: By ridge regression, we can get The optimal solution is as follows: Where λ2 represents the regularization parameter and I represents the identity matrix; Step 6-7: Introduce a collaborative training mechanism to train the model, that is, use these two classifiers to predict the unlabeled data in the target field after each round of training, and take out p samples with the highest confidence and add them to the training set to participate in the next round of training.

Images