A Cross-Domain Sentiment Analysis Method and Storage Medium Based on Knowledge Transfer

Abstract

The present invention discloses a cross-domain sentiment analysis method and a storage medium based on knowledge transfer. To solve the problem of difficult data annotation, it is usually necessary to transfer knowledge from other domains to the target domain. There are many ways of transfer, including model-based, instance-based, and relationship-based, which have been greatly developed in the context of deep learning. However, in the process of transfer, the role of external knowledge bases is always ignored. Therefore, trying to use the knowledge of external knowledge bases to enhance the effect of transfer learning has become a new method. The method of the present invention includes: constructing a semantic network module, a feature extraction module, and a domain adversarial training module. Process five product review data sets of Amazon, extract some words with relatively high frequencies of occurrence and connect them to ConceptNet to form a semantic network. Then use RGCN to process the semantic network and extract feature representations. Finally, add domain adversarial training to improve the robustness of the model.

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning, natural language processing, and cross-domain sentiment analysis. In particular, external common sense knowledge is added to the domain adaptation module, enhancing the discriminative ability and robustness of the cross-domain sentiment analysis model. Background Art

[0002] Traditional sentiment analysis methods include dictionary-based and non-neural network classifier methods. The dictionary-based method determines the sentiment polarity by constructing a sentiment dictionary, extracting sentiment values, etc., and classifies sentences according to the dictionary. The dictionary-based method is easy to operate, but it highly depends on the sentiment dictionary and ignores the positional relationship between words. Non-neural network classifiers belong to supervised machine learning methods. The most common classifiers are Naive Bayes (NB), maximum entropy, and support vector machines. These classifiers perform well in different classification fields, such as image classification and text classification.

[0003] Deep learning plays an important role in many fields such as artificial intelligence, computer vision, and the Internet of Things. With the great development of deep learning in the field of natural language processing, the application of deep learning in sentiment analysis is becoming more and more extensive. Neural network models for sentiment analysis are divided into three categories: one is the CNN-based model, the second is the RNN-based model, and the hybrid neural network model. However, deep learning methods require a large amount of labeled data, and it is difficult to label a large amount of data in a field in practice. At this time, transfer learning emerged.

[0004] Transfer learning can obtain shared knowledge from the source task and then apply it to the learning of the target task, thereby improving the generalization performance of the model and reducing the time and resource consumption caused by the large amount of labeled data required for the target task. And the appropriate introduction of an external knowledge base can improve the accuracy and stability of the transfer model. Summary of the Invention

[0005] The present invention aims to solve the above problems of the prior art. A cross-domain sentiment analysis method based on transfer learning using an external knowledge base is proposed. This method uses ConceptNet (a large-scale knowledge graph) to inject common sense knowledge into the domain adversarial framework to further improve the effect of the model.

[0006] The present invention includes three modules: a semantic network construction module, a semantic graph feature extraction module, and a domain adversarial training module. When learning knowledge in different domains, domain adversarial training is used to reduce the domain difference and improve the robustness of the model. The technical solution of the present invention is as follows:

[0007] A cross-domain sentiment analysis method based on knowledge transfer, comprising the following steps:

[0008] Construct a semantic network, combine the refined text with ConceptNet to form a graph, which is called a "semantic graph", and obtain a semantic network that reflects the relationships between some concepts in various fields.

[0009] Extract semantic features, process the semantic network through RGCN, and obtain the feature representation of the graph.

[0010] Domain adversarial training, including a feature extractor, a domain classifier, and a class predictor, performs transfer learning on the features between two unrelated domains. To reduce the differences between domains, when the feature representations between two unrelated domains are too different, the effect of transfer learning may be very poor. Therefore, domain adversarial training is needed to reduce the domain differences.

[0011] Furthermore, the construction of the semantic network specifically includes:

[0012] 1.1 Represent the graph with a triple <node1, relation, node2>, extract the key information of the comment set, and set it as a node, and observe the relationship with other nodes;

[0013] 1.2 Process the original data set, use a word segmentation tool to divide it into phrases, and ignore the stop words to obtain the first refined text information;

[0014] 1.3 Continue to refine and extract the first refined text information in step 1.2, and extract the words with higher frequencies in the text to obtain the refined text;

[0015] 1.4 Construct a heterogeneous graph network with the refined text and ConceptNet to obtain a semantic network.

[0016] Furthermore, the semantic network is essentially a heterogeneous graph network. In a heterogeneous graph, there are multiple relationships between nodes and between edges. Therefore, using a graph convolutional neural network to process the heterogeneous graph at this time will obviously not achieve good results. At this time, it can be considered to use RGCN, that is, a relational graph convolutional neural network, to process this semantic graph and extract the key information in it. The structure of RGCN is as Figure 2 shown: The RGCN divides a heterogeneous network into multiple homogeneous networks for processing, takes the nodes pointing in and out as adjacent nodes, and at the same time adds self-loop features for feature fusion and participates in node update.

[0017] Define a simple propagation model for calculating the forward propagation update of the entity or node represented by v in the graph i as follows:

[0018]

[0019] Where:

[0020] Denote the node v i The hidden state at the l-th layer;

[0021] σ represents the activation function;

[0022] W represents the weight matrix;

[0023] Denote the set of adjacent indices of node i under the relation r;

[0024] C i,r Denote a problem-specific normalization constant, which can be pre-learned or selected.

[0025] From Figure 2 The structure of RGCN, the processing steps of RGCN are as follows:

[0026] (1) Calculate the state of a single node in the graph according to formula (1);

[0027] (2) Collect the activations (d-dimensional vectors) from adjacent nodes, and then transform them respectively for each relation type (for input and output edges), resulting in 2N relation types. For example, re1_1(in) represents the input edge of relation type 1.

[0028] (3) Cumulate the representations of various relation types and the self-loop features obtained in the form of a normalized sum, and pass them through an activation function (such as ReLU) to obtain the final feature representation. In this way, the update of each node can be calculated in parallel with the shared parameters in the whole graph.

[0029] Furthermore, the purpose of domain adversarial training is to learn the shared features of source domain and target domain instances, so that the classifier trained for the source domain can be applied to the target domain. Domain adversarial training includes a feature extractor, a domain classifier, a class predictor, and the structure is as Figure 3 shown.

[0030] The said feature extractor maps and mixes the source domain samples and target domain samples, and transmits the extracted information to the domain classifier and the class predictor;

[0031] The said domain classifier judges whether the incoming information comes from the source domain or the target domain, and calculates the loss;

[0032] The said class predictor classifies the data from the source domain.

[0033] 1. Feature extractor

[0034] As Figure 3 shown, the feature extractor maps the extracted text information of the source domain and target domain through a multi-layer neural network to obtain the text feature representation M(xs / t ) Then, the extracted feature representations are respectively input into the class predictor and the domain classifier. The work of the feature extractor is completed by the above-mentioned feature extraction module.

[0035] 2. Class Predictor

[0036] As Figure 3 shown, the class predictor classifies the text information extracted by the feature extractor to distinguish the text information in different domains as much as possible. Since the source domain samples are labeled, when extracting features, not only the situation of the subsequent domain discriminator needs to be considered, but also the labeled samples in the source domain are used for supervised training to take into account the classification accuracy.

[0037] The loss function adopted by the class predictor is:

[0038]

[0039] Where:

[0040] K is the total number of labeled samples;

[0041] M(x s ) is the sample information of the source domain;

[0042] C is the trained classifier, a neural network model using softmax as the activation function.

[0043] 3. Domain Classifier

[0044] As Figure 3 shown, the domain classifier classifies the text information extracted by the feature extractor to determine whether the input information comes from the source domain or the target domain and calculates the loss. The training objective of the domain classifier is to classify the input information into the correct domain category as much as possible, which is exactly opposite to the training objective of the feature extractor (due to the existence of the gradient reversal layer). The purpose of the features extracted by the feature extractor is to make the domain discriminator unable to correctly determine which domain the data comes from, thus forming an adversarial relationship.

[0045] The loss function adopted by the domain classifier is the cross-entropy loss function, and the cross-entropy is mainly used to measure the difference between two probability distributions.

[0046] L domain = -logD adv (M(x s )) - log(1 - D adv (M(x t ))

[0047] Where:

[0048] M(x t ) is the sample information of the target domain;

[0049] D adv (M(x s )) represents the probability of the sample in the source domain;

[0050] D adv (M(x t )) represents the probability of the sample in the target domain;

[0051] Therefore, in summary, the model parameters finally trained should satisfy the following relationship:

[0052] θ = arg min max(L label + λL domain )

[0053] Where:

[0054] λ represents the hyperparameter.

[0055] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the cross-domain sentiment analysis method based on knowledge transfer as described above is implemented.

[0056] The advantages and beneficial effects of the present invention are as follows:

[0057] Introduce an external common sense knowledge base to expand semantics:

[0058] Although most works using deep networks rely on automatic feature learning, they rely heavily on annotated data. Therefore, using common sense knowledge in the form of syntactic information or lexical forms as additional input provides a new idea for transfer learning.

[0059] Common sense knowledge includes facts that everyone should know, but the currently trained word embeddings do not utilize these trivial but important knowledge that may improve language understanding. Therefore, the application of common sense knowledge has become a new trend in modern natural language processing research.

[0060] Use RGCN to process the semantic graph and extract key features.

[0061] Add a domain adversarial training network to improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is the semantic graph illustration of the present invention;

[0063] Figure 2 is the RGCN structure illustration of the present invention;

[0064] Figure 3 is the structure illustration of the domain adversarial training module;

[0065] Figure 4 This is the flowchart of the method of the present invention. Detailed implementation manners

[0066] As Figure 4 shown, implementing the method of the present invention requires the following steps:

[0067] I. Construct a semantic network module

[0068] 1. Data set and data preprocessing

[0069] (1) Download the product review data sets of Amazon, including 5 fields: DVA, electronic products, kitchen utensils, music, and books. Divide the data sets in each field into two types: labeled and unlabeled. And for the labeled data sets, divide them into positive and negative classes according to their sentiment polarities;

[0070] (2) Use the jieba word segmentation tool to scan out all the words that can form words in the sentence and remove the stop words;

[0071] (3) Count and sort the number of occurrences of repeated words in the labeled data sets in each field, and retain the top 5000 words and their occurrence frequencies;

[0072] (4) Screen out the proper nouns, nouns, adjectives, and adverbs in the above-mentioned high-frequency vocabulary and save them to a list. These saved words are the "pivots".

[0073] 2. Construct a semantic network conceptGraph with the "pivots" and conceptNet

[0074] (1) Use networkx to convert conceptNet into an overview graph conceptMap

[0075] networkx, as an open-source package in Python, is convenient for users to create, operate, and learn complex networks. Using networkx, networks can be stored in standardized and non-standardized data formats, various random networks and classical networks can be generated, network structures can be analyzed, network models can be established, new network algorithms can be designed, and network drawing can be performed, etc.

[0076] Use networkx to read conceptNet and store it in a matrix in the format of a directed graph.

[0077] The essence of conceptNet is a triple: <starting node, relationship label, ending node>. For example, "a dog has a tail" can be expressed as: <dog, HasA, tail>. Such triples use a graph structure to represent the relationships between words, as Figure 1 shown.

[0078] (2) Construct a semantic network conceptGraph by combining the overview diagram conceptMap with the "pivot".

[0079] II. Feature Extraction Module

[0080] Create a PyG subgraph

[0081] Create a list of nodes to be used as seeds for constructing the PyG subgraph (using k-hops). The goal is to start only from recipes and ingredients, as all nodes passing through the GNN will be classified, and these are our main targets.

[0082] Define some metrics required for layer sizes

[0083] Perform semi-supervised node classification using PyTorch Geometric<9000204>

[0084] For the text classification task, a small portion of the source data has true labels for the text. Use the model to train on this labeled data.

[0085] Train the network to obtain the optimal weights of the model.

[0086] III. Domain Adversarial Training Module

[0087] 1. Use the feature extractor to map and mix the source domain samples and the target domain samples, making it impossible for the domain discriminator to distinguish which domain the data comes from.

[0088] 2. Use the label predictor to classify the data from the source domain and try to distinguish the correct labels as much as possible.[[ID=]]

[0089] 3. Use the domain classifier to classify the data in the feature space and try to distinguish which domain the data comes from as much as possible. )]]

[0090] IV. Model Evaluation

[0091] Use the trained model to perform sentiment polarity classification on the text in the target domain and evaluate the running results of the model. The selected evaluation metric is accuracy (Accuracy).

[0092]

[0093] Among them:

[0094] TP + TN represents the number of samples predicted correctly;

[0095] K represents the total number of samples.

[0096] Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cross-domain sentiment analysis method based on knowledge transfer, characterized in that It includes the following steps: Construct a semantic network, combine the refined text with ConceptNet to form a semantic network; Extract semantic features, process the semantic network through RGCN to obtain the feature representation of the graph. The RGCN divides a heterogeneous network into multiple homogeneous networks for processing, treats the nodes pointing inside and outside as adjacent nodes, and adds self-loop features at the same time for feature fusion and participates in node update; Define a propagation model for calculating the forward propagation update of the entity or node represented by v in the graph i as follows: Where: Denote the node v i The hidden state at the l-th layer; σ represents the activation function; W represents the weight matrix; Denote the set of adjacent indices of node i under relationship r; C i,r represents a problem-specific normalization constant, pre-learned or selected; The processing steps of RGCN are as follows: (1) Calculate the state of a single node in the graph according to formula (1); (2) Collect the activations from adjacent nodes, and then perform transformations for each relationship type respectively, resulting in 2N relationship types; (3) Cumulate the representations of various relationship types and self-loop features obtained in a normalized sum form and pass through the activation function to obtain the final feature representation. In this way, the update of each node is calculated in parallel with the shared parameters in the whole graph; Domain adversarial training includes a feature extractor, a domain classifier, and a class predictor, and performs transfer learning on the features between two unrelated domains; The feature extractor obtains the text feature representation M(x s / t ) by mapping the text information of the source domain and the target domain extracted through a multi-layer neural network, and then respectively inputs the extracted feature representations into the class predictor and the domain classifier; The class predictor classifies the text information extracted by the feature extractor to distinguish the text information in different domains. The loss function adopted by the class predictor is: Among them: K is the total number of labeled samples; M(x s ) is the sample information of the source domain; C is the trained classifier, a neural network model using softmax as the activation function; The domain classifier classifies the text information extracted by the feature extractor to judge whether the incoming information comes from the source domain or the target domain and calculates the loss. The loss function adopted by the domain classifier is the cross-entropy loss function, and cross-entropy is used to measure the difference between two probability distributions; L domain = -logD adv (M(x s )) - log(1 - D adv (M(x t )) Among them: M(x t ) is the sample information of the target domain; D adv (M(x s )) represents the probability of the sample in the source domain; D adv (M(x t )) represents the probability of the sample in the target domain; Therefore, in summary, the model parameters finally trained satisfy the following relationship: θ = arg min max(L label + λL domain ) Where: λ represents the hyperparameter.

2. The cross-domain sentiment analysis method based on knowledge transfer according to claim 1, wherein: The construction of the semantic network specifically includes: 1.1 Represent the graph with a triple <node1, relationship, node2>, extract the key information of the comment set, set it as a node, and observe the relationship with other nodes; 1.2 Process the original data set, use a word segmentation tool to divide it into phrases, and ignore the stop words to obtain the first refined text information; 1.3 Continue to refine and extract the first refined text information in step 1.2, and take out the words with high frequency in the text to obtain the refined text; 1.4 Construct a heterogeneous graph network with the refined text and ConceptNet to obtain a semantic network.

3. The cross-domain sentiment analysis method based on knowledge transfer according to claim 1, characterized in that: The feature extractor maps and mixes the source domain samples and target domain samples and passes the extracted information to the domain classifier; The domain classifier judges whether the incoming information comes from the source domain or the target domain and calculates the loss; The class predictor classifies the data from the source domain.

4. A computer-readable storage medium, characterized in that: A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the cross-domain sentiment analysis method based on knowledge transfer as described in any one of claims 1-3.

Images