Cross-Domain End-to-End Aspect-Level Sentiment Analysis Method Based on Shared Syntactic Representations

Abstract

The present invention relates to a cross-domain end-to-end aspect-level sentiment analysis method based on shared syntactic representations, belonging to the technical field of natural language processing. The present invention includes the steps of: obtaining an original syntactic dependency tree on the unlabeled texts of the source domain and the target domain by using the text processing tool spaCy; pruning and reshaping the original syntactic dependency tree through syntactic rules to construct a simplified syntactic dependency tree, and at the same time obtaining the corresponding part-of-speech tag sequence and simplified syntactic tag sequence of the text; using the part-of-speech and simplified syntactic information as self-supervised signals to fine-tune BERT to obtain a domain-invariant representation integrating syntactic information; training a word-level classifier to obtain word-level weights, performing a weighted aspect-level sentiment analysis task on the labeled texts of the source domain, and optimizing through a weighted cross-entropy loss; training a cross-domain end-to-end aspect-level sentiment analysis model. The present invention can effectively shorten the differences existing between the source domain and the target domain, and provide technical support for aspect-level sentiment analysis of the target domain lacking labels.

Description

Technical Field

[0001] The present invention relates to a cross - domain end - to - end aspect - level sentiment analysis method based on shared syntactic representations, belonging to the technical field of natural language processing. Background Art

[0002] End - to - end aspect - level sentiment analysis (E2E - ABSA) is usually defined as a supervised sequence labeling problem. However, models based on supervised learning usually can only achieve good performance in a specific domain. To expand their use to different domains, the most direct solution is to collect a large amount of additional data in other domains for model training. However, the fine - grained tasks increase the difficulty of annotation, and it is usually very difficult to obtain data in unknown domains. The proposal of cross - domain aspect - level sentiment analysis can train a model for the target domain lacking labels based on a large amount of labeled data in the source domain, thereby reducing the cost of labeled data.

[0003] Aspect - level sentiment analysis models within a single domain have achieved good performance. However, in real - world scenarios, data from multiple domains are often involved. Aspect terms from different domains usually have significant differences, and the previously trained models may not have prior knowledge of common terms in other domains. Re - collecting labeled data is expensive, and retraining the model with this data is time - consuming. To reduce the cost of implementing cross - domain aspect - level sentiment analysis tasks, existing research mostly uses domain adaptation methods to achieve cross - domain sentiment analysis tasks. In particular, feature - based domain adaptation aims to learn domain - independent representations for aspect - level sentiment analysis tasks. Its core idea is structure - correspondence learning, and using structure - correspondence can effectively narrow the gap between domains.

[0004] In cross - domain sentiment analysis tasks, most existing work focuses on the coarse - grained level for predicting sentence - level or document - level sentiment polarities. In contrast, due to the difficulty of fine - grained domain adaptation, only a few methods are used for cross - domain aspect - level sentiment analysis. Most existing research uses feature - based domain adaptation to solve cross - domain aspect - level sentiment analysis problems. Syntactic relations, as the fulcrum of cross - domain fine - grained tasks, play an important role in the information propagation between aspect terms and opinion terms. For example, Wang and Pan use the syntactic relations of the domain to construct auxiliary tasks to narrow the gap between domains. However, existing research only considers the syntactic relations between words and ignores that in aspect - level sentiment analysis tasks, words with the same syntactic distance from aspect terms are not necessarily equally important. Directly using the initial syntactic parsing results will lead to inaccurate prediction of sentiment polarities. To address the above problems, a method for simplifying syntactic dependency trees is proposed to shorten the syntactic distance between aspect terms and their corresponding opinion terms, enhance the ability to perceive the association between aspect terms and opinion terms, and obtain more effective domain - invariant representations for cross - domain aspect - level sentiment analysis tasks. Summary of the Invention

[0005] To solve the above problems, the present invention provides a cross-domain end-to-end aspect-level sentiment analysis method based on shared syntactic representations. The present invention can shorten the syntactic distance between aspect terms and their corresponding opinion terms, enhance the ability to perceive the association between aspect terms and opinion terms, and obtain domain-invariant representations that are more effective for cross-domain aspect-level sentiment analysis tasks.

[0006] The technical solution of the present invention is: a cross-domain end-to-end aspect-level sentiment analysis method based on shared syntactic representations. The specific steps of the method are as follows:

[0007] Step 1, on the unlabeled text W of the source domain and the target domain, use the text processing tool spaCy to obtain the original syntactic dependency tree G d ;

[0008] Step 2, prune and reshape the original syntactic dependency tree G d through syntactic rules to obtain a simplified syntactic dependency tree G ′ d and the corresponding simplified syntactic label sequence D of the text ′ W and part-of-speech label sequence P W ;

[0009] Step 3, encode the text to obtain a context representation H that fuses syntactic information;

[0010] Step 4, use part-of-speech and simplified syntactic information as self-supervised signals to fine-tune BERT to obtain a domain-invariant representation that fuses syntactic information;

[0011] Step 5, train a word-level classifier to obtain word-level weights α i , perform weighting on the labeled text of the source domain to implement the aspect-level sentiment analysis task, and optimize through weighted cross-entropy loss;

[0012] Step 6, train a cross-domain end-to-end aspect-level sentiment analysis model, and use the trained model for sentiment analysis.

[0013] As a further solution of the present invention, in step 2, the specific steps of pruning and reshaping the original syntactic dependency tree through syntactic rules to obtain a simplified syntactic dependency tree and the corresponding simplified syntactic label sequence and part-of-speech label sequence of the text are as follows:

[0014] Step 2.1, traverse the original syntactic dependency tree G d starting from the root node to obtain syntactic arcs that connect the core word to the dependent word and carry syntactic and part-of-speech information;

[0015] Step 2.2: Make a judgment according to the syntactic rules. If the core word and the dependent word conform to the rules, add a syntactic arc from the parent node of the core word to the dependent word. The syntactic relationship is the same as the dependency relationship between the core word and its parent node. At the same time, disconnect the syntactic arc from the core word to the dependent word to obtain the simplified syntactic dependency tree G ′ d ;

[0016] Step 2.3: According to the simplified syntactic dependency tree, assign part-of-speech tags and simplified syntactic tags to each word in the sentence to obtain the part-of-speech tag sequence P W and the simplified syntactic tag sequence D ′ W 。

[0017] As a further solution of the present invention, in step 3, first convert the unlabeled text sequence into a continuous word embedding e including word embedding, paragraph embedding, position embedding, part-of-speech embedding, and simplified syntactic embedding. Among them, the word embedding, paragraph embedding, and position embedding are initialized by pre-trained BERT; the part-of-speech embedding and simplified syntactic embedding are randomly initialized using the part-of-speech tags and simplified syntactic tags as the embedding matrices, and are trained using the unlabeled data from the source domain and the target domain. Then, the word embedding is transformed into a context representation H through a multi-layer transformer, and the calculation formula is H = transformer(E).

[0018] As a further solution of the present invention, in step 4, select 25% of the words from the unlabeled text, and use [MASK] to replace the words, part-of-speech tags, and simplified syntactic tags in the original sentence at the same time to construct two auxiliary tasks of mask-based part-of-speech prediction and syntactic prediction.

[0019] Assume that the word masked by [MASK] is w j , and in the part-of-speech prediction task, the corresponding part-of-speech tag is predicted using the following formula:

[0020]

[0021] Then use cross-entropy loss for optimization, and the loss function is as follows:

[0022]

[0023] Where I(i) is an indicator function, which is equal to 1 if replaced by [MASK], otherwise equal to 0.

[0024] Similarly, assume that the word masked by [MASK] is w j , there is a syntactic arc connecting the core word w i to the dependent word w j , first obtain the core word representation and the dependent word representation, and the calculation formula is as follows:

[0025]

[0026]

[0027] The final syntactic dependency representation is where ; represents the concatenation operation of vectors, and ⊙ represents the multiplication operation of vectors

[0028] In the syntactic prediction task, the corresponding syntactic labels are predicted using the following formula:

[0029]

[0030] Then, cross-entropy loss is used for optimization, and the loss function is as follows:

[0031]

[0032] where I(j) is an indicator function, which is equal to 1 if [MASK] is replaced, and 0 otherwise.

[0033] Through two auxiliary tasks of syntactic label prediction and part-of-speech prediction, perform feature-based domain adaptation L feature = λL dep + L pis , where λ is used to control the contribution ratio of the two auxiliary tasks.

[0034] As a further solution of the present invention, in step 5, a word-level classifier is trained to obtain the domain distribution of each word, and its loss function is L d , and the ratio of the target domain probability to the source domain probability is used as the weight α of each word i , to implement the aspect-level sentiment analysis task, and it is optimized through weighted cross-entropy loss:

[0035]

[0036]

[0037] The overall loss L of the aspect-level sentiment analysis task ABSA = L d + L as ;

[0038] As a further solution of the present invention, in step 6, the two auxiliary tasks of part-of-speech prediction and syntactic prediction and the aspect-level sentiment analysis task are jointly trained in a multi-task learning manner, L = L feature + L ABSA .

[0039] In step 2, the original syntactic dependency tree is pruned and reshaped through syntactic rules to shorten the syntactic distance between aspect terms and opinion terms, and then part-of-speech tags and simplified syntactic tags are assigned to each word in the sentence to learn domain-invariant representations that are more conducive to cross-domain aspect-level sentiment analysis tasks.

[0040] The present invention can summarize syntactic rules through statistical analysis of the dataset and prune and reshape the original syntactic dependency tree according to the syntactic rules.

[0041] The beneficial effects of the present invention are:

[0042] 1. The present invention uses text processing tools to simplify the syntactic tree according to the task, capture the direct association between aspect and perceived aspect opinion terms, eliminate the syntactic distance, and establish a domain-invariant structure of simplified syntax.

[0043] 2. The present invention obtains domain-invariant representations integrating syntactic information by constructing part-of-speech prediction and syntactic prediction auxiliary tasks, and further shortens the differences between different domains by using word-level weighting, which helps the model to achieve cross-domain end-to-end aspect-level sentiment analysis tasks and provides technical support for aspect-level sentiment analysis of the target domain lacking labels. Brief Description of the Drawings

[0044] Figure 1 is the overall flowchart of the present invention;

[0045] Figure 2 is a schematic diagram illustrating the changes in the syntactic tree structure and the corresponding syntactic tags of words before and after the simplification of the syntactic dependency tree. Detailed Description of the Invention

[0046] Example 1: As Figure 1 - Figure 2 shown, the cross-domain end-to-end aspect-level sentiment analysis method based on shared syntactic representations includes:

[0047] Step 1: Obtain the original syntactic dependency tree. On the unlabeled text W = {w1,..., w N} of the source domain and the target domain, use the text processing tool spaCy for syntactic parsing to obtain the original syntactic dependency tree G d (W, A, R), where A is used to indicate whether there is a syntactic arc connection between two words, and R represents the dependency relationship between the two words connected by the syntactic arc;

[0048] Step 2: Simplify the syntactic dependency tree. First, statistically analyze a large amount of data to summarize syntactic rules; then traverse the original syntactic dependency tree starting from the root node to obtain syntactic arcs connecting the core words to the dependent words with syntactic and part-of-speech information; then, make a judgment according to the syntactic rules. If the core word and the dependent word conform to the rules, add a syntactic arc from the parent node of the core word to the dependent word, and the syntactic relationship is the same as the dependency relationship between the core word and its parent node. At the same time, disconnect the syntactic arc from the core word to the dependent word to obtain the simplified syntactic dependency tree G′ d (W, A′, R′), according to the simplified syntactic dependency tree, sequentially assign part-of-speech tags and simplified syntactic tags to each word in the sentence to obtain the part-of-speech tag sequence P W ={P1,…,P N} and the simplified syntactic tag sequence D′ W ={D′1,…,D′ N}.

[0049] Step 3: Text encoding. First, convert the unlabeled text sequence into continuous word embeddings e = {e1, e2, …, e N}, where each word embedding e i consists of five embeddings. The word embedding, paragraph embedding, and position embedding are initialized using pre-trained BERT. The part-of-speech embedding and simplified syntactic embedding are randomly initialized using the part-of-speech tag and simplified syntactic tag as the embedding matrix respectively, and are trained using the unlabeled data from the source domain and the target domain. Then, transform the word embeddings into context representations H = {h1, …, h N} through multiple layers of transformers, and the calculation formula is H = transformer(E).

[0050] Step 4: Construct auxiliary tasks for part-of-speech prediction and syntactic prediction. Select 25% of the words from the unlabeled text, and use [MASK] to replace the words, part-of-speech tags, and simplified syntactic tags in the original sentence at the same time to construct two auxiliary tasks for part-of-speech prediction and syntactic prediction based on masking. Assume the word masked by [MASK] is w j , and use softmax to predict the part-of-speech tag corresponding to w j

[0051]

[0052] where W p and b p are both learnable parameters.

[0053] Then use cross-entropy loss for optimization, and the loss function is as follows:

[0054] ​

[0055] Where I(i) is an indicator function that equals 1 if [MASK] is replaced and 0 otherwise.

[0056] Similarly, assume the word masked by [MASK] is w j , there exists a syntactic arc connecting the head word w i to the dependent word w j . First, obtain the head word representation and the dependent word representation

[0057]

[0058]

[0059] The final syntactic dependency representation is where ; represents the concatenation operation of vectors, and ⊙ represents the multiplication operation of vectors

[0060] Use softmax to predict the syntactic label of w j corresponding to the syntactic label

[0061]

[0062] Then, use cross-entropy loss for optimization. The loss function is as follows:

[0063]

[0064] where I(j) is an indicator function that equals 1 if [MASK] is replaced and 0 otherwise.

[0065] Execute feature-based domain adaptation L through two auxiliary tasks: syntactic label prediction and part-of-speech prediction feature = λL dep + L pos , where λ is used to control the contribution ratio of the two auxiliary tasks.

[0066] Step 5: Obtain word-level weights. Train a word-level classifier using unlabeled data in the source domain and the target domain to obtain the word domain distribution probability. The distribution probability and the calculation process of the loss function are as follows

[0067]

[0068]

[0069] Use the ratio of the target domain probability to the source domain probability as the weight α of each word i , implement the aspect-level sentiment analysis task, and optimize it through weighted cross-entropy loss:

[0070]

[0071]

[0072] The overall loss L of the aspect-level sentiment analysis task ABSA = L d + L as ;

[0073] Step 6, model training. Perform predictions on the target test instances to obtain the results of cross-domain aspect-level sentiment analysis. First, through two auxiliary tasks of syntactic label prediction and part-of-speech prediction, perform feature-based domain adaptation L feature = λL dep + L pos , where λ is used to control the contribution ratio of the two auxiliary tasks. Then, for the aspect-level sentiment analysis task, dynamically learn word-level weights and jointly update the loss L d and L as . Finally, jointly train the two auxiliary tasks and the aspect-level sentiment analysis task in the form of multi-tasking L = L feature + L ABsA .

[0074] A large amount of analysis was carried out on the dataset to summarize the syntactic rules, as shown in Table 1:

[0075] Table 1 is a summary of syntactic rules

[0076]

[0077]

[0078] A represents an aspect term composed of a single word; O represents an opinion term composed of a single word; W represents a certain word in an aspect term composed of multiple words.

[0079] Select datasets in four domains of Laptop, Restaurant, Device, and Service to construct ten transfer pairs from the source domain to the target domain. For BERT, the BERT-base-uncased model was adopted, all BERT layers were fine-tuned, Adam was used for optimization, the batch size was set to 32, and the learning rate was set to 3·10 -5 , for the training of auxiliary tasks, complete words instead of sub-words were masked during the [MASK] process to reduce the impact on the tokenizer; λ was 0.1, and the Adam optimizer was used. At a learning rate of 2·10 -5 , 3·10 -5 and 5·10 -5, grid search algorithm was used to tune the parameters with batch sizes of 16, 32, and 64, and the Micro-F1 score was used to evaluate the performance of the model. Finally, the parameter combination with the smallest error on the validation set was selected. All experiments were repeated 5 times, and the average results of the 5 runs were finally reported.

[0080] Among the existing deep learning frameworks, a cross-domain model relevant to the task and representative was selected as the baseline for comparison. The comparison results are shown in Table 2:

[0081] Table 2 shows the comparison results of the cross-domain end-to-end aspect-level sentiment analysis task

[0082]

[0083]

[0084] The baseline models included therein are Heir-Joint, RNSCN, AD-SAL, BERT-base, BERT-DANN, BERT-UDA, and AHF. Obviously, the method proposed in the present invention achieved the optimal result with an average Micro-F1 value of 38.58% on all ten transfer pairs composed of four datasets relative to all baseline models under all datasets. Compared with the AHF model, it was improved by 0.92%.

[0085] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments, and various changes can be made without departing from the spirit of the present invention within the scope of knowledge possessed by those of ordinary skill in the art.

Claims

1. An end-to-end cross-domain aspect-level sentiment analysis method based on shared syntactic representations, characterized in that, Including: Step 1. On the unlabeled text W in the source domain and the target domain, use the text processing tool spaCy to obtain the original syntactic dependency tree G d ; Step 2: Take the original syntactic dependency tree G d Through pruning and reshaping by syntactic rules, obtain the simplified syntactic dependency tree G ′ d and the corresponding simplified syntactic label sequence D of the text ′ W and the part-of-speech label sequence P W ; Step 3: Encode the text to obtain the context representation H that integrates syntactic information; Step 4: Use part-of-speech and simplified syntactic information as self-supervised signals to fine-tune BERT to obtain a domain-invariant representation that integrates syntactic information; Step 5: Train a word-level classifier to obtain word-level weights α i , perform weighting on the labeled text in the source domain to achieve aspect-level sentiment analysis, and optimize it using weighted cross-entropy loss; Step 6: Train a cross-domain end-to-end aspect-level sentiment analysis model and use the trained model for sentiment analysis; The specific steps of step 2 are as follows: Step 2.

1. Traverse the original syntactic dependency tree G d starting from the root node, to obtain syntactic arcs that connect the core words to the dependent words and carry syntactic and part-of-speech information; Step 2.

2. Judge according to syntactic rules. If the core word and the dependent word conform to the rules, add the parent node of the core word to the syntactic arc of the dependent word, and the syntactic relationship is the same as the dependency relationship between the core word and its parent node. At the same time, disconnect the syntactic arc from the core word to the dependent word to obtain the simplified syntactic dependency tree G ′ d ; Step 2.

3. Assign part-of-speech tags and simplified syntactic tags to each word in the sentence according to the simplified syntactic dependency tree, and obtain the part-of-speech tag sequence P corresponding to the text W and the simplified syntactic tag sequence D ′ W .

2. The cross-domain end-to-end aspect-level sentiment analysis method based on shared syntactic representations according to claim 1, characterized in that: In step 3, first convert the unlabeled text sequence into a continuous word embedding e that includes word embedding, paragraph embedding, position embedding, part-of-speech embedding, and simplified syntactic embedding, and then convert the word embedding into the context representation H through multiple layers of transformers.

3. The cross-domain end-to-end aspect-level sentiment analysis method based on shared syntactic representations according to claim 1, characterized in that: In step 4, select 25% of the words from the unlabeled text, use [MASK] to replace the words, part-of-speech tags, and simplified syntactic tags in the original sentence, and construct two auxiliary tasks of mask-based part-of-speech prediction and syntactic prediction.

4. The cross-domain end-to-end aspect-level sentiment analysis method based on shared syntactic representations according to claim 1, wherein: In step 5, train a word-level classifier to obtain the domain distribution of each word, use the ratio of the target domain probability to the source domain probability as the weight of each word, implement aspect-level sentiment analysis, and optimize it through weighted cross-entropy loss.

5. The cross-domain end-to-end aspect-level sentiment analysis method based on shared syntactic representations according to claim 1, wherein: In step 6, jointly train the two auxiliary tasks of part-of-speech prediction and syntactic prediction and the aspect-level sentiment analysis task in a multi-task learning manner.

Images