Sentiment quadruple extraction method based on boundary token labeling

Abstract

The application relates to the technical field of natural language processing, and relates to a sentiment quadruple extraction method based on boundary token labeling, which comprises the following steps: 1) using a pre-trained language model as a backbone to generate token representation of semantic context; 2) introducing a syntax dependency tree into the token representation by using GCN; 3) enumerating all possible token pairs, and respectively inputting the corresponding token representations in 1) and 2) into CLN to respectively generate representations of all token pairs with context semantics and syntax dependency; 4) fusing the two token pair representations through a linear layer to obtain a final token pair representation, and using a multi-label classifier to predict labels; and 5) restoring structured sentiment quadruples from the labels. The application can preferably perform sentiment quadruple extraction.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically, to a method for extracting sentiment quadruples based on boundary token annotation. Background Technology

[0002] Structured Sentiment Quadruple Extraction (SSQE): All the aforementioned studies have neglected the extraction of the opinion holder, which plays a crucial role in opinion expression. Structured Sentiment Quadruple Extraction, a recently proposed subtask by ABSA, extracts four times the holder, aspect target item, corresponding opinion item, and expressed sentiment. Some researchers have employed a token-based bilexical dependency parsing model, extracting span and relation by establishing relationships between token pairs. However, the model suffers from an imbalance in the label ratio between span prediction and span relation prediction, and some overlapping sentiment quadruples cannot be identified by the model. Other researchers have extracted four times the sentiment by introducing a novel labeling strategy containing two sets of token pair labels: a basic label set and a complete label set. Although they captured various token relations through the complete label set and gently applied these relations to the hidden layer, the process still suffers from a label ratio imbalance. Summary of the Invention

[0003] The present invention provides a sentiment quadruple extraction method based on boundary token annotation, which can overcome some or all the defects of the prior art.

[0004] The sentiment quadruple extraction method based on boundary token annotation according to the present invention includes the following steps:

[0005] 1) Use a pre-trained language model as the backbone to generate token representations of semantic context;

[0006] 2) Introduce syntactic dependency trees into token representations using GCN;

[0007] 3) Enumerate all possible token pairs, and input the corresponding token representations from 1) and 2) into CLN respectively to generate representations of all token pairs with contextual semantics and syntactic dependencies;

[0008] 4) The two token pair representations are fused through a linear layer to obtain the final token pair representation, and a multi-label classifier is used to predict the label;

[0009] 5) Reconstruct the structured emotional quadruple from the tags.

[0010] Preferably, step 1) specifically involves:

[0011] Given an input sentence S = {x1, x2, ..., x...} nThe word sequence is input into a pre-trained language model to generate a sequence of context-labeled embeddings. Where d h This indicates the dimension of the token.

[0012] Preferably, step 2) specifically involves: performing dependency parsing of the sentence and generating a syntactic dependency tree, then converting it into an adjacency matrix.

[0013]

[0014] a ij This indicates whether there is a dependency arc between the i-th word and the j-th word in the matrix;

[0015] Then, the token representation and adjacency matrix output by the pre-trained language model are input into the graph convolutional network GCN to obtain a token representation with syntactic dependency information;

[0016] Specifically, for node i at level l, its hidden representation is obtained by the following equation:

[0017]

[0018] Among them W l b is the trainable weight matrix of the l-th layer. l It is the offset value of the current weight; when l=1, It is the embedding of the j-th token generated by the encoder; It is the embedding of the j-th token generated in the l-th iteration of GCN; N represents the number of word tokens.

[0019] Preferably, in step 3), specifically: the tokens with context representations can be combined into n*n token pairs, which can represent an n*n matrix. Here, to save on GPU memory overhead during training, only the token pairs in the upper triangular matrix are used, and they are flattened into a sequence. Where V k Let L and d represent the lengths of the sequence, where L represents the k-th tag pair. h These are the length of the sequence and the dimension of its representation, respectively.

[0020] word pairs (x i ;x j The representation of V) k regarded as x i The context of h i and x j The context of h j A combination of x, where the combination should mean x j It is x i As a condition; V is calculated through a conditional regularization layer.k :

[0021]

[0022] Among them W α and W β These are trainable weights; b α and b β It is a trainable offset; μ and σ represent the mean and standard deviation over h, respectively;

[0023] h j The standard deviation of an element is calculated as follows:

[0024]

[0025] Where h jq h j The qth dimension;

[0026] Similarly, syntactic dependency features are input into the CLN to generate a representation of syntactic dependency token pairs. in

[0027] U k =CLN(g i g j )

[0028] g i g j They represent h respectively i ,h j Embedded after GCN.

[0029] Preferably, in step 4), linear interpolation is used to combine the two types of token pair representations to generate the final token pair representation. Z k The definition is as follows:

[0030] Z k =αU k +(1-α)V k

[0031] Where α∈[0;1] is an adjustable weighting parameter.

[0032] Preferably, in step 4), a linear layer is used to compute the label score matrix S of the final label pair in order to predict the labels. k ∈R V×|L| The expression is represented as follows: where V is the sequence length represented by the tag pairs, and |L| is the number of tag types.

[0033] Specifically, S k The calculation is as follows:

[0034] Sk =W T Z k +b

[0035] Where W is the trainable weights, b is the bias; S k Each component corresponds to a label; when S k The r-th dimension The k-th token pair is marked with the corresponding label.

[0036] As a preferred option, step 5) specifically involves:

[0037] First, decode all types of opinion holders-opinion pairs and opinion-aspect pairs;

[0038] Then, retain the holder-opinion pairs, where the holder and opinion pair appear in the first obtained role set, and filter out pairs that do not meet the criteria; use the same method for opinion-opinion pairs.

[0039] Finally, the two types of pairs are combined into a complete quadruple, which requires that the two different types of pairs cannot have the same opinion role.

[0040] As a preferred method, the cross-entropy loss function is used for multi-label classification during training:

[0041]

[0042] in These represent the positive and negative classes of the k-th token pair, respectively.

[0043] Unlike existing methods, this invention transforms the task of extracting sentiment quadruples into a joint task of extracting entities and relations. This invention proposes a labeling framework for extracting sentiment quadruples using boundary token pairs that only label entities, thereby avoiding the problem of imbalanced label ratios. Using a multi-label classifier, this invention can identify overlapping sentiment quadruples. Furthermore, this invention uses GCN to introduce syntactic dependency information and CLN to generate high-quality token pair representations, enhancing the recognition of entities and relations. Attached Figure Description

[0044] Figure 1 This is a flowchart of a sentiment quadruple extraction method based on boundary token annotation in an embodiment. Detailed Implementation

[0045] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0046] Example

[0047] like Figure 1 As shown, this embodiment provides a sentiment quadruple extraction method based on boundary token annotation, which includes the following steps:

[0048] 1) Use a pre-trained language model as the backbone to generate token representations of semantic context;

[0049] 2) Introduce syntactic dependency trees into token representations using GCN;

[0050] 3) Enumerate all possible token pairs, and input the corresponding token representations from 1) and 2) into CLN respectively to generate representations of all token pairs with contextual semantics and syntactic dependencies;

[0051] 4) The two token pair representations are fused through a linear layer to obtain the final token pair representation, and a multi-label classifier is used to predict the label;

[0052] 5) Reconstruct the structured emotional quadruple from the tags.

[0053] In step 1), specifically:

[0054] Given an input sentence S = {x1, x2, ..., x...} n The word sequence is input into a pre-trained language model to generate a sequence of context-labeled embeddings. Where d h This indicates the dimension of the token.

[0055] In step 2), specifically:

[0056] Syntactic relationships exist between different words in a sentence, which helps in identifying entities and relations. Attention mechanisms are frequently used to handle these relationships. However, attention mechanisms can fail when faced with complex syntactic structures. Generally, we have found that dependency trees shorten the distances between components in the sentiment quadruple of a sentence, capture syntactic relationships between words, and provide differentiated syntactic paths for any sentence to propagate information up the tree. Similarly, an adjective modifier dependency exists between the words "economy" and "opinion situation," representing the modifying relationship between "economy" and "situation." Syntactic dependency knowledge can help the proposed BTPL better identify opinion roles and the relationships between them.

[0057] To enhance the token pair representation with syntactic dependency information, we first use Stanza (official website, https: / / github.com / stanfordnlp / stanza) to perform dependency parsing of the sentence and generate a syntactic dependency tree, which is then converted into an adjacency matrix:

[0058]

[0059] a ij This indicates whether there is a dependency arc between the i-th word and the j-th word in the matrix;

[0060] Then, the token representation and adjacency matrix output by the pre-trained language model are input into the graph convolutional network GCN to obtain a token representation with syntactic dependency information;

[0061] Specifically, for node i at level l, its hidden representation is obtained by the following equation:

[0062]

[0063] Among them W l b is the trainable weight matrix of the l-th layer. l It is the offset value of the current weight; when l=1, It is the embedding of the j-th token generated by the encoder; It is the embedding of the j-th token generated in the l-th iteration of GCN; N represents the number of word tokens.

[0064] In step 3), specifically:

[0065] Since the purpose of this embodiment is to identify labels between token pairs, generating a large number of high-quality token pairs is crucial. Tokens with context representations can be combined into n*n token pairs, which can represent an n*n matrix. To save on GPU memory during training, only the token pairs in the upper triangular matrix are used, and they are flattened into a sequence. Where V k Let L and d represent the lengths of the sequence, where L represents the k-th tag pair. h These are the length of the sequence and the dimension of its representation, respectively.

[0066] word pairs (x i ;x j The representation of V) k regarded as x i The context of h i and x j The context of h j A combination of x, where the combination should mean x j It is x i As a condition; V is calculated through a conditional regularization layer. k :

[0067]

[0068] Among them W α and W βThese are trainable weights, b α and b β It is a trainable offset; μ and σ represent the mean and standard deviation over h, respectively;

[0069] h j The standard deviation of an element is calculated as follows:

[0070]

[0071] Where h jq h j The qth dimension;

[0072] Similarly, syntactic dependency features are input into the CLN to generate a representation of syntactic dependency token pairs. in

[0073] U k =CLN(g i g j )

[0074] g i g j They represent h respectively i ,h j Embedded after GCN.

[0075] In step 4), linear interpolation is used to combine the two types of token pair representations to generate the final token pair representation. Z k The definition is as follows:

[0076] Z k =αU k +(1-α)V k

[0077] Where α∈[0;1] is an adjustable weighting parameter.

[0078] In step 4), intuitively, the sentiment quadruple can be decomposed into (i) holder, (ii) aspect, (iii) opinion, (iv) polarity, (v) the relationship between holder and opinion, and (vi) the relationship between opinions. Therefore, this embodiment designs a labeling strategy that can simultaneously extract opinion roles and their relationships. In this embodiment, thirteen labels are used, which is not the minimum setting required to extract four times the sentiment. For example, since we already have the labels "holderhead-holder tail" and three opinion head-opinion tail pairs, only the labels "holderhead-opinion head" and "holdertail-opinion tail" are retained to extract the relationship between holders and opinion items. However, simultaneously retaining "holderhead-opinion head" and "holdertail-opinion tail" helps in the mutual verification stage of predicted labels during decoding, thus improving the robustness of the extraction.

[0079] Using the thirteen tags above, we can effectively identify the polarity of holders, aspects, opinions, and sentiments, and their corresponding relationships. For example, since "most" and "people" are the fifth and sixth tags in the sentence, they are the head (first) and tail (last) tags for the majority of holders, respectively. That is, the tag holder head-holder tail is marked in the fifth row and sixth column of the matrix. Similarly, the most concerned are the fifth and eighth tokens in the sentence, which are the head tokens for the most concerned and most worried opinion words, respectively. Therefore, the tag holder head-opinion head is marked in the fifth row and eighth column of the matrix. Only by using an upper triangular matrix and converting it into a sequence can we effectively reduce computational complexity.

[0080] However, when the opinion item appears before the holder or the aspect item appears before the opinion item, the sentence label will be located in the lower triangular matrix. To solve this problem, we use four labels: opinion head-holderhead, opinion tail-holder tail, aspect head-opinion head, and aspect tail-opinion tail. In this way, the label will always appear in the upper triangular matrix.

[0081] In step 4), in order to predict the labels, a linear layer is used to compute the label score matrix S of the final label pairs. k ∈R V ×|L| The expression is represented as follows: where V is the sequence length represented by the tag pairs, and |L| is the number of tag types.

[0082] Specifically, S k The calculation is as follows:

[0083] S k =W T Zk +b

[0084] Where W is the trainable weights, b is the bias; S k Each component corresponds to a label; when S k The r-th dimension The k-th token pair is labeled with the corresponding tag. Since token pairs can be assigned to use multiple tags in this way, we solve the component overlap problem of sentiment quadruples.

[0085] In step 5), specifically:

[0086] First, decode all types of opinion holders-opinion pairs and opinion-aspect pairs;

[0087] Then, retain the holder-opinion pairs, where the holder and opinion pair appear in the first obtained role set, and filter out pairs that do not meet the criteria; use the same method for opinion-opinion pairs.

[0088] Finally, by combining the two types of pairs (holder-opinion and opinion-aspect) into a complete quadruple, it is required that the two different types of pairs cannot have the same opinion role.

[0089] During training, the cross-entropy loss function is used for multi-label classification:

[0090]

[0091] in This represents the positive and negative classes of the k-th token. This represents the score of the r-th relation label of the k-th token pair.

[0092] This embodiment outperforms existing models in most aspects of extracting individual components and complete quadruples, achieving optimal results. Furthermore, ablation experiments were conducted on GCN and CLN structures. Syntactic dependency information and the CLN module significantly improved the performance of span extraction (overall span F1) and sentiment quadruple extraction (SF1). When dependency information was not fused, the overall span F1 and SF1 decreased significantly, with average decreases of 1.8% and 1.6%, respectively. This indicates that syntactic dependency information can effectively enhance token pair representations, thereby improving the extraction performance of sentiment quadruples. In addition, the performance of CLN was examined. When token pair representations were not generated by CLN, the overall span F1 and SF1 decreased by 2.0% and 1.3%, respectively. CLN applies one token as a condition to another to generate token pair representations. Compared to ordinary concatenation methods, it allows for deeper information interaction between token pairs and generates higher-quality token pair representations. Therefore, removing the CLN module significantly reduced the model's performance in extracting opinion roles and quadruples.

[0093] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A sentiment quadruple extraction method based on boundary token annotation, characterized by: Includes the following steps: 1) Use a pre-trained language model as the backbone to generate token representations of semantic context; 2) Introduce syntactic dependency trees into token representations using GCN; 3) Enumerate all possible token pairs, and input the corresponding token representations from 1) and 2) into CLN respectively to generate representations of all token pairs with contextual semantics and syntactic dependencies; In step 3), specifically: the tokens with context representations are combined into n*n token pairs, representing an n*n matrix. Only the token pairs in the upper triangular matrix are used for representation, and they are flattened into a sequence. V k Let L and d represent the sequence of the k-th tag pair. h These are the length of the sequence and the dimension of its representation, respectively. word pairs (x i ; x j The representation of V) k regarded as x i The context of h i and x j The context of h j A combination of x, where the combination should mean x j It is x i As a condition; V is calculated through a conditional regularization layer. k : ； in and These are trainable weights; and It is a trainable offset; These represent the mean and standard deviation over h, respectively; The standard deviation of an element is calculated as follows: ； in express The q dimension; Similarly, syntactic dependency features are input into the CLN to generate a representation of syntactic dependency token pairs. ,in ； They represent Embedded after GCN; 4) The two token pair representations are fused through a linear layer to obtain the final token pair representation, and a multi-label classifier is used to predict the label; 5) Reconstruct the structured emotional quadruple from the tags.

2. The sentiment quadruple extraction method based on boundary token annotation as described in claim 1, characterized in that: In step 1), specifically: Given an input sentence S = { x 1, x 2, ..., x n The word sequence is input into a pre-trained language model to generate a sequence of context-labeled embeddings. H = { h 1 , h 2 , ..., h n } ,in d h Indicates the dimension of the token.

3. The sentiment quadruple extraction method based on boundary token annotation as described in claim 2, characterized in that: In step 2), specifically: perform dependency parsing of the sentence and generate a syntactic dependency tree, then convert it into an adjacency matrix. ； Represents the first in the matrix i The word and the first j Are there dependency arcs between the words? Then, the token representation and adjacency matrix output by the pre-trained language model are input into the graph convolutional network GCN to obtain a token representation with syntactic dependency information; Specifically, for the first l Layer nodes i Its hidden representation is obtained by the following equation: ； in It is the first l The trainable weight matrix of the layer, It is the offset value of the current weight; when l When = 1, It is the first one generated by the encoder. j Embedding of individual tokens; It is the embedding of the j-th token generated in the l-th iteration of GCN; N represents the number of word tokens.

4. The sentiment quadruple extraction method based on boundary token annotation as described in claim 3, characterized in that: In step 4), linear interpolation is used to combine the two types of token pair representations to generate the final token pair representation. Z k The definition is as follows: ； in It is an adjustable weighting parameter.

5. The sentiment quadruple extraction method based on boundary token annotation as described in claim 4, characterized in that: In step 4), a linear layer is used to compute the label score matrix for the final label pair in order to predict the labels. Let V be the sequence length represented by the tag pairs, and |L| be the number of tag types. Specifically, The calculation is as follows: ； Where W is the trainable weight and b is the bias; Each component corresponds to a label; when The r-th dimension If the value is greater than 0, the k-th token pair is marked with the corresponding label.

6. The sentiment quadruple extraction method based on boundary token annotation as described in claim 5, characterized in that: In step 5), specifically: First, decode all types of opinion holders-opinion pairs and opinion-aspect pairs; Then, retain the holder-opinion pairs, where the holder and opinion pair appear in the first obtained role set, and filter out pairs that do not meet the criteria; use the same method for opinion-opinion pairs. Finally, the two types of pairs are combined into a complete quadruple, which requires that the two different types of pairs cannot have the same opinion role.

7. The sentiment quadruple extraction method based on boundary token annotation as described in claim 6, characterized in that: During training, the cross-entropy loss function is used for multi-label classification: ； in , where represent the positive and negative classes of the k-th token pair, respectively.

Images