An end-to-end attribute-level sentiment analysis method based on sentence component-aware attention mechanism

Abstract

The application relates to an end-to-end attribute-level sentiment analysis method based on a sentence component perception attention mechanism and belongs to the technical field of natural language processing. The application comprises the following steps: firstly, a pre-training language model is used to train a word vector. Secondly, a sentence component perception attention mechanism C-ATT is used to divide sentence components, words belonging to the same component in a sentence divided by the C-ATT pay attention to each other, and words of different components do not pay attention to each other, so that aspect words pay more attention to opinion words divided into the same component. Finally, a linear classifier is used to predict a joint label; experiments are conducted on four public sentiment analysis data sets, and the experimental results show that the end-to-end attribute-level sentiment analysis method based on the sentence component perception attention mechanism is effective.

Description

Technical Field

[0001] This invention relates to an end-to-end attribute-level sentiment analysis method based on a sentence component awareness attention mechanism, belonging to the field of natural language processing technology. Background Technology

[0002] Attribute-based sentiment analysis (ABSA) aims to identify people's emotional inclination towards specific aspects of the same entity in comment text. For example, given a comment "so what if the laptops look chic and cool the after sales support is terrible.", the aspect words mentioned are "look" and "after sales support," corresponding to positive and negative sentiment polarities, respectively. In recent years, ABSA has gradually deepened into two sub-tasks: aspect terms extraction (ATE) and aspect sentiment classification (ASC). ATE aims to extract aspect terms mentioned in the comment text, essentially a sequence labeling task, and related research is extensive. ASC aims to predict the sentiment polarity of aspect terms (e.g., positive, neutral, negative). It is essentially a classification task, and related research is also numerous. However, all of these studies treat the two sub-tasks as separate tasks, either processing them sequentially in a pipeline or focusing only on one sub-task, thus failing to fully utilize the joint information between the two sub-tasks. Based on this, researchers proposed end-to-end aspect-based sentiment analysis (E2E-ABSA) to compensate for this deficiency.

[0003] E2E-ABSA is a subtask of ABSA, aiming to simultaneously perform ATE and ASC in an end-to-end manner. Two research questions for E2E-ABSA are: first, how to improve the model's ability to handle ATE and ASC simultaneously. Related research includes how to obtain boundary information of aspect words in the ATE subtask and how to maintain sentiment consistency in the ASC subtask. Second, how to obtain the connection between ATE and ASC. As mentioned above, ATE and ASC are essentially different tasks; the former is a sequence labeling task, and the latter is a classification task. Obtaining the correlation between the two is key to achieving joint extraction of aspect words and sentiment polarity. Summary of the Invention

[0004] This invention provides an end-to-end attribute-level sentiment analysis method based on a sentence component awareness attention mechanism, for the effective prediction of end-to-end attribute sentiment polarity.

[0005] The technical solution of this invention is: an end-to-end attribute-level sentiment analysis method based on a sentence component awareness attention mechanism, the specific steps of which are as follows:

[0006] Step 1: Obtain a text vectorization representation containing contextual information using a BERT pre-trained model. Where L is the number of layers in the Transformer network in BERT, and T is the text length;

[0007] Step 2: A sentence component awareness coefficient is introduced using a sentence component awareness attention mechanism. This coefficient assigns high weights to words belonging to the same component in the sentence and low weights to words between different components. At the same time, an adjacency attention mechanism is introduced to determine which words in the sentence belong to the same component.

[0008] Step 3: Then, vectorize the text containing contextual information into H. L The input sentence component awareness attention mechanism divides the sentence components and obtains a text representation containing aspect word boundary information and the corresponding emotional polarity features of the aspect words;

[0009] Step 4: Use a simple linear classifier to jointly extract aspect words and their corresponding sentiment polarity predictions;

[0010] Step 5: Use the cross-entropy function to calculate the loss during model training.

[0011] As a further aspect of the present invention, in Step 2, the input consists of three matrices Q, K, and V, each composed of d-dimensional vectors; the input word vectors are transformed into initial query vector q, key vector k, and value vector v through a linear transformation; then, the dot product between q and each k is calculated as the weight coefficient of each v; finally, vectors q, k, and v are combined to form matrices Q, K, and V respectively; specifically, q is multiplied by the dot product of each k and then multiplied by a scaling factor. Finally, the attention weights are obtained through a softmax function. The sentence component-aware attention mechanism C-ATT differs from traditional attention mechanisms in that it introduces a sentence component awareness coefficient Cattention. i,j This coefficient assigns a high weight to words belonging to the same component in a sentence, and a low weight to words belonging to different components.

[0012] As a further aspect of the present invention, the specific steps of Step 2 are as follows:

[0013] Step 2.1: Vectorize the obtained text containing contextual information and perform a linear transformation to obtain the initial query vector q, key vector k, and value vector v. Then, calculate the dot product between q and each k as the weight coefficient for each v. Finally, combine vectors q, k, and v to form matrices Q, K, and V, respectively. Specifically, perform a dot product operation between q and each k, and then multiply by a scaling factor. Finally, the attention weights are obtained by applying a softmax function; the calculation process is as follows:

[0014]

[0015] Step 2.2: Compared to traditional attention mechanisms, the sentence component awareness attention mechanism C-ATT introduces a sentence component awareness coefficient C. i,j This coefficient assigns higher weights to words belonging to the same grammatical component in a sentence, and lower weights to words belonging to different grammatical components. Words belonging to the same grammatical component are mutually influenced, while words belonging to different grammatical components are not influenced. The implementation process is as follows:

[0016]

[0017] Where C is an N×N matrix, ⊙ represents the element-wise multiplication operation, and matrices Q and K are composed of the corresponding q vector and k vector, respectively; in equation (2), the constituent elements of C are C i,j , representing the word w i with w j The probability that they belong to the same component, C i,j When the value is small, it means that the word at position i and the word at position j belong to different components and are not related to each other.

[0018] As a further aspect of the present invention, the calculation process of the sentence component perception coefficient is as follows:

[0019] First, for the input sentence sequence X, calculate w for any word in X. i The two words adjacent to it, w i-1 w i+1 The probability of words belonging to the same component is calculated using a method consistent with traditional attention mechanisms, hence the name "adjacent attention mechanism." This mechanism determines which words in a sentence belong to the same component, and the sentence component perception coefficient C... i,j The algorithm then determines which words in the sentence are closely related and assigns them corresponding high weights; the calculation process of the adjacent attention mechanism is as follows:

[0020]

[0021]

[0022] Where, qi For target word w i The corresponding d model k-dimensional query vector i-1 k i+1 w i Two adjacent words w i-1 w i+1 d model 3D key vector;

[0023] Then, to prevent the model from classifying all words into the same component when segmenting sentence elements later, a softmax function is used to constrain this, as follows:

[0024] p i,j =softmax(p i,i-1 ,p i,i+1 (5)

[0025] Where, p i,i-1 p i,i+1 The results were obtained from formulas (3) and (4) respectively, and p i,i-1 +p i,i+1 =1, in addition, the sentence component perception coefficient C i,j The word w i with w j The probability of belonging to the same component, when p i,i-1 >p i,i+1 At that time, w i It belongs to the same grammatical element as its left neighbor; otherwise, it belongs to the same grammatical element as its right neighbor.

[0026] Equation (5) yields the probability that adjacent words in a sentence belong to the same component. Since p i,i+1 and p i+1,i If the calculation results are not equal, the adjacent attention weights are calculated by averaging, as defined below:

[0027]

[0028] Thus, the weight sequence a = {a1, ..., a1} of the adjacent attention mechanism is obtained. i ,…,a N}, where N is the length of the sentence, a i For target word w i Its right neighboring word w ii1 The probability of belonging to the same component;

[0029] For the target word w i , or with the word w on the left i-1 They belong to the same grammatical component, or are related to the word on the right. i+1If words belong to the same component, then at most only two words can belong to the same sentence component. To achieve the goal of classifying multiple words into the same sentence component, a hierarchical constraint is set. That is, the C-ATT in the same layer shares a weight sequence 'a' of the adjacent attention mechanism. The sentence components classified in the current layer l (l≥0) are used as the basic unit for the classification of sentence components in the next layer. This allows the number of words belonging to the same sentence component to increase as the network deepens. The hierarchical constraint is defined as follows:

[0030]

[0031] in, Calculated from the adjacent attention of layer l-1, The constraints calculated from the current layer l guarantee that words belonging to the same component in layer l-1 must also belong to the same sentence component in layer l.

[0032] The adjacent attention weight sequence of layer l is obtained from formula (7). Then, the sentence component perception coefficient C is calculated according to formula (8). i,j (c i≤k＜j );

[0033]

[0034] Calculate C using a cumulative multiplication method i,j This utilizes the property that multiplying a small probability by another small probability still results in a small probability, thereby improving the model's sensitivity in perceiving breakpoints when segmenting sentence components. In the implementation process, to prevent the obtained probability from being too small, equation (7) is transformed into:

[0035]

[0036] c k This is the weight sequence after adding linear constraints.

[0037] As a further aspect of the present invention, in Steps 3 and 4, the sentence is represented After linear transformation, matrices Q and K are obtained and used as input to C-ATT. After segmenting the sentence into components, the sentence representation is output. Finally, H X Input a linear classifier to predict the label of E2E-ABSA; the calculation process is as follows:

[0038] H C =C-ATT(Q,K) (10)

[0039] P(y i |x i = softmax(W) o H C +bo (11)

[0040] Where W0 and b0 are the parameters that the linear layer needs to learn, y i It is the word x in the input sentence sequence X. i The corresponding output labels.

[0041] As a further aspect of this invention, in Step 5, the cross-entropy function is used to calculate the model training loss, resulting in faster convergence. Furthermore, backpropagation involves multiple productives, thus accelerating the update of the entire weight matrix. Additionally, the derivative of the multi-class cross-entropy loss is simpler, as the loss depends only on the probability of the correct class. Moreover, the derivative of the loss with respect to the input of the softmax activation layer is also simpler.

[0042] This invention addresses two research problems in end-to-end attribute-level sentiment analysis, proposing an end-to-end attribute-level sentiment analysis model based on Sentence Constituent-aware Attention (C-ATT). Specifically, C-ATT divides the input sentence into different components. This paper uses a sentence component-aware coefficient to make aspect words pay more attention to opinion words belonging to the same component, thus assisting in E2E-ABSA. The C-ATT process involves two important algorithms: first, an adjacency attention mechanism, which determines whether two adjacent words in a sentence belong to the same sentence component; and second, hierarchical constraints, which allow the same sentence component to encompass more words. After sentence component division, each word in the sentence is assigned boundary information, and different aspect words and their corresponding opinion words are divided into different components. This prevents aspect words from locating opinion words belonging to other sentence components, thereby ensuring the sentiment consistency within aspect words. The theory and technology were validated on two datasets: product reviews and restaurant reviews. The experimental results fully demonstrate the effectiveness of the proposed method.

[0043] The beneficial effects of this invention are as follows: This invention treats the two subtasks of ABSA as a single sequence labeling task, achieving joint extraction of aspect words and sentiment polarities; this invention uses the pre-trained model BERT to train word vectors containing contextual information; then, it uses the sentence component awareness attention mechanism C-ATT to segment sentence components, obtaining text representations containing aspect word boundary information and corresponding sentiment polarity features; finally, it uses a simple linear classifier to jointly extract aspect words and their corresponding sentiment polarities, predicting the joint label of aspect words and sentiment polarities; comparative experimental results show that the proposed model C-ATT outperforms the baseline model, proving that the operation of segmenting the input sentence into different sentence components is effective. Attached Figure Description

[0044] Figure 1 This is a diagram illustrating the overall architecture of the method of the present invention;

[0045] Figure 2 This is a schematic diagram illustrating the probability calculation of adjacent words belonging to the same component using the adjacent attention mechanism in this invention.

[0046] Figure 3 The impact of the number of layers in C-ATT on its sensitivity to perceived sentence components;

[0047] Figure 4 A visual diagram of the sentence component perception coefficient matrix. Detailed Implementation

[0048] Example 1: As Figures 1-4 As shown, an end-to-end attribute-level sentiment analysis method based on a sentence component awareness attention mechanism is described. The specific steps of the method are as follows:

[0049] Step 1: Obtain a text vectorization representation containing contextual information using a BERT pre-trained model. Where L is the number of layers in the Transformer network in BERT, and T is the text length;

[0050] Specifically, the segment embedding, position embedding, and word embedding of the input sentence sequence X are first integrated (concatenated) as the input H of BRET. 0 ={e1,…,e T Then, the H layer is processed through an L-layer Transformer network. 0 Perform bidirectional deep semantic encoding to obtain a text vectorization representation rich in contextual information.

[0051] This invention uses four public datasets for model training and evaluation. Basic information about the datasets is shown in Table 1. Specifically, the Laptop14 dataset is the publicly available Laptop dataset from SemEval challenge 2014 Task 4; and the Rest14, Rest15, and Rest16 datasets are the publicly available Restaurant datasets from SemEval challenge 2014 Task 4, SemEval Challenge 2015 Task 12, and SemEval Challenge 2016 Task 5, respectively. Here, Sent represents the total number of sentences, and Aspect represents the number of aspect term labels.

[0052] Table 1 Experimental Dataset

[0053]

[0054] Step 2: The sentence component awareness attention mechanism (C-ATT) is used to introduce a sentence component awareness coefficient, which gives high weight to words that belong to the same component in the sentence and low weight to words that belong to different components. At the same time, the adjacent attention mechanism is introduced to determine which words in the sentence belong to the same component.

[0055] As a further aspect of the present invention, in Step 2, the input consists of three matrices Q, K, and V, each composed of d-dimensional vectors; the input word vectors are transformed into initial query vector q, key vector k, and value vector v through a linear transformation; then, the dot product between q and each k is calculated as the weight coefficient of each v; finally, vectors q, k, and v are combined to form matrices Q, K, and V respectively; specifically, q is multiplied by the dot product of each k and then multiplied by a scaling factor. Finally, the attention weights are obtained through a softmax function. The sentence component-aware attention mechanism C-ATT differs from traditional attention mechanisms in that it introduces a sentence component awareness coefficient Cattention. i,j This coefficient assigns a high weight to words belonging to the same component in a sentence, and a low weight to words belonging to different components.

[0056] As a further aspect of the present invention, the specific steps of Step 2 are as follows:

[0057] Step 2.1: Vectorize the obtained text containing contextual information and perform a linear transformation to obtain the initial query vector q, key vector k, and value vector v. Then, calculate the dot product between q and each k as the weight coefficient for each v. Finally, combine vectors q, k, and v to form matrices Q, K, and V, respectively. Specifically, perform a dot product operation between q and each k, and then multiply by a scaling factor. Finally, the attention weights are obtained by applying a softmax function; the calculation process is as follows:

[0058]

[0059] Step 2.2: Compared to traditional attention mechanisms, the sentence component awareness attention mechanism C-ATT introduces a sentence component awareness coefficient C. i,j This coefficient assigns higher weights to words belonging to the same grammatical component in a sentence, and lower weights to words belonging to different grammatical components, such as... Figure 1 As shown, in the Sentence Constituents module, arrows represent Self-Attention, and rectangles represent the segmented sentence components. Words of the same component pay attention to each other, while words of different components do not pay attention to each other. The implementation process is as follows:

[0060]

[0061] Where C is an N×N matrix, ⊙ represents the element-wise multiplication operation, and matrices Q and K are composed of the corresponding q vector and k vector, respectively; in equation (2), the constituent elements of C are C i,j , representing the word w i with w j The probability that they belong to the same component, C i,j When the value is small, it means that the word at position i and the word at position j belong to different components and are not related to each other.

[0062] As a further aspect of the present invention, the calculation process of the sentence component perception coefficient is as follows:

[0063] First, for the input sentence sequence X, calculate w for any word in X. i The two words adjacent to it, w i-1 w i+1 The probability of words belonging to the same component is calculated using a method consistent with traditional attention mechanisms, hence the name "adjacent attention mechanism." This mechanism determines which words in a sentence belong to the same component, and the sentence component perception coefficient C... o,j The algorithm then determines which words in the sentence are closely related and assigns them corresponding high weights; the calculation process of the adjacent attention mechanism is as follows:

[0064]

[0065]

[0066] Where, q i For target word w i The corresponding d model k-dimensional query vector i-1 k i+1 w i Two adjacent words w i-1 w ii1 d model Dimensional key vector; note the difference between the scaling factor in formulas (3) and (4) and formula (2);

[0067] Then, to prevent the model from classifying all words into the same component when segmenting sentence elements later, a softmax function is used to constrain this, as follows:

[0068] p i,j =softmax(p i,i-1 ,p i,i+1 (5)

[0069] Where, p i,i-1 p i,i+1The results were obtained from formulas (3) and (4) respectively, and p i,i-1 +p i,i+1 =1, in addition, the sentence component perception coefficient C i,j The word w i with w j The probability of belonging to the same component, when p i,i-1 >p i,i+1 At that time, w i It belongs to the same grammatical element as its left neighbor; otherwise, it belongs to the same grammatical element as its right neighbor.

[0070] Equation (5) yields the probability that adjacent words in a sentence belong to the same grammatical component, such as... Figure 2 As shown in Layer 3, due to p i,i+1 and p i+1,i The calculation results are not equal, such as p. 2,3 ≠p 3,2 The neighbor attention weights are calculated using an averaging method, and are defined as follows:

[0071]

[0072] Thus, the weight sequence a = {a1, ..., a1} of the adjacent attention mechanism is obtained. i ,…,a N}, where N is the length of the sentence, a i For target word w i Its right neighboring word w i+1 The probability of belonging to the same component;

[0073] For the target word w i , or with the word w on the left i-1 They belong to the same grammatical component, or are related to the word on the right. i+1 If words belong to the same component, then at most only two words can belong to the same sentence component. To achieve the goal of classifying multiple words into the same sentence component, a hierarchical constraint is set. That is, the C-ATT in the same layer shares a weight sequence 'a' of the adjacent attention mechanism. The sentence components classified in the current layer l (l≥0) are used as the basic unit for the classification of sentence components in the next layer. Thus, as the network deepens, the number of words belonging to the same sentence component increases. For example... Figure 2 In this context, Layer 1 can have at most two words belonging to the same sentence component, while Layer 2 can have three. The hierarchical constraints are defined as follows:

[0074]

[0075] in, Calculated from the adjacent attention of layer l-1, Calculated from the current layer l, the constraints at this layer guarantee that words belonging to the same component in layer l-1 must also belong to the same sentence component in layer l; initialization is as follows: Figure 2 As shown in the Initialize section, different words belong to different grammatical components. The initial value is 0.

[0076] The adjacent attention weight sequence of layer l is obtained from formula (7). Then, the sentence component perception coefficient C is calculated according to formula (8). i,j (c i≤k＜j );

[0077]

[0078] Calculate C using a cumulative multiplication method i,j This utilizes the property that multiplying a small probability by another small probability still results in a small probability, thereby improving the model's sensitivity in perceiving breakpoints when segmenting sentence components. In the implementation process, to prevent the obtained probability from being too small, equation (7) is transformed into:

[0079]

[0080] c k This is the weight sequence after adding linear constraints.

[0081] Step 3: Linear transformation and linear classifier to obtain sentiment polarity prediction: Then, the text containing contextual information is vectorized into a representation H. L The input sentence component awareness attention mechanism divides the sentence components and obtains a text representation containing aspect word boundary information and the corresponding emotional polarity features of the aspect words;

[0082] Step 4: Use a simple linear classifier to jointly extract aspect words and their corresponding sentiment polarity predictions;

[0083] As a further aspect of the present invention, in Step 3 and Step 4, as follows: Figure 1 As shown, the sentence is represented After linear transformation, matrices Q and K are obtained and used as input to C-ATT. After segmenting the sentence into components, the sentence representation is output. Finally, H C Input a linear classifier to predict the label of E2E-ABSA; the calculation process is as follows:

[0084] H C =C-ATT(Q,K) (10)

[0085] P(y i |x i = softmax(W) o HC +b o (11)

[0086] Where W0 and b0 are the parameters that the linear layer needs to learn, y i It is the word x in the input sentence sequence X. i The corresponding output labels.

[0087] Step 5: Use the cross-entropy function to calculate the loss during model training.

[0088] Use the cross-entropy function to calculate the loss during model training.

[0089]

[0090] Where K is the total number of samples, M is the number of labels, and in this paper M = 13, y ij To predict labels, g ij For sample x i The corresponding one-hot encoding of the gold label.

[0091] To verify the performance of the patented model on the E2E-ABSA task, it was compared with some baseline models. One type of baseline model is the classic E2E-ABSA model; the other type is a BERT-based model, both of which use BERT as the embedding layer to obtain the above representation.

[0092] Base model+BG+SC+OE: This model, proposed by Bailin Wang and Wei Lu (2018), implements E2E-ABSA in a collapsed manner. The model consists of a backbone framework and three components. The backbone framework is composed of two stacked Bi-LSTMs; the three components are the boundary guidance component (BG), the sentiment consistency component (SC), and the opinion-enhanced target word detection component (OE).

[0093] DOER: This model is the one proposed by Zheng Li, Ying Wei, Yu Zhang, Xiang Zhang, and Xin Li (2019). It implements E2E-ABSA in a joint manner. Two recurrent neural networks (RNNs) are used: one to extract aspect words and the other to predict the sentiment polarity corresponding to those aspect words. A cross-sharing unit then connects the two.

[0094] LM-LSTM-CRF: This model was proposed by Yunlong Liang, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou (2021). Because of its superior performance in sequence labeling tasks, Li et al. applied it to the E2E-ABSA task and provided experimental results.

[0095] IMN: This model is proposed by Xin Li, Lidong Bing, Piji Li, and Wai Lam (2019a). It treats E2E-ABSA as a multi-task approach and introduces an information transfer mechanism to associate different tasks.

[0096] BERT+E2E-ABSA Layer: This model, proposed by Lu Xu, Lidong Bing, Wei Lu, and Fei Huang (2020), implements E2E-ABSA in a collapsed manner. After obtaining the sentence representation using BERT, Li et al. designed various E2E-ABSA layers, namely linear layers, gated recurrent units (GRUs), self-attention networks (SANs), transformers (TFMs), and conditional random fields (CRFs), and provided experimental results.

[0097] Table 2 shows the experimental results for the classic E2E-ABSA model, Base model+BG+SC+OE, DOER, LM-LSTM-CRF, IMN, and BERT+E2E-ABSA Layer. In Table 2, the data in upright text uses the Restaurant dataset, which is the merged dataset of Rest14, Rest15, and Rest16; the data in italics uses the three independent datasets officially released. Because the merged datasets would cause some overlap between the training and test sets, this invention uses the officially released independent datasets.

[0098] As shown in Table 2, the performance of BERT-based models outperforms all other existing models. Even the current best model, DOER, falls short of BERT+Linear. Specifically, BERT+Linear outperforms by 0.08% on the Laptop14 dataset and by 0.08% on the Restaurant dataset, demonstrating the necessity of using BERT as the embedding layer. This is because models like Word2Vec or GloVe, when used as embedding layers, produce context-independent word vector representations, while BERT can learn context-rich word vector representations. This also proves that introducing BERT to obtain text representations containing contextual information significantly improves the performance of our model on the E2E-ABSA task.

[0099] Table 2 shows a comparison of the Micro-F1 values ​​of the baseline models, with the best results highlighted in bold.

[0100]

[0101] Step a4: Experimental analysis was conducted. According to the italicized data in Table 2, among all BERT-based models, the BERT+Linear model performed the worst. On the Laptop14 dataset, the F1 score of the BERT+Linear model was only 60.43%, on the Rest14 dataset only 72.61%, on the Rest15 dataset only 60.29%, and on the Rest16 dataset only 69.67%. This may be because the other models used GRU, SAN, TFM, and CRF respectively to obtain task-specific token representations.

[0102] As shown in Table 2, firstly, compared to all baseline models, the proposed model BERT+C-ATT performs best on the Rest14 and Rest15 datasets. Especially compared to BERT+Linear, our model shows a significant advantage on all four datasets, with F1 scores improved by 0.19%, 2.19%, 1.41%, and 0.68%, respectively, fully demonstrating the effectiveness of our model. BERT+Linear does not involve any auxiliary information when jointly extracting aspect words-sentiment polarity, while C-ATT divides sentences into different components, assigning positional information to each word in the sentence, and making aspect words focus more on opinion words belonging to the same sentence component. Therefore, our model stands out. Secondly, compared to models BERT+GRU, BERT+SAN, and BERT+TFM, BERT+C-ATT consistently leads in F1 scores on three datasets. Specifically, compared to BERT+SAN, BERT+C-ATT improves the F1 score by 0.13%, 1.12%, and 1.80% on the Laptop14, Rest14, and Rest15 datasets, respectively, while slightly decreasing by 0.16% on the Rest16 dataset. The superior performance of BERT+C-ATT is attributed to C-ATT's segmentation of aspect words and their corresponding opinion words into the same sentence component, ensuring the sentiment consistency of aspect words and effectively reducing the probability of misclassified sentiment polarity. Furthermore, BERT+C-ATT achieves higher F1 scores than BERT+CRF on two separate occasions: 1.63% higher on the Rest14 dataset and 1% higher on the Rest15 dataset. These comparisons demonstrate the effectiveness of sentence component segmentation for the E2E-ABSA task. Moreover, unlike most previous methods that relied on semantic parsing tools, our model directly obtains semantic information by segmenting sentence components using C-ATT, making its performance independent of any parsing tools.

[0103] This paper analyzes the experimental results of the proposed C-ATT model with hierarchical constraints. To ensure that C-ATT encompasses appropriate words when segmenting sentence components, hierarchical constraints were adopted. If the number of layers in the C-ATT network is too low, different words in the sentence belong to different components, failing to achieve the effect of classifying each aspect word and its corresponding opinion word into the same component; if the number of layers is too high, all words in the same sentence belong to the same component, making the proposed model indistinguishable from traditional self-attention mechanisms. Therefore, the influence of the C-ATT's layers on its sensitivity to perceiving sentence components was explored. Experiments were conducted on four datasets with a layer count ranging from 2 to 10 and a stride of 2. The experimental results are as follows: Figure 3 As shown.

[0104] Table 3 shows the part-of-speech statistics.

[0105]

[0106] like Figure 3 As shown, let L be the number of layers in C-ATT. Overall, the model performs better on the Rest14 dataset with an average performance of 74.32%. This is partly due to the amount of data in the dataset. As shown in Table 1, the Rest14 dataset contains a total of 4715 sentences, far exceeding other datasets, indicating that increasing the amount of data can improve model performance. On the other hand, it is related to the model's understanding ability. As shown in Table 3, we statistically analyzed the proportion of the four most common parts of speech (nouns, verbs, adjectives, and adverbs) in the data presented in this paper. The data in Table 3 shows that more than half of the words in the Rest14 and Rest16 datasets are nouns and adjectives. This indicates that the model in this paper focuses more on nouns and adjectives during training, thus resulting in better performance on the Rest14 and Rest16 datasets. This also aligns with the pattern in the E2E-ABSA task where aspect words are mostly nouns and opinion words are mostly adjectives, proving the rationality of using the model in this paper to handle the E2E-ABSA task.

[0107] from Figure 3 The data trends show that, except for the Rest16 dataset, the F1 score of C-ATT peaks at L=4. Therefore, the optimal number of layers is L=4. The highest F1 scores of our model on all four datasets occur when L≥2, indicating that when the number of layers is too small, different words in the sentence belong to different components, resulting in weak connections between aspect words and opinion words, thus reducing model performance. This experimental phenomenon demonstrates that increasing the number of C-ATT layers is beneficial to the model within a certain range. However, when L≥8, the model performance decreases instead of increasing, indicating that when the number of layers is too large, all words in the sentence belong to the same component, and further increasing the number of layers is clearly not beneficial. At this point, C-ATT distracts aspect words from opinion words, making C-ATT no different from traditional attention mechanisms.

[0108] To more intuitively understand the role of C-ATT, this invention visualizes the sentence component perception coefficient matrix C. Taking a sentence from the Realt16 dataset as an example, with a layer number of 4, the example sentence is "The duck confit is always amazing and the foie gras terrine with figs was out of this world." The visualization result of matrix C is as follows... Figure 4 As shown, each color block represents C. ij The results show that, firstly, C-ATT can effectively detect sentence breaks. (From...) Figure 4It can be seen that the attention weights for the words "and" and "amazing" are very small, and C-ATT identifies a breakpoint between them. Secondly, C-ATT pays more attention to words belonging to the same component. For example... Figure 4 As shown, the words "is," "always," and "amazing" belong to the same sentence component, and C-ATT assigns them all high weights. Furthermore, C-ATT is indeed effective on the E2E-ABSA task. The words "amazing" and "out of this world" are assigned relatively high weights, proving that C-ATT correctly captures the opinion words in the sentence, which is beneficial for E2E-ABSA.

[0109] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. An end-to-end attribute-level sentiment analysis method based on sentence component-aware attention mechanism, characterized in that: The specific steps of the method are as follows: Step 1, obtaining text vectorization representation containing context information through BERT pre-training model; Step 2, introducing a sentence component perception coefficient using sentence component perception attention mechanism, which gives high weight to words belonging to the same component in the sentence and low weight to words between different components, and introducing adjacent attention mechanism to determine which words in the sentence belong to the same component; Step 3, then input the text vectorization representation containing context information into the sentence component perception attention mechanism to divide the sentence components, and obtain the text representation containing aspect word boundary information and aspect word corresponding sentiment polarity features; Step 4, using a simple linear classifier to realize joint extraction of aspect words and their corresponding sentiment polarity prediction; Step 5, using cross-entropy function to calculate the loss of model training.

2. The end-to-end attribute-level sentiment analysis method based on sentence component-aware attention mechanism according to claim 1, characterized in that: In Step 2, the input is three matrices Q, K, and V, each composed of d-dimensional vectors; the input word vector obtains the initial query vector q, key vector k, and value vector v through a linear transformation; then, the dot product between q and each k is calculated as the weight coefficient of each v; finally, the vectors q, k, and v are combined to form matrices Q, K, and V, respectively; specifically, q is multiplied by each k through dot product operation, and then multiplied by a scaling factor Finally, the attention weight is obtained through a softmax function; The sentence component-aware attention mechanism C-ATT differs from the traditional attention mechanism in that a sentence component-aware coefficient C is introduced i,j which gives high weight to words in the same component and low weight to words between different components.

3. The end-to-end attribute-level sentiment analysis method based on sentence component-aware attention mechanism according to claim 1, characterized in that: The specific steps of Step 2 are as follows: Step 2.1: Vectorize the obtained text containing contextual information and perform a linear transformation to obtain the initial query vector q, key vector k, and value vector v. Then, calculate the dot product between q and each k as the weight coefficient for each v. Finally, combine vectors q, k, and v to form matrices Q, K, and V, respectively. Specifically, perform a dot product operation between q and each k, and then multiply by a scaling factor. Finally, the attention weights are obtained by applying a softmax function; the calculation process is as follows: Step 2.2, compared with the traditional attention mechanism, the sentence component-aware attention mechanism C-ATT introduces a sentence component-aware coefficient C i,j , which gives high weight to the words in the same component in the sentence and low weight to the words between different components, and the words in the same component are mutually concerned, and the words in different components are not concerned. The implementation process is as follows: where C is an N x N matrix, e is a matrix identity element corresponding multiplication operation, and matrix Q, K are composed of corresponding q vectors and k vectors respectively; in equation (2), the constituent elements of C i,j represent the probability that word w i belongs to the same component as w j , and the value of C i,j indicates that the word located at position i and the word located at position j belong to different components and are not concerned with each other.

4. The end-to-end attribute-level sentiment analysis method based on sentence component-aware attention mechanism according to any one of claims 1-3, characterized in that: The calculation process of the sentence component perception coefficient is as follows: First, for the input sentence sequence X, the probability that any word w i and its two adjacent words w i-1 , w i+1 belong to the same component is calculated, which is consistent with the traditional attention mechanism, so it is called adjacent attention mechanism; the adjacent attention mechanism determines which words in the same sentence belong to the same component, and the sentence component awareness coefficient C i,j determines which words in the sentence are closely related and gives them the corresponding high weight; the calculation process of the adjacent attention mechanism is as follows: Where, q i For target word w i The corresponding d model k-dimensional query vector i-1 k i+1 w respectively i Two adjacent words w i-1 w i+1 d model 3D key vector; Then, in order to prevent the model from tending to divide all words into the same component when dividing the sentence components later, the softmax function is used to limit it, as follows: p i,j = softmax(p i,i-1 ,p i,i+1 ) (5) where p i,i-1 , p i,i+1 are calculated from equations (3), (4) respectively, and p i,i-1 +p i,i+1 = 1, and further, the sentence component awareness coefficient C i,j represents the probability that the word w i belongs to the same component as w j , where p i,i-1 >p i,i+1 indicates that w i belongs to the same component as its left neighbor, otherwise it belongs to the same component as its right neighbor; The probability that the words in a sentence are in the same component is given by p i,i+1 and p i+1,i are not equal, the average is used to compute the adjacent attention weight, which is defined as follows: Thus, the weight sequence a = {a1, ..., a1} of the adjacent attention mechanism is obtained. i ,…,a n }, where N is the length of the sentence, a i For target word w i Its right neighboring word w i+1 The probability of belonging to the same component; For the target word w i either belongs to the same component as the word w i-1 on the left, or belongs to the same component as the word w i+1 on the right, thus, at most, two words belong to the same sentence component, in order to realize the division of multiple words into the same sentence component, a hierarchical constraint condition is set, that is, the C-ATTs in the same layer share a weight sequence a of an adjacent attention mechanism, and the sentence component divided by the current layer l is used as a basic unit for the division of the next layer sentence component, where l≥0, so as to realize that the words belonging to the same sentence component increase with the deepening of the network, and the hierarchical constraint condition is defined as follows: wherein, computed by the adjacent attention of layer l-1, computed by the current layer l, the hierarchical constraint guarantees that words belonging to the same component at layer l-1 must also belong to the same sentence component at layer l; From equation (7), the adjacent attention weight sequence of the l-th layer is obtained Then, the sentence component awareness coefficient C is calculated according to equation (8) i,j , c i≤k＜j ; C is calculated in a multiplicative way i,j The model improves the sensitivity of the breakpoint perception when dividing the sentence components by using the characteristic that multiplying small probabilities still results in a small probability. In the implementation process, to prevent the obtained probability from being too small, the formula (7) is modified as follows: c k is the weight sequence after adding linear constraints.

5. The end-to-end attribute-level sentiment analysis method based on sentence component-aware attention mechanism according to claim 1, characterized in that: In Step 3 and Step 4, the sentence representation is After linear transformation, matrix Q and K are obtained as the input of C-ATT. After dividing the sentence components, the sentence representation is output Finally, H C An input linear classifier is used to predict the label of E2E-ABSA; the calculation process is as follows: H C = C - ATT(Q, K) (10) P(y i |x i ) = softmax(W o H C +b o ) (11) where W0 and b0 are parameters that the linear layer needs to learn, y i is the word x i in the input sentence sequence X corresponds to the output label.

Images