Aspect-level sentiment analysis method of self-attention routing capsule network

Abstract

The application discloses an aspect-level sentiment analysis method based on a self-attention routing capsule network, generates a training set and a test set, carries out word vectorization on a sentence and corresponding entities through a pre-trained Glove model, calculates enhanced representation of the entities, encodes the enhanced representation of the entities and word vectors of the entities through a pre-trained BERT Encoder model, obtains context representation and entity representation, calculates entity-related weights of context capsules, takes the obtained context capsules as each classification capsule in a first layer of classification capsule layers, connects classification capsule layers of adjacent layers through a self-attention routing layer, and finally outputs a final sentiment label classification result. The application constructs enhanced representation of the entities through position bias weights, limits related information between the entities and the context, and can pay more attention to specific entity features.

Description

Technical Field

[0001] This invention relates to the field of neural network classification technology, specifically to an aspect-level sentiment analysis method using self-attention routing capsule networks, applicable to sentiment label recognition of one or more entities mentioned in a sentence. Background Technology

[0002] Aspect-based sentiment classification (ASC) aims to identify the sentiment labels of one or more entities mentioned in a sentence, a key fine-grained variant subtask of aspect-based sentiment analysis (ABSA). For example, in the sentence "The weather is nice, but the traffic is congested," the entities "weather" and "traffic" have corresponding sentiment polarizations of positive and negative, respectively. Compared to document-level sentiment classification (DSC), the ASC task aims to predict the sentiment polarity of specific aspects rather than the entire sentence.

[0003] Traditional sentiment classification methods mostly utilize machine learning algorithms to construct sentiment classifiers with carefully extracted features, which requires significant time and resources for data collection. In recent years, various deep neural networks (DNNs) have been used to model the ASC task due to their remarkable results, primarily including attention-based methods, grammar-based methods, and methods based on pre-trained BERT. For example, LSTM-based models utilize attention mechanisms to model the dependencies between aspect features and LSTM hidden units. Graph-based models are used to propagate grammatical neighborhood word information to aspect information. BERT-based methods benefit from the powerful representational capabilities of pre-trained models.

[0004] Furthermore, capsule networks have been used for ASC tasks due to their capsule vectors and dynamic routing methods for proper storage and clustering of features. Compared to distributed neural networks, capsule networks not only replace scalar output feature detectors with vector output capsules but also retain additional information, such as the frequency and location of phase. For example, TransCap combines capsule networks with inductive transfer learning to successfully address the data scarcity problem in ASC tasks. CapsNet utilizes a novel aspect-aware normalization method to select the master capsule to avoid feature loss. However, a problem remains to be solved in ASC tasks using capsule networks: the original dynamic routing mechanism in capsule networks suffers from low training efficiency due to the iterative routing process. Summary of the Invention

[0005] The purpose of this invention is to address the aforementioned problems in the prior art by providing an aspect-level sentiment analysis method using self-attention routing capsule networks.

[0006] To achieve the above objectives, the present invention employs the following technical measures:

[0007] An aspect-level sentiment analysis method based on self-attention routing capsule networks includes the following steps:

[0008] Step 1: Select a benchmark dataset, preprocess the benchmark dataset and generate a training set and a test set. Each sample in the training set includes a sentence and an entity in the sentence.

[0009] Step 2: Use a pre-trained GloVe model to vectorize the sentence and its corresponding entities into word vectors, obtaining the word vectors of the sentence and the word vectors of the entities;

[0010] Step 3: Based on the relative positional information t between the word vectors of the sentence and the context of the entity... i The augmented representation of the computational entity P = {p1, p2, ..., p n}, where n is the word length of the sentence;

[0011] Step 4: Augment the entity representation P = {p1, p2, ..., p} using the pre-trained BERT Encoder model. n The word vectors of the entities are encoded to obtain the context representation H = [h1, h2, ..., h...]. n ] and entity representation h α ;

[0012] Step 5: Based on the context, represent H = [h1, h2, ..., h n ] and entity representation h α Building a context capsule c i And entity capsule a, and capsule c according to the context i Calculate the entity-related weights o of the context capsule and entity capsule a. i , i∈{1~n};

[0013] Step 6: Obtain the context capsule u i Each category capsule in the first category capsule layer is connected to adjacent category capsule layers through a self-attention routing layer, and the last category capsule layer outputs the final sentiment label classification result.

[0014] As mentioned above in step 3, the positional deviation weight t of the i-th word relative to the entity i Based on the following formula:

[0015]

[0016] Where m and n represent the word lengths of the entity and sentence, respectively, and γ is the index of the first word of the entity.

[0017] p i =t i ·e i

[0018] Where i∈{1~n}, e i The word vector of the sentence is E = {e1, e2, ..., e}. n The word vector of the i-th word in}.

[0019] In step 5 as described above, the context capsule c is constructed based on the following formula. i And solid capsule a:

[0020] c i =squash(W h h i +b h )

[0021] a = squash(W a h α +b a )

[0022] Among them, W h b h W a and b a All of these are learning parameters, and squash() is a non-linear squash function.

[0023] In step 5 as described above, the entity-related weights of the context capsule are calculated based on the following formula. i :

[0024]

[0025] Among them, W o These are learning parameters, o i These are the entity-related weights of the context capsule, where D is the number of capsules in the predefined context capsule layer, and D is less than n.

[0026] As described above, in step 5, the context capsule u related to the entity i =c i ·o i .

[0027] As described in step 6 above, the (l+1)th layer of the classification capsule U l+1 Based on the following formula:

[0028]

[0029] in, The prediction result is calculated for the l-th layer classification capsule, where T is the transpose operation and C is the result of the transpose operation. l It is the coefficient matrix, B l The logarithmic prior matrix is ​​obtained through random initialization.

[0030]

[0031] Among them, A l n is the coupling coefficient. l n l+1 These represent the number of capsules in the l-th and l+1-th layers, respectively.

[0032]

[0033] in, This is the balance coefficient.

[0034] As described above, the last layer of the sorting capsules in step 6 consists of sorting capsules. Composition, where j is the sentiment tag y of the entity. j Serial number, classification capsule activation probability

[0035] The loss of the last layer of classified capsules is C represents the total number of sentiment labels. The Adam optimizer is used to optimize the network parameters of the classification capsule, with the goal of minimizing the loss L.

[0036] L j The loss function for each classification capsule in the last layer:

[0037] L j =T j max(0,m + -||v j ||) 2 +λ(1-T j max(0,||v) j ||-m - ) 2

[0038] Where, λ, m + and m - All are hyperparameters; if the v of the j-th classification capsule in the last layer... j When the value is at its maximum, the corresponding T j =1, otherwise T j =0.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] 1. An enhanced representation of the entity was constructed using positional bias weights, which fully demonstrates the effectiveness of this mechanism in limiting the relevant information between the entity and the context;

[0041] 2. A novel classification capsule based on a self-attention routing mechanism is proposed, which overcomes the disadvantage of low training efficiency and can focus more on specific entity features;

[0042] 3. A pre-trained BERT Encoder model is introduced to leverage the advantages of large-scale pre-training. Experimental results and detailed analysis demonstrate the effectiveness of the mechanism in this invention. Attached Figure Description

[0043] Figure 1 This is the complete principle and system architecture of the present invention;

[0044] Figure 2 This is a schematic diagram of the self-attention mechanism of the present invention;

[0045] Figure 3 This is an example image of the first five dimensions extracted from the 300-dimensional GloVe word vectors of "Hello" and "World".

[0046] Figure 4 This is a specific representation of the positional deviation weight used in this invention on an example sentence;

[0047] Figure 5 This is a diagram illustrating the effect of the number of self-attention routing layers on the performance of the present invention; where (a) is a schematic diagram of the effect of the number of layers on accuracy; and (b) is a schematic diagram of the effect of the number of layers on the macro-averaged F1 score. Detailed Implementation

[0048] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to examples. The implementation examples described herein are only for illustration and explanation and are not intended to limit the present invention.

[0049] The aspect-level sentiment analysis method for self-attention routing capsule networks includes the following steps:

[0050] Step 1: Select a benchmark dataset, preprocess the benchmark dataset to generate a training set and a test set. The samples in the training set and the test set are the benchmark data in the benchmark dataset. Each benchmark data corresponds to a sentiment label. The benchmark dataset involves multiple sentiment labels.

[0051] The experiments in this invention were conducted using a PyTorch environment based on Python 3.6, PyCharm 2020.2.2 (Professional Edition), with the benchmark datasets being cuda v11.1, cuDNN v8.0.4, and PyTorch v1.7.

[0052] The benchmark datasets used in this embodiment are three ABSA benchmark datasets: the SemEval2014 Laptop dataset, the SemEval2014 Restaurant dataset, and the Twitter dataset. In this embodiment, the sentiment labels involved in the benchmark datasets include positive, neutral, and negative. The three benchmark datasets are arranged in ascending order of size as follows: Twitter dataset, SemEval2014 Restaurant dataset, and SemEval2014 Laptop dataset. Furthermore, the dataset configuration strictly follows that of previous studies. The statistical data for the datasets are shown in Table 1. These datasets are widely used benchmarks in the field of relation extraction.

[0053] Table 1. Sentiment Labels for Samples in the Training Set

[0054]

[0055] The sentiment labels for the samples in the training set include: Positive, Negative, and Neutral. The sentiment labels for the samples in the training and validation sets are shown in Tables 1 and 2, respectively.

[0056] Table 2. Emotional Labels for Samples in the Test Set

[0057]

[0058] The evaluation metrics used in the experiment are based on Accuracy and macro-averaged F1 score. Accuracy and macro-averaged F1 score are the official evaluation metrics of the benchmark dataset and are used to judge the quality of the model.

[0059] One of the samples in the restaurant dataset is shown in Table 3. In the benchmark dataset, a sample consists of a sentence and an entity in the sentence and its location information. Because there are multiple entities in a sentence, each entity and sentence combination can constitute a separate sample for different entities in a sentence. An entity may be a word or a word composed of multiple consecutive words.

[0060] Table 3 Sample Examples

[0061]

[0062] In each sample of the benchmark dataset, a separate array of positional indices records the relative positions of the context (words) and entities.

[0063] (2) Data preprocessing

[0064] This invention preprocesses the SemEval-2014 Restaurant & Laptop and Twitter standard datasets to adapt them for subsequent code execution, converting them into a special JSON file format for annotation.

[0065] Table 4. Examples of Preprocessed Data Samples

[0066]

[0067] The sentence is segmented and irrelevant symbols and spaces are removed to obtain the segmented sentence. The segmented sentence is then annotated using a specific JSON format. The specific steps are as follows: 1. Each word in the segmented sentence is treated as a corresponding word list, with each word in the segmented sentence as an element in the list. The position indices of each word in the segmented sentence are used as a corresponding position index list, corresponding one-to-one with the words in the sentence. Then, entities are extracted from the sentence individually, and their start and end positions in the segmented sentence are recorded. The processing results are shown in Table 4. A sentence processed in this step and one entity together form a sample; multiple entities and their corresponding sentences together form multiple samples. The preprocessed data is a sample set composed of such sentence and entity pairs.

[0068] Step 2: The sentence and its corresponding entity are vectorized using the pre-trained Glove-300d model in the input layer to obtain the word vectors of the sentence and the word vectors of the entity.

[0069] Input layer

[0070] At the input layer, the preprocessed sentences and entities are vectorized. During preprocessing, the positions of entities in the context are labeled. Entities are the targets of aspect-level sentiment classification tasks; they are part of the original input sentence and typically consist of one or more words. This process generates complete sentence and entity mapping tables for subsequent processing. The encoding of the sentence and entity mapping tables is then modified to UTF-8. Duplicate sentences in both tables are removed, and word vectors for the corresponding sentences and entities are further generated.

[0071] The pre-trained GloVe-300d model aims to convert the semantic and entity information of the input sentence into word vectors.

[0072] The input sentence is represented by S = {w1, w2, ..., w n} represents the length of the words constituting the sentence in the sample, where n is the length of the words in the sample, and w1, w2, ..., w n These are the individual words in the sentence.

[0073] A={w γ ,w γ+1 ,…,w γ+m-1 Let} represent an entity in the sentence, where γ is the index of the word in which the entity first appears in the input sentence, and m is the total number of words that make up the entity in the sample.

[0074] To enable the pre-trained Glove-300d model to capture more accurate semantic information, the pre-trained Glove-300d model in this application uses a dimension of d. w The GloVe word embedding pre-trained language model significantly improves various natural language tasks. Each word corresponds to a vector, and these word vectors are static and unchanging. Through this layer, the word vectors E = {e1, e2, ..., e...} of a sentence can be obtained. n} and the word vector e of the entity α Where, e1, e2, ..., e n They are w1, w2, ..., w n The corresponding word vectors.

[0075] It is worth noting that, since an entity may consist of more than one word, the entity embedding representation e α The calculation result is the average pooled index of the word embeddings that make up the entity, where the embeddings are word vectors obtained by mapping each word in the entity using GloVe.

[0076] During the experiment, the dimension of the words in this sentiment analysis model must be consistent with the dimension of the word vectors in GloVe. Example sentences are as follows: Figure 1 As shown.

[0077] The parameters related to the word vectors in the experiment are shown in Table 5.

[0078] Table 5. Word Vector Related Parameters

[0079] Hyperparameters Description Value d e ]]> Word vector dimension size 300 vocabs Size of the vocabulary 2.2M

[0080] Step 3, based on the relative positional information t between the word vectors of the sentence and the context of the entity. i , compute the enhanced representation P of the entity;

[0081] Position bias weight

[0082] To obtain entity augmentation input for capsule networks, relative positional information is used to compute the weights of context words. A novel positional bias weighting method is employed to represent the relative positional information t between entities and their context. iSpecifically, it is described as calculating the positional bias weight of context words using the relative positional information between the entity and the context. This weight is used to reduce the importance of words that are relatively far from the entity and increase the importance of words that are closer to the entity.

[0083]

[0084] Where m and n represent the word lengths of the entity and sentence, respectively, and γ is the index of the first word of the entity. i This represents the positional deviation weight of the i-th word relative to the entity. Therefore, an enhanced representation of the entity, P = {p1, p2, ..., p...}, can be refined and generated. n}, p i It can be calculated as: p i =t i ·e i , where e i The word vector of the sentence is E = {e1, e2, ..., e}. n The word vector of the i-th word in}. A sentence and an entity constitute a single sample, so a sample has only one γ.

[0085] Because the ASC task necessitates entity feature extraction, the positional information between the entity and its context words is crucial to the overall model performance. Context words far from the entity have a relatively smaller impact on the entity, while context words close to the entity play a more significant role in conveying the entity's sentiment label.

[0086] The positional bias mechanism is based on this fact. It calculates weights for entities and their context words, giving greater weight to context words that are relatively closer to the entity and less weight to context words that are relatively farther away from the entity. This adds certain positional information to the sentence and further improves the model's information acquisition ability. Figure 2 For an example sentence and the calculated positional deviation weight t of each word relative to the entity. i .

[0087] According to formula (1), we know that the word length n of the example sentence is 12, the length m of the entity "food" is 1, and its starting subscript γ is 5. Therefore, according to the formula, we can obtain the following... Figure 2 The positional deviation weight t of each word relative to the entity is shown. i .

[0088] The parameters of the position offset mechanism are described in Table 6.

[0089] Table 6. Parameter description of the position offset mechanism

[0090]

[0091] Step 4: Augment the entity representation P = {p1, p2, ..., p} using the pre-trained BERT Encoder model. n The word vectors of the entities are encoded to obtain the context representation H = [h1, h2, ..., h...]. n ] and entity representation h α ;

[0092] BERT-Encoder model

[0093] The pre-trained model trained on a large-scale corpus demonstrates strong semantic representation capabilities. To leverage the pre-trained model, the encoder layer utilizes a pre-trained BERT encoding layer. In this case, the BERT encoding layer takes "[CLS] + sentence embedding representation + [SEP] + entity embedding representation + [SEP]" as input Seg, where the sentence embedding representation is the entity augmentation representation P, and the entity embedding representation is the entity word vector e. α [CLS] and [SEP] represent the start and end markers, respectively. Therefore, this invention can utilize the additional knowledge provided by the pre-trained BERT Encoder model to construct the context representation H = [h1, h2, ..., h...]. n ] and entity representation h α :

[0094] H,h α =BERT(Seg) (2)

[0095] This invention uses a pre-trained BERT to extract and encode information from the output enhanced with positional bias. BERT is a bidirectional encoder-decoder structure based on Transformer. Previous pre-trained models were limited by unidirectional language models (from left to right or right to left), thus restricting their representational capabilities and limiting them to acquiring only unidirectional contextual information. BERT, however, utilizes MLM for pre-training and employs deep bidirectional Transformer components to construct the entire model, ultimately generating deep bidirectional language representations that can integrate left and right contextual information.

[0096] The pre-trained BERT version used in this invention is BERT-base, and its relevant parameters are shown in the table below:

[0097] Table 7 BERT-base parameters

[0098] Hyperparameters Description Value L Number of Transformer encoders 12 A Number of multi-headed attentions 12 H Hidden layer dimension 762

[0099] Step 5: Based on the contextual semantics, represent H = [h1, h2, ..., h nThe entity-related weights of the context capsules are calculated, and the context capsule u is used to calculate the entity-related weights. i According to entity representation h α Calculate entity capsule a;

[0100] Context capsule layer

[0101] In this layer, linear transformations and nonlinear squeezing functions are used to transform the context representation H = [h1, h2, ..., h2] obtained from the BERT Encoder model. n This is converted into a context capsule. Furthermore, to construct entity-related context capsules, an entity-related routing method is used to select important context capsules from the entity capsules.

[0102] The nonlinear squeezing function is a nonlinear active function that restricts the length of the capsule to [0, 1] and can be represented as the probability of the current input semantics of the capsule. The definition of the nonlinear squeezing function is shown in equation (3), where z is the input of the nonlinear squeezing function.

[0103]

[0104] Therefore, context capsule c can be constructed according to equations (4) and (5) respectively. i and solid capsule a, where h i The context representation H = [h1, h2, ..., h is obtained through the BERT Encoder in step 4. n Semantic representation of each word in ]:

[0105] c i =squash(W h h i +b h (4)

[0106] a = squash(W a h α +b a (5)

[0107] W h h i +b h and W a h α +b a Each is taken as the input z of the nonlinear compression function, where W h b h W a and b aAll are learnable parameters. Using fusion convolution operations to calculate the aspect ratio weights of context words cannot update the weights, leading to instability in the training process. To solve this problem, this invention utilizes the "aspect-related routing method" to modify the weights of related routes using the following formula to calculate the entity-related weights of the context capsule in equation (6). i :

[0108]

[0109] Among them W o It is a learnable parameter, o i This represents the entity-related weights of the context capsules, where D is the number of capsules in the predefined context capsule layer. D is less than the length n of the input context representation and is used to further extract information. Based on this, the entity-related context capsule u can be obtained. i =c i ·o i .

[0110] In convolutional neural networks (CNNs), convolutional kernels are used to perform convolution operations on images to produce an n*n result. This can be seen as the result obtained by repeatedly extracting the content of each local part of the image from the convolutional kernels. Then, pooling is used to obtain the activity (scalar) of the neurons, achieving viewpoint invariance. In this process, pooling loses a lot of information in the data, and even if some adjustments are made to the image, similar results may be produced. Capsule networks were proposed to solve the problems existing in CNNs. Unlike CNNs, which use scalars to record local information, capsule networks use important information of vector feature states. The context capsule layer converts the input data into vector feature states through the squash activation function shown in formula (3). The parameter settings of the context capsule layer are shown in Table 8.

[0111] Table 8. Relevant parameters and characteristic parameter settings for the self-attention mechanism.

[0112] Hyperparameters Description Value n_epoch Number of training epochs 25 n_stop Number of epochs to stop early 5 dim_context Context capsule dimension 16

[0113] Step 6: Obtain the context capsule u i Each category capsule in the first category capsule layer is connected to adjacent category capsule layers via self-attention routing. The final category capsule layer outputs the final sentiment label classification result.

[0114] Classification capsule layer

[0115] The original dynamic routing mechanism in classic capsule networks works like a self-iterative process, which makes training inefficient.

[0116] In this layer, the invention utilizes a novel self-attention routing mechanism to route activity capsules to the target classes they should belong to. Specifically, self-attention routing can maintain the existence of multiple capsule layers, stacked on top of each other to create deeper layers. Figure 3 As shown, the structure of the target capsule layer is very similar to a fully connected network, with key points being additional branches obtained through a self-attention algorithm. To predict the next layer, the classification capsule set U of the l-th layer... l Each category of capsules With a randomly initialized weight matrix W l Matrix multiplication is used to calculate the prediction results that the l-th layer classification capsule can obtain. Where l represents the layer number of the classification capsule, U l This represents the set of all category capsules in layer l. In the category capsules of layer l connected to the context capsule layer, the category capsules of layer l are the respective context capsules u. i W l This represents the weight matrix used for calculating the classification capsule at layer l; the parameters for layer l+1 are similar.

[0117]

[0118] in, The prediction result is calculated for the l-th layer classification capsule, where T is the transpose operation. Furthermore, the classification capsule U for the next layer can be calculated using equation (8). l+1 :

[0119]

[0120] Among them B l It is a log-prior matrix obtained through random initialization, containing all weights as other learnable weights, which helps in creating preferences for activity capsules. l This is a coefficient matrix obtained from a self-attention algorithm that dynamically assigns detected entity features to target capsules represented in a specific sensor. The coupling coefficients A are calculated from the self-attention tensor using equation (9). l :

[0121]

[0122] Among them, the balance coefficient Used to stabilize training and maintain a balance between coupling coefficients and logarithmic priors. Each self-attention tensor A l The scores for each classification capsule prediction combination are consistent, therefore they can be used to calculate all coupling coefficients. Equation (10) is used to calculate the coupling coefficient C. lThe coupling coefficient can be used in equation (8) to obtain all capsules U of the (l+1)th layer. l+1 .

[0123]

[0124] Where n l n l+1 These represent the number of capsules in the l-th and l+1-th layers, respectively.

[0125] The last layer of categorized capsule collection U last It consists of classified capsules Composition, where j is the sentiment tag y of the entity. j The serial number, used in the last layer, indicates the serial number of the classification capsule in the last layer. Apply a nonlinear squeezing function to obtain the final representation v j :

[0126]

[0127] v j The length is limited to [0,1] to represent the classification capsule. The activation probability.

[0128] In summary, self-attention routing is a multi-layered structure, similar to a fully connected layer. Its additional branch is the calculation of self-attention weights, which can transmit important capsule information to the next layer while simultaneously closing off the transmission of unimportant capsules.

[0129] The optimal self-attention routing parameter settings are shown in the table.

[0130] Table 9 Self-Attention Routing Parameter Settings

[0131] Hyperparameters Description Value dim_class Classification capsule dimension 24 n_class Number of classification capsules 3 attn_layer Number of self-attention routing layers 2

[0132] The impact of self-attention layers on model performance is illustrated in the figure. To investigate the effect of the number of self-attention routing layers, ASCaps-BERT models with one to six layers were evaluated on Twitter, laptop, and restaurant datasets. Figure 4 As shown, this embodiment uses two self-attention routing layers. Clearly, when the number of layers is too high, the model becomes unstable due to information redundancy and vanishing gradients. When the number of layers is low, volume features and sentence representations cannot propagate far, leading to incomplete training.

[0133] VI. Margin Loss Function

[0134] In this embodiment, the loss function used is the MarginLoss loss function. The length of each classification capsule is used to represent the probability of the sentiment label. Therefore, the loss function L for each classification capsule in the last layer can be obtained. j y, where j is the emotional polarity of the entity. j The serial number is used to indicate the serial number of the classification capsule in the last layer:

[0135] L j =T j max(0,m + -||v j ||) 2 +λ(1-T j max(0,||v) j ||-m - ) 2 (12)

[0136] Among them, the missing sentiment label is controlled by the hyperparameter λ. If the v of the j-th classification capsule in the last layer is... j When the value is at its maximum, the corresponding T j =1, otherwise T j =0. In this embodiment, the hyperparameters λ and m + and m - The values ​​are set to 0.5, 0.9, and 0.1 respectively. At this point, the loss for the last classification capsule is calculated as follows: The Adam optimizer is used to optimize the network parameters of the classification capsule with the goal of minimizing the loss L. Here, C represents the total number of sentiment labels, i.e., y. j The number of capsules is 3 in this embodiment, representing positive, negative, and neutral, respectively. The process of closing unimportant capsules is the process of calculating the possible size of the sentiment label of the last layer of classification capsules. The sentiment label with the highest activation probability of the last layer of classification capsules will be used as the prediction result.

[0137] Experimental comparison:

[0138] During the experiment, the following classification methods were compared with the present invention.

[0139] ATAE-LSTM:

[0140] ATAE-LSTM utilizes aspect embedding and attention mechanisms in aspect-level sentiment classification. It is an attention-based neural network model that has achieved top-tier performance on several benchmark datasets.

[0141] IAN:

[0142] IAN uses two LSTMs and an interactive attention mechanism to generate representations of volumes and sentences.

[0143] RAM:

[0144] RAM uses multi-attention and memory networks to learn sentence representations.

[0145] MGAN:

[0146] MGAN designs a multi-granularity attention mechanism to capture word-level interactions between aspects and context.

[0147] ASGCN:

[0148] GCN was the first to propose using GCN to learn aspect-specific representations for aspect-based sentiment classification.

[0149] BiGCN:

[0150] BiGCN uses a hierarchical graph structure to integrate word co-occurrence information and dependency type information.

[0151] CDT:

[0152] CDT uses GCN on a dependency tree to learn aspect representations with syntactic information.

[0153] BERT-SPC:

[0154] BERT-SPC builds upon the basic BERT model by designing a more refined prediction and classification layer.

[0155] BERT-PT:

[0156] BERT-PT is a vanilla-BERT model that makes predictions by taking sentence body pairs as input and using the [CLS] representation.

[0157] CapsNet-BERT:

[0158] CapsNet-BERT was the first to introduce capsule networks into aspect-level sentiment classification tasks.

[0159] Table 10 Comparison of classification methods and the present invention (Figure)

[0160]

[0161] Experimental results show that the proposed ASCaps-BERT model outperforms grammar enhancement and attention-based methods on Twitter, restaurant, and laptop datasets. These results demonstrate that ASCaps-BERT can capture positional bias information in sentences and effectively select important entity features. Furthermore, the proposed ASCaps-BERT accurately fits datasets containing formal, informal, or complex reviews. Compared to attention-based methods (such as MGAN, IAN, RAM, and ATAE-LSTM), the ASCaps-BERT model utilizes positional bias information to establish dependencies between aspects and context, thus providing grammatical information and overcoming noise from the attention mechanism. Additionally, grammar enhancement methods, such as CDT, BiGCN, and ASGCN, outperform all attention-based methods, but they ignore semantic relevance between aspects. However, when considering informal or complex sentences, using only syntactic knowledge leads to poor performance. In Table 3, the results in the last group show that the basic BERT outperforms most models based on RNN encoders and static text embeddings. Furthermore, based on BERT, our ASCaps BERT achieves even better performance.

[0162] It should be noted that the specific embodiments described in this invention are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains can make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. An aspect-level sentiment analysis method based on self-attention routing capsule networks, characterized in that, Includes the following steps: Step 1: Select a benchmark dataset, preprocess the benchmark dataset and generate a training set and a test set. Each sample in the training set includes a sentence and an entity in the sentence. Step 2: Use a pre-trained GloVe model to vectorize the sentence and its corresponding entities into word vectors, obtaining the word vectors of the sentence and the word vectors of the entities; Step 3: Based on the relative positional information t between the word vectors of the sentence and the context of the entity... i The augmented representation of the computational entity P = {p1, p2, ..., p...} n }, where n is the word length of the sentence; Step 4: Augment the entity representation P = {p1, p2, ..., p...} using the pre-trained BERT Encoder model. n The word vectors of the entities are encoded to obtain the context representation H = [h1, h2, ..., h...]. n ] and entity representation h α ; Step 5: Based on the context, represent H = [h1, h2, ..., h n ] and entity representation h α Building a context capsule c i And entity capsule a, and capsule c according to the context i Calculate the entity-related weights o of the context capsule and entity capsule a. i , i∈{1~n}; Step 6: Obtain the context capsule u i Each category capsule in the first category capsule layer is connected to adjacent category capsule layers via a self-attention routing layer. The last category capsule layer outputs the final sentiment label classification result. As mentioned above in step 3, the positional deviation weight t of the i-th word relative to the entity i Based on the following formula: Where m and n represent the word lengths of the entity and sentence, respectively, and γ is the index of the first word of the entity. p i =t i ·e i Where i∈{1~n}, e i The word vector of the sentence is E = {e1, e2, ..., e...} n The word vector of the i-th word in} In step 5 as described above, the context capsule c is constructed based on the following formula. i And solid capsule a: c i =squash(W h h i +b h ) a=squash(W a h α +b a ) Among them, W h b h W a and b a These are all learned parameters; squash() is a non-linear squash function. In step 5 as described above, the entity-related weights of the context capsule are calculated based on the following formula. i : Among them, W o These are learning parameters, o i These are the entity-related weights of the context capsule, where D is the number of capsules in the predefined context capsule layer, and D is less than n. As described above, in step 5, the context capsule u related to the entity i =c i ·o i .

2. The aspect-level sentiment analysis method based on self-attention routing capsule networks according to claim 1, characterized in that, As described in step 6 above, the (l+1)th layer of the classification capsule U l+1 Based on the following formula: in, The prediction result is calculated for the l-th layer classification capsule, where T is the transpose operation and C is the result of the transpose operation. l It is the coefficient matrix, B l The logarithmic prior matrix is ​​obtained through random initialization. Among them, A l n is the coupling coefficient. l n l+1 These represent the number of capsules in the l-th and l+1-th layers, respectively. in, This is the balance coefficient.

3. The aspect-level sentiment analysis method based on self-attention routing capsule networks according to claim 2, characterized in that, As described above, the last layer of the sorting capsules in step 6 consists of sorting capsules. Composition, where j is the sentiment tag y of the entity. j Serial number, classification capsule activation probability 4. The aspect-level sentiment analysis method based on self-attention routing capsule networks according to claim 3, characterized in that, The loss of the last layer of classified capsules is C represents the total number of sentiment labels. The Adam optimizer is used to optimize the network parameters of the classification capsules, with the goal of minimizing the loss L. L j The loss function for each classification capsule in the last layer: L j =T j max(0,m + -||v j ||) 2 +λ(1-T j )max(0,||v j ||-m - ) 2 Where, λ, m + and m - All are hyperparameters; if the v of the j-th classification capsule in the last layer... j When the value is at its maximum, the corresponding T j =1, otherwise T j =0.

Images