A text sentiment classification method based on attention static routing capsule network

Abstract

The application discloses a text sentiment classification method based on an attention static routing capsule network, collects unannotated text data of a target language; word vector representation of the target language is trained by using the unannotated text data and a word2vec method; text data with labels of the target language is collected; a classification model based on the attention static routing capsule network is constructed; the classification model is supervisedly trained by using the text data with labels of the target language; the trained classification model is evaluated by using accuracy, precision, recall and F1Score, and a text sentiment classification model meeting requirements is obtained; and the input text is classified by using the text sentiment classification model meeting the requirements. The application can improve the text feature extraction capability and the relationship modeling capability between text features, and finally improve the text sentiment classification precision.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and text sentiment classification, and particularly relates to a text sentiment classification method based on an attention static routing capsule network. BACKGROUND

[0002] Text sentiment classification is one of the most basic and important tasks in the field of machine learning. Tradically, term frequency-inverse document frequency (tf-idf) is used as the feature representation of text, and then a general classifier such as support vector machine (SVM) or logistic regression is used for text sentiment classification.

[0003] However, in recent years, the continuous development of deep learning methods has made it possible to find distributed representations of words and documents in an efficient way, which has further improved the accuracy of text sentiment classification. The main deep learning models used in the field of text sentiment classification are mainly based on convolutional neural networks (CNN) and recurrent neural networks (RNN) and the hot Transformer architecture in recent years. In 2017, Hinton proposed a capsule network out of consideration of the shortcomings of convolutional neural networks, and applied it to the field of image processing, proving its effectiveness in understanding the spatial relationships of high-level data. After that, researchers tried to apply capsule networks to text processing and achieved good results, proving that capsule networks also have advantages for text information processing. The information transmission between different layers of capsules in traditional capsule networks adopts a dynamic routing mechanism, which needs to dynamically calculate the weights of different capsules each time, which is very time-consuming.

[0004] Therefore, how to reduce the time spent on routing between capsules in the capsule network while not reducing the accuracy of the model has become a problem to be solved in this field. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a text sentiment classification method based on an attention static routing capsule network, in order to improve the extraction ability of text features and the relationship modeling ability between text features, and ultimately improve the accuracy of text sentiment classification.

[0006] In order to achieve the above purpose, the technical solution adopted by the present application is:

[0007] A text sentiment classification method based on an attention static routing capsule network, comprising the following steps:

[0008] Step 1, collect unannotated text data of a target language; the target language is the language used to finally complete the text sentiment classification task;

[0009] Step 2, use the unlabeled text data in step 1 and the word2vec method to train the word vector representation of the target language;

[0010] Step 3, collect the target language labeled text data;

[0011] Step 4, build a classification model based on the attention static routing capsule network;

[0012] Step 5, use the target language labeled text data obtained in step 3 to supervise the training of the classification model in step 4;

[0013] Step 6, use accuracy, precision, recall and F1 Score to evaluate the classification model trained in step 5, and obtain a text sentiment classification model that meets the requirements, and use the text sentiment classification model to classify the input text.

[0014] In one embodiment, the step 1, collect text data and clean it, remove useless text and non-text content, and obtain unlabeled text data, wherein the total number of words in the unlabeled text is not less than 1 million words.

[0015] In one embodiment, the step 2, use the continuous bag-of-words (CBOW) model of word2vec for word embedding pre-training to obtain real number vectors of all words in the target language, i.e. the word vector representation.

[0016] In one embodiment, the step 3, collect text data and clean it, remove useless text and non-text content, and then manually label the sentiment of each text.

[0017] In one embodiment, the model components of the classification model include: a word2vec word embedding layer, a two-dimensional convolution layer, a fully connected layer, a squeeze pooling layer, a primary capsule layer, a middle capsule layer, a high-level capsule layer, and a classification capsule layer.

[0018] The word2vec word embedding layer is used to map text into a word vector sequence; the word vector sequence forms a real number matrix and is input into the two-dimensional convolution layer as a single input channel picture; the two-dimensional convolution layer uses multi-scale convolution to extract multi-scale features of the text and converts them into vector capsules.

[0019] The fully connected layer is used to unify the dimensions of the multi-scale features extracted by the two-dimensional convolution layer, and then perform feature fusion on the multi-scale features with unified dimensions based on attention weights;

[0020] The squeeze pooling layer is used to compress the fused features into vectors with a module length of 0-1, and then input them into the primary capsule layer;

[0021] The primary capsule layer, intermediate capsule layer, advanced capsule layer, and classification capsule layer are used to pass the most primitive semantic information extracted by the convolutional layer to the model output step by step using static attention routing, thereby obtaining the category of text sentiment.

[0022] In one embodiment, the training process in step 5 is as follows:

[0023] 1) The text data to be classified is T = {w1, w2, ..., w...} n The input is fed into the word2vec word embedding layer, and each word is then processed. i Mapped to a real vector v i ∈R d This transforms the entire text into a matrix D = {v1, v2, ..., v...} n}∈R d×n , where d is the dimension of the word vector and n is the length of the text;

[0024] 2) Input matrix D as a single-channel image into a two-dimensional convolutional layer, and use multi-scale convolutional kernels to extract features from matrix D to obtain multi-scale features. The formula for the output shape is as follows;

[0025]

[0026] in This indicates rounding down to the nearest integer, n h Let k be the vertical length of matrix D. h p is the vertical length of the convolution kernel. h For vertical filling, s h This refers to the longitudinal stride.

[0027] 3) The dimensions of the multi-scale features are made the same through the fully connected layer to obtain the multi-scale output features g. i ;

[0028] 4) Multi-scale output features g i Weighted fusion based on attention weights is performed on the same output channel to obtain the fused feature s. i ;

[0029] 5) In the squeeze pooling layer, the fused feature s i The capsule is compressed into a vector c with a modulus of 1 using an extrusion operation. This vector is then input into subsequent capsule layers. The extrusion operation formula is shown below:

[0030]

[0031] 6) The primary capsule layer, intermediate capsule layer, advanced capsule layer and classification capsule layer are fully connected, and the routing between capsules adopts the attention static routing mechanism.

[0032] In one embodiment, the two-dimensional convolutional layer has five multi-scale convolutional kernels with sizes of 1×d, 3×d, 5×d, 7×d, and 9×d, and a vertical stride of s. h =1, vertically fill p h =0, all output channels are 256, and the calculated output shapes of the multi-scale convolution are: o1∈R n×1 o2∈R (n-2)×1 o3∈R (n-4)×1 o4∈R (n-6)×1 ,o5∈R (n-8)×1 The fully connected layers are respectively: W1∈R e×n W2∈R e×(n-2) W3∈R e×(n-4) W4∈R e×(n-6) W5∈R e×(n-8) The dimension is unified as: W i o i =g i ∈R e×1 , where g i is the multi-scale output feature with the same dimension; e is the dimension after the multi-scale feature is unified.

[0033] In one embodiment, the weighted fusion method is as follows:

[0034] The multi-scale output features are m vectors g in each channel. i ∈R e , let g i =k i =v i ∈R e Define a query vector q∈R to query the importance of semantic features. e and m key-value pairs (k1, v1), ..., (k m ,v m The multi-scale feature fusion based on attention weights can be expressed as follows:

[0035]

[0036] s∈R e

[0037] Let g i =k i =v i ∈R e q represents query, k represents key, and v represents value; q, k1…k m ,v1…v i It is a function input, and the function relationship is...

[0038] where q and k i The attention weight a(q, k i ) is calculated by an attention scoring function The vector q and k i are mapped into a scalar, and the real weight between 0 and 1 is obtained by the softmax operation, and the calculation formula of a(q, k i ) is as follows:

[0039]

[0040] a(q, k i ) ∈ R

[0041] The attention scoring function is calculated by additive attention, given the vector q ∈ R e , the vector k i ∈ R e , the learnable parameter matrix W q ∈ R e×e , the learnable parameter matrix W k ∈ R e×e , the learnable parameter vector w v ∈ R 1×e , the result of the matrix multiplication of the matrix W q and the vector q and the matrix multiplication of the matrix W k and the vector k i is added, input to the tanh function for nonlinear transformation, and the transpose of the vector w v is multiplied by the nonlinear transformed result, and finally the attention score is obtained, which is a real number, and the calculation formula is as follows:

[0042]

[0043]

[0044] In an embodiment, the attention static routing mechanism relies on a learnable parameter matrix and an attention mechanism to assign weights to each vector, and the low-level capsules 1, 2 and 3 output vectors v1, v2 and v3 respectively, and the additive attention scoring function is used to score each output vector, the attention score is input to the softmax operation to obtain the corresponding weight, the weight is weighted and summed with v1, v2 and v3 to obtain the vector y, and the vector y is squeezed to obtain a vector v i with a length between 0 and 1, which is input into the next layer of capsules.

[0045] Compared with the prior art, the beneficial effects of the present application are:

[0046] Firstly, the application designs a new model structure: Capsule network based on attention static routing (CapsNet-ASR), the overall structure of the network model is: word embedding layer, convolution layer, initial capsule layer, intermediate capsule layer, high-level capsule layer and classification capsule layer. Secondly, the dynamic routing between the capsule layers is replaced with the unique attention static routing mechanism of the application, the network automatically learns how to assign the weight size of the routing for the bottom capsule during the training stage, thereby improving the routing efficiency. Thirdly, the convolution layer in the model uses multi-scale convolution kernels to better extract text information, and the multi-scale convolution features are weighted and fused on the same output channel using the attention mechanism. Finally, the squeeze operation is used instead of the pooling operation commonly used in convolutional neural networks to improve the modeling ability of the relationship between semantic features. Through these improvements, the text feature extraction capability can be effectively improved, the relationship modeling capability between text features can be improved, and finally the precision of text sentiment classification can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a classification model structure diagram based on attention static routing capsule network.

[0048] Figure 2 is a multi-scale feature fusion based on attention weight diagram.

[0049] Figure 3 is an attention static routing diagram. DETAILED DESCRIPTION

[0050] The embodiments of the application will be described in detail below with reference to the accompanying drawings and examples.

[0051] The application is a text sentiment classification method based on attention static routing capsule network. Compared with the existing text sentiment classification method, the dynamic routing process of the capsule network is replaced by a static routing based on the attention mechanism, and the network can automatically learn how to assign the weight size of the routing for the bottom capsule during the training stage. And the squeeze operation is used instead of the pooling operation commonly used in convolutional neural networks. The overall structure of the network model is: word embedding layer, two-dimensional convolution layer, fully connected layer, squeeze pooling layer, initial capsule layer, intermediate capsule layer, high-level capsule layer and classification capsule layer.

[0052] Specifically, the application includes the following steps:

[0053] Step 1, collect and arrange unannotated text data in the target language.

[0054] This step mainly collects the corresponding text data according to the specific task, and the target language is the language used to finally complete the text sentiment classification task. The purpose of word embedding is to map each word into a real number vector so that the subsequent neural network can recognize the similarity between words. The word embedding process only needs to rely on the context information of each word, so only unlabeled text in the target language needs to be collected.

[0055] For example, if the target task is sentiment analysis of Weibo comments, then a large number of unlabeled Chinese comment information in Weibo is collected and sorted. Exemplarily, in this step, text data is collected and cleaned to remove useless text and non-text content such as hyperlinks, symbols, and emojis, obtaining unlabeled text data. For example, various articles on the Internet or various texts on Weibo (topic texts, comment texts, etc.) are collected, and then irrelevant hyperlinks, symbols, emojis, etc. in the text are removed. In order to ensure the accuracy of word vectors, the more unlabeled text used for pre-training word vectors, the better. Generally speaking, the total number of words is not less than 1 million words.

[0056] Step 2: Use the unlabeled text data in Step 1 and the word2vec method to train and obtain the word vector representation of the target language.

[0057] Specifically, perform word frequency statistics on the unlabeled text data collected and sorted in Step 1, such as Weibo comment texts, and establish a vocabulary. Each word in the vocabulary corresponds to a real number vector w to be trained i ∈R d . Then word embedding training is carried out. The word2vec method includes two types, the Skip-Gram model and the Continuous Bag of Words (CBOW) model. The present invention adopts a self-supervised training method and uses the Continuous Bag of Words (CBOW) model of word2vec for word embedding pre-training to obtain the real number vectors of all words in the target language, that is, the word vector representation.

[0058] The Continuous Bag of Words model assumes that the center word is generated based on its surrounding context words in the text sequence. For example, in the text sequence "I", "we", "love", "ourselves", "our", "motherland", when "love" is the center word and the context window is 2, the Continuous Bag of Words model considers the conditional probability of generating the center word "love" based on the context words "I", "we", "ourselves", "our", that is: P("love"|"I", "we", "ourselves", "our").

[0059] With the goal of maximizing this conditional probability, use maximum likelihood estimation to train all words in the sorted text, and continuously update and iterate the word vector w i ∈R d .

[0060] Step 3, collect and organize the target language labeled text data.

[0061] In order to realize the task of text sentiment classification, the model needs to be supervised training, so it is necessary to collect labeled text data, each training sample is a text and a category label. For example, the comment text and the corresponding sentiment label in the microblog comment sentiment classification.

[0062] For example, similar to step 1, this step also collects text data and cleanses useless text and non-text content, after which the sentiment of each text (positive, negative, neutral) is manually labeled. For example, collect microblog comment text, remove non-text content such as emoticons and hyperlinks, and manually label the sentiment of each comment text. Finally, each comment becomes a "text-sentiment label" pair. For example, the collected comment information is "This blogger is really young and capable, we should learn from you", and the sentiment label is manually labeled as "positive sentiment". In this way, a piece of labeled text data in the field of microblog comment sentiment classification is obtained.

[0063] Step 4, build a classification model based on attention static routing capsule network.

[0064] The classification model of the present application can use the PyTorch framework widely used in the academic field to write model code. Referring to Figure 1 , the model components include: a word2vec word embedding layer, a two-dimensional convolution layer, a fully connected layer, a squeeze pooling layer, a primary capsule layer, a secondary capsule layer, a high-level capsule layer, and a classification capsule layer, wherein:

[0065] The word2vec word embedding layer is used to map the text into a word vector sequence; the word vector sequence forms a real matrix and is input into the two-dimensional convolution layer as a single input channel, the two-dimensional convolution layer uses multi-scale convolution to extract multi-scale features of the text and converts them into vector capsules;

[0066] The fully connected layer is used to unify the dimensions of the multi-scale features extracted by the two-dimensional convolution layer, and then the multi-scale features with unified dimensions are fused based on attention weights;

[0067] The squeeze pooling layer is used to compress the fused features into a vector with a modulus length of 0-1, which is then input into the primary capsule layer;

[0068] The primary capsule layer, the secondary capsule layer, the high-level capsule layer, and the classification capsule layer are used to pass the most original semantic information extracted by the convolution layer to the model output through attention static routing at each level, so as to obtain the category of text sentiment.

[0069] The overall sequence of the classification model during training is:

[0070] 1) Input the text data to be classified into the word2vec word embedding layer. The input is T = {w1, w2, ..., w...} n}, for each word w i Mapped to a real vector v i ∈R d This transforms the entire text into a matrix D = {v1, v2, ..., v...} n}∈R d ×n , where d is the dimension of the word vector and n is the length of the text.

[0071] like Figure 1 As shown, the text "This blogger is truly young and promising" is mapped to a real matrix D∈R after passing through the word embedding layer. d×n The text has a length of n = 10 and a hyperparameter d = 64.

[0072] 2) Input matrix D as a single-channel image into a two-dimensional convolutional layer, and use multi-scale convolutional kernels to extract features from matrix D to obtain multi-scale features. The output shape is determined according to the following formula:

[0073]

[0074] in This indicates rounding down to the nearest integer, n h Let k be the vertical length of matrix D. h p is the vertical length of the convolution kernel. h For vertical filling, s h This refers to the longitudinal stride.

[0075] For example, there are 5 multi-scale convolutional kernels with sizes of 1×d, 3×d, 5×d, 7×d, and 9×d. The vertical stride is s. h =1, vertically fill p h =0, all output channels are 256. The calculated output shapes of the multi-scale convolution are: o1∈R n×1 o2∈R (n-2)×1 o3∈R (n-4)×1 o4∈R (n-6)×1 ,o5∈R (n-8)×1 In the text above, “This blogger is truly young and promising,” the output shape is o1∈R. 10×1 o2∈R 8×1 o3∈R 6×1 o4∈R 4×1 ,o5∈R 2×1 .

[0076] 3) Because the output shape of multi-scale features in each output channel is different, it is necessary to change the dimension of multi-scale features to the same through a full connection layer to obtain multi-scale output features g i The specific method is as follows:

[0077] There is a full connection layer W1∈R e×n , W2∈R e×(n-2) , W3∈R e×(n-4) , W4∈R e×(n-6) , W5∈R e×(n-8) (in the above text: “This blogger is really young and capable”, W1∈R e×10 , W2∈R e×8 , W3∈R e×6 , W4∈R e×4 , W5∈R e×2 ). The dimension is unified as: W i o i =g i ∈R e×1 , where g i is the multi-scale output feature of the same dimension; e is the dimension of the unified multi-scale feature. Because the result of matrix and vector multiplication W i o i =g i ∈R e×1 is always an e-dimensional vector, for example, W1 is an e*n matrix, and o1 is an n*1 vector, so W1o1 obtains an e*1 vector.

[0078] 4) The multi-scale output feature g i after changing the dimension is weighted and fused based on the attention weight in the same output channel to obtain the fused feature S i .

[0079] Referring to Figure 2 , the multi-scale output feature in each channel is m vectors g i ∈R e , let g i =k i =v i ∈R e . Assuming that there is a query vector q∈R e used to query the importance of semantic features and m key-value pairs (k1,v1),…,(k m ,v m ), the multi-scale feature based on the attention weight fusion can be expressed as the following formula:

[0080]

[0081] s∈R e

[0082] Where q represents query, k represents key, and v represents value; q, k1…k m ,v1…v i It is a function input, and the function relationship is... q and k i Attention weights α(q,k) i ) is the attention scoring function Let vectors q and k i Mapping to scalars, then using softmax to obtain real-valued weights between 0 and 1. Attention weights α(q,k) i The calculation formula for ) is as follows:

[0083]

[0084] α(q,k i )∈R

[0085] Attention scoring function The computation employs additive attention. Given a vector q∈R e Vector k i ∈R e Learnable parameter matrix W q ∈R e×e Learnable parameter matrix W k ∈R e×e Learnable parameter vector w v ∈R 1×e , matrix W q Matrix multiplication with vector q and matrix W k With vector k i The results of matrix multiplication are summed and then input into the tanh function for nonlinear transformation. The vector w v The attention score is obtained by multiplying the transpose of the expression with the result of the nonlinear transformation. The attention score is a real number. The calculation formula is as follows:

[0086]

[0087]

[0088] 5) In the squeeze pooling layer, the fused feature s i The compression operation is used to compress the vector into a vector c with a modulus of 1. The vector is then input into subsequent capsule layers. The compression operation formula is shown below.

[0089]

[0090] 6) The primary capsule layer, intermediate capsule layer, advanced capsule layer and classification capsule layer are fully connected, and the routing between capsules adopts the attention static routing mechanism.

[0091] Traditional dynamic routing mechanism relies on iterative way to assign weights to each low-level capsule's output vector, while attention static routing mechanism relies on learnable parameter matrix and attention mechanism to assign weights to each vector, as shown in Figure 3 Low-level capsules 1, 2, 3 output vectors v1, v2 and v3 respectively, use additive attention scoring function Score each output vector, input attention score into softmax operation to get corresponding weight, weight the weight with v1, v2 and v3 to get vector y, and then pass vector y through squashing operation to get vector v with length between 0 and 1 i Into the next layer of capsules.

[0092] Step 5, use the target language labeled text data obtained in step 3 to supervise the training of the classification model in step 4. For example, the sentiment category of the text "the blogger is really young and capable" has been manually annotated as "positive". The prediction result obtained by supervised training Calculate the loss with the actual category "positive", and update the model parameters using the backpropagation algorithm.

[0093] Step 6, use accuracy, precision, recall and F1 Score to evaluate the classification model trained in step 5. After model training, use part of the test data set that is not used for training to test the model. The results of model testing are used to evaluate the model with accuracy, precision, recall and F1 Score, and finally get the required text sentiment classification model. Using the text sentiment classification model that meets the requirements, the input text can be classified by sentiment.

[0094] Accuracy Accuracy refers to the proportion of all correctly predicted samples to all samples, and the calculation formula is as follows:

[0095]

[0096] Precision Precision indicates how many of the samples predicted as positive are truly positive samples, and the calculation formula is as follows:

[0097]

[0098] Recall Recall indicates how many positive samples in the sample are correctly predicted, and the calculation formula is as follows:

[0099]

[0100] F1 Score is a statistical measure used to evaluate the accuracy of a binary classification model. It takes into account both the precision and recall of the classification model. F1 Score can be seen as a harmonic mean of precision and recall, reflecting the robustness of the model. Its maximum value is 1, and its minimum value is 0. The F1 Score calculation formula is as follows:

[0101]

[0102] In order to obtain the accuracy, precision, recall, and F1 Score, it is necessary to first draw a confusion matrix for statistics, and obtain TP, TN, FP, and FN. In the classification task, there are four different combinations between the predicted results and the actual results, which form a confusion matrix, as shown in the following table:

[0103]

[0104] In the confusion matrix, TP (True Positive) represents the number of samples that are actually positive samples among the samples predicted as positive samples; FP (False Positive) represents the number of samples that are not actually positive samples among the samples predicted as positive samples; FN (False Negative) represents the number of samples that are not actually negative samples among the samples predicted as negative samples; TN (True Negative) represents the number of samples that are actually negative samples among the samples predicted as negative samples.

[0105] The positive and negative examples mentioned above are relative. For example, in the sentiment classification task, the sentiment category of a sentence can be positive, neutral, and negative. If positive is chosen as positive, then neutral and negative are collectively referred to as negative.

[0106] The model CapsNet-ASR of the present application was used to conduct experiments on Chinese sentiment text data sets ASAP, ChnSentiCorp, NLPCC14-SC, and SE-ABSA16, and compared with traditional models in terms of accuracy. The comparison results are shown in the following table, which shows that the present application has significantly improved the performance in the field of Chinese text sentiment classification.

[0107] ASAP ChnSentiCorp NLPCC14-SC SE-ABSA16 RNN 75.9 84.4 83.8 83.1 LSTM 80.3 85.7 84.5 89.5 ordinary capsule network 81.2 88.9 87.5 90.8 CapsNet-ASR 84.5 92.2 91.9 91.5

[0108] In addition, since the CapsNet-ASR of the application adopts a static routing mechanism, while the ordinary capsule network adopts a dynamic routing mechanism, the training time of the application is less in theory. Therefore, the two kinds of models are trained on the datasets ASAP, ChnSentiCorp, NLPCC14-SC and SE-ABSA16 respectively, and both the two kinds of models are trained for 60 epochs. The experimental results show that the training time of the CapsNet-ASR model is significantly lower than that of the ordinary capsule network. The experimental results are shown in the following table, and the numbers in the table are the training time of the model under the dataset, with the unit of hours.

[0109] ASAP ChnSentiCorp NLPCC14-SC SE-ABSA16 ordinary capsule network 8 14 16 9 CapsNet-ASR 3 8 10 6

Claims

1. A text sentiment classification method based on an attention static routing capsule network, characterized in that, The method comprises the following steps: Step 1, collecting unlabelled text data in a target language; the target language is the language used for finally completing a text sentiment classification task; Step 2, training word vector representation of the target language by using unlabelled text data in step 1 and a word2vec method; Step 3, collecting text data with labels in the target language; Step 4, constructing a classification model based on an attention static routing capsule network; The model components of the classification model comprise a word2vec word embedding layer, a two-dimensional convolution layer, a full connection layer, a squeeze pooling layer, a primary capsule layer, an intermediate capsule layer, a high-level capsule layer and a classification capsule layer; The word2vec word embedding layer is used for mapping text into a word vector sequence; the word vector sequence constitutes a real number matrix and is input into the two-dimensional convolution layer as a single input channel picture, the two-dimensional convolution layer extracts multi-scale features of the text by using multi-scale convolution and converts the multi-scale features into a vector capsule; The full connection layer is used for unifying the dimensions of the multi-scale features extracted by the two-dimensional convolution layer, and then performing feature fusion on the multi-scale features with unified dimensions based on attention weights; The squeeze pooling layer is used for compressing the fused features into a vector with a module length of 0-1, and then taking the vector as an input of the primary capsule layer; The primary capsule layer, the intermediate capsule layer, the high-level capsule layer and the classification capsule layer are used for transmitting the most original semantic information extracted by the convolution layer to the model output by using attention static routing in stages, so as to obtain the category of text sentiment; The primary capsule layer, the middle capsule layer, the high-level capsule layer and the classification capsule layer are fully connected, and the routing mode between the capsules adopts an attention static routing mechanism, which relies on a learnable parameter matrix and an attention mechanism to assign weights to each vector, and the low-level capsules 1, 2 and 3 respectively output vectors , and An additive attention scoring function is used to score each output vector, the attention score is input into a softmax operation to obtain the corresponding weight, the weight is weighted and summed with , and to obtain a vector , and then the vector is input into the next layer of capsules through a squashing operation to obtain a vector with a length of 0 to 1. Step 5, performing supervised training on the classification model in step 4 by using text data with labels in the target language obtained in step 3; Step 6, evaluating the classification model trained in step 5 by using accuracy, precision, recall and F1 Score, obtaining a text sentiment classification model meeting requirements, and classifying input text by using the text sentiment classification model meeting requirements.

2. The text sentiment classification method based on the attention-based static routing capsule network according to claim 1, characterized in that, In step 1, text data is collected and cleaned to remove useless text and non-text content, and unlabelled text data is obtained, wherein the total number of words of the unlabelled text data is not less than 1 million words.

3. The text sentiment classification method based on the attention-based static routing capsule network according to claim 1, characterized in that, In step 2, word embedding pre-training is performed by using a bag-of-words model of word2vec, and a real number vector of all words in the target language is obtained, that is, the word vector representation.

4. The text sentiment classification method based on the attention-based static routing capsule network according to claim 1, characterized in that, In step 3, text data is collected and cleaned to remove useless text and non-text content, and then the sentiment tendency of each text is manually labeled.

5. The text sentiment classification method based on the attention-based static routing capsule network according to claim 4, characterized in that, In step 5, the training process is as follows: 1) Input the text data to be classified to the word2vec word embedding layer, which maps each word to a real-valued vector , thus turning the entire text into a matrix where is the dimension of the word vector, is the length of the text; 2) matrix As a single input channel picture, input to the two-dimensional convolution layer, with multi-scale convolution kernel on the matrix Feature extraction is performed to obtain multi-scale features, and the formula of the output shape is as follows; wherein denotes rounding down, is a matrix is a vertical length of the matrix is a vertical length of the convolution kernel, is vertical padding, is a vertical stride; 3) change the dimension of multi-scale features to the same through the full connection layer, to obtain multi-scale output features ; 4) the multi-scale output features are fused based on the attention weights on the same output channel to obtain fused features are fused based on the attention weights on the same output channel to obtain fused features ; 5) In the squash and pool layer, the fused features Compress to a vector of length 1 with a squash operation The vector is then input to the subsequent capsule layer, with the squash operation formula shown below:

6. The text sentiment classification method based on the attention-based static routing capsule network according to claim 5, characterized in that, The two-dimensional convolution layer has five multi-scale convolution kernels, and the sizes are respectively: , , , and , the longitudinal stride is , the longitudinal padding is , and the output channel is 256; the calculation obtains the output shape of the multi-scale convolution respectively as: , , , , ; the full connection layer is respectively: , , , , ; the dimension is unified as: , wherein is the multi-scale output feature with the same dimension; is the dimension of the unified multi-scale feature.

7. The text sentiment classification method based on the attention-based static routing capsule network according to claim 6, characterized in that, The weighted fusion method is as follows: The multi-scale output features are one vector , let ; Setting a query vector for querying the importance of semantic features and key-value pairs The multi-scale feature representation based on attention weight fusion is expressed as follows: Let , represent query, represent key, represent value; , , is the function input, the function relationship is ; in, and Attention weights It is determined by the attention scoring function vector and Map the values ​​to scalars, then use softmax to obtain real weights between 0 and 1. The calculation formula is as follows: Attention scoring function The calculation of the attention scoring function employs additive attention, given a vector , a vector , a learnable parameter matrix , a learnable parameter matrix , a learnable parameter vector , the result of matrix multiplication of the matrix and the vector , and the result of matrix multiplication of the matrix and the vector are added, input to a tanh function for nonlinear transformation, the result of multiplication of the transpose of the vector and the nonlinear transformation result is obtained, and the attention score is a real number, and the calculation formula is as follows: 。

Images